Psychometrics 101: Test Quality

Posted on
Share on Google+Share on LinkedInShare on FacebookShare on RedditTweet about this on TwitterEmail this to someone

Everyone needs to know about tests these days. If you are an IT certification candidate, you have lots of questions about the tests you see and their quality. As parents, we wonder about the SAT and other admissions tests, and about the tests teachers create for our children. We also need to take tests to get our driver’s licenses. There is no doubt about it: Tests are a part of our lives.

So, what are the most important things to know about tests? Here is what I think. You want to know what makes a test a good test. Like anything else you buy, you want to know that you are buying something of quality. Who buys a car without reading consumer reports about it? Or a computer? Or a plane ticket? As consumers, we come to expect that what we buy has a specific level of quality that satisfies us. We also expect that we can understand why it provides that quality.

Unfortunately, in the testing industry, we have made an art form of keeping the consumer in the dark. We either believe that what we do is so complicated that lowly test-takers will never be able to understand, or that we need to keep the mystery in it all to support our pricing. Probably both. But that’s going to change right now! I’m going to open the kimono a bit. Not so much that you’ll get dizzy and ill, but enough to give you confidence and an appetite for more. Then, I’ll follow up in later columns.

The first lesson deals with test quality. What makes a test good? That’s the question I want to answer. It boils down to two characteristics of a test: good content and enough content.

Good content means that the test has questions that reasonably measure the appropriate skill or knowledge. The questions can be multiple-choice or some other type. That doesn’t matter so much, as long as they capture what the test-taker knows or is able to do. For example, if the test questions are supposed to determine whether or not the person can pick out a triangle among other shapes, then a multiple-choice question that presents four different shapes (including a triangle) and asks the test-taker to choose the triangle will do. A question that asks how many sides a triangle has, while related, does not measure the specific knowledge required. Unfortunately, many examinations fail to measure the abilities they claim to test. In IT certification, which is heavily dependent on job skills, the tests tend to use questions that measure memorized information, asking for facts and definitions rather than a demonstration of skills. Thus, some tests appear to be irrelevant and overpriced.

Even when the test questions are good, there still needs to be a sufficient quantity to get a reliable score. Just asking a person one question, even if it’s a great question, won’t work. Maybe the person will guess it correctly. You can see why more than one question is necessary.

How many questions are needed? That depends on the design of the test. Roughly, a test needs between 50 and 70 questions. Too few, and the test can’t produce a good score. Too many questions, and money and time are wasted because a more reliable score is produced than is needed. Imagine a factory inspector who is inspecting a pair of pants. What would be the point of inspecting the pair of pants five extra times?

If the questions are good and there are enough of them—and if the test-taker didn’t cheat in any way—then the score is valid. The score accurately represents the test-taker’s ability, and it can be used with confidence to make a certification decision.

One added benefit of a good test is the straightforward nature of training and learning the content that the test covers. If the test measures knowledge, then grabbing books and manuals should be helpful in preparing. If the test measures a job skill, then on-the-job experience, working with simulations and gathering other types of experience will be the most effective way to prepare.

This isn’t all there is to psychometrics, the science that underlies testing, but question quantity and question quality are two very important standards of testing that are required for testing to work. If an IT certification test is bad, it is likely because of one or both of these reasons.

David Foster, Ph.D., is president of Caveon ( and is a member of the International Test Commission, as well as several measurement industry boards.


Share on Google+Share on LinkedInShare on FacebookShare on RedditTweet about this on TwitterEmail this to someone


Posted in Archive|