The Most Important Thing to Know about Tests

Posted on
Share on Google+Share on LinkedInShare on FacebookShare on RedditTweet about this on TwitterEmail this to someone

A certification test is like a piece of sophisticated equipment, such as a cell phone. We usually don’t know what’s inside the phone or what goes into producing it, but we like that it works. And we do have some basic understanding of its function, which helps. It’s the same with tests. Their purpose is clear, but how they are built and why are generally mysteries to those who take the tests, including IT certification candidates.


Because the tests have such profound effects on us, we need to understand them better. Not completely, of course. That would take more time and effort than it would be worth. But enough to get some basic questions answered. Here are a few of the more important questions:



  • How is a passing score set? How can a single score separate people who are competent from those who aren’t?
  • Why are there so many stupid questions in tests?
  • Why aren’t the tests shorter? Or longer?
  • Why does it cost so much to take a test?
  • What percentage of people pass the test?
  • Why are the tests mostly multiple-choice?
  • Aren’t there better ways to test someone?
  • How is the test scored?
  • How is the time limit set?
  • Why doesn’t the test cover all of the material I studied?


These are all good questions. And many others could be on the list. It’s a good idea to cover these questions in this column, which I’ll do from time to time. Today, however, there is space to answer only one of them, and I picked the last one on the list: Why doesn’t the test cover all the material I studied?


The answer to this has to do with the test’s design. So first I need to contrast the terms “test” and “test form.” If you register to take a test at a testing center, you are actually signing up to take one of several possible test forms. A test form for a typical IT certification exam contains questions sampled from the entire domain of skills. If you were able to gather them and list all of the questions from all of the test forms, you would see all of the domain’s topics represented. If all of the questions were part of only one big test form, it could be several hundred questions long and take you several hours to complete.


Programs divide the entire set of questions more or less randomly into test forms. The purpose is to build test forms of sufficient length to measure you well, at the same time covering a reasonable and representative sample of the domain of skills. And part of the test design makes sure that the forms are not so large that they take too much time and cost more to take.


One important additional point about test forms: They are built and validated to produce equivalent scores for any test taker. So it doesn’t matter which one is randomly selected for you.


The test taker—you in this case—should be unaware of the exact test form and therefore the exact questions to be presented that day. This will properly motivate you to study all of the material, complete the full training course, gather as much experience as possible, etc. Obviously, that’s not a bad thing for you or the certification program. And it’s a good thing for anyone who hires someone who passed the test and got certified.


To reinforce the point, if you were given the subset of topics covered in a specific test form you were scheduled to take, you would probably only prepare to answer questions on those topics. If every test taker did the same thing, there would be no consistency in the quality of those who are certified. And each person’s capability would be too narrow to be effective on the job.


To summarize, a test as published by an IT certification program contains all of the questions that cover the domain. The test forms (what you will see) contain reasonable and equivalent samples of that content.


Here’s another, somewhat dubious, advantage: Say you study all of the material you can, but you still fail the test. The good news is that you already have a head start on preparing for the retake.


David Foster, Ph.D., is president of Caveon ( and is a member of the International Test Commission, as well as several measurement industry boards.

Share on Google+Share on LinkedInShare on FacebookShare on RedditTweet about this on TwitterEmail this to someone


Posted in Archive|