Where Do Good Questions Come From?

Posted on
Share on Google+Share on LinkedInShare on FacebookShare on RedditTweet about this on TwitterEmail this to someone

Think of a test question as a product, like a disposable razor, but a lot more expensive and with more long-term impact on your life. The razor, made of plastic and metal, was molded, cut, sharpened, assembled, inspected, packaged, distributed and, finally, bought. You probably never thought much about the process that brought that razor to you, but that’s because it’s not all that important.

But a test question, that’s different. It’s important to you, at least at the moment you are trying to answer it correctly. And you probably try to understand it and evaluate it from the moment you see it until you move on to the next one.

Does a test question go through a development process similar to that of a razor, from raw material to useful product? How was the question originally written? Or better yet, why was it written? What reviews and changes did it go through? How many people like you actually read it and agreed that it should be on the test? These are great questions (no pun intended) and deserve to be answered.

First of all, a question can’t be written until a job skill has been identified. For example, a job skill might be: The test-taker must be able to install a router. (Job skills are usually identified by interviewing experts through a process known as a job task analysis.) Once the skill is identified, one or more questions can be written with the goal of measuring the skill as well as possible.

An expert in the subject matter (SME) who also has some experience writing test questions is the first person to actually produce the first draft of the question, which may include graphics as well. Often the SME will get help from colleagues to make sure the question is accurate. All questions, but especially multiple-choice questions, require that the SME follow specific format rules for such questions.

After the initial authoring, the question, along with all the others produced, is sent to an editor. The editor is not an SME, but does understand the rules of language, style and the formatting of questions. The editor will fix the language and design problems with the sole goal of reducing ambiguity. For example, if the editor notices that, because of wording, two choices of a multiple-choice question are correct (when only one should be), he will rework one of them or alert the original SME to the problem. The result is a better question.

The question is returned to a group of SMEs who review each one for technical accuracy, representation and relevance. Does the question really measure the test objective? Is it an important question, measuring an important skill? Does the test “need” the question to be balanced across the content domain? Is the question accurate, including a correct answer? The question is usually changed (and may even be deleted) at this stage.

The question is returned to the editor again because changes produced during the technical review have added or changed text. The editor will fix any obvious errors introduced by the technical review.

When all questions have been refined in this way, they are subjected to an actual “field test” of their quality. In what is called a beta test, questions are answered by actual certification candidates in circumstances that mimic the motivation and environment of a real certification test. The beta test provides test results that are subjected to statistical analysis. The analysis will catch those questions that aren’t performing properly. They are then removed from further consideration. Obviously, the question you see on the certification test survived the beta process.

Finally, the final set of questions is published as the actual certification exam. Before the test is released to any candidate, it goes through a series of quality assurance steps. While these steps are focused on the actual functioning of the test, the questions are reviewed once more.

These several steps make sure that each test question, while not perfect, is as good as it can be at measuring the identified job skill. With enough of these great questions, it is possible to produce a reliable and valuable test score that indicates whether a person should be certified or not.

David Foster, Ph.D., is a member of the International Test Commission and sits on several measurement industry boards.


Share on Google+Share on LinkedInShare on FacebookShare on RedditTweet about this on TwitterEmail this to someone


Posted in Archive|