Multiple-Choice Questions Are the Answer
It doesn’t take much research to determine that multiple-choice questions represent the majority of questions in any high-stakes test, including IT certification tests. I’m not telling you anything new—according to my recent number crunching, this question type accounts for more than 95 percent of the questions on IT exams. There are alternative question styles, but even those share common traits with the multiple-choice question we all know and love. For example, a question where the testtaker simply moves the mouse pointer over a graphic and clicks to select an area is really just a multiple-choice question without the text. The same goes for a question where the task is to drag one or more objects on the screen to the correctly corresponding destinations. It might seem like you’re doing something different, but these are fundamentally multiple-choice questions.
A multiple-choice question is described simply as providing all necessary information and asking the test-taker to make a choice, usually with the keyboard or mouse. All of the choices can be read or viewed. The candidate doesn’t need to produce anything, except to choose one or more of the answers. Because everything is there on the screen, the response is usually relatively quick, as the test-taker only has to recognize the correct choice. The ease of presenting information and collecting the answer is the power behind the multiple-choice question.
A true drawback is that a multiple-choice question can be guessed correctly. But if an exam is well written, guessing is limited and reduced by the number of choices available, and is taken into account in the scoring and where the pass-fail mark is set. Writing a larger number of choices and incorporating more than one correct answer are popular ways to reduce the chances of guessing a question correctly.
Besides its efficiency in allowing easy reading and quick responding, the multiple-choice question lends itself well to computerized testing. The computer can present the questions quickly, one at a time. After the response, the answer can be scored immediately, and after the last item, the overall score can be immediately provided. So-called “performance” questions, such as essay questions, are much more difficult to present, collect the answer and score. The test-taker often has to wait weeks before the final score is reported.
Over the years, the multiple-choice question has been criticized for its inability to measure complex human performance. It is generally believed that the multiple-choice question can only measure the simple memorization of facts. For example, the question may ask who the first president of the United States was and provide four or more good choices, including the correct answer, George Washington. But the criticism is incorrect and unfair.
Here’s an example of how higher-level skills and performance can be tested with a simple multiple-choice question: Several years ago, while at Novell working in the certification program, we came upon a problem: how to measure network engineers’ ability to effectively use the technical manuals and support encyclopedias stored on CDs. We decided to install CD players in all of the testing centers and distribute the CDs to them. We then authored multiple-choice questions that asked very technical questions that could only be answered by successfully launching and navigating the information on the CDs. During the question, a competent test-taker was able to get into the CD, find the information quickly, return to the test and answer the question by selecting the correct answer from a list of choices.
What Novell was measuring was the higher-order cognitive skill of planning and conducting an efficient search for information. Those who could do it answered the questions correctly. Others guessed, and their lower scores reflected that. The candidates knew they were being tested on skills they used every day to solve network problems and responded favorably to the new test. While a bit more complex, the question was still multiple-choice.
Because of their advantages and few problems, multiple-choice questions will continue to dominate the testing landscape for years to come. As audio, video, simulations and graphics become more common components of multiple-choice questions, they will become even more effective in measuring many IT certification skills.
David Foster, Ph.D., is president of Caveon (www.caveon.com) and is a member of the International Test Commission, as well as several measurement industry boards. He can be reached at firstname.lastname@example.org.