Multiple Choices: Not as Simple as You Think
Multiple-choice questions have been indicted and maligned as a too-simple way to test for IT knowledge and skill. According to some, what you can learn from answering a bunch of multiple-choice questions doesn’t seem to match up against the complex set of skills used by experienced people in most IT jobs.
One reason they may not appear to work well in some exams is that they are not written very well. Multiple-choice questions are very difficult to produce. There are no less than 40 strict rules to adhere to in order to create a good multiple-choice question. I can’t list them all here, but let me refer you to a document titled “Item Writing Rules,” compiled and edited by Dr. David Lauret of Prometric and found at www.galton.com.
Instead of reprinting the rules, I thought I’d provide a good question as well as a couple that miss the mark.
Objective: Given the lengths of any two sides of a right triangle, and which side is the hypotenuse, the student can determine the length of the third side.
Question: One side of a right triangle has a length of 10 cm. Another side, the one opposite the right angle, has a length of 15 cm. What is the length (rounded to 2 decimal places) of the third side?
Here is another well-written question that is written to the test objective, but does not measure the objective:
Question: A triangle has one angle of 120 degrees and two others of 30 degrees each. What kind of triangle is it?
A. right triangle
B. isosceles triangle
C. obtuse triangle
D. equilateral triangle
This next question neither measures the objective, nor is it well written. In addition to the fact that it isn’t congruent with the objective, there are at least eight other flaws. Can you find them?
Question: How do you know if a triangle is a right triangle. Choose one of the answers below:
A. It looks like a right triangle.
B. It has a couple of angles that are smaller than the other one.
C. One of the angles is a right angle.
D. One or more of the angles is a right angle.
E. None of the above.
In producing this second question, the author assumed that since the objective mentioned a right triangle, it was acceptable to create a general question about right triangles. Obviously the question doesn’t match the test objective.
Perhaps you have noticed the other flaws already:
- The question should end with a question mark.
- The second sentence in the actual stem of the question is unnecessary. The person has no choice, particularly if this were a computerized test question, except to choose one of the options.
- Option B should replace “a couple of” with “two.” Slang should be avoided.
- While Option C is the correct answer, the specific wording of Option D, and the use of the word “or” in particular, makes it technically correct as well.
- Option E has three problems. First, it does not attempt to even answer the question. Second, it is grammatically dissimilar from the others, not being a complete sentence. And third, options such as “none of the above” or “all of the above” are used less often in multiple-choice questions and are prohibited in most testing programs.
- Option A is a “throw-away” option that no one would select and is therefore not useful.
Unfortunately, items such as this one are often written for IT certification exams. If the author is not trained, if a good test objective has not been previously articulated, if there isn’t a good process for editing and refining the question, or any number of other causes, it’s possible that questions like this one could end up in the test you are taking. This happens way too often, even in high-stakes certification exams.
Don’t give up on multiple-choice questions. Instead, when communicating with your certification program manager, insist that such questions be used responsibly and created according to existing guidelines. Send them a copy of Dr. Lauret’s paper.
David Foster, Ph.D., is a member of the International Test Commission and sits on several measurement industry boards.