It’s a Putter
Golfing, like most sports, has finely tuned its equipment for specific purposes. For example—and this is not going to be a surprise—a putter is very useful for putting on greens. It’s not the club of choice in other golfing situations. Nor are the other clubs that useful for putting. Depending on weather conditions, distance to the hole, the lie of the ball and more, there is a club that fits the need in every golf bag.
You never really hear anyone ask, “Why can’t I hit the ball farther with a putter?” If someone did ask the question, the answer would be, “Because it’s not designed to hit far.” Anyone looking at a putter would agree.
But I received an analogous question recently following a speech on testing and certification I gave at a Sun certification conference in Colorado Springs. After the speech, one of the attendees approached me with a question. He told me that he recently took his first Sun exam and had failed by a small margin. He asked, “Why didn’t the exam give me more information as to why I failed? I’m not sure which areas to focus my preparation.”
Part of my answer was that the certification test was not designed to discover and report on strengths and weaknesses. Just like a putter, it was built for a specific purpose: to determine a pass or fail decision asking the fewest number of questions possible. A pass decision would indicate that the examinee is competent to perform on the job described by the certification. A fail decision means that the person is less than minimally competent. The content, the difficulty and the number of questions, as well as where the cut score is set are all specific elements of the design of a certification test. The certification test, like the putter, is designed for a single purpose, and if built well, it does that job very effectively.
If the test has a different purpose, say, one designed to diagnose a person’s strengths and weaknesses, it is designed differently. A diagnostic test covering the same content areas would need to be much longer. Probably four times the number of questions would be required. There would need to be more variety in the difficulty of the questions. Plus, there would not be a need for a cut score and pass/fail decision.
So why not just make the certification test longer and provide diagnostic information at the same time the certification test is given? Good question, although it’s somewhat like asking to change the putter to make it serve as a driver as well. Here’s the answer.
Even though it is possible, there are several reasons why a diagnostic test design is not combined with a certification test design. First, because it needs more questions, the test would take longer to create, causing it to be available later than it should. Certification tests in IT are required to be as current as the newest technology. Second, besides being more expensive to create, taking a test at a professional testing center would add cost by quadrupling the time required to take the test. Third, most test-takers actually pass certification tests the first time and have no need of diagnostic information. And fourth, a diagnostic test is not high-stakes and can be delivered less expensively and more conveniently in other ways—on the Web for example.
I appreciate the person’s question, though. It addresses the basic need to know more about our own competence than what we can find out by taking a certification exam. It’s too bad that the first real evidence of competence most candidates get is a certification exam result. A fail decision is motivating, but it doesn’t provide direction. And for that small amount of information, the fee has been mostly wasted. It makes more sense to me that a certification candidate already knows all about his competence before taking the certification test and that passing is a foregone conclusion. This knowledge can be obtained from other types of tests—cheaper ones.
Certification programs would find it useful to create low-stakes exams that diagnose a candidate’s strengths and weaknesses, and that can be used to place a person appropriately into a course of study or work experience. When preparation is complete—and can be verified by a brief, cheap test—the candidate is ready to lay down the bucks and confidently take the certification test.
So far, all we have are putters in our IT certification bag. Everyone is forced to pull one of them out and use it regardless of the shot.
David Foster, Ph.D., is president of Galton Technologies Inc., a past president of the Association of Test Publishers (ATP) and a member of the National Council for Measurement in Education and the American Educational Research Association. E-mail David at firstname.lastname@example.org.