Measuring Performance on a Certification Exam
In a move I applaud wholeheartedly, Microsoft recently announced and launched certification exams that use simulations to measure a candidate’s ability to do the work. There is a demonstration of how the simulations work at www.microsoft.com/learning/mcpexams/simulations/.
The demo itself is not that impressive: a plodding narration accompanying a forced step-through. That is, when in the actual simulation, none of the incorrect options are available. A description of the simulation question explained that during the real certification exams, some ways of completing the task were not available because the intent of the simulation is to test the candidate’s ability to complete the task in specific ways. The demo was helpful in getting a feel for how the simulation might look, but that’s about all.
I sincerely hope that the certification test doesn’t work the way the demo suggests. After reading some reviews from the first few to try the real test, there appears to be more flexibility in the simulation than in the demo. That’s good, but I think it would be better to provide a demo that allowed a candidate to try to answer correctly rather than baby-step them through it. Someone who doesn’t know how to do the task would quickly realize the value of such a test question, because it weeds out the book learners and good guessers. It would also motivate the candidate to gain experience along with technical knowledge.
Almost 10 years ago, we created a similar simulation for Novell’s certification exams. Throughout the exams, the candidate would be asked to complete many NetWare tasks by clicking on a NetWare simulator. Back then, it cost about $250,000 to program the simulator and a similar amount to implement the tool in the testing centers. At the time, we hoped the cost was worth it.
The NetWare simulator was built to mimic about 95 percent of the actual NetWare software, allowing the candidate to try the multiple and typical ways that Windows lets you do something. The simulator was built to be useful for training purposes as well. It was offered to Novell’s training partners and licensed to a test-preparation company at the time. This had the benefit of creating more relevant NetWare training and practice for the tests.
I bring up that bit of history to give you confidence that I can reasonably predict the value of this simulation to the Microsoft certification program and to the IT certification community. I also can describe the results of some interesting research Novell conducted on the validity of the test with simulations and on customers’ satisfaction with the effort.
First, a little background. We built the simulator because we were criticized heavily for producing “paper CNEs.” Whether the criticism was valid or not, we felt compelled to respond by producing a test that actually measured a candidate’s NetWare skills—a test that favored someone with experience over one who had only picked up a lesson manual. The test questions that used the simulations comprised about 60 percent of the test. The non-simulation questions were mostly multiple-choice. All of the questions were randomly presented throughout the test.
Adding simulation questions to the test had the effect we were looking for. The audience loved the new exams, except for the few inexperienced candidates who could no longer pass the test without a few hours using NetWare. Research showed that:
- The simulation questions performed better statistically than other question types, were easier to author and were judged by certification candidates as more acceptable.
- As a way to evaluate the validity of the simulation-based test, the new certified individuals were judged by themselves, their peers and their supervisors to be more competent on the job than those who weren’t certified.
As an added benefit, the “paper-CNE” criticism seemed to vanish overnight. (It should be noted that while Novell no longer uses simulations, the program has kept up the tradition of testing performance directly using virtual machine technologies.)
I want to congratulate Microsoft on this extended use of simulations and to ask its certification candidates to have patience as the company implements them to greater effectiveness. Putting such technology in place takes time and is very expensive. It shows Microsoft’s commitment to those who have been certified, to current candidates and to the future of the program, in general.
David Foster, Ph.D., is president of Caveon and is a member of the International Test Commission, as well as several measurement industry boards. He can be reached at firstname.lastname@example.org.