Microsoft is getting ready to introduce simulation questions, or performance-based testing, to more of its exams.
The company currently has five exams with a performance-based testing (PBT) component, with the sixth expected to hit the market within the next three months. Following that, many more PBT exams are expected to be introduced during the next 15 to 18 months. “We see [performance-based testing] as a major part of our program and reinforced in the exams that we release to the market,” said Rob Linsky, group product manager for IT pro and developer certification at Microsoft.
“In a performance-based testing environment, you actually have to use the technology to demonstrate your skills; you can’t get by with reading and memorizing something,” Linsky said. “It actually discovers and validates real skills.”
PBTs mean an improvement in exam security, as well. Traditional exam items such as multiple-choice, matching, drag and drop, true/false and listing often are stolen and pirated to people who resell that information. But a performance-based testing environment makes it almost impossible to do that. “You might be able to pirate a question, but at the end of the day, in order to get the question right on a performance-based exam, you still have to show that you can perform the task by using the technology,” Linsky said.
Linsky feels that forcing candidates to actually perform tasks is a much more relevant way to measure skills.
Reliability is one of the variables that Microsoft considers when analyzing the performance of its exams. “Microsoft has an in-house analyst who’s constantly reviewing our performance. And our traditional exams, in terms of reliability, have always been very high,” Linsky said. “Add in performance-based items and the reliability goes up even more; [it] makes the whole exam more relevant.”
That spells huge benefits for IT companies, as it makes candidates more marketable. “If you know that someone can maneuver a performance-based testing experience on their way to a certification, then you as a hiring manager have a much more reliable indicator of their skill level,” Linsky said. “We’re going to spend a lot of time and energy [indicating to] hiring managers that this is something they need to start looking for and have to be searching out and recruiting against,” he said.
Just because performance-based testing is more realistic doesn’t mean it’s designed to trick people. “Microsoft Learning’s role is to get an accurate measurement of a candidate’s true skill level to a hiring manager [or] an IT manager,” Linsky said. “If you know how to manipulate the technology, you should do well; if you don’t, you’re going to struggle. We think of our exams as testing someone’s ability to go do a job.”
Feedback from the usability study indicates that test-takers seem excited about performance-based testing. “It mirrors real-life situations that they’re going to run into rather than traditional multiple-choice answers, so they definitely prefer much more of a performance-based, lab-based, hands-on experience because it’s extremely relevant to what they’re going to be doing from day-to-day,” said Bill Wall, director of certification at Microsoft. “[Test-takers] also like the fact that after they pass, they feel like they’ve achieved something, set a higher bar, and perhaps over time it does a better job of eliminating those people who shouldn’t have the certification — who don’t have the skills.”
While performance-based testing is a more realistic gauge of a candidate’s skills, the scoring doesn’t differ from the traditional section, except that instead of having a choice of four or five answers, they are left to complete an open-ended scenario to find the correct answer using the technology. “If you’re using the [user interface] — using the command line — you have to go to the command line and enter in the proper information, the proper string of commands,” Linsky said. “So rather than having the answers displayed for you, you’re given a scenario and you’re given access to the technology and you’re asked to work through the technology. And to find the correct commands and components of the technology will allow you to get to a right answer.”
While Microsoft is not allowed to reveal passing scores, it can reveal some changes that will be made to the score report candidates receive. Each Microsoft exam typically consists of five to seven functional groups, or topic areas. When Microsoft first released simulation questions, the limitations of the technology made it difficult to give candidates a traditional exam-section score by functional group. The new score reports are laid out by functional group, thereby allowing candidates to assess their specific areas of weakness.
“We’re going to be involving a lot of our current [Microsoft Certified Professionals] who will be testing and validating the new performance-based testing technology we’re working on that incorporates Microsoft’s own virtualization technology,” Linsky said. “A lot of people think we’ve been giving it lip service, but it’s a complex thing to integrate into your program. We’re a leading certifier of people in the IT state, [and] we couldn’t wade into it haphazardly. We’ve had to really think about how to incorporate it correctly into our program.”