Performance Testing Council pursues mandate to improve exams
There’s a running debate in certification circles as to whether it’s more effective to test certification candidates using traditional multiple-choice exams or performance-based tests that simulate (or duplicate) real world IT conundrums. Multiple choice tests are convenient, inexpensive and straightforward. Many have argued, however, that such tests don’t accurately or fully assess whether a certification candidate has job-ready IT skills and knowledge.
Supporting, furthering and exploring the side of the debate that favors performance-based exams is a large part of the mission of the Performance Testing Council, a global organization that’s been advocating for performance-based testing since 2003. PTC has summit meetings twice a year, including Wednesday and Thursday this week at Hotel Park City in the (currently offseason) resort town of Park City, Utah. In a pre-summit interview with Certification Magazine, Performance Testing Council executive director Julie Timmcke said that the meetings are not restricted to PTC members.
“We try to bring people together who are doing interesting things in the industry,” Timmcke said. Groups and individuals make presentations at the summit, but attendees are encouraged to share and discuss their own reactions and ideas. “It’s definitely not a situation where you walk into a room, listen for 45 minutes and then leave,” Timmcke said.
The Performance Testing Council’s most important goal is to facilitate and increase the use of performance-based testing, and not just in IT certification circles. Because there’s an ongoing concern about certification testing, the certification realm is a natural fit for PTC, but Timmcke said that the group embraces many different career and educational realms where testing is integral, including aviation, construction, medicine, and many other disciplines.
Many performance-based tests involved computerized simulations, but Timmcke said that’s far from the only way to measure skills and aptitude. There are some things, after all, that can’t be effectively gauged using a simulation. For example, Timmcke said, tests often feature a “gating item,” an all-or-nothing performance measure that’s essential to the test. If you can’t pass the gating item, then it doesn’t matter how well you do on the rest of the test.
For operators of large-scale construction cranes, an important gating item is what you might call the ability to access the workplace. “If you can’t climb all the way up to the top of the crane,” Timmcke said, “you’re probably not going to be a good crane operator.”
One topic to be featured at the summit is exam security. Timmcke said it’s often assumed that performance-based tests are more secure than traditional multiple choice exams. “The thinking was, ‘If you can do it, you can do it,’ so of course it would be more secure,” she said. Exam cheating has become increasingly sophisticated, however, and organizations that conduct testing have had to increase vigilance to keep pace. One such group, the nonprofit Linux Foundation, recently announced two new certification exams available via online, proctored testing. Linux Foundation representatives will be at the Performance Testing Council summit to share ideas for improving exam security.
The Performance Testing Council has held more than 22 summits since 2005, and while there have been a handful of prior meetings in Park City, the summit events are held in cities across the United States. If you don’t have a travel budget, you can still get a sense of what’s going on: In 2013, PTC launched a series of webinars to further its discussions.