Performance Testing: The Next Generation
Skeptics claim that performance-based tests are too expensive, lack scalability and are simply too complicated to run in a testing environment. So why are performance tests gaining ground? Because they provide an elevated value proposition, offer real-world advantages and reduce opportunities to cheat. Even studying with actual equipment—the best preparation method—is good for everyone involved. As certification candidates tinker with live or simulated systems, pulling them apart and putting them back together like some great electronic puzzle, they soak up the skills that will enable them to function capably at work.
Performance tests measure skills using real or simulated equipment. A successful candidate can’t fulfill one task to the detriment of the entire system and pass. Who would you rather hire? Someone whose credential measured competence with the actual equipment and software used in your company, or someone who passed by guessing correctly via the process of elimination?
Performance Test Pioneers: Cisco & Red Hat
Red Hat has been using performance-based training and certification since 1999 when the Red Hat Certified Engineer (RHCE) program began. The test is always delivered on a live system, and individuals who successfully earn their credentials have proof of competency that they can perform job tasks in a live environment. After the exam, the system is tested and measured to see whether the assigned tasks were executed correctly. If the system performs as specified, the candidate passes. If it doesn’t, better luck after more study.
“Certification went through a bad period in the late ’90s and is still not fully recovered because many individuals and companies invested in multiple-choice certifications and didn’t get the result,” said Peter Childers, vice president of global learning services for Red Hat. “They didn’t get a proof of competency. Companies that want to restore value to certification are going to add higher-fidelity modes of testing. Simulations are among them, but the absolute highest fidelity that you can get is to have the test subject execute job tasks in as realistic as possible a scenario, and live systems can’t be beat. Performance-based approaches are growing despite resistance from some in the testing and training industry who still believe that these approaches are harder to bring to market—that they can’t be profitable or scalable. Red Hat has really proved those skeptics wrong in what we’ve been doing for the last six years.
“Only a few companies have led the way with performance testing,” Childers added. “One of the most notable is Cisco with their Cisco Certified Internetwork Expert (CCIE). That has been performance-based on a live system since it was introduced in 1990 or thereabouts, and CCIE is regarded as one of the highest-value certifications in IT.”
Those who hold the CCIE credential are capable of handling some of the most challenging jobs in the IT field. In fact, the credential is so difficult that less than 3 percent of Cisco-certified professionals become CCIEs. That said, it doesn’t quite make sense that a testing center, employer or other interested party might think that performance-based testing does not pose advantages great enough to outweigh any additional expenses—not when you look at the big picture. Even in the short term, training takes place on the real equipment. If you have a training room equipped to teach someone, chances are high that there is equipment available to use for testing as well, Childers said. “It’s not always the case. Sometimes you need higher quality or more controlled quality of equipment to run tests. You need it in a secure environment, but on the face of it, there’s no reason why it shouldn’t be profitable.”
Profit isn’t really the point, anyway. Certification was originally touted as a way for a third party to assess a potential employee’s skill and value in the workplace. Performance tests on live systems are ideally suited to diagnostics and troubleshooting, teaching and testing critical skills such as the ability to recover malfunctioning systems. “It’s almost impossible to test that in the multiple-choice arena,” Childers said. “Testing on live systems can be a learning experience. It’s hard to learn from a multiple-choice test experience because you don’t always come away with a clear idea of what you know or don’t know, your weaknesses and strengths. A multiple-choice test is a knowledge-based test. If you only need to test knowledge—and there are many professions where that is the primary need—then that’s fine. Multiple-choice testing has its place. We use it as a pre-assessment tool for our training courses. It’s online. It’s automated. But, its purpose is simply to provide an individual with an indicator of the proper course for them to start training based on their experience.”
Performance testing is suitable for the higher-value job roles and high-value credentials, Childers said. This makes sense, since a host of professionals, such as doctors, pilots, military and other fields where life or death is measured by a person’s skill and responsiveness, actively use simulations for training. An RHCE certification holder just might save the company thousands of hours and dollars by leveraging his skill and savvy on a recalcitrant operating system used daily throughout the organization. “There’s no division between the activities that will get you a pass on the exam and the activities that will make you valuable to your employer,” Childers said. “It’s unimpeachable. Performance testing is only considered unusual in IT. It’s not unusual in many other well-established professions. It is the most valuable and meaningful form of testing. If an IT company can possibly do it, they should. It’s beneficial for their customers.”
All of Red Hat’s credentials, the RHCE, the Red Hat Certified Technician (RHCT) and the Red Hat Certified Architect (RHCA), are performance tests. The RHCA is the highest level of certification the company offers, with curriculum based on storage management, directory services, performance tuning and software management across the enterprise. A candidate must pass five endorsement exams to earn this credential.
Practicum: Novell & Oracle
Novell offers several highly valued credentials that are respected across the IT industry. The Novell Practicum exam is built on a series of open-source technologies that also leverage virtual machine technology to provide a proprietary practical assessment of Novell technologies and services. Novell’s customers have access to the Practicum exam directly, as well as through authorized training partners. Customers gain access to the exam via a browser and a PC configuration with a minimum set of requirements that are communicated and identified for the test provider. Customers then are confronted with a minimum of two hours of real-world scenarios in which they are asked to perform a series of tasks on a network or network services. As they perform the tasks, the Practicum infrastructure keeps track of all steps, choices and selections to ensure that test-takers don’t jeopardize another part of the system. At the end of the exam, a full health check is performed incorporating the scenarios and challenges presented, and the test-taker is awarded a pass or fail grade.
“While the student is there behind the engine or under the hood, you might say they’re accessing Novell directly,” said Daniel Veitkus, vice president of worldwide training services for Novell. “Our network operations where we herd the series of machines and server farms clustered together serves up a variety of scenarios, each one being different. We have a pass rate of roughly 40 percent, which is about where we want it for this type of exam. We expect it to be because pass rates for traditional exams that have the advantage of test preps, cheat sites, etc., ar