Raising the Bar With Performance Testing
The IT certification industry has experienced a significant downturn over the past three to four years. An IT certification was once a credential that enabled individuals to get a job, change careers or enhance an existing career. Now we hear plenty about the declining value and credibility of certification, and prospective candidates wonder whether exams are worth their time and money. However, not all is lost—performance testing can help raise the bar.
Traditionally, certification exams have fallen short as true measurements of an individual’s ability to perform a task. Instead they are memory comprehension exercises. A high percentage of candidates who prepare for certification employ self-study material that does not include any “hands-on” component. Unfortunately, those candidates who don’t consider integrity and honesty an issue can purchase actual exam questions from a plethora of test-stealing renegades. The end result is that someone can prepare to pass a certification exam without ever having touched the product or performed a task. The time has come for certification sponsors to step up to the plate and ensure candidates receive credentials that are valued by their professional colleagues, hiring managers and themselves.
Performance testing takes certification to a new level by moving away from standard multiple-choice questions to situations that measure ability by requiring completion of a task. If a candidate reads through an exam question and at the end is confronted with four or five choices, it is not a performance-based question. On the other hand, if a candidate is required to analyze a problem or situation, determine the course of action, select the appropriate tool (or tools) and perform the steps to solve the problem, we have a performance-based situation. Performance tests measure the understanding of the big picture and the ability to proactively prevent problems from occurring, and encapsulate the testing of knowledge, skills and experience.
In a recent performance test that I deployed, candidates were tested on their ability to recover from a database failure without a loss of data. To complete this task successfully, the candidate had to know how to configure his environment to ensure there was no data loss in the event of a failure. To recover the database, the candidate had to determine the appropriate course of action, select the tool to use and verify the recovery. Throughout the test event a candidate could encounter any one of a multitude of failure scenarios at any given time. Therefore, the experience is actually testing more than the ability to recover the database. It’s about testing the candidate’s ability to actively manage the environment and to be prepared for a failure.
Building a performance test is not an easy task and is challenging from a development-cost-and-scalability perspective. Live application performance testing is only one method of testing for performance. Performance-based testing also comprises the use of simulations, emulations and other innovative testing methodologies that test for higher-order cognitive thinking, skills demonstration and competency. Integrating performance testing into certification paths is a task that certification sponsors must undertake in order to positively effect the value and credibility of certification. Certification candidates continually ask for relevant, “real world” tests. New people coming into the industry want to ensure the time and money they invest will pay off. Existing certified professionals want avenues to help differentiate themselves from other certification holders. Hiring managers want a high level of confidence that prospective employees have demonstrated technical skills.
Certification sponsors understand the increasing need for performance testing as a means to increase the value of certification. As a testimony to the interest in performance testing, a consortium of industry leaders in certification recently formed the Performance Testing Council to help promote the design, development and advancement of performance testing. A number of certification sponsors now have high-end, live application performance tests. The development and deployment of simulations in high-volume exams is beginning to appear in a number of certification programs. Certification sponsors and other leaders in the industry understand the important role that performance testing plays in revitalizing the value of certification.
The certification world has changed. A robust credential does not have to be all about performance testing. However, a robust, valuable and credible certification should at some point employ methods to ensure candidates have demonstrated their ability to perform relevant job tasks. Performance testing helps raise the bar and is a service to the certified professional who holds the credential.
James A. DiIanni is the director of assessment and certification exam development at Microsoft Learning and supports the Microsoft Certified Professional program. His experience with performance testing started in 1986 developing simulators for the U.S. Navy, and he has been involved in the IT certification industry since 1997.