Live Application Performance Tests
The Information Technology (IT) certification industry has a number of high-end live application performance tests. Many of these tests are very rigorous and challenge candidates by requiring them to complete complex tasks. The structure of these performance tests incorporates the use of knowledge, technical skills, experience and the application of best practices into the testing event. The pass rate for first-time test-takers is often low, and the investment that candidates make is high in terms of time and money.
A candidate who successfully passes a live application performance test gains the notoriety of being an expert in his or her profession. Passing a live performance test is a testimony to having demonstrated key competencies associated with the job role. Consequently, integrating a live application performance test into a certification program helps raise the value and the credibility of a certification credential.
Making the decision to design and build a live application performance requires numerous considerations by the test sponsor. A few of these factors include: the cost to build, maintain and deliver, scalability, operations, security, scoring, tracking, design, localization and pricing. This is not an exhaustive list, but I’ll briefly address some of these factors in this column.
The task of moving into live application performance testing can be an overwhelming endeavor for a test sponsor. However, the return on the investment can boost the value of a certification program. And live application performance testing is not limited to the IT certification industry. It has application in any situation where candidates are required to demonstrate their competency at performing tasks that are representative of the real world in which they will apply them.
Regardless of all the factors that must be considered, my first recommendation for moving forward with live application performance testing is to ensure that adequate resources for time and labor are allocated to the project. In the past, I’ve seen situations where test sponsors approach the development of a live application performance test in the same manner they use to develop standard knowledge-based tests. There is not a one-to-one ratio in costs, and while I don’t have specific numbers, the development of live application performance tests is considerably more expensive and time-consuming than knowledge-based exams. The overall success of the effort is dependent upon having sufficient and dedicated resources. Coming out of the gate with a half-baked, low-quality test is not the way to introduce a high-end test in the market. Remember that a live application performance test is an investment in a certification program. Therefore, it takes diligence to plan and manage the process properly.
My second recommendation is to consider how you want to position the live application performance test in your program. If you have a program with a large number of candidates, you may end up targeting only a small percentage of those candidates. Some factors here include price, availability, candidate preparedness and delivery. Not all candidates will be willing to pay a considerably higher price to take a live application performance test. Availability may be limited because delivery of a live application performance test may be limited to specific geographic sites. Candidates will have to make more of an investment in their preparation and, in some cases, will have to set up a practice environment that is representative of the testing environment. Delivery will require additional resources because, unlike standard knowledge-based tests, there is often a limited delivery channel with live application performance tests. Therefore, ask yourself whether you are in this to generate revenue or to recover your costs in order to provide a high-end credential for your candidates and subsequently add value to your program.
Demands on the certification industry continue to increase, particularly with IT certification programs, where we’ve seen the value and credibility of programs decline year-over-year for the last four years. There is also a much higher demand to ensure that certification programs do in fact test for candidate’s ability to perform tasks. Live application performance testing is one avenue toward increasing the value and credibility of credentials and validating candidates’ competencies. Making the decision to move forward with a live application performance test is the first step in the process. I’ll discuss more about the details of the various factors I mentioned in future columns.
James A. DiIanni is the director of assessment and certification exam development at Microsoft Learning and supports the Microsoft Certified Professional program. His experience with performance testing started in 1986 developing simulators for the U.S. Navy, and he has been involved in the IT certification industry since 1997. He can be reached at email@example.com.