High-Stakes Testing: A Behind-the-Scenes Look
Somewhere between 1.5 and 2 million high-stakes IT certification exams are taken each year at more than 5,000 testing centers in more than 130 countries. What goes on behind the scenes when you step through the door at a testing center, whether it’s in Bombay, Beijing or Biloxi, and take an IT certification exam? After reading this article, you may view testing centers with a new appreciation for the global infrastructure that delivers and reports on your certification exam.
Putting the Infrastructure into Perspective
Close to a million people take one or more IT certification exams in an average year. The IT certification testing infrastructure not only delivers, scores and reports, but it also:
- Works with the certification sponsors to add new exams to the system.
- Manages exam revision control.
- Schedules each exam.
- Provides exams in any of more than a dozen designated languages.
- Ensures proper currency transactions.
- Applies vouchers and special promotions.
- Collects demographic and survey information.
- Maintains the security and integrity of the exams.
- Updates testing center software.
Test centers communicate with the hub system at the testing companies multiple times per day to update scheduling, receive downloaded and encrypted exams and upload exam results. These connections result in more than 25,000 network interactions a day and in excess of 750,000 each month from around the world. This global infrastructure is supported by only about 1,000 men and women employed by the testing companies.
Note that these figures reflect IT certification testing for technicians, network and database administrators, programmers and all those on the technical end of IT. If we included other professions and academia, which also use computer-based testing, the numbers would be much greater.
Once Upon a Time
The first significant large-scale commercial use of computer-based testing started in about 1978 when the National Association of Security Dealers introduced its first stockbroker tests on computer. Built on the PLATO computer-based training system owned by Control Data Corp., the testing system was mainframe-based with dumb graphics terminals as the interface devices. With a few small exceptions, between 1978 and 1988 that was about all there was in the fledgling computer-based testing industry.
In 1988, Novell began experiencing explosive business growth and needed to expand its support infrastructure to manage that growth. In response, Control Data worked with Novell to develop the Certified NetWare Engineer certification, a new credential that would ensure that resellers and third-party support organizations were qualified to provide authorized service.
Control Data began delivery of Novell exams at 50 PLATO centers across the United States using its mainframe-based system. Shortly thereafter, Control Data sold the testing business and most of the training business, and the testing industry as we know it today began to evolve.
The Backbone of Test Delivery
The first generation of computer-based testing delivery, from 1978 through 1988, used mainframes connected interactively with graphics terminals. The second generation developed for Novell and the rest of the IT industry used personal computers that dialed into a hub system via modem to download exams and upload results. Tests were delivered in a stand-alone mode, without relying on the hub connection. This stage of the industry had centralized scheduling functions and downloading of exams to the testing centers.
The third generation came in the late 1990s. The big step forward for generation three was using the Internet as the means of communication between the hub and the testing centers. That decision to use the Internet seems obvious today, but at the time it was an innovative and risky move.
Secure Internet connections gave the computer-testing system much more flexibility. Testing centers could use their existing Internet connections to manage test delivery without the cost and complexity of a special-purpose dial-up connection. Candidates benefited as well: Tests could now be scheduled via the Web, which was convenient and fast.
The third generation was also designed to empower the testing centers. Registration software allowed them to register their students right from the testing site, giving them the opportunity for after-class and same-day scheduling and testing.
The testing centers were also given the tools to manage their own resources, hours of operation and personnel. For example, if a center wanted to open on a Saturday because it scheduled a large group of students for a number of exams, the center had the software tools available to amend its hours of operation and run the exams. This wasn’t an easy option in generation two. Centers could also, at peak testing periods, easily turn entire classrooms into temporary testing centers.
The Three-Quarters You Don’t See
You schedule the exam, arrive at the testing center, sweat out the exam and, with knowledge, mastery and hard work, walk out having earned your certificate. This is the one-quarter of the process that is visible to the IT candidate. It’s the three-quarters that you don’t see that manages the exams, protects your results and maintains the integrity of the certification. Here are some of the things going on behind the scenes.
The process begins with the certification sponsors–Microsoft, Cisco, CompTIA, Novell and hundreds of others–developing the exams and sending them to the testing companies. Once received, the exams are “compiled” to a deliverable format, encrypted and placed on a hub system where they are ready for downloading to the testing center. At the same time, the testing company defines such exam parameters as:
- Languages in which the exam is offered—currently more than 15 used.
- Pricing, which might be a single worldwide price or unique country-based pricing.
- How long a candidate has to take the exam.
- Retake policies.
- Vouchers and discounts offered in connection with the exam.
- Demographic information collected from the candidates.
Each time a candidate registers for an exam, the testing center not only locks in the time, date, location and exam requested, but it also determines or assigns a unique identifier to the candidate. The identifier helps to ensure that all of the candidate’s certifications with a particular sponsor are recognized as being from the same person. This helps eliminate duplicate records and is especially helpful when multiple exams add up to a master certification like the Microsoft Certified Systems Engineer (MCSE), Cisco Certified Network Professional (CCNP) or Master Certified Novell Engineer (MCNE).
The encrypted exam file is sent to the testing center a few days before the exam is to be delivered. We check and then double-check that the proper exam has been downloaded to the testing center prior to the candidate’s arrival. A late revision by a client might result in an extra download shortly before the candidate arrives.
As the candidate takes the exam, the test-delivery software decrypts one exam question–one item–at a time; there is never more than one item in clear text at a time. A test file usually contains several times the number of items that will actually be presented to any one candidate. The test-delivery software can support a number of item presentation strategies. It may, for example, randomly select questions from an item pool and then deliver them sequentially, or it may randomly order the items that are delivered. There are also more advanced strategies, such as adaptive testing, in which each item is selecte