DELIVERY: CBT Offers More Delivery Benefits
Paper-and-pencil tests are very common in state, national and international organizations, but computer-based testing (CBT) is gaining ground quickly. David Meissner, executive director of Testing System Strategy, Thomson Prometric, has written a white paper titled “Conversion: The Benefits and Best Practices of Computer-Based Testing” to help organizations understand the benefits and importance of converting to a computer-based test-taking format.
In the past 10 of his 18 years of testing experience, Meissner said that discussions have shifted between now outmoded worries over usability and comparability between the two modalities to more sophisticated discussions on how to best leverage the computer environment. “There’s issues of security,” Meissner said. “Whenever you’re dealing with paper, whether it’s paper testing or paper in an office, there’s the concern that you may lose the papers or they may get stolen or misplaced. There are also issues of readiness. Candidates can potentially gain access to content when testing in a large hall with hundreds of people testing simultaneously in a paper-and-pencil environment. The other piece is, at the heart of any testing program is the desire to come up with valid and reliable measurement about a particular candidate’s knowledge, skills and abilities. So there’s a great deal of interest now around what are the testing methodologies, what are the approaches that we can use, what’s the navigation that we can employ to not only minimize security but have the items be much more engaging, much more representative of real world activities.”
There are numerous benefits for computer-based technology testing delivery, including data-rich test results, increased security and enhanced navigation and presentation capabilities. “In the past in a paper-and-pencil environment, at best you’d know whether the candidate got the item right or wrong,” Meissner said. “In a computer-based environment you have the ability to see exactly how long they spent on each item, so you’re able to make judgments about time-on-task activities, as well as use that information as a way of looking at trends to see if there’s undue item exposure, which might prompt candidates to respond more quickly than they would otherwise to items. Also, if a set of candidates routinely take longer than you expected to answer an item, that might indicate that either the item itself is misleading or answer options are cumbersome or difficult to understand.”
Also, in a CBT format there’s a great deal of information you can gather on traditional item types that isn’t visible in a paper-and-pencil-based environment, such as the ability to capture essay content, simulate activities such as word processor functionality or simulate the interaction of a candidate with a tool used on the job. “You can capture each and every keystroke, so you’re able to not only get information about that candidate’s specific activity, but you can collect that information for longitudinal analysis, look at trends and do a bit of forecasting that you wouldn’t be able to do in a paper-and-pencil environment,” Meissner said.
Security is more controllable in a CBT environment. Paper-and-pencil testing offers many potential points of failure: test books and materials often are printed outside the testing agency; independent organizations such as Federal Express or a bonded courier deliver materials to testing locations; on-site security guards keep an eye on the materials before and after testing delivery; part-time proctors are typically hired to monitor testing activity. Then, all of the same steps are taken to distribute the materials back to the test sponsor, creating lots of opportunities to gain unauthorized access to the content, Meissner said.
In a CBT environment, test data is transmitted electronically, often fragmented and encrypted so that even if someone gains access to it they would have to spend time and money deciphering the content. Dedicated, full-time professional proctors who have a high level of accountability and are familiar with potential security problems also can help combat breaches, as does material presented in an electronic format one item at a time so the entire test or significant pieces of the test are not available at once. Additionally, the ratio of candidates to proctors in a CBT environment is typically much lower than in a paper-and-pencil environment. A CBT environment may have dozens of different testing programs running at once, so an opportunity to look at your neighbor’s material is pointless.
Navigation and presentation also are enhanced in a CBT environment. In paper-and-pencil tests, every item is presented in exactly the same sequence for a particular form of the test. In a CBT environment, you can present items in a randomized presentation. “You have the ability to look at more advanced delivery mechanisms like linear on-the-fly, which focuses on testlets,” Meissner said. “I might be presented with one subset of items, whereas you would be presented with a different one. Or there are various forms of adaptive testing in which the item I am presented is driven by how I responded to previous items I was exposed to. Only if someone performed in exactly the same way as me would you have the potential to have the same content.”
The only way to enjoy that level of security with paper-and-pencil testing would be to have everyone take a different test, which is not cost-effective and creates logistical concerns. CBT is usually more cost-effective because of the number of processes that have to be performed with paper tests. Paper has a lot of upfront costs: printing the booklets, score sheets, distribution of materials, security put in place to monitor materials before and after test event, etc.
“One of the things that I think is often underestimated is the degree to which the test sponsors can leverage what’s been done by their predecessors,” Meissner said. “In 2004, our company alone has converted well over 100 major testing programs and certainly far more of the smaller programs, from paper-and-pencil to CBT. We find that there’s typically two reasons why a client or test sponsor would be interested in converting. They’re either looking to take advantage of some of the security and delivery benefits associated with CBT, or they’re planning a substantial change to their testing program and they want to take that opportunity to move the program from paper-and-pencil to computer. They don’t want to spend a lot of time and money coming up with new content and methodologies if they stay within the older paper-and-pencil structure.”
CBT also offers significant time savings. Candidates have to show up at a paper-and-pencil facility up to an hour and a half in advance, and there are mandatory breaks and lunches to prevent people going in and out of the testing room at different times. In a CBT environment, conditions are much more fluid, offering more streamlined, automated processes. A candidate can opt to take breaks or shorten lunch, and doesn’t have to show up at the site so far in advance. CBT testing also can be offered every day, which gives candidates greater flexibility since they don’t have to wait months for the next testing opportunity, as is frequently the case in a paper-and-pencil testing environment. And it takes far less time to get the scores. Paper test scores take as long as six to 12 weeks to process. CBT scores can be given immediately.
Kellye Whitney is associate editor for Certification Magazine. E-mail Kellye at firstname.lastname@example.org.