Find and Fix Now: A Future Look
A long time ago when I was a graduate student, and about 100 miles from Phoenix where a conference was being held, the engine-temperature-warning light on my friend’s car came on. We immediately pulled over and found steam coming from the radiator. We were towed into the nearest town and spent the rest of the day and part of the next getting the thermostat and hoses replaced. We were on our way within 24 hours and were able to attend most of the conference.
What would have happened without the engine-monitoring-and-warning system? Probably the engine would have overheated to the point of destroying more costly engine parts, resulting in a much longer repair period and much more expense. And we would have missed the conference.
Monitoring of important systems is common and sensible in our world. We use thermostats to monitor the heat or cold in our home. Some teachers send grade reports home to help us monitor our children’s progress in school. The TV meteorologist lets us know how the weather is so we can decide what to wear and how to plan our day. And traffic reports let us know which routes to avoid. Continuous monitoring leads to quicker and more accurate decision-making, better safety and efficient use of time and money.
However, things are different in the world of testing, even with the use of the Internet to register and pay for tests, wonderful computers to develop and administer them and state-of-the-art Web-based communication systems to retrieve test results. Even though tests can be taken “on demand” and are available continuously for months and years, as a certification industry, we have no mechanism for monitoring their continuous quality or effectiveness. Therefore, we obviously have no way of responding in case of an emergency.
Well, maybe that’s OK. Maybe nothing can go wrong, and therefore there is no need to monitor the test or put systems in place to fix problems. Right! And, of course, nothing ever goes wrong with electrical grids, the economy, foreign relations or marriages either. Ouch! That’s too much sarcasm even for me. So let me tell you straight out what could (and usually does) go wrong with tests. Here are five things that I’ve experienced:
- A parameter of a test could be inadvertently set wrong from the first day of the test’s release, such as the score needed to pass or fail the test. Such errors are not uncommon and have drastic effects.
- In IT certification tests, technology may change in the real world, rendering entire sections of the test immediately inaccurate.
- The makeup of the population of test-takers changes quickly over time due to training, program decisions, marketing promotions and who knows what else. The characteristics of the test were developed using a particular audience to validate it. The test probably will not work well for a different audience.
- The theft of the test questions, and their subsequent exposure to others, would certainly cause a dramatic shift in the test’s performance.
- Finally, tests undergo daily wear and tear, just like car engines. By being taken by hundreds and thousands of test-takers over time, the test naturally becomes overexposed and worn-out. It doesn’t perform as it did when brand-new.
Each of these, and other reasons, require that tests be monitored on a daily basis to make sure that they haven’t changed—that they still perform as they have been designed. Unfortunately, such a system is not in place today.
Nor is there a system that would allow a reasonably quick fix. Today, if a test is discovered to have a problem, it may take several weeks or months to fix it and another several weeks or months to simply replace the broken one. If auto repair operated this way, I would probably still be in Phoenix.
Things have to change. So I predict that things are going to change. Using what we have more effectively, we will begin to monitor our tests more closely. New statistical tools will help to detect changes in performance of individual questions and the test overall. Reserve pools of questions will be created and held to be able to immediately replace poorly performing questions. Technology will help us collect and analyze test results at lightning speed. And test-publishing systems will respond to the need with immediate publication.
These things are possible and necessary. The auto mechanics know this. What’s wrong with us?
David Foster, Ph.D., is president of Caveon (www.caveon.com) and is a member of the International Test Commission, as well as several measurement industry boards.