The Ins and Outs of the Beta Test
The first time I participated in the creation of a certification, I thought that publishing and offering the beta exam would be the easiest steps in the whole process. After all, I had survived the creation of the job-task analysis, had fared well enough through the objectives and exam blueprint stage, and had weathered the writing of the exam items. I figured that I could just sail right into the beta exam and then move on to publishing the final version. Pretty quickly, I found that most of the surprises in exam development occur at the beta exam stage. If you take this stage of the process too lightly, you are likely to run into a few nasty surprises. Having a truly successful beta exam required considerably more effort than I expected. So, what does it take to manage the process?
Justifying the Beta Exam: Certification Purists and Business Interests
From a certification purist’s point of view, there are two reasons to have a beta exam:
- To give exam items and forms a “shakedown cruise” before they go live.
- To obtain data that helps psychometricians set a valid cut score, if you use the borderline method of psychometric analysis.
But there is another perspective that you will need to consider when it comes to beta exams.
I was once asked to create a high-stakes certification exam proposal for a large corporation. I proposed using the borderline method and a beta exam to set the cut score. During my presentation, a couple of executives hit me with a flurry of questions. I could tell right away that they were experiencing sticker shock. As you might suspect, their concerns came down to time and money. “Why even have a beta?” I was asked. Here are the reasons:
- Exam quality: A beta exam helps you gather essential feedback from industry professionals about each question. Although subject experts have helped develop items, your exam needs to be seen in the cold light of day. It’s a shakedown cruise, as it were. In short, a beta exam ensures that a company won’t embarrass itself and will be held in high esteem. After the beta exam, all parties can then move forward with confidence, knowing that experts have tested the items.
- Legal defensibility: Certification exams are often used in hiring decisions and can lead to college credit. If someone can argue that your exam was improperly generated, your company may be held liable.
- Academic endorsement: For me, the most exciting growth area in terms of assessment and certification is working with academia. One of the first things academic programs look for when adopting a certification program is some sort of evidence that the program was created properly. The data you get from a beta exam ensures that your exam has the proper pedigree, and your program is likely to receive wider adoption.
Certification purists reading this article might wonder why I didn’t use the phrases “ensure psychometric quality” or “obtain a valid cut score” on this list. One of the reasons is that some psychometric cut-score methods (e.g., modified Angoff and bookmark) don’t absolutely require beta exams. However, the primary reason I didn’t use those phrases is that I’m a recovering certification purist. or perhaps a certification purist who has fallen from the pure faith. I have found over the years that most certification exams exist because they fulfill a particular business need. If you use phrases like “psychometric analysis” and “cut score” in an executive meeting, eyes glaze over and executives begin shifting in their seats almost instantaneously. If you use words such as “profitability,” “endorsement” and “defensibility,” executives readily understand the value of a beta exam.
Identifying the Tasks
Once the project has been approved, you must:
- Set reasonable dates.
- Assemble properly created items.
- Prime the candidate pool and monitor participation.
- Proof the files used for the beta exam.
- Publish the exam.
Setting Reasonable Dates
As you set a date for the beta exam, properly pad each step of the process by a few days. Vacations, holidays and sick days can cause delays. I have found that just one day’s delay in delivering a file can have a ripple effect, causing a significant delay in the release date of the exam. Such delays can be embarrassing at the least, and disastrous at the worst, especially if it means that you can’t get enough people to take the beta exam.
If it takes the test publisher a week to create the beta forms, estimate a week and a half. If you are told that you have a day to proof the exam files, budget three or four days. Don’t forget that it takes time to set up a marketing campaign. Work with your marketing team to set the proper dates. Remember, if things go faster than you expected, you can always slow down. Generally, it is hard to make up for lost days.
Assembling Properly Created Items
Your beta items need to be written as clearly as possible. Like many in the industry, I contract with external subject-matter experts to create exam items. I work hard to make sure that these experts are competent, respected professionals. Still, although these people may know their job roles and Web technologies very well, they often know very little about how to present those ideas in a clear, understandable way.
Before I publish a beta exam form, I have all the items reviewed by a professional editor who also knows the technology reasonably well. This essential step reduces nasty surprises and increases quality.
Once you assemble your edited exam items, you need the following:
- A valid exam blueprint: Your blueprint determines how many items will be written to a particular objective. It also determines how many forms you will write, and how many questions you need. The exam publisher needs this form for the beta.
- At least one-third more items than will be used in the actual exam: Generally, roughly one-third of the items are dropped during the beta exam. Sometimes, items are dropped because they are simply wrong. Other times, items that are technically valid will still get dropped because they were deemed either too easy or too difficult. Remember, the point of a certification exam is to create an industry-verified standard of competency for a job role.
Priming the Candidate Pool and Monitoring Participation
Make sure that you get enough people to take the beta. You want a minimum of 50 candidates to take your beta—200 or more would be best. The typical beta exam period is between three and five weeks. To ensure that you attract enough qualified candidates, consider the following strategies:
- E-mail blasts.
- Telephone calls.
- Announcements on company Web pages.
- Announcements to industry and community Web pages and blog sites.
You want candidates of all skill levels to take the beta exam. A good candidate mix ensures a good variety of statistical data that can be used by psychometricians to determine the cut score. Still, the majority should be knowledgeable enough to have a good chance of passing. Don’t worry if you have overqualified or underqualified candidates taking the exam.
Monitor the exam rate regularly during the beta exam period. Some companies want to provide statistics only on a weekly basis. I have found that obtaining statistics two or three times a week helps ensure sufficient participation.
Proofing the Files Used for the Beta Exam
The exam bank isn’t the only item you will deliver to your exam publisher. If you choose the borderline method to set your cut score, you will need to provide a survey. You may also need to provide disclaimers and security agreements.
Edit and review all submissions before and after you submit them to your test publisher. If your experien