Calculate the Bottom Line to Understand Risk
Do you know your bottom line? What’s the amount you invested in the creation of your program’s certification or licensure exam? It is important to calculate this cost so that you can determine the amount that should be spent to protect the investment. If you don’t protect your investment and your exam content is stolen, you also need to be able to document the cost of development if you want to seek damages.
Let me walk you through the key exam development phases and what costs should be included, along with a few estimates of the actual cost. For our purposes here, all figures are based on the following assumptions about the exam:
- High-stakes certification or licensure exam.
- New exam, not a replacement.
- Standard exam development processes.
- Multiple-choice question formats.
- Initial release of the exam meets industry standards for reliability and validity.
Content Analysis: The first step in the creation of a new certification or licensure program is to conduct a job analysis to determine the knowledge, skills and abilities that should be measured. The key factors that impact cost are as follows:
- Size of the domain to be measured (narrow versus complex).
- Newness of the domain (new versus well-established job roles).
- Availability of people that actually perform the job role (also known as subject-matter experts).
If the domain is very narrowly defined and many people have been practicing in the field for a number of years, a project of this nature may cost as little as $25,000. The price rises to $100,000 or more as the complexity of the domain increases. Once the job analysis is complete, however, it can serve as the basis for a certification or licensure program for many years. In some cases where multiple exams are required to complete a program, individual exam blueprints also may need to be developed at an additional cost of $5,000 or more per exam.
Item Authoring: Once the content domain is defined, it’s time to create the raw test questions. Subject-matter experts (SMEs) are either hired to perform this task, or volunteers are recruited. Often, hired SMEs also receive training on how to write good test questions. If volunteer SMEs are utilized, then professional test developers are paid to transform the questions. Either way, you can estimate that the cost per item is $600 or more per question during this phase of development. If more advanced question formats are used, the cost increases.
Item Review: Now it’s time to bring in a new set of SMEs and your editing team to verify the accuracy and clarity of the test questions. You can estimate another $100 to $150 per test question for this phase of development.
Field Test Administration: Once the test questions are complete, it’s time to determine which questions will stand up to rigorous statistical analyses to make the final cut for the exam. Depending on the program, SMEs may or may not be compensated for their role as a “beta” examinee during this phase. You also need to include the cost of administering the exam. For a computer-based test (CBT), this fee would likely be based on the number of testing events. In the case of a paper-and-pencil exam, the fee would include distribution and test administration costs.
Exam Production and Release: Whether CBT or paper-and-pencil, there are costs associated with putting the exam into operational or production mode and disseminating it per program specifications. This cost varies depending on factors such as the number of test questions, the complexity of the questions and the test format (fixed-form versus computer-adaptive).
Overhead Costs: Don’t forget to include the cost of the program staff when calculating your bottom line. Estimate the time spent by each staff member on exam development and prorate accordingly.
What does our calculator show? Bottom line, industry experts estimate that the average cost for a single test question these days is around $1,000.
Now you know how expensive it is to develop an exam (from a low of $80,000 to hundreds of thousands of dollars). You also need to consider projected revenue from the exam, the estimated shelf life and the cost of redevelopment of content if a breach occurs. You make the call. What are you willing to pay to protect your exam content?
Cyndy Fitzgerald, Ph.D., is co-founder and senior security director at Caveon Test Security (www.caveon.com) and is a member of Association of Test Publishers. Address any test security questions or recommendations to Cyndy via e-mail at email@example.com.