The Price of IT Certification Testing
At the 2004 conference for the Association of Test Publishers, representatives from the American Institute of Certified Public Accountants (AICPA) described how they had spent $20 million to create a new certification test. Some of that money was allocated to convert the test from a paper-and-pencil format to a computerized format, while some was budgeted to create performance-based questions involving computer simulations and accounting tasks.
Regardless of such innovations, creating new, high-stakes tests is expensive. Certification tests, including those for IT, are no different. After all is said and done, a typical IT certification test can easily hit the $100,000 mark, and a few enhancements can push it up to $500,000 or more.
A select few IT certification tests out there have required millions of dollars to build and maintain. How do testing programs spend so much money? Why can’t the tests be created more inexpensively? These are both good questions.
- Content Analysis: Testing programs conduct extensive research to decide which knowledge and skills are important and should be measured. Specifically, these programs bring together subject-matter experts (SMEs) to determine what is important for the performance of IT jobs. After an extensive discussion and review, an outline of the test’s content is determined. In some cases, a program can save money by omitting this step, but the authoring of test questions will be somewhat unorganized. Estimated cost: $20,000 to $40,000.
- Item Authoring: Once the content domains or testing objectives have been identified, another group of SMEs is asked to write corresponding test questions. These people need training, as well as mentoring, as they attempt to produce test questions according to specific rules. If they are authoring multiple-choice questions, each SME may be able to write only 10 to 20 per day. Three hundred questions might be the typical goal for a test, so the numbers add up quickly—and don’t forget travel and other expenses. Estimated cost: $30,000 to $50,000.
- Item Review: Once drafts of test questions are produced, the SMEs and expert editors have their turn at them in order to make them more understandable and accurate. Estimated cost: $10,000 to $15,000.
- Field Test: At the conclusion of the authoring and review processes, questions are beta-tested, or field-tested, using real certification candidates to ensure that they work as intended. The test results are then analyzed statistically, eliminating questions that have not performed well, generating enough information to create the final test and determining the score needed to pass the test. Ordinarily, enough volunteers agree to take a beta test to provide the needed information, but the testing program still has to pay for each administration of the beta test, and the cost is added to the price tag of the statistical analyses. Estimated cost: $15,000 to $20,000.
- Test Publication: Organizations that administer the test for the certification program typically charge to build and verify the final version of the test, and to publish it to testing centers worldwide. Estimated cost: $5,000 to $10,000.
The total estimate of the typical test-creation process from these areas is $80,000 to $135,000, not including many indirect and hidden costs. Some examples of these include salaries and overhead, security, equipment, travel expenses, certification program management and translation costs. Security expenses alone should probably account for 5 percent to 10 percent of the total program budget. If we assume conservatively that this category of hidden and indirect costs represents 25 percent additional expense, we have a grand total of $100,000 to $170,000 per test.
The important conclusion that can be drawn from this cost analysis is that properly built IT certification tests are expensive—and for good reason. They show a commitment by the producing organization or company to support a certification program that meets the test-takers’ needs. As a certification candidate, you now can better understand why tests cost what they do, why it is important for testing programs to recoup expenses through testing fees and why programs are protecting their investment with stronger security procedures.
David Foster, Ph.D., is president of Caveon (www.caveon.com) and is a member of the International Test Commission, as well as several measurement industry boards. He can be reached at firstname.lastname@example.org.