Building a better mousetrap: The science of certification exams

Posted on
Share on Google+Share on LinkedInShare on FacebookShare on RedditTweet about this on TwitterEmail this to someone

Certification is hard, man.Certification can help enhance your skills and increase your earning potential. If you want to understand networking, or security, or a specific vendor’s software more completely than you already do, or if you’re in line for a promotion or considering a new job (hopefully with a better salary), then certification can make a dramatic difference.

Before any of that can happen, however, you have to study for, and pass, a certification exam. And certification exams probably aren’t quite like any other test you’ve taken. You’ll probably be better prepared, and achieve a better performance, if you have a better understanding of how certification exams are created and what makes them work.

If you think back to your days in school (for some of you, that might not be too far back), you undoubtedly have fond (or not so fond) memories of endless tests — quizzes, midterms, finals. How do tests that a teacher might write for a class differ from a certification test?

Certification tests differ from tests created by teachers in three main ways:

Ÿ● Development process

Ÿ● Statistical analysis

Ÿ● Intended use

Development process

Certification tests differ from other tests commonly encountered in classrooms by the way they are made. In a classroom setting, the teacher is typically in charge of determining what to test, then sits down to write questions. In some cases, a group of teachers who have similar classes might work together to come up with a standard test used for all classes.

Certification tests are typically developed following a more rigorous process. Individual test questions are written, reviewed and quite often reviewed again. Processes are followed not only in the writing of questions, but also in early stages of test construction, where it is decided what needs to be tested. An individual teacher working alone might sometimes construct a test of very high quality. Following the test development process, however, provides consistently high-caliber results, relying less on the skill of a single individual to determine the overall exam quality.

Statistical analysis

Certification tests undergo extensive statistical analysis. This analysis looks at not only the quality of individual questions, but also the entire test as a measurement instrument. Statistical analysis for tests typically focuses on looking for evidence that the exam has two important qualities:

ŸThe exam is valid: Test validity means that the test actually measures what it is supposed to measure. Poorly written questions, questions that are not applicable to the content area, or questions that score incorrectly negatively affect the validity of a test. Statistical analysis helps to identify questions that do not accurately assess the target skills.

ŸThe exam is reliable: Reliability means that the test will yield consistent results. For example, test scores should be accurate regardless of the time of day or where the test is taken. Results should be consistent for all individuals who possess the target skills, regardless of race, gender, or other non-essential characteristics.

Statistics used by teachers on exams might include scoring the exam, looking at the average score on the exam in the class, or possibly looking at individual questions upon occasion to throw out a badly written question. At times, teachers might “grade on a curve”, meaning they adjust the passing rate based on how well the entire class did on a test. The need to adjust the passing score on the test based on class performance suggests that the test itself might not be an accurate measure of what was taught or what should have been learned, or that the content covered in class did not match well what was being tested.

Intended use

Tests used in a traditional classroom are typically given to measure how well an individual remembers or can use what was taught in class. These tests are often content mastery checks, evaluating retention, familiarity, or mastery of specific content. Tests apply directly to what was taught in class, with many teachers often adjusting tests in subsequent semesters based on what content was covered.

The purpose of a certification test is to evaluate individual performance of a specific set of job skills, concepts and tasks. While classroom tests typically measure how well someone memorized the material, certification tests compare skills and abilities to a predefined standard or against a consistent set of objectives. Classroom tests are meant to apply only to members within the class, but certification tests can be used to measure anyone’s fluenct with respect to the target skills and abilities. Certification tests are often used as national or international measurements.

We’ve compared certification tests to tests that might be taken in a school setting, now let’s define a certification in the context of two things a certification is not — certificates and badges.


In an educational setting, a certificate is something that provides evidence of successful completion of a course of study. The requirements for obtaining the certificate vary depending on the organization that grants the certificate — the only requirement might be as simple as attending all sessions of the training, or it might include doing assignments or tests with scores at a “passing” level.

Certificates might be granted for a single class or course, or might require completion of multiple classes that together make up a program of study. In higher education, certificates are similar to diplomas or degrees, typically requiring a year or less of study to complete.

Certifications differ from certificates in that the certification is independent from the training. Certificates are tied to a specific course of study offered by a school or an organization. By contrast, a certification is only concerned with measuring knowledge, skill and ability, regardless of the source of training (if any). Based on international standards, certifications are not to include any specific course of study as a requirement to obtaining the certification.


Badges are recognitions of skill, achievement, or accomplishment. Badges might be awarded for having passed a class or course of study, for having mastered a skill, or even for having passed a test. But badges can also be awarded for reasons as varied as having an interest in a specific subject, for attendance and effort, or even as a recognition of attitudes or commitment. The use of badges has become more popular as a means of not only recognizing accomplishment but also a method of motivation (i.e. as students are awarded badges for small accomplishments, they might be more motivated to continue their efforts in order to attain more recognition).

Badges can be used to identify competency on one or multiple skills. For example, instead of granting a certification, an organization could simply give a badge that identifies a similar range of skills. Badges can also be useful in identifying abilities on a more granular level than a certification is capable of doing. For example, rather than certifying that an individual is a qualified network administrator, individual badges can identify skill levels in specific tasks such as cable management, switch administration, DHCP configuration, etc.


The purpose of a certification is to validate knowledge, skills, and abilities, typically in a work-related set of tasks. Certification exams are carefully designed to be reliable and valid measures of these skills. While tests, certificates, and even badges can give some indication of one’s educational or work background, certifications are meant to be a higher measurement of specific skills used on the job.

Share on Google+Share on LinkedInShare on FacebookShare on RedditTweet about this on TwitterEmail this to someone
Don Whitnah


Don Whitnah is vice president of research and development for TestOut Corporation and a veteran of the certification industry. Among the many certifications he has earned are CNA, CNE, MCP, MCTS, MCITP Server and Enterprise Administrator, A+, Network+, Security+, CCNA and CISSP.

Posted in Certification|