Fairness and Cheating
Fairness is a topic with a long history in testing. Tests are supposed to be fair. They are not supposed to be biased for or against any particular group or individual. For example, if you compared the scores from a test of equally competent males and females, you shouldn’t see higher scores for either group—if the test is fair. If it isn’t fair—and there could be many reasons—then one of the groups on the whole would score better. Some individuals would get a higher score, even though they are not more knowledgeable or experienced. The increased score would have nothing to do with what is being measured by the test, and therefore, the test would be judged as unfair. There are several statistical and analytical procedures that are used to determine fairness.
Typical sources of unfairness are gender, age, race, ethnic background, language capability and disability. Tests are supposed to measure your knowledge, skill, ability and competence.
I’d like to focus the rest of the column on an implication of Item No. 3 in the Code of Fair Testing Practices in Education, and add another source of unfairness, one that is very common in IT certification exams, but has received very little attention.
That source is cheating. Many IT certification tests, in how they are developed and administered, allow cheaters to win, making these IT certification tests unfair. I’ll talk about how this happens next month. But let me continue with my point.
Let’s say that 5,000 people are certified during a period of time for a particular IT certification program. Of those 5,000, let’s assume that 10 percent achieved the certification by cheating (using brain dumps, cheat sheets, electronic devices, a proxy test-taking service, etc.). That’s 500 certified people who appear more competent than they really are. That’s 500 people who receive or keep jobs they don’t deserve, or may even be promoted.
But let’s look at the flipside: There are also 500 competent individuals who are blocked out of those same opportunities. The job market is limited. These people can’t get jobs, because they are already filled by certified incompetents. The competent individuals had their opportunities literally stolen from them by cheaters, brain-dump users and others—individuals getting ahead by climbing over others.
It’s clear what the effect is on certificants, but what is the overall effect on IT certification? Let me suggest the answer by referring to a similar problem in another industry.
I have a son who likes to play online games, and he’s pretty good at it. Diablo II is one of his favorite games. When a game like Diablo II is released, an immediate effort ensues by individuals and companies to create “cheats” for the game. While some cheats are fun and productive and make the game more interesting, others are not. These other cheats are programs that can be run to help a person gain an advantage by stealing from or killing off an opponent outside of the normal rules. For example, one player will spend a great deal of effort to obtain a rare weapon that can then be stolen, and his character can be killed.
Talking about the impact of such cheating on the online gaming industry, Dave Becker writing in Popular Science (August 2002) said that fans complain that a game becomes unplayable once cheat tools become widespread, and that “analysts and industry executives agree that cheating is one of the leading issues likely to limit the growth of online gaming.” He also quotes Mark Jacobs, president of Mythic Entertainment: “You come into a game, you get killed repeatedly because someone is using a cheat weapon—why would you stick around?”
The growth of IT certification has been hit hard by cheaters. Why would you stick around?
It’s just not fair.
David Foster, Ph.D., is president of Caveon (www.caveon.com) and is a member of the International Test Commission, as well as several measurement industry boards.
Code of Fair Testing Practices in Education
In 1994, the Joint Committee on Testing Practices provided specific language called The Code of Fair Testing Practices in Education. Here is part of that Code:
Test developers should strive to make tests that are as fair as possible for test takers of different races, gender, ethnic backgrounds or handicapping conditions. Test developers should:
- Review and revise test questions and related materials to avoid potentially insensitive content or language.
- Investigate the performance of test takers of different races, gender and ethnic backgrounds when samples of sufficient size are available.
- Enact procedures that help to ensure that differences in performance are related primarily to the skills under assessment rather than to irrelevant factors.
- When feasible, make appropriately modified forms of tests or administration procedures available for test takers with handicapping conditions.