This feature first appeared in the Spring 2018 issue of Certification Magazine. Click here to get your own print or digital copy.

In a quiet announcement last year, cybersecurity industry association (ISC)² made a major change to its long-established flagship testing program. Starting in December of 2017, the Certified Information Systems Security Professional (CISSP) exam moved to a new exam format: adaptive testing. This change took effect in December for all test-takers sitting for the English-language version of the CISSP exam.

Social media immediately lit up with concern, as test-takers tried to figure out what effect the announcement would have on their test-taking experience, as well as how the changes to the CISSP exam would affect the difficulty and content of the exam. Let’s take a look at adaptive testing in general, discuss ways that you might prepare for an adaptive certification exam, and then talk about how this change might affect the IT certification industry over the coming years.

What is adaptive testing?

Adaptive testing is a time-honored approach for delivering certification exams. Traditional exams, otherwise known as “linear format” exams, consist of a fixed number of questions delivered in an order that is predetermined before a candidate sits for the exam. Each candidate may receive different questions, possibly in a different order, but the computer selects those questions before the exam and the candidate sits down and progresses through those questions in order.

In an adaptive test, the questions are not predetermined when you sit down to take the exam. Instead, the computer analyzes your response to each question and determines how well you’re performing on the exam. It then uses a mathematical algorithm to choose the next question that you’ll receive.

If you’re doing well in a particular domain of knowledge, then the algorithm may decide not to ask you any additional questions on that domain, once it’s satisfied that you’ve mastered the material. Similarly, if you’re struggling in a particular area, then you may receive additional questions on that subject as the exam software attempts to determine your scope of knowledge.

From a test-taker’s perspective, adaptive testing may lead to a somewhat frustrating experience. As you continue to answer questions correctly, the difficulty of future questions will increase. This is by design, as the algorithm already determined that you’re capable of answering the easier questions and is trying to identify the limits of your knowledge.

As you sit taking the exam, you may feel like the test is extremely difficult and that you’re not doing well. Rest assured that the fact that the questions are increasing in difficulty is actually a good sign. If it appeared that the exam were getting easier as you moved on, that could be an indication that you’d been answering questions incorrectly.

How are adaptive tests scored?

The scoring process for an adaptive exam is a little unusual. Rather than the typical percentage of questions answered correctly, adaptive tests must take the nature of those questions into account. For example, the adaptive test scoring process used on the CISSP exam follows three rules:

Confidence Interval Rule — This rule draws its name from the statistical concept of confidence intervals. It says that once you’ve answered the minimum number of questions, which is 100 on the CISSP exam, the algorithm will stop the exam when it is 95 percent sure that you’ve either passed or failed.

Maximum Length Rule — This says that the exam will stop when you’ve answered the maximum number of questions. For the CISSP exam, this means that you’ve answered 150 questions.

If you make it to question 150, the algorithm looks at the last 75 questions that you answered. If your score was above the passing threshold for the entire time that you answered those questions, you will pass. If your score dips below the passing threshold, even once, you will fail the exam.

Run Out Of Time (ROOT) Rule — This rule says that the exam will stop when you’ve used up all of the time available to you. On the CISSP exam, the exam will stop once three hours have elapsed. If you’ve answered at least 75 questions, the algorithm applies the same procedure as the maximum length rule:

If your score was above the passing threshold for the last 75 questions, then you will pass the exam. If your score dipped below the passing threshold at least once during the last 75 questions, then you will fail the exam. If you do not answer 75 questions, then you automatically fail the exam.

Those scoring rules are fairly complicated and, in reality, you shouldn’t worry about them as a test-taker. It’s not worth your time to sit and try to guess what the algorithm is doing. Just tackle each question to the best of your ability as it is presented to you.

This scoring process also means that you will no longer receive a straightforward score on the exam. Remember, the algorithm is not trying to determine exactly how well you know the material. It’s simply trying to ask you enough questions so that it can be reasonably certain that you’ve either passed or failed the exam.

There are no gold stars for earning an A on the CISSP exam! Instead of a scaled score, you will simply receive a notice that you’ve either passed or failed the test. If you fail, you’ll also receive advice on the areas of the exam where you did not perform well. This can help you focus your study for future exams.

How should I approach an adaptive exam?

I’ve heard from many candidates preparing for the CISSP exam over the past few months, all asking a similar question: how should I change my preparation for the new exam? Do I need to purchase new books, or watch new courses online?

The short answer is no. Adaptive testing is just a different way of determining the questions that you’ll receive on the exam. The content remains the same, as do the questions. You should prepare for an adaptive exam exactly the same way you would for a traditional linear exam.

In addition to the shorter exam length, the biggest change that you’ll see when you sit down for the exam is that you can no longer move back and forth between questions during the exam. This particular quirk of adaptive testing does warrant a change in test-taking strategy.

I used to recommend that candidates take at least three passes through the exam, quickly answering the easy questions, moving back to tackle the more difficult questions, and then reviewing their work. Without a back button, however, this approach is no longer possible. Once you click the button to advance to the next question, your answer is locked in and you can’t see the question again.

This means that it’s now very important to pace yourself as you sit for the exam. Don’t spend an inordinate amount of time stuck on a single question, but be sure to read the question carefully and check your work before moving forward. You won’t have another chance to correct any mistakes.

One final note on the CISSP exam. While it’s true that the move to adaptive testing in Dec. 2017 didn’t change the content on the exam, there is a revision to the CISSP Body of Knowledge coming out in April. If you’re taking the exam in April 2018 or later, then be sure that you’re using study materials that were revised to cover the 2018 version of the test!

What’s Next for Adaptive Testing?

Computerized adaptive tests aren’t new. They’ve been used for over a decade across a variety of fields. Notably, the Graduate Record Exam (GRE) and Graduate Management Admissions Test (GMAT) have used adaptive testing for several years in a very high-stakes environment.

Adaptive testing also isn’t new to the IT certification field. Novell was using adaptive testing for certification exams as early as 1999, and Microsoft rolled out a series of adaptive tests about a decade ago. While those exams have since gone by the wayside, it’s not surprising that adaptive testing is making a comeback, because it shortens the amount of time used in the testing center.

I’m certain that other IT certification providers are closely watching the CISSP adaptive testing experiment. If this process works well for (ISC)², we will probably see others follow their lead and adopt this technology.

While that may seem like an intimidating change, I am confident that the change to adaptive testing benefits everyone involved. Candidates get to take shorter exams, testing centers can fit in more students, and certification providers lower their costs. Stay tuned!