This feature first appeared in the Fall 2022 issue of Certification Magazine. Click here to get your own print or digital copy.

When a certification exam is created and deployed properly, the impact is felt across the entire spectrum of stakeholders: exam takers, exam administrators, everyone involved at the company or organization that sponsors the exam, and more.

A well-designed exam validates the IT skills and knowledge of individuals on both sides of the exam: those who created the exam and those who use the exam to become certified. Not only that, but the exam and its associated credential increase the value of the skills and training associated with it. In this sense, certification truly is at the top of the food chain in the IT training sphere.

For the program owner or sponsoring company/organization, reviewing the results of a good exam will clearly identify the most important areas to test on, and in turn point out key areas of focus for training material. This goes right to the objective level: Training courses and content can go far beyond what an exam blueprint would cover, but it is a must to cover the objectives on the exam.

When we say "well-designed" or properly put together, we are talking about creating scenario-based questions for an entire exam. Scenario-based questions are the highest form of question writing and the hardest to create, but the impact of this effort carries forward in many ways.

First though, let’s get back to the impact of training content — then we will talk about the impact to the question writers. If you ever have the opportunity to participate in an exam development workshop, please jump at that chance. We will talk about what to expect during such a workshop later in this article.

Training content and scenario-based questions

Without training content, there is no training program. Without some kind of job and audience analysis, there is no foundation to work from. If a company simply has an idea about training and slaps some content together, it can make for a long haul to get to a world-class content level.

The good news, in this respect, is that putting together a certification and creating the accompanying exam will force you to do the analysis. It will also lead you to focus on your product's most important job roles and the skills and knowledge required to fill those roles. The end result, the exam blueprint, will identify holes in the training which can then be addressed.

Another point about technical training is that hands-on labs are a key way to gain the real world knowledge needed to succeed both when taking certification exams and, more importantly, on the job. Hands-on labs can feed directly into scenario-based questions, which are based on real world experience.

This is vital. Exam candidates must perform critical thinking to successfully pass an exam that is all scenario-based questions. Critical thinking is needed to succeed in any field, and is especially important in the IT sphere. If you have someone who has passed such an exam, it makes your recertification efforts appreciably easier.

In this case, recertification should only require absorbing new product information, the type of training one can pick up at a conference, or which can easily fill up continuing education units. This is all because the original exam was scenario-based and focused on the real world. It circumvents the need for continually updating an exam based on product cycles, and focus on job roles instead.

Get the real-world perspective

One topic to include on every certification exam is troubleshooting. You will need subject matter experts from your technical customer support team to write these questions. Often, there will be no training in place to cover what is being called in to the support team. The services or implementation group will also encounter areas to troubleshoot.

Troubleshooting questions can flow backward from your exam design team and be worked into your training content, including courseware, knowledge-based articles, and labs. Thus, if you have strong content, you will get a better exam, but the exam can also inform — and strengthen — your training content. That's a worthwhile feedback loop.

Exam development workshops can also help you develop training courseware. Labs are an important key here, and labs should always involve hand-son application of knowledge and skills. You should always try to involve partners from the company’s ecosystem in exam development: Each of them will bring in different real world troubleshooting scenarios.

With their additional input, you can cook up between six and 12 case studies, instead of focusing on one or two. This is exactly what exam candidates are looking for. The bar for training content has been raised. Better content means workers who are better prepared to support your products, which means faster resolution of real-world problems.

Mixing workplace professionals and SMEs

For any subject matter expert who writes exam questions, participating in an exam development workshop can be priceless. An assembled team of experts from both the sponsoring company and the partner ecosystem can have many benefits. The informal network that forms is by itself invaluable. The subject matter experts (SMEs) learn from each other.

A good SME will also learn how to be clearer and more concise with their written questions after participating in an exam development workshop. All SMEs will learn how to write effective scenario-based question. SMEs who come from the education team, or at least instructional and course designers, will be getting a master class experience in how to write questions.

Better SMEs, of course, means a much higher quality of practice question content included in end-of-course assessments. And that will also better prepare exam candidates to succeed both in passing your certification exam and dealing with your products in the real-world workplace.

Another benefit of participating in exam development workshops is that your SMEs will learn how to deconstruct (and reconstruct) poorly written questions. If a certification program has poorly written questions, it reflects badly on your program and on the company and its products.

The new world of workshops

Earlier, we mentioned what to expect during an exam development workshop from a test writer’s perspective. Prior to the onset of the COVID-19 pandemic, most (if not all) exam development workshops were conducted in person. The process would take maybe four-and-a-half days to complete.

Once your scenario-based questions have been written, they must be tech reviewed. Tech review is a systematic and disciplined process for the team to look at congruity, relevance, technical accuracy, and readability. Some development teams have begun returning to this type of in-person workshop.

Exam development conducted remotely does take more time than when you have everyone in the same room, at the same time. The process can stretch out over a few weeks, rather than being compressed into a few days. When carrying out this type of effort, it's tougher to keep everyone engaged and quality sometimes suffers.

Inside the exam development workshop

Whether working remotely or in-person, if an SME has been prepped ahead of the workshop and asked to write a few questions ahead of time, they will be a half-day ahead of the others in getting solid items written. A prep session also helps everyone become familiar with the exam blueprint, as well as the key guidelines to follow when writing questions.

You can also get a jump on the review process. If an SME sends in questions ahead of the workshop, they can be reviewed, edited, and, more importantly, sent back with suggestions. A test writer can only get good at writing questions by writing questions — and having the development team tech review those questions.

Day One: On the first day of the workshop, after one last quick look at the exam blueprint, it is time for the SMEs to sign up for test writing by putting their name next to exam objectives. Those SMEs who are experienced will get up and sign up first, picking the topics they are best suited for.

It is then time for writing questions. Each SME is on their own to start. After each SME has written at least one question, it is time for the initial review. My advice to SMEs is, “Don’t take it personally,” as the process is always about the quality of the questions. (You do need to soothe SME egos and some hurt feelings every now and then.)

Tell your SMEs to take no more than 20 minutes on any one question, and to be open to suggestions from the group. Other SMEs will also have strong ideas. At the end of the first day, if there are between 10 and 15 questions that can be deemed "keepers" — meaning they passed the criteria and will appear on a test — then you are off to a good start.

Day Two: The overall process of the second day is the same as the first. The goal for the second day is to strive to double the number of "keepers" that were finished on the first day.

Day Three and Day Four: The third and fourth days are simply focused on getting to the exam bank goal. The exam bank goal for a 60-question certification exam can vary between 90 and 110. The end of the fourth day can be a struggle as the questions can be more challenging to write, but many workshops do make the goal at this time.

The last half-day is for any final clean up and to enter the questions into the test development tool. The last part of Day 4 might require some collaboration to get to the finish line. Same goes for whatever time is needed on a fifth day (and beyond).

Many an SME has had dreams about writing questions after the second day. After Day 3 or 4, it is time to take everyone to dinner. The whole week is always quite the bonding experience, and the dinner helps to clinch it. As someone who has led, sponsored, or facilitated these types of workshops, I can tell you the bond remains for many years. It is one of the best benefits from my perspective.

Onward

SMEs learn a great deal from each other and raise their game. They also go back to their peers and let them know to be on their game when they take the new exam. The SMEs become evangelists for the new certification exam and that is the best type of marketing you can have.

The marketplace will recognize the value of a world-class exam and certification credentials will be proudly displayed. This in turn impacts the visibility of the sponsoring company and the partners who participate and earn credentials.

As an additional note, scenario-based questions are difficult to cheat on, raising the integrity of the exam. Exam security is vital to delivering a satisfying and effective exam experience. Everybody wins, across the board, when a properly created and delivered certification exam is in place.

‍