Challenges of Global Delivery
Program managers charged with the delivery of certification exams for IT professionals, CPAs, sports medicine personnel or anyone who must certify or license their skills for a job have a tall order to fill. They must maintain and provide a consistent and fair method of testing delivery for all candidates, often over a large body of testing centers in far-flung locations.
Pearson VUE has a domestic network of more than 200 Pearson Professional Centers and more than 3,500 Pearson VUE authorized testing centers worldwide. “The largest challenge is managing such a large channel in so many countries,” said Mark Poole, director of testing quality and security, Pearson VUE. “We have in excess of 3,500 third-party test centers, and they’re located in well over 100 countries worldwide. The challenge lies in being able to manage such a diverse and widespread delivery channel without necessarily being physically there. Another challenge related to that is the language and cultural issues, the ability to maintain consistency, as well as security in that type of environment. If you own 100 testing centers here in the U.S., you can control every aspect of what happens every day, and you can have an extremely high degree of assurance, close to 100 percent, that every test delivered in every center is going to be delivered under exactly the same conditions. When you’re dealing with IT certification that’s delivered around the globe, the challenge is determining how you can ensure that, for instance, a Microsoft test delivered to a candidate in Katmandu is delivered under the same type of conditions and with the same degree of fairness as a Microsoft or some other test delivered in a large New Horizons site in New York City, for example.”
Program managers need to be aware of efforts that ensure testing fairness and consistency across large and disparate networks of testing centers. Poole said you must have and exercise a comprehensive quality control and security management program, which includes pieces such as integrity shopping. “It’s sometimes referred to as ‘secret shopping,’ where you send undercover candidates to test centers,” Poole said. “They are professionally trained to understand how the test centers should be operating and what procedures should be followed. You send them in and they take a test as a Microsoft, CompTIA, Cisco or IBM student, and they prepare a report after the testing experience, which is then analyzed and responded to.”
To help combat delivery challenges, such as testing inconsistency or misconduct, Pearson VUE sends integrity shoppers to close to a quarter of its testing centers every year. Test centers are aware that there may be shoppers, but they don’t know when they may be ‘shopped.’ “That’s a very effective tool because it allows us to do a spot audit without the site knowing that you’re doing it. We get lots of really good feedback,” Poole said. “Another thing we do is survey candidates. We send out a voluntary questionnaire to candidates in most programs, asking about their experience at the test center. We’re looking for feedback on compliance with policies and procedures and consistent test administration conditions. They also have the ability to provide any tips or other feedback as well. For instance, sometimes you may hear of possible misconduct through a response on a candidate survey. Some of the certification programs have a tips line where if you think something unfair is happening anywhere in the testing channel or with their certification program, you can call that line or e-mail that e-mail address. We do the same thing. We allow that vehicle through our survey. We ask, ‘Is there anything else you want to tell us about your experience at the testing center?’ On some occasions, you’ll get feedback about something that’s going on that we will then investigate.”
To ensure compliance and fairness, Pearson VUE and its certification sponsors also engage in various forms of data analysis on test results. “We look at patterns in the testing data. We look for things like anomalous pass rates. We look for other interesting patterns in the test data that may suggest something unusual going on. All of these things can be indicators of poor consistency or inconsistent application of quality standards, or it could be indicative of misconduct of some kind or another,” Poole said.
-Kellye Whitney, firstname.lastname@example.org.