What Is the Goal of Your Program?
One of the most important questions for test design and development is “What is the goal of your program?” This is an important question because I often see tests selected as the de facto means-to-an-end when organizations are interested in assessing or validating a skill set, verifying competencies, or demonstrating performance for a select employee population within an organization. An example is determining whether or not a sales person has the appropriate level of readiness to be effective.
As one example, many organizations are increasingly leaning toward implementing internal programs that assess and validate employee skills, competencies, and ultimately performance. These programs have considerable merit as they significantly benefit both employees and organizations by helping to identify skill gaps, determine readiness, and increase the overall knowledge base of the target population. Getting back to my sales person example, a program that effectively measures readiness and helps bridge skill gaps can directly impact a company’s bottom line.
The planning phase is the right place to determine the ultimate goal of an assessment and/or testing program therefore it is important to fully consider the factors that will ensure the success of the program. Another question to ask is whether you need to assess or test?
A fine line exists between defining an assessment versus a test. There are undoubtedly numerous academic definitions of each and I have little doubt that there would be considerable dialogue about the definition as well as the use of each as measurement tools. I think the line between the two of them continues to become more clouded but that is a topic for another discussion.
For purposes of this discussion I consider assessments to be measurement tools that identify a gap between what an individual should know and how they score on an assessment. I consider assessments in the context of this discussion to be formative measurement tools where a knowledge, skill, or competency gap is identified. Based on the results individuals are directed to the appropriate training or education that will help them gain the knowledge, skills, or competencies that will help narrow these gaps. Additionally, assessments are low stakes in the sense that they are delivered in non-proctored environments and typically do not have significant hiring, promotional, salary, or performance consequences.
On the other hand, I consider tests to be summative measurement tools that result in individuals passing or failing a test where the passing score is based on a cut score that was established using psychometric methods. Many current certification programs are based on high-stakes tests where the tests have undergone a sufficient degree of rigor in the definition, design, development, and implementation phases of construction to ensure the tests are valid and reliable. For additional clarity on high-stakes testing you can reference my column entitled “Increasing Value through High Stakes Testing” in the December 2003 issue of Chief Learning Officer magazine at: http://www.clomedia.com/content/templates/clo_feature.asp?articleid=327&zoneid=34.
It boils down to this: Before jumping on a testing and certification band wagon as a means to support internal programs be clear on whether or not the program is intended to certify individuals or assess their knowledge, skills, competency, or performance. Are you looking for a definitive measurement or are you determining readiness where you use that measurement to establish training and education plans? The difference between a certification program and an assessment program can be significant with respect to resources and dollars for test design, development, and implementation.
There are numerous challenges to consider when implementing assessment or testing programs for internal audiences. One major consideration is that often the target population of individuals to whom the program is intended is small. This limits the number of individuals who can be involved in a job or competency analysis, the number of individuals from whom test statistics are collected, and the amount of time that individuals have to participate in beta programs since they are busy with their day job. These factors also make it difficult to accurately estimate realistic time lines for fully implementing internal programs. Realistic time lines are imperative to ensure the success of the program.
When thinking about the program you want to establish, be clear with respect to the overall program goals. Think about assessing versus testing, summative versus formative, and whether or not certification is necessary. You can derive your own definition of assessments and tests but ensure whatever you use supports your program goals. Consider what it will take to effectively design, development, and implement a program given the size and nature of your target population. Your success is highly dependent upon the success the program you are implementing.
James A. DiIanni is the director of assessment and certification exam development at Microsoft Learning and supports the Microsoft Certified Professional program. His experience with performance testing started in 1986 developing simulators for the U.S. Navy, and he has been involved in the IT certification industry since 1997. He can be reached at firstname.lastname@example.org.