In many fields, you can't practice your profession without a license. In others, staying employed, getting ahead, or improving your earning potential requires certification. Whatever the case, you expect the required exams to measure your knowledge accurately, fairly, and reliably.
Castle Worldwide, in Morrisville, N.C., carries the burden of meeting those expectations for examinees globally.
In fact, it has a legal obligation to do so, said Robert Pedigo, the company's vice president for client services. "When you have a situation in which an individual's career, income, or capacity to practice is on the line... you have to make sure the examinations measure what they say they measure and, as far as the state of test development will permit, they are reasonably free of bias. The courts have reinforced this very clearly."
A social obligation compels the company as well, added Pedigo, pointing to public safety as a concern. "Do you really want anesthesiologists who haven't absolutely proven that they're competent in the field? Of course not," he said.
These obligations circle the globe. Castle, a leading certification and licensure testing company, said it counts more than 80 government agencies, corporations, educational institutions, and trade and professional associations as clients. It conducts tests at more than 1,000 centers in more than 500 cities in 70 countries.
In a recent phone interview, Pedigo, along with Andrew Dwyer, senior psychometrician, shared how statistics and analytics tools enable the company to keep on top of the testing programs. For example, statistics come into play for:
- Vetting new tests. Before it launches a new test, Castle might gather a group of examinees to go through the test. It gathers statistics on the test as a whole, as well as on individual questions or, as known in testing parlance, items.
"There are statistics that tell you whether or not each item is performing well, statistics that tell you whether the test as a whole is doing its job, statistics that tell you whether or not classification -- pass/fail -- decisions are being made as accurately as possible," Dwyer said.
- Evaluating current tests. Castle will run analysis to make sure individual test questions are performing correctly. If they aren't, then it'll swap the items out or take other corrective action, Dwyer said. "So a lot of the statistics we use are to evaluate the performance of current exams and to make them better going forward or to refresh them."
- Refreshing test questions. This is done not only for regular maintenance but also to diminish the opportunity for cheating, which is prone to happen, given the high stakes involved in receiving a license or certification. The statistical analysis done here is called equating, which is the process of making adjustments to new tests to make sure scores are fair or a true reflection of the candidate's ability and not a reflection of how hard the test form was. "If we take out an easy question and put in a hard question, we don't want to disadvantage that new examinee" or, conversely, provide an advantage if replacing a hard question with an easier one, Dwyer said.
- Reporting to accreditation agencies. Many Castle clients must report to accreditation agencies on the quality of exams, number of test takers, number passing the tests, and so on. "We put together large statistical reports that clients pass on to these agencies to demonstrate they're meeting standards with respect to the quality of the exams and the processes they're following," Dwyer said.
In the last six months, Castle has gotten much more efficient and flexible in how it handles its statistical processes, Dwyer said. That's because it has brought in business analytics software from SAS (this site's sponsor), which, among other benefits, allows it to link directly to the database and get data out.
The statistical process previously required ordering the data from IT, then scrubbing the data and loading it into a particular package -- operationally straightforward but time-consuming procedures. With the SAS tool, Castle has been able automate the workflow and operate more quickly.
This becomes particularly beneficial in disciplines, such as IT, where tests change frequently, Pedigo said. "If we can shave a month off that workflow, we suddenly can save clients a significant amount of money and reduce a variety of business risk for them by increasing the speed of test production."
Castle has scored big by applying an analytical tool to its statistics processes. As Pedigo said: "The benefits are operational and very substantial."
How is your company benefiting from analytical tools? Share below.