In many fields, you can't practice your profession without a license. In others, staying employed, getting ahead, or improving your earning potential requires certification. Whatever the case, you expect the required exams to measure your knowledge accurately, fairly, and reliably.
Castle Worldwide, in Morrisville, N.C., carries the burden of meeting those expectations for examinees globally.
In fact, it has a legal obligation to do so, said Robert Pedigo, the company's vice president for client services. "When you have a situation in which an individual's career, income, or capacity to practice is on the line... you have to make sure the examinations measure what they say they measure and, as far as the state of test development will permit, they are reasonably free of bias. The courts have reinforced this very clearly."
A social obligation compels the company as well, added Pedigo, pointing to public safety as a concern. "Do you really want anesthesiologists who haven't absolutely proven that they're competent in the field? Of course not," he said.
These obligations circle the globe. Castle, a leading certification and licensure testing company, said it counts more than 80 government agencies, corporations, educational institutions, and trade and professional associations as clients. It conducts tests at more than 1,000 centers in more than 500 cities in 70 countries.
In a recent phone interview, Pedigo, along with Andrew Dwyer, senior psychometrician, shared how statistics and analytics tools enable the company to keep on top of the testing programs. For example, statistics come into play for:
Vetting new tests. Before it launches a new test, Castle might gather a group of examinees to go through the test. It gathers statistics on the test as a whole, as well as on individual questions or, as known in testing parlance, items.
"There are statistics that tell you whether or not each item is performing well, statistics that tell you whether the test as a whole is doing its job, statistics that tell you whether or not classification -- pass/fail -- decisions are being made as accurately as possible," Dwyer said.
Evaluating current tests. Castle will run analysis to make sure individual test questions are performing correctly. If they aren't, then it'll swap the items out or take other corrective action, Dwyer said. "So a lot of the statistics we use are to evaluate the performance of current exams and to make them better going forward or to refresh them."
Refreshing test questions. This is done not only for regular maintenance but also to diminish the opportunity for cheating, which is prone to happen, given the high stakes involved in receiving a license or certification. The statistical analysis done here is called equating, which is the process of making adjustments to new tests to make sure scores are fair or a true reflection of the candidate's ability and not a reflection of how hard the test form was. "If we take out an easy question and put in a hard question, we don't want to disadvantage that new examinee" or, conversely, provide an advantage if replacing a hard question with an easier one, Dwyer said.
Reporting to accreditation agencies. Many Castle clients must report to accreditation agencies on the quality of exams, number of test takers, number passing the tests, and so on. "We put together large statistical reports that clients pass on to these agencies to demonstrate they're meeting standards with respect to the quality of the exams and the processes they're following," Dwyer said.
In the last six months, Castle has gotten much more efficient and flexible in how it handles its statistical processes, Dwyer said. That's because it has brought in business analytics software from SAS (this site's sponsor), which, among other benefits, allows it to link directly to the database and get data out.
The statistical process previously required ordering the data from IT, then scrubbing the data and loading it into a particular package -- operationally straightforward but time-consuming procedures. With the SAS tool, Castle has been able automate the workflow and operate more quickly.
This becomes particularly beneficial in disciplines, such as IT, where tests change frequently, Pedigo said. "If we can shave a month off that workflow, we suddenly can save clients a significant amount of money and reduce a variety of business risk for them by increasing the speed of test production."
Castle has scored big by applying an analytical tool to its statistics processes. As Pedigo said: "The benefits are operational and very substantial."
How is your company benefiting from analytical tools? Share below.
@adwyer, given some other discussion here at AllAnalytics.com (namely our Point/Counterpoint debate on establishing an Analytics Center of Excellence), I have to ask: As user of analytics tools crosses out of the psychometric department and into others, how do you see analytical processes shaping up? Will your department spearhead direction? Will you establish a formalized cross-departmental team for analytics? Will everybody act independently, more or less? Are you guys giving this any thought at this point or is it still too early days yet?
@PC Good question. As you correctly guessed, we are currently interested in making our current workflows more efficient (within the psychometric department). But I definitely see opportunities for new workflows and new analyses both inside and outside our department. For example, we have several test developers who are not statisticians or SAS programmers, but their job requires them to be users of those statistics. We are just starting to explore the Microsoft Add-in as a way for these non-SAS users to access real-time statistics for their clients using stored processes (in Enterprise Guide) that are created by us, the psychometricians (i.e., statisticians). Also, we are also in the process of updating our organization's databases. Once that is in place, I feel like there will be tremendous opportunities for cross-departmental analyses to help improve our business operations.
@adwyer - thanks for taking the time to respond to questions here. You're a good sport.
Beth's article describes how you're seeing the benefits of improved efficiency and flexibiltiy with the business analytics software. It's good that the existing work flows better - that might be enough, but I want to take it one step further.
Do you anticipate ways that the new software can enable new workflows? In other words, making the same process we've always had work faster is one kind of improvement. Another kind of improvement is changing the things we do - allowing us to do things we couldn't even think of before the change.
At this point, I don't think any of our clients have essay questions on their exams. When we did, I think we only used human raters, because I don't think Castle has developed any computer algorithms to automatically score essays. I think it's a fascinating area with pros and cons, but I haven't had any personal experience with that issue.
@seth It is not surprising, Seth, that you haven't heard of us (Castle Worldwide), as we are a medium-sized testing vendor that specializes in certification and licensure testing. There are several other larger testing vendors (e.g., Educational Testing Service (ETS), Pearson, Prometric) that deal with a larger variety of tests (e.g., k-12 achievement tests, college admissions tests). You might have heard of one or more of those companies.
We do not own any of the tests we work with. We have a psychometric department that consults with various organizations (our clients) to develop their tests, and we also provide test administration services both nationally and internationally. Many testing programs use a testing vendor for development, but amost all require a vendor for administration services, because developing a national testing network is very cost prohibitive, as you can imagine.
A very small number of testing programs, such as the LSAT (law school admissions), develop their own tests AND have their own test delivery network (i.e., accredited law schools across the country).
As a side note, since this is a SAS sponsored blog, I'll mention that not that long ago, I know we (Castle) considered putting in a bid to become the testing vendor for SAS's certification program. I'm not sure if we ever did, however.
@adwyer, yes people tend not to say anything unless they find a problem. Sometimes problems only come to the attention of people during scoring. A while back, one of the SAT essay topics was reality television. At some point during scoring, we (scorers) got a message to forward any responses in which students complained about not being able to answer the question because they do not watch television. The questions ususally are more general than that with no assumptions about student leisure activities.
@ariella I did hear about the talking pineapple question on the NY state educational exam, and I think another commenter below also mentioned it. Actually, being in the testing industry, I see these kinds of things more frequently than most, probably. It's funny/sad what slips through the cracks, but it is not that surprising. State departments of education have enormous testing responsibilities (hundreds of thousands of students tested in multiple subjects across multiple grades), in addition to the other roles that state DOEs play, and most states have limited budgets.
Regarding the pineapple question, it is possible that it was pretested and exhibited very satisfactory item statistics, even though it appears to be a weird and confusing question. I'm very sure that item also went through several other review processes, but like I said, sometimes poor items do not get flagged. And when that happens, the media is usually all over it. For the testing industry, no news is good news :)
@beth The development timeline really depends on the organization and resources ($$ and the availability of subject matter experts to write questions, etc...), but generally speaking, it takes anywhere from 6 months to a year to develop a test from scratch.
Once a testing program has launched, the schedule for introducing new forms depends on a wide variety of factors. If the field is changing rapidly (e.g., the IT industry), then the test needs to be updated often to reflect those changes. In addition, if a test has a large testing population or particularly high stakes, there is a larger opportunity or higher motivation for cheating, which usually means the test is refreshed more often. The financial resources of the test owner also plays a role. There are other factors as well, but those are the big ones.
LEADERS FROM THE BUSINESS AND IT COMMUNITIES DUEL OVER CRITICAL TECHNOLOGY ISSUES
The Current Discussion
Visual Analytics: Who Carries the Onus? The Issue: Data visualization is an up-and-coming technology for businesses that want to deliver analytical results in a visual way, enabling analysts the ability to spot patterns more easily and business users to absorb the insight at a glance and better understand what questions to ask of the data. But does it make more sense to train everybody to handle the visualization mandate or bring on visualization expertise? Our experts are divided on the question. The Speakers: Hyoun Park, Principal Analyst, Nucleus Research; Jonathan Schwabish, US Economist & Data Visualizer
To save this item to your list of favorite AllAnalytics content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Dynamic data visualizations let analysts and business users interact with the data, changing variables or drilling down into data points, and see results in a flash. Advance your use of data visualization with tools that support features like auto-charting, explanatory pop-ups, and mobile sharing.
No doubt your enterprise is amassing loads of data for fact-based decision-making. Hand in hand with all that data comes big computational requirements. Can traditional IT infrastructure handle the increasing number and complexity of your analytical work? Probably not, which is why you need a backend rethink. Big data calls for a high-performance analytics infrastructure, as Fern Halper, a partner at the IT consulting and research firm, Hurwitz & Associates, discusses here.
Redbox's bright-red DVD kiosks are all but ubiquitous these days, located in more than 28,000 spots across the country. Jayson Tipp, Redbox VP of Analytics and CRM, provides an insider's look at how the company has accomplished its phenomenal nine-year growth.
InterContinental Hotels Group (IHG), a seven-brand global hotelier, has woven analytics into the fabric of its operations. David Schmitt, director of performance strategy and planning, shares IHG's analytics story and his lessons learned.
Elizabeth Barth-Thacker, a BI and informatics technology manager at Humana, tells us how her team is creating data transparency and building engagement with the business – with the help of an internal collaboration portal called Humanalytics.
Speaking at SAS Global Forum Executive Conference, Rajeev Kaul, SVP of pricing at OfficeMax, uses a Chinese proverb to explain one of the reasons he's deploying SAS Visual Analytics.
In an All Analytics interview, Mike Cavaretta, technical leader, predictive analytics at Ford Research & Advanced Engineering, shares how big-data is fueling vehicle decisions.
Analytics professionals and SAS executives share how organizations can get on with their work so much faster when working in a high-performance and visual analytics environment.
Analytics professionals who attended SAS's recent Executive Briefing in New York share how they think visual analytics might help their organizations get better value from data.
At Boeing, effective decision making comes down to this simple formula: QxA=E, as executive Jerry Allyne explained at the recent INFORMS analytics conference.
Whether working in major league sports, financial services, or healthcare, analytics, and data, professionals are checking out how visual analytics and high-performance technologies can help them optimize their environments, shrink their cycle times, and improve decision making, as attendees at the recent SAS Executive Briefing in New York share with us.
SAS CEO Jim Goodnight speaks with us at a recent SAS Executive Briefing about getting a feel for what's in your big-data and other new realities powered by advanced analytics.
Jim Davis, SVP and CMO at SAS, talks with us at a recent SAS Executive Briefing about how high-performance analytics and visual analytics take away the concerns over big-data and let companies get down to business with their data.
Tweet This
24 
14 
