Surviving the Big Data HR Test

I've known a bunch of different human resources professionals throughout my career and have tended to have good working relationships with them. In fact, I've found that I could be honest with them when suggesting ideas or airing complaints, maybe because I wouldn't tell them something that I wouldn't tell the big boss to their face.

Credit: Pixabay
Credit: Pixabay

One thing I have learned about those HR people is that they aren't gamblers. Their idea of risk is limited to decisions such as whether to move to a new forms processing system or to do the "trust your coworker when you fall backward" exercise at the company meeting.

So I was shocked to read about the position HR leaders are taking when it comes to big data and personnel decisions. Let's face it, in using big data to hire people or even support decisions on promotions, there are privacy considerations and even legal worries that can come up.

But by golly, those HR pros are for it, warts and all.

The website Human Resource Executive Online reported on an Equal Employment Opportunity Commission hearing on big data. Reading some of the comments from government officials, HR leaders, and attorneys who testified, the bottom line seems to be that big data-driven HR systems can't be any worse than the already flawed systems employers have been using for decades.

Plus, they said a technology and big data-based system makes it easier to defend against complaints about fairness in personnel decisions because the data is more tangible than human opinions.

"Big data can eliminate existing major problems, and while it may create some new unexpected issues, we're making progress," said Marko Mrkonich, a shareholder in the Minneapolis office of law firm Littler. "I think that's the right way to look at this."

Other speakers advised that HR teams start by using big data in a test-bed environment so they can look for holes such as subtle bias, in their models, and that independent experts should be called in to provide an additional sanity check for fairness.

The EEOC hearing was designed to explore the fairness of big data HR systems and how to avoid designs that would have a disproportionate impact on a protected group -- the so-called "disparate impact" theory. It's a concern that has been voiced about some other big data applications, including those that retailers use to do promotional pricing or lenders use to set interest rates. Do those systems represent even unintentional bias?

Big data has been edging its way into HR for a while now, approaching in stealth mode. Criminal record checks and credit reports once were used only for filling a handful of sensitive jobs. Not any more; those have become standard for many hires today. That's big data.

Closer to the big data model is the background check that delves into a candidate's social media activity, whether it is done by HR or by a hiring manager. That's where the candidate learns that the party photo with said candidate holding a bong or wearing a two-can beer hat might not have been a good idea.

As that type of review becomes more common, it also will have to become more structured, which raises the old issue of how to give structure to unstructured data. Does the company do social media analyses for every candidate? Just for young people? Just for certain jobs? Just for people who raise red flags about politics in the interview process?

An even bigger issue is how managers quantify what they find. Does bad credit count for a certain number of negative points in the overall evaluation? Does that recent drinking party pic carry less weight for a new college grad than it does for a 40-something middle management candidate? Does a candidate get positive points if their social media activity highlights charitable work or community leadership? If you assign point values to what you find on social media, do you do the same for professional experience and industry awards?

Big data can serve a valuable role in hiring and other personnel decisions. Yet we are still applying metrics to human beings. These models will have to be tested, refined, tested again, and modified over time. Despite all that data, in the end someone will have to decide whether that person over there is right for the job. What we have to see is just how deep the data goes in our decision processes.

Share a comment below: Would you be comfortable if HR and hiring managers did big data analysis of your past activities?

James M. Connolly, Editor of All Analytics

Jim Connolly is a versatile and experienced technology journalist who has reported on IT trends for more than two decades. As editor of All Analytics he writes about the move to big data analytics and data-driven decision making. Over the years he has covered enterprise computing, the PC revolution, client/server, the evolution of the Internet, the rise of web-based business, and IT management. He has covered breaking industry news and has led teams focused on product reviews and technology trends. Throughout his tech journalism career, he has concentrated on serving the information needs of IT decision-makers in large organizations and has worked with those managers to help them learn from their peers and share their experiences in implementing leading-edge technologies through publications including Computerworld. Jim also has helped to launch a technology-focused startup, as one of the founding editors at TechTarget, and has served as editor of an established news organization focused on technology startups and the Boston-area venture capital sector at MassHighTech. A former crime reporter for the Boston Herald, he majored in journalism at Northeastern University.

Big Data Success Essentials: Tech, People, and Process

While there is an increasing focus on the role of people alongside technology in analytics initiatives, let's not forget that process -- business rules -- play an important role in big data success.

Why You Should Remember Equifax

The Equifax breach raises multiple concerns about how companies respond to hacks, but also how they handle third-party consumer data.

Re: Am I good witch or a bad witch?
  • 11/6/2016 5:09:31 PM

@Tricia. Actually, I didn't think about people who do charity work during their workday (except where the company allots time for it). Right, penality points for the ones who devote their work days to unsanctioned charity work (or in one case I recall, writing a sports book). I was thinking of the people who are active in the community on their own time.

Am I good witch or a bad witch?
  • 11/4/2016 3:57:37 PM


You bring up some good points.  You asked if charitable work should be considered a bonus?

I have worked with people who basically spent half of their time working for an outside organization.  This one lady in particular worked with the Girl Scouts of America. It was hard to schedule meetings with her because she had  "Girl Scout calls" and she was constantly using the copier/printer for those activities.

So ... I might deduct points for that activity.  :-)

But I'm really just furthering your point - not all managers are going to see things the same way.  If you are running a Sales organization - maybe the volunteer organization is a great way to meet people who want to buy your product.  So who cares if we have to spend a day a week helping them out?

That's the problem with the data - I don't know how it's going to be used and what criteria is being applied.