Customer relationship management and marketing remain the top fields for data miners, according to a recent survey conducted by Rexer Analytics, a Boston analytics and CRM consulting firm.
Forty-one percent of data mining professionals polled for Rexer's 4th Annual Data Miner Survey reported working in the CRM/marketing field. Rexer said 735 data miners from 60 countries participated in its 50-question survey, conducted in early 2010.
CRM/marketing has remained the top field of respondents since the polling began four years ago. Other sectors employing large numbers of data miners include financial, academic, insurance, and telecommunications.
Source: 2011 Data Miner Survey, Rexer Analytics
Other conclusions from the survey:
Most data miners use decision trees, regression, and cluster analysis, though in the latest survey, 22 percent of respondents reported using ensemble models, which made their first appearance in the Rexer survey.
Data miners tend to prefer models with fewer variables. A third of the respondents reported building final models with 10 or fewer variables, while only 28 percent built models with more than 45 variables.
The majority of data miners reported using open-source tools and working on laptops or desktops with data stored locally.
The biggest challenges for data miners remain dirty and difficult-to-access data, as well as the ability to explain their work to others effectively.
Data quality and analytics capabilities continue to prove problematic. Only 13 percent of data miners rated their company’s analytical capabilities as excellent, and only 8 percent had confidence in the strength of company data.
Despite the challenges, the survey clearly shows an increase in the demand for data mining, a cause for general optimism in the field. More than 75 percent of data miners anticipate more projects at their companies in the near future. These are over and above increases reported in the previous survey.
In addition, Rexer said it has measured the use of the term “data mining” in a variety of online job ads and found strong requirements.
Have you seen an increase in the number of data mining projects and a rise in skill requirements in your company or organization? What fields do you believe make the greatest use of data mining today?
@Seth: from my experience, dirty and difficult data is a problem because it is difficult to convince stakeholders of the importance of cleaning it up. So we either publish untrustworthy results quickly, or we anger those who pay our salaries by taking too long. Tools are good, but this is time consuming and expensive and not always seen as valuable.
On the other hand, explaining the work effectively includes the above conundrum but goes far beyond it. There is a tremendously deep and wide set of significantly complex science involved in data analytics, but explaining this to anyone at all is challenging. It is hard enough to find qualified people to do the work competently. If you want them to communicate effectively also, you're reducing your talent pool even further.
I have definitely seen an increase in the requirments for analytics in many job postings and like the chart says, in marketing. Like others, I am surprised that CRM/marketing is number one, but I guess that makes sense as it's more profitible to retain and market to a current customer.
I would like to know more about the reasons behind "The biggest challenges for data miners remain dirty and difficult-to-access data, as well as the ability to explain their work to others effectively." Is it the lack of tools or the lack of understanding?
Yes, especially insurance given the potential importance of data mining in risk assessment. But I have to honestly say that I was also surprised by the distance that medical and Internet are behind, two fields where it is obvious to see the benefits of data mining.
Cordell, Thanks for the clarification of standard practice on credit scores. I see your point that it is not the variable in itself that is the issue, it whether is has any significant meaning to the question(s) at hand.
@dataguy. Certainly terms get jumbled a lot. I recall talking to a lot of people about "analytics" which took to mean predictive analytics and they were typically talking about reporting or BI. Some really confusing conversations! Nevermind people passing off analytics that aren't. See our discussion on sentiment analysis that's just keyword search.
@louis. I can confirm that most credit scoring models don't have more than about 15 characteristics though they may start with hundreds. It's not the management of variables that's the problem it's that after about 15, new variables add little to the model. Of course credit scores have been around for a long time and over the years the strongest variables are fairly well known so there's a good starting point already.
I totally agree about communication. If findings can't be communicated accurately then what's the point! This is an ongoing skill set deficit. Tough to find analysts that are good communicators. It's a stereotype I know but it seem to be directionally correct at least.
2015 Visual Analytics Interactive RoadshowSAS(r) experts are coming to a city near you in a series of live, interactive workshops focused on SAS Visual Analytics, including how to prepare your data for VA, the integration of VA with Office Analytics and a Visual Statistics demo.
January 22: King of Prussia, PA
February 24: Austin, TX
March 26: Redwood City, CA
April 22: NYC, NY (1st of 2 stops)
May 13: Seattle, WA
June 18: Minneapolis, MN
July 21: Rockville, MD
August 18: Chicago, IL
September 24: Irvine, CA
October 9: Cary, NC (during SAS Championship)
October 21: NYC, NY (2nd of 2 stops)
November 17: Orlando, FL
December 8: Atlanta, GA
LEADERS FROM THE BUSINESS AND IT COMMUNITIES DUEL OVER CRITICAL TECHNOLOGY ISSUES
The Current Discussion
Visual Analytics: Who Carries the Onus? The Issue: Data visualization is an up-and-coming technology for businesses that want to deliver analytical results in a visual way, enabling analysts the ability to spot patterns more easily and business users to absorb the insight at a glance and better understand what questions to ask of the data. But does it make more sense to train everybody to handle the visualization mandate or bring on visualization expertise? Our experts are divided on the question. The Speakers: Hyoun Park, Principal Analyst, Nucleus Research; Jonathan Schwabish, US Economist & Data Visualizer
The hospitality industry gathers massive amounts of customer data, and mining that data effectively can yield tremendous results in terms of improved CRM, better-targeted marketing spend, and more efficient back-end processes. Roger Ares, vice president of analytics at Hyatt Corp., discusses the ways he and his staff use big data.
Charged with keeping track of travel assets, including employees, iJET International relies on data management best-practices and advanced analytics to keep its clients in the know on current and potential world events affecting travel, Rich Murnane, Director of Enterprise Data Operations & Data Architect, told All Analytics in an interview from the 2014 SAS Global Forum Executive Conference.
Jason Dorsey, chief strategy officer for the Center for Generational Kinetics and keynote speaker at last month's SAS Global Forum 2014, describes how Gen Y professionals are enhancing the makeup of multigenerational analytics organizations.
From analytics talent development to the power of visual analytics, All Analytics found a variety of common themes circulating throughout the exhibition floor and session discussions at the 2014 SAS Global Forum and SAS Global Forum Executive Conference events held last month in Washington, DC.
Talking with All Analytics live from the 2014 SAS Global Forum Executive Conference, Eric Helmer, senior manager of campaign design and execution for T-Mobile, discussed the importance of customer data -- starting internally -- in devising the mobile operator's marketing plans.
The big-data analytics market can be a confusing place. Among the vendors vying for your dollars are traditional database management providers, Hadoop startup services, and IT giants. In this video, All Analytics editors Beth Schultz and Michael Steinhart sit down in a Google+ Hangout on Air with Doug Henschen, executive editor of InformationWeek. Henschen discusses use cases for big-data analytics, purchase considerations, and his recent roundup of the top 16 big-data analytics platforms.
At the National Retail Federation BIG Show last month, All Analytics executive editor Michael Steinhart noted a host of solutions for tracking and analyzing customer activity in retail stores. From Bluetooth beacons to RFID tags to NFC connections to video analytics, retailers must find the right combination of tools to help optimize the shopper experience, streamline operations, and boost revenues.
The days when historical shipment trends and gut feelings were enough to forecast retail demand accurately are long over. SAS chief industry consultant Charles Chase outlines the benefits of pulling real-time sales information from point-of-sale and product scanner systems, then flowing that data into dynamic forecasting tools from SAS.
With today's advanced visual analytics tools, you can stream data into memory for real-time processing, provide users the ability to explore and manipulate the data, and bring your data to life for the business.
Dynamic data visualizations let analysts and business users interact with the data, changing variables or drilling down into data points, and see results in a flash. Advance your use of data visualization with tools that support features like auto-charting, explanatory pop-ups, and mobile sharing.
No doubt your enterprise is amassing loads of data for fact-based decision-making. Hand in hand with all that data comes big computational requirements. Can traditional IT infrastructure handle the increasing number and complexity of your analytical work? Probably not, which is why you need a backend rethink. Big data calls for a high-performance analytics infrastructure, as Fern Halper, a partner at the IT consulting and research firm, Hurwitz & Associates, discusses here.
Redbox's bright-red DVD kiosks are all but ubiquitous these days, located in more than 28,000 spots across the country. Jayson Tipp, Redbox VP of Analytics and CRM, provides an insider's look at how the company has accomplished its phenomenal nine-year growth.
InterContinental Hotels Group (IHG), a seven-brand global hotelier, has woven analytics into the fabric of its operations. David Schmitt, director of performance strategy and planning, shares IHG's analytics story and his lessons learned.