Text analytics is a new horizon for data-driven decision making in municipal government.
Why text analytics? If there's one thing local governments have in plentiful supply, it's text. Records of everyday government activities are loaded with it -- police reports, court proceedings, license applications, school records, and other documents from routine government business generate a sea of text.
Text sources can be rich in valuable information that does not exist in any other form. Consider the gems found in the notes of police reports -- details about weapons, behavior, past encounters with suspects or victims, and so forth. Locating relevant information in these records is a challenging process that has depended on memory and manual review in the past.
Now police officers in Alexandria, Va. -- a Washington, DC, suburb with a population of 150,000 -- are using text mining to track gang and drug-related violence. Text mining facilitates the identification of key concepts within things like police and crime reports, tips from the public, and news stories to help officers conduct research quicker and easier. Faster and more thorough searches mean better information for detectives and patrol officers.
Municipal 311 systems are fertile ground for text analytics opportunities. Records of past inquiries provide a tremendous resource for learning the kinds of information constituents want, what difficulties they face in finding it, and what their most frequent questions are. Manually organizing that information from thousands or millions of records is tough, but text analytics makes the process faster and more consistent.
Back to the roots
The exploration of text analytics by city governments is not just a US phenomenon. Hong Kong is using text analytics for its call center and paying particular attention to complaints, which represent about 10% of those calls. Why complaints? They carry hidden messages about root causes of problems with government services.
Identifying root causes is a unique value proposition for text analytics in government. It's one thing to know something happened -- a crime, a missed garbage collection, a school expulsion -- and another to understand where the problem started. Conventional data often lacks clues about causes, but text reveals a lot.
Chris Bowman, president of Educational Analytics and Logistics in Louisiana, explained to us how root-cause analysis works, using student expulsions as an example. "You would read that a student was in trouble for disrespect for authority, but the incident started as a question about not having a shirt tucked in, or a silly comment in class. How could a molehill become the Himalayas?" He realized that, if he saw this pattern several times in individual reviews, there would have to be more, but it was impossible for him to read tens of thousands of student records to find out. That's how he became one of the education industry's earliest adopters of text analytics.
These stories are just the beginning. Every city has problems, and every city has lots and lots of text. In the next few years, we'll be hearing many more tales of text and the city.
Is your city exploring text analytics? Do you wish it would? Share your stories.
There is more data to be captured than text alone. 311 can see location data and while 911 will see much more info like the exact adress or name of cell phone user. A caller will be identified and I think more can and will come of text analytics.
I don't know how much time those guys actually spend at U of C. Brett Goldstein is also CTO at Grosvenor Capital, and I've seen Rayid Ghani at a speaking gig that was obviously paid, so clearly he has other irons in the fire.
Hi Meta, I see I was right that you'd have some thoughts to share on Chicago and its data strategy. I had the chance to hear Goldstein do a presentation a while back and he did seem a sharp and determined guy with some good ideas to build on. But, being a native Chicagoan, I must confess to have been thinking the whole time I heard him speak, and during a quick followup afterwards, "Ya, right." As you say, Chicago government is still Chicago government -- and change certainly doesn't come easy.
Seems U of C's Harris School of Public Policy is amassing the local data talent; I believe that's where Rayid Ghani, the Obama campaign's chief data scientist, is too -- at least in part.
I wonder what became of Goldstein's efforts to share data insights across muncipalities. I recall him trying to team up with NYC and Phillie data guys for some lessons learned and best practices sort of undertakings. I'll have to check it out.
As Mary Jander has acknowledged, analysis only means as much as what you're prepared to do with it. I wish I could say that my city is all about data-driven decision making, but there's a lot of evidence to the contrary.
Within the Chicago police force, it's my understanding that there has been serious investment in analytics, and that there are people who are serious about using the information. Brett Goldstein, the former Chief Data Officer of Chicago whom Mary also mentioned in her post, came to that role after working for the Chicago police. So, there are certainky spots of serious use of data in Chicago government.
The city established a new role - Chief Data Officer. And they hired a very capable person for the job. But they didn't provide him with money for staff, software, or much of anything else.
Brett Goldstein is a resourceful guy. He managed to attract a lot of expert volunteer labor. And he used open source software. So he made a lot from a little. Now the city has a portal for datasets. And it has some interesting analyses performed by talented and generous volunteers.
But, so what?
I have seen some good data journalism, interesting ways that reporters have integrated data (especially crime data) from the city's portal into stories and visualizations, sometimes enriching these with additional data sources. What I have not seen is one credible example of the city doing anything positive as a result of establishing the portal, or hiring a Chief Data Officer, or claiming that Chicago government will be transparent.
Chicago government is still Chicago government.
Nothing revolutionary is happening here. I'm all for making data available, and making it easily accessible. But it's important to recognize that the data being shared was already public information. Goldstein's emphasis on open source software should not be taken as evidence of superiority of open source for municipal purposes. I have personally heard him sing the praises of certain commercial software. He went with open source because he had no money and no choice. Brett Goldstein has left the job. No doubt most of the volunteers have flown the coop, too.
While the Chief Data Officer and the data portal were getting lots of press, here's a story that didn't. The Metro Chicago Information Center (MCIC), a nonprofit that collected demographic data and performed research - exactly the type of data and research that are appropriate to inform public policy decisions - folded.
If Chicago is ever to make decisions driven by honest and competent analysis of data, we'll need fundamental political change. That kind of change is hard to come by, especially in Chicago.
Hi Meta, I came across this blog today about Chicago's data use and, since this post inspired the writer to take a look at what's going on here, I thought I'd share. If you have a chance to read it, let me know if your impressions match up with hers! Chicago's Data Projects: Overblown or On Target?
@Meta. you touch on a good point. The whole notion that intel services failed to "connect the dots" before 9/11 was in part a myth created by the administration to point fingers elsewhere. The truth was, they were being told that specific terrorists posed a threat. They were told that terrorists planned to smash planes into buildings. And they chose to make their priorities elsewhere.
It's not for no reason that there are certain boundaries on information sharing between agencies. The NSA has access to information I strongly feel they should not have in the first place. And though we hear about that, it's only one of the places where there is room for debate about what data the government is able to obtain about citizens without any particular reason - commercial data vendors offer all sorts of goodies to law enforcement agencies.
It's not the volume of data, but the relevance of data that's important. And yes, the willingness and ability to take proper action.
Is more data, or more data sharing, the key to preventing terrorism? I don't believe so.
Remember that the director of the CIA warned Condoleezza Rice of the looming Al Qaeda threat before the September 11, 2001 attacks. The information was available to the right people at the right time, without the Patriot Act, and without any new policies for interagency information sharing. But it did not matter.
The NSA had its monster database well before the Boston bombings. And that didn't matter, either.
If those big old databases were of real value, we'd hear about a lot of convictions for crimes related to terrorist conspiracies. But the federal courthouse near me is busy with trials for politcal corruption and tax evasion, not terrorism.
I know that these cities are trying new things, and that's something I appreciate. But they are not necessarily avoiding the major mistake we discussed in the broadcast - which was the failure to begin with a viable plan to use the information in a way that could be realistically expected to yield positive returns.
The Alexandria police have been deep in data analysis for a number of years, and they have publically discussed some examples of actually taking action based on data analysis and having measurable results to show. For example, I recall a case where they used predictive analytics to determine hotspots for gun crimes, and used that information for police dispatching on New Year's Eve, resulting in a significant decrease in that particular type of crime (shooting guns as a form of celebration) on an occasion when it was known to be a problem. Perhaps one key element of that story is that it was the police force themselves who invested in data analysis.
On the other hand, I can't say that I see equivalent commitment in the example of my home, Chicago. Although our police force does have a significant involvement in anallytics, and has for some time, the new data initiatives put in place by Mayor Rahm Emmanuel may be little more than window dressing. Brett Goldstein, a former police officer who was first to take on the city's Chief Data Officer role, has already left the job. He went in with nothing more than volunteers for staff, and no budget for tools. The city has established a data portal, but as for stories of properly measured results from successful applications - please let me know if you hear of any such reports.
I just want to say I wan't arguing the value of the call text just that if they are shifting focus away from current metrics I think that they need to be careful. The text could man a deeper look at the problems but for the high level if a public service isn't catching caller info and tracking problems by type they are doing some things very wrong.
Diego Klabjan, chair of the INFORMS University Analytics Program Committee and program director for Northwestern University's Master of Science in Analytics program, gives his advice for figuring out where to get an advanced analytics degree.
What Works: Open Source Analytics Software International Institute for Analytics WebinarOn Wednesday, Sept. 24, join IIA CEO and Co-Founder Jack Phillips, along with featured guest Gary Spakes, as we explore the five modernization stages that analytics hardware/software have experienced. We will discuss the considerations when calculating total cost of ownership of the analytics ecosystem.
2014 VA Interactive Roadshow -- Cary, NCThe 2014 VA Interactive Roadshow will feature SASŪ Data Management and SASŪ Visual Analytics experts covering topics like prepping data for VA and VA integration with SASŪ Office Analytics. This year's events will keep presentations at a minimum and focus on giving attendees hands-on exposure to the latest version of VA.
Essential Practice Skills for Analytics Professionals Drawing on best practices from the field, this INFORMS course helps analytics professionals add value from beginning to end: listening to clients, framing the central problem, scoping a project, defining metrics for success, creating a work plan, assembling data and expert sources, selecting modeling approaches, validating and verifying analytical results, communicating and presenting results to clients, driving organizational change, and assessing impact.
Analytics 2014 The Analytics 2014 Conference is a two-day, educational event for anyone who is serious about analytics. This annual event brings together hundreds of professionals, industry experts and leading researchers in the field of analytics. All Analytics members save $500 on conference fees by using promo code ACAA.
Premier Business Leadership Series 2014 The Premier Business Leadership Series is an exclusive event for senior executives and decision makers that focuses on solving the current issues that affect governments and businesses globally. The Series is a unique learning and networking experience focused on the most innovative leadership strategies and analytic solutions for competing in todayâs global economy.
2014 VA Interactive Roadshow -- BostonThe 2014 VA Interactive Roadshow will feature SASŪ Data Management and SASŪ Visual Analytics experts covering topics like prepping data for VA and VA integration with SASŪ Office Analytics. This year's events will keep presentations at a minimum and focus on giving attendees hands-on exposure to the latest version of VA.
Data Exploration & Visualization Get hands-on training that focuses on the critical steps in the process of analyzing data: accessing and extracting data, cleaning and preparing data, exploring and visualizing data. This INFORMS course will use several of the most popular software tools intensively, and provide an overview of the range of software options.
Foundations of Modern Predictive Analytics In this INFORMS course, learn about modern predictive analytics, the science of discovering and exploiting complex data relationships. This course will give participants hands-on practice in handling real data types, real business problems and practical methods for delivering business-useful results.
2014 VA Interactive Roadshow -- AtlantaThe 2014 VA Interactive Roadshow will feature SASŪ Data Management and SASŪ Visual Analytics experts covering topics like prepping data for VA and VA integration with SASŪ Office Analytics. This year's events will keep presentations at a minimum and focus on giving attendees hands-on exposure to the latest version of VA.
LEADERS FROM THE BUSINESS AND IT COMMUNITIES DUEL OVER CRITICAL TECHNOLOGY ISSUES
The Current Discussion
Visual Analytics: Who Carries the Onus? The Issue: Data visualization is an up-and-coming technology for businesses that want to deliver analytical results in a visual way, enabling analysts the ability to spot patterns more easily and business users to absorb the insight at a glance and better understand what questions to ask of the data. But does it make more sense to train everybody to handle the visualization mandate or bring on visualization expertise? Our experts are divided on the question. The Speakers: Hyoun Park, Principal Analyst, Nucleus Research; Jonathan Schwabish, US Economist & Data Visualizer
The hospitality industry gathers massive amounts of customer data, and mining that data effectively can yield tremendous results in terms of improved CRM, better-targeted marketing spend, and more efficient back-end processes. Roger Ares, vice president of analytics at Hyatt Corp., discusses the ways he and his staff use big data.
Charged with keeping track of travel assets, including employees, iJET International relies on data management best-practices and advanced analytics to keep its clients in the know on current and potential world events affecting travel, Rich Murnane, Director of Enterprise Data Operations & Data Architect, told All Analytics in an interview from the 2014 SAS Global Forum Executive Conference.
Jason Dorsey, chief strategy officer for the Center for Generational Kinetics and keynote speaker at last month's SAS Global Forum 2014, describes how Gen Y professionals are enhancing the makeup of multigenerational analytics organizations.
From analytics talent development to the power of visual analytics, All Analytics found a variety of common themes circulating throughout the exhibition floor and session discussions at the 2014 SAS Global Forum and SAS Global Forum Executive Conference events held last month in Washington, DC.
Talking with All Analytics live from the 2014 SAS Global Forum Executive Conference, Eric Helmer, senior manager of campaign design and execution for T-Mobile, discussed the importance of customer data -- starting internally -- in devising the mobile operator's marketing plans.
The big-data analytics market can be a confusing place. Among the vendors vying for your dollars are traditional database management providers, Hadoop startup services, and IT giants. In this video, All Analytics editors Beth Schultz and Michael Steinhart sit down in a Google+ Hangout on Air with Doug Henschen, executive editor of InformationWeek. Henschen discusses use cases for big-data analytics, purchase considerations, and his recent roundup of the top 16 big-data analytics platforms.
At the National Retail Federation BIG Show last month, All Analytics executive editor Michael Steinhart noted a host of solutions for tracking and analyzing customer activity in retail stores. From Bluetooth beacons to RFID tags to NFC connections to video analytics, retailers must find the right combination of tools to help optimize the shopper experience, streamline operations, and boost revenues.
The days when historical shipment trends and gut feelings were enough to forecast retail demand accurately are long over. SAS chief industry consultant Charles Chase outlines the benefits of pulling real-time sales information from point-of-sale and product scanner systems, then flowing that data into dynamic forecasting tools from SAS.
With today's advanced visual analytics tools, you can stream data into memory for real-time processing, provide users the ability to explore and manipulate the data, and bring your data to life for the business.
Dynamic data visualizations let analysts and business users interact with the data, changing variables or drilling down into data points, and see results in a flash. Advance your use of data visualization with tools that support features like auto-charting, explanatory pop-ups, and mobile sharing.
No doubt your enterprise is amassing loads of data for fact-based decision-making. Hand in hand with all that data comes big computational requirements. Can traditional IT infrastructure handle the increasing number and complexity of your analytical work? Probably not, which is why you need a backend rethink. Big data calls for a high-performance analytics infrastructure, as Fern Halper, a partner at the IT consulting and research firm, Hurwitz & Associates, discusses here.
Redbox's bright-red DVD kiosks are all but ubiquitous these days, located in more than 28,000 spots across the country. Jayson Tipp, Redbox VP of Analytics and CRM, provides an insider's look at how the company has accomplished its phenomenal nine-year growth.
InterContinental Hotels Group (IHG), a seven-brand global hotelier, has woven analytics into the fabric of its operations. David Schmitt, director of performance strategy and planning, shares IHG's analytics story and his lessons learned.