Text & the City: Municipalities Discover Text Analytics

Text analytics is a new horizon for data-driven decision making in municipal government.

Why text analytics? If there's one thing local governments have in plentiful supply, it's text. Records of everyday government activities are loaded with it -- police reports, court proceedings, license applications, school records, and other documents from routine government business generate a sea of text.

Text sources can be rich in valuable information that does not exist in any other form. Consider the gems found in the notes of police reports -- details about weapons, behavior, past encounters with suspects or victims, and so forth. Locating relevant information in these records is a challenging process that has depended on memory and manual review in the past.

Now police officers in Alexandria, Va. -- a Washington, DC, suburb with a population of 150,000 -- are using text mining to track gang and drug-related violence. Text mining facilitates the identification of key concepts within things like police and crime reports, tips from the public, and news stories to help officers conduct research quicker and easier. Faster and more thorough searches mean better information for detectives and patrol officers.

Chicago city limits
Large cities are not about to let themselves be outdone by the suburbs. Chicago, for example, began exploring Twitter analysis to complement its 311 system more than a year ago. It now mines 311 calls to predict and prevent rat infestations.

Municipal 311 systems are fertile ground for text analytics opportunities. Records of past inquiries provide a tremendous resource for learning the kinds of information constituents want, what difficulties they face in finding it, and what their most frequent questions are. Manually organizing that information from thousands or millions of records is tough, but text analytics makes the process faster and more consistent.

Back to the roots
The exploration of text analytics by city governments is not just a US phenomenon. Hong Kong is using text analytics for its call center and paying particular attention to complaints, which represent about 10% of those calls. Why complaints? They carry hidden messages about root causes of problems with government services.

Identifying root causes is a unique value proposition for text analytics in government. It's one thing to know something happened -- a crime, a missed garbage collection, a school expulsion -- and another to understand where the problem started. Conventional data often lacks clues about causes, but text reveals a lot.

Chris Bowman, president of Educational Analytics and Logistics in Louisiana, explained to us how root-cause analysis works, using student expulsions as an example. "You would read that a student was in trouble for disrespect for authority, but the incident started as a question about not having a shirt tucked in, or a silly comment in class. How could a molehill become the Himalayas?" He realized that, if he saw this pattern several times in individual reviews, there would have to be more, but it was impossible for him to read tens of thousands of student records to find out. That's how he became one of the education industry's earliest adopters of text analytics.

These stories are just the beginning. Every city has problems, and every city has lots and lots of text. In the next few years, we'll be hearing many more tales of text and the city.

Is your city exploring text analytics? Do you wish it would? Share your stories.

Related posts:

Meta S. Brown, Business Analytics Consultant

Meta S. Brown is a consultant, speaker, and writer who promotes the use of business analytics. A hands-on analyst who has tackled projects with up to $900 million at stake, she is a recognized expert in cutting-edge business analytics. She has conducted more than 4,000 hours of presentations about business analytics, and written guides on neural networks, quality improvement, statistical process control, and many other statistical methods. Meta's seminars have attracted thousands of attendees from across the US and Canada, from novices to professors.

Tell Me a Story: Why Data Analysts Must Be Storytellers, Too

Data alone won't make an analyst's work memorable or actionable in the eyes of a business executive. A story puts it into perspective.

It's the Data, Stupid

When it comes to acquiring the data that will feed your analytics initiative, "free" isn't always the best approach.

Text & the City
  • 1/24/2014 10:12:53 AM

There is more data to be captured than text alone. 311 can see location data and while 911 will see much more info like the exact adress or name of cell phone user. A caller will be identified and I think more can and will come of text analytics.  

Re: The City of Big Data
  • 1/23/2014 6:41:14 PM

I don't know how much time those guys actually spend at U of C. Brett Goldstein is also CTO at Grosvenor Capital, and I've seen Rayid Ghani at a speaking gig that was obviously paid, so clearly he has other irons in the fire.

Re: The City of Big Data
  • 1/23/2014 5:23:11 PM

Hi Meta, I see I was right that you'd have some thoughts to share on Chicago and its data strategy. I had the chance to hear Goldstein do a presentation a while back and he did seem a sharp and determined guy with some good ideas to build on. But, being a native Chicagoan, I must confess to have been thinking the whole time I heard him speak, and during a quick followup afterwards, "Ya, right." As you say, Chicago government is still Chicago government -- and change certainly doesn't come easy. 

Seems U of C's Harris School of Public Policy is amassing the local data talent; I believe that's where Rayid Ghani, the Obama campaign's chief data scientist, is too -- at least in part.

I wonder what became of Goldstein's efforts to share data insights across muncipalities. I recall him trying to team up with NYC and Phillie data guys for some lessons learned and best practices sort of undertakings. I'll have to check it out.



Re: The City of Big Data
  • 1/23/2014 3:03:09 PM


Thanks for pointing this out!

As Mary Jander has acknowledged, analysis only means as much as what you're prepared to do with it. I wish I could say that my city is all about data-driven decision making, but there's a lot of evidence to the contrary.

Within the Chicago police force, it's my understanding that there has been serious investment in analytics, and that there are people who are serious about using the information. Brett Goldstein, the former Chief Data Officer of Chicago whom Mary also mentioned in her post, came to that role after working for the Chicago police. So, there are certainky spots of serious use of data in Chicago government.

The city established a new role - Chief Data Officer. And they hired a very capable person for the job. But they didn't provide him with money for staff, software, or much of anything else.

Brett Goldstein is a resourceful guy. He managed to attract a lot of expert volunteer labor. And he used open source software. So he made a lot from a little. Now the city has a portal for datasets. And it has some interesting analyses performed by talented and generous volunteers.

But, so what?

I have seen some good data journalism, interesting ways that reporters have integrated data (especially crime data) from the city's portal into stories and visualizations, sometimes enriching these with additional data sources. What I have not seen is one credible example of the city doing anything positive as a result of establishing the portal, or hiring a Chief Data Officer, or claiming that Chicago government will be transparent.

Chicago government is still Chicago government.

Nothing revolutionary is happening here. I'm all for making data available, and making it easily accessible. But it's important to recognize that the data being shared was already public information. Goldstein's emphasis on open source software should not be taken as evidence of superiority of open source for municipal purposes. I have personally heard him sing the praises of certain commercial software. He went with open source because he had no money and no choice. Brett Goldstein has left the job. No doubt most of the volunteers have flown the coop, too.

While the Chief Data Officer and the data portal were getting lots of press, here's a story that didn't. The Metro Chicago Information Center (MCIC), a nonprofit that collected demographic data and performed research - exactly the type of data and research that are appropriate to inform public policy decisions - folded.

If Chicago is ever to make decisions driven by honest and competent analysis of data, we'll need fundamental political change. That kind of change is hard to come by, especially in Chicago.

The City of Big Data
  • 1/23/2014 1:26:22 PM

Hi Meta, I came across this blog today about Chicago's data use and, since this post inspired the writer to take a look at what's going on here, I thought I'd share. If you have a chance to read it, let me know if your impressions match up with hers! Chicago's Data Projects: Overblown or On Target?

Re: Red Flags
  • 1/20/2014 9:13:28 PM

@Meta. you touch on a good point. The whole notion that intel services failed to "connect the dots" before 9/11 was in part a myth created by the administration to point fingers elsewhere. The truth was, they were being told that specific terrorists posed a threat. They were told that terrorists planned to smash planes into buildings. And they chose to make their priorities elsewhere.

Re: Red Flags
  • 1/20/2014 10:29:28 AM

It's not for no reason that there are certain boundaries on information sharing between agencies. The NSA has access to information I strongly feel they should not have in the first place. And though we hear about that, it's only one of the places where there is room for debate about what data the government is able to obtain about citizens without any particular reason - commercial data vendors offer all sorts of goodies to law enforcement agencies.

It's not the volume of data, but the relevance of data that's important. And yes, the willingness and ability to take proper action.

Is more data, or more data sharing, the key to preventing terrorism? I don't believe so.

Remember that the director of the CIA warned Condoleezza Rice of the looming Al Qaeda threat before the September 11, 2001 attacks. The information was available to the right people at the right time, without the Patriot Act, and without any new policies for interagency information sharing. But it did not matter.

The NSA had its monster database well before the Boston bombings. And that didn't matter, either.

If those big old databases were of real value, we'd hear about a lot of convictions for crimes related to terrorist conspiracies. But the federal courthouse near me is busy with trials for politcal corruption and tax evasion, not terrorism.


Re: A measure of success
  • 1/20/2014 10:10:34 AM


I know that these cities are trying new things, and that's something I appreciate. But they are not necessarily avoiding the major mistake we discussed in the broadcast - which was the failure to begin with a viable plan to use the information in a way that could be realistically expected to yield positive returns.

The Alexandria police have been deep in data analysis for a number of years, and they have publically discussed some examples of actually taking action based on data analysis and having measurable results to show. For example, I recall a case where they used predictive analytics to determine hotspots for gun crimes, and used that information for police dispatching on New Year's Eve, resulting in a significant decrease in that particular type of crime (shooting guns as a form of celebration) on an occasion when it was known to be a problem. Perhaps one key element of that story is that it was the police force themselves who invested in data analysis.

On the other hand, I can't say that I see equivalent commitment in the example of my home, Chicago. Although our police force does have a significant involvement in anallytics, and has for some time, the new data initiatives put in place by Mayor Rahm Emmanuel may be little more than window dressing. Brett Goldstein, a former police officer who was first to take on the city's Chief Data Officer role, has already left the job. He went in with nothing more than volunteers for staff, and no budget for tools. The city has established a data portal, but as for stories of properly measured results from successful applications - please let me know if you hear of any such reports.



Re: Monitoring text
  • 1/20/2014 8:20:10 AM

I just want to say I wan't arguing the value of the call text just that if they are shifting focus away from current metrics I think that they need to be careful.  The text could man a deeper look at the problems but for the high level if a public service isn't catching caller info and tracking problems by type they are doing some things very wrong.

Re: Red Flags
  • 1/18/2014 11:53:47 AM

Ah yes...let's give the government even more data and save it the trouble of stealing it without our consent

Page 1 / 2   >   >>