Turning Chicago Into the City of Big-Data

Long ago, poet Carl Sandburg etched Chicago into the world's mind as a hard-working, gritty, tenacious kind of a place:

HOG Butcher for the World,
Tool Maker, Stacker of Wheat,
Player with Railroads and the Nation's Freight Handler;
Stormy, husky, brawling,
City of the Big Shoulders...

Were he writing today, the description would surely be much different. Adjectives like "data rich" and "open architected" would find way into his characterization, as would "City of Big-Data." Consider, as two quick examples, that the city hosts one of the largest -- if not the largest -- crime datasets in the world and opens its data stockpiles though a public portal, data.cityofchicago.org.

Chicago, then, makes the perfect backdrop for Predictive Analytics World (PAW), taking place here yesterday and today. Of all the cities across the world, Chicago is making a name for itself in the way it treats its data (remember Predictive Modeling Keeps Chicago Beachgoers in Safe Waters?).

That Chicago so openly and aggressively works on its data modeling and information management strategies is a bit of an anomaly, admitted Brett Goldstein, acting commissioner of the just-weeks-old Department of Innovation and Technology -- a role he comes to from his previous posts as the city's chief data officer and, before that, director of the city's Counterterrorism and Intelligence Division's predictive analytics group -- during a PAW presentation on Monday. But he said he is working on changing that, noting creation of an intercity analytics working group among colleagues in Boston, Chicago, and Philadelphia.

In his presentation, "Lessons From Year One: Predictive Analytics in Government," Goldstein said: "Analytics and government -- everybody here knows predictive analytics and data mining are commonly used in the private sector, and everyone says [municipal government will] always be 10 years behind in technology. I think that's crap. That's not OK. We need to get our best out to the community."

Goldstein is all about openness -- among data professionals and between the city and its denizens. For one, there's that intercity analytics group in which he participates. For another, Goldstein said drawing on Chicago's engineering community is essential for furthering his data goals and so he has created an engineering cohort for project work. It's all about force multiplication: "We don't need to do government IT and research work in the traditional ways."

Goldstein's call for openness extends to the data architecture and data modeling, too. That's another must if cities are to deliver value. "People say, 'Oh analytics is too expensive for government' " -- and if you think about it in a traditional way, it can be. If your budget deficit reaches into the hundreds of millions, or even billions, then you really can't be paying $14,000 a seat for analytics software, wonderful as it might be, Goldstein said.

Rather, you have to think outside the box and bring in more cost-friendly options, like the R open-source programming language. You don't get a GUI, but it's still robust. Besides R, Goldstein likes programming with Python, and the city has successfully used MongoDB, an open-source noSQL database, and Hadoop, open-source software for distributing computing, to support various analytics projects.

For example, Goldstein's team recently built data visualizations for real-time situational awareness during the NATO Summit that took place without incident last month. It wrote a series of extract, transform, and loads (ETL) to grab data in a MongoDB database and create a spatial dataset. Then it mapped that data and delivered up a user-friendly visualization.

Goldstein shared lots of other cool data analytics projects during his presentation, and I'll be doing the same over time. But here's one critical takeaway from him for today: "One lesson to leverage from the past year is the importance of data integration throughout the enterprise. You need strong data science leadership, governance, and strategy going forward -- and I'm always looking for people who have an interest in ways to transform government."