NYC Probation Department Puts Open-Source BI to Good Use

Not too long ago, the New York City Department of Probation used Excel to analyze data on probationers and passed out pdf reports to managers, officers, and branch chiefs. Exchange of documents was anything but instantaneous, and errors might easily go undetected.

That was before the department launched a business intelligence and data integration initiative that changed everything. The department opted, not for traditional BI and data integration software from big-name players, but for an open-source option featuring ease of operation at a fraction of the cost.

Anibal Sempertegui, IT director, software, shared the department's BI strategy with the community yesterday during a live e-chat. The department uses the open-source BI platform from Pentaho to track 35,000 convicted felons for whom it is responsible and provide information to the 200 probation officers who supervise them.

“[Our] biggest gain is speed to return reports,” Sempertegui said.

Before the department deployed the BI software, IT sometimes took longer than a day to turn reports around. With the BI software, it has cut that time to just three hours on average, depending on the need to correct and approve data.

“Also we now allow users to create their own simple reports ad hoc,” said Sempertegui, adding that another great benefit of the new deployment is the overall efficiency with which the department is now able to share data.

While the department had initially looked at a variety of traditional BI and data integration software packages, all were out of its budget range. The open-source-based suite cost about 80 percent less, yet offered ease of operation, technical support, and other surprising functionality.

For example, the BI package provides the ability to “cleanse” data, allowing IT to identify what Sempertegui called “anomalies” in standard reports. That bad data gets resubmitted for verification and correction. Specifically, he said the software features “open fields” where data can await any necessary correction or validation before being forwarded for calculation and manipulation purposes.

Implementing and operating the open-source BI platform has been easy, not necessarily requiring extra technical knowledge on the part of staff, Sempertegui said. A week of training with a consultant and hands-on interaction was all the department needed to feel comfortable with the software. Afterwards, staff set up the project and had the system deployed within three months.

More analysis and data mining are likely in the department's future, he said.

What has been your company or organization’s experience with open-source BI or analytics software? Do open-source options compare well with traditional software? Please share your thoughts on the message board below.

Shawn Hessinger, Community Editor

Shawn Hessinger is a community manager, blogger, social media and tech enthusiast, journalist, and entrepreneur based in Northeastern Pennsylvania. He serves as community manager and blogger for, a business news and information Website, and contributes regularly to the online business news source, Small Business Trends. He is the founder of, an online content and media community, and has provided blogging and social media services and consulting for companies all over the world. He researches and writes on a variety of business, Internet-related, and other tech topics including business intelligence and analytics. He is also keenly interested in computer-aided data management as it relates to his various online ventures. A newspaper journalist with more than 11 years experience as a reporter and then managing editor, Shawn began blogging in 2006 and now provides a variety of consulting and outsourcing services in Search Engine Optimization, Web development, and online marketing to companies large and small. He is a strong advocate for the use of BI and related computer data management in business decision making, whether using software as a service (SaaS), cloud, or other applications, and in the opportunity these technologies provide to transform small startups and larger established businesses alike.

BCBSNC, SAS Team on Advanced Analytics

The key to improving heathcare outcomes is to look at individual needs, the companies say.

Spoofing, Privacy Greatest Barriers for Biometrics

In Wednesday's e-chat, we discussed the analytics of identification and whether the technology might find a bigger role one day in marketing intelligence.

  • 9/30/2011 4:34:56 PM

Hi, Shawn.

Thanks for this post.  It seems that open-source BI software offers a great deal of promise in government, education, and other "bureaucracies" in providing greater flexibility and efficiency in environments that are often inflexible and inefficient.

Was Sempertegui able to offer any other specific figures on the ROI they've realized?

Open source
  • 9/30/2011 10:30:25 AM


Shawn thanks for the article it’s a great example of using open source in the real world. I think open source has many benefits depending the organization and its needs. This is a great example.

Re: Great example of open source benefits
  • 9/29/2011 12:06:20 PM

@Shawn - Great post!!

Its great to see government using analytics for such important cause. With decisions as important as this, that have time sensitivity, its great to see them using analytics to make more informed choices at a greater speed.

Re: Great example of open source benefits
  • 9/29/2011 9:28:54 AM

Hi Pierre,

Yes, productivity was the main gain here as opposed to any spectacular new functionality. And certainly that productivity includes the ease of reporting which increases the overall speed at which users in the department can communicate and interact. Another component of this, of course, is that at some point in the future some sort of data mining will be possible and that will take the department's BI functionality to a new level, but at the moment not having to pass around PDFs is a huge step forward in simply making communications easier.

Great example of open source benefits
  • 9/29/2011 3:08:13 AM



This is a great example of open source BI benefits for a government agency, particularly regarding productivity. I really liked that the department found a way to develop ad-hoc reporting.