NYC Probation Department Puts Open-Source BI to Good Use

Not too long ago, the New York City Department of Probation used Excel to analyze data on probationers and passed out pdf reports to managers, officers, and branch chiefs. Exchange of documents was anything but instantaneous, and errors might easily go undetected.

That was before the department launched a business intelligence and data integration initiative that changed everything. The department opted, not for traditional BI and data integration software from big-name players, but for an open-source option featuring ease of operation at a fraction of the cost.

Anibal Sempertegui, IT director, software, shared the department's BI strategy with the AllAnalytics.com community yesterday during a live e-chat. The department uses the open-source BI platform from Pentaho to track 35,000 convicted felons for whom it is responsible and provide information to the 200 probation officers who supervise them.

“[Our] biggest gain is speed to return reports,” Sempertegui said.

Before the department deployed the BI software, IT sometimes took longer than a day to turn reports around. With the BI software, it has cut that time to just three hours on average, depending on the need to correct and approve data.

“Also we now allow users to create their own simple reports ad hoc,” said Sempertegui, adding that another great benefit of the new deployment is the overall efficiency with which the department is now able to share data.

While the department had initially looked at a variety of traditional BI and data integration software packages, all were out of its budget range. The open-source-based suite cost about 80 percent less, yet offered ease of operation, technical support, and other surprising functionality.

For example, the BI package provides the ability to “cleanse” data, allowing IT to identify what Sempertegui called “anomalies” in standard reports. That bad data gets resubmitted for verification and correction. Specifically, he said the software features “open fields” where data can await any necessary correction or validation before being forwarded for calculation and manipulation purposes.

Implementing and operating the open-source BI platform has been easy, not necessarily requiring extra technical knowledge on the part of staff, Sempertegui said. A week of training with a consultant and hands-on interaction was all the department needed to feel comfortable with the software. Afterwards, staff set up the project and had the system deployed within three months.

More analysis and data mining are likely in the department's future, he said.

What has been your company or organization’s experience with open-source BI or analytics software? Do open-source options compare well with traditional software? Please share your thoughts on the message board below.