What is going on here when half of us can't even get access to the data that defines our companies and then over half of those with access don't even both to use this data? Don't we live in an age of analytics, big data, machine learning, automation, and digital transformation? Aren't data architects, data analysts, and data scientists the most important employees in the company? Then why isn't anyone using the data and accessing the analytic results?
Getting past the buzzwords, one of the key challenges in analytic adoption goes back to understanding how humans prioritize their own behavior and how analytic development has mostly ignored the human nature of using data.
Consider the development of analytics in context of Maslow's Hierarchy of Needs. This framework, first proposed in 1943, consists of five layers of prioritization used to define human motivation:
- Physiological needs
- Safety needs
- Belongingness and love
From a business perspective, these five needs match up with
- Basic operational functionality
- Security and Governance
- Collaboration, Team work, and project work
- Business accomplishments
- Holistic excellence
Together, the comparisons of personal and business motivations look like this:
Now, consider how business intelligence is typically supported within an organization. There is a lot of focus on building a data repository, supporting data integration and quality, building reports, scaling infrastructure, defining data models, and providing core metrics for the business. This work is all challenging, but ultimately never gets past the initial "physiological" and "safety" needs of supporting basic analytic functionality. As long as analytics and business intelligence focuses on how to manage and govern models, data marts, and report, adoption will always stay in the 15-20% range that market research has tracked for the past decade.
To move forward and develop higher level adoption, businesses must build on the decades of analytic experience that they have developed and start focusing on the motivations that drive higher levels of usage, adoption, and satisfaction. This requires building both a collaborative BI environment and business-based performance indicators that define the success of analytics above and beyond data quality initiatives. I recommend using a FEAST framework to enable end users to effectively adopt data and analytics: Find, Explore, Augment, Statistics, Trust
Finding the right data: Data must be aligned to each knowledge worker community in the organization. There is no reason that employees should not be able to find and access the information that is relevant to their jobs. This means that the business alignment of data ends up being an important stage of driving effective adoption. To make this happen, each department looking for data needs to assign at least one employee to work with a data analyst on defining data requirements, sources, and attributes that are needed. Top down approaches without significant business input will only lead to a status quo adoption of BI where only those who already are comfortable with older generations of business intelligence technologies will end up using these tools.
Exploration of data: Once relevant data is available, end users must be able to explore the data. The concept of data discovery transformed business intelligence because it allowed savvy business users to test hypotheses and directly access enterprise data. But a dirty secret of the BI world is that data discovery tools are still not all that easy for line-of-business users to learn. Drill-down visualizations, natural language interfaces, wizards that guide users towards data and visualization, and emerging machine learning capabilities that figure out what users are looking for over time are all necessary to support a greater number of business intelligence users with a guided and contextualized exploration of data. Empower employees to move past the static report and drill into relevant financial, operational, and transactional data.
Augmenting data: Once the initial exploration takes place, employees will start thinking about new data sources that should be brought in and linked to existing enterprise data. It could be the combination of Operational Technology with Information Technology, the use of third-party data sources such as weather and census data, or basic combinations of hierarchical and transactional data that have previously been disconnected. Give employees the power to connect the dots through a combination of self-service data linkages, enterprise integration, and curated "data lakes" where information may be brought together en masse.
Statistical analysis: The initial exploration of data will inevitably lead to questions of whether certain data relationships are "significantly" connected or related. End users need access to statistical analysis and models. This can include access to statistical libraries for a programmatic approach to statistical analysis or a more automated approach to suggest statistical models, relationships, and correlations to the user. As users get access to statistical analysis, their relationship with data will become more powerful. However, in taking this step, Amalgam recommends that every knowledge-based employee should be trained with a basic understanding of how to read and label charts, the concepts of false positives and negatives, and the dangers of overfitting and extrapolating data. This doesn't have to be a formal statistics course, per se, but employees need to understand the limitations and challenges associated with using statistics improperly so that they have proper context for the results that they receive.
Trusting the Results: With business-identified data, user-driven exploration, data enhancement, and a combination of statistical analysis and machine learning to further validate results and drivers, employees are more likely to get accurate results. But the true test of analytic adoption is whether users can actually trust these enhanced results. It is easy to look at a straightforward chart of production, revenue, or service ticket management and figure out what a "good" month or a "bad" month is. But it is harder to augment the data, use statistical analysis, and look at a result that is not consistent with your world view or your experience. For instance, what happens if the data states that working fewer hours leads to greater overall productivity? Users who are savvy enough to either see that the data is self-selecting or to find that the data has provided a counterintuitive insight will be able to "trust" the results best because they will understand how to properly contextualize data within their working environments. And trust leads to greater adoption.
By building a FEAST framework for every employee, organizations can increase both the personal and business motivation to look at BI and analytics as more than just a compliance tool or necessary evil. By aligning analytics to corporate business goals and making enterprise data a trusted partner for key decisions, data architects and analysts play a vital role in expanding the adoption of analytics and data in their organizations.