Text Analytics: What It Is, Why You Need It


Gartner defines text analytics as "the process of deriving information from text sources." .

Text analytics are used for trying to find the key content across a larger body of information, analyzing the sentiments of individual writers, or classifying information into various topical areas.

To derive business insights from plain text, analytical software uses a combination of natural language processing in which the program actually deciphers and understands the human language that is being spoken; speech tagging in which the various words contained within a text are tagged based upon the part of speech they are; syntactic parsing in which strings of letters are analyzed and grouped into word phrase structures; and entity recognition, which identifies the names of persons, organizations, locations, etc. These text analytics programs can also use artificial intelligence to analyze sets of information and deduce a subjective expression or sentiment behind the words.

"Text analytics can help businesses listen to the right stories by extracting insights from free text written by or about customers, combining it with existing feedback data, and identifying patterns and trends," Terry Lawlor, a product manager at Confirmit, which provides solutions that focus on understanding the customer experience, wrote in a post for TDWI. However, Lawlor also acknowledges what others have noted in the industry -- that text analytics is still in early stages of adoption at most companies.

One reason adoption has been slow is that companies are still getting their arms around how to best plumb the depths of their unstructured data, including voice and text. Second, businesses have to develop internal competencies or find knowledgeable business partners so they can learn how to exploit text analytics to best advantage.

What are the use cases where text analytics is showing promise? Legal eDiscovery is one area where text analytics is achieving significant traction.

(Image: Happy Stock Photo/Shutterstock)

(Image: Happy Stock Photo/Shutterstock)

"In civil litigation, both sides of the dispute have the obligation to provide documentation," said John Tredennick, an attorney and CEO of Catalyst Repository Systems, a cloud-based eDiscovery service. "Years ago as a trial lawyer, when I first got into the discovery process of a civil litigation, we were looking at document populations of perhaps 30,000 documents. However, with the growth of digital documentation, in a major litigation this document population could expand to 20 million or 30 million documents...Now we use text analysis and predictive analytics in the document review process. By using this process, we often find that 75% to 80% of the relevance in a litigation can be found in a population of the most 6,000 highly ranked documents for relevance...When you're talking about an average cost of $2 per document for a manual review and you have 1.5 million documents to review, this can save companies a lot of money."

Libraries, universities and research institutes use text analysis to research trends and help explore the depths of vast repositories of data.

In one case, the U.S. Army Combat Readiness/Safety Center collected information on the circumstances of military vehicle crashes so it could determine if vehicle technology could have prevented the crashes. It used text analytics tools to review the text of 3,944 military vehicle crash narratives.

How do you decide if text analytics is for you?

1. You are a text-intensive business

If your business is life sciences, library research, marketing, sales, customer service, media or legal research, an analytics tool that helps automate information searches and narrows down the amount of text you have to look at can really help.

2. You want to understand how your customers feel

If your company wants to understand customer sentiment during sales and service, along with what customers are saying about you in social media tweets and comments, text analytics can give you this visibility.

3. You have the ability to start small

Starting any new phase of analytics is best done on small scale where you can see if you company is going to get the value out of the text analytics that you think it will. If you and your end users don't see a value, it's relatively easy to pull the plug. If the analytics work out great for the business, you can expand your work into new areas.

Mary E. Shacklett,

Mary E. Shacklett is President of Transworld Data, a technology research and market development firm. Prior to founding the company, Mary was Senior Vice President of Marketing and Technology at TCCU, Inc., a financial services firm; Vice President of Product Research and Software Development for Summit Information Systems, a computer software company; and Vice President of Strategic Planning and Technology at FSI International, a multinational manufacturing company in the semiconductor industry.  Mary has business experience in Europe, Japan and the Pacific Rim. She has a B.S. degree from the University of Wisconsin and an M.A. degree from the University of Southern California, where she taught for several years. Mary is a noted technology analyst and commentator who is listed in Who's Who Worldwideand in Who's Who in the Computer Industry.  She is a keynote speaker, and has over 1,000 articles, research studies and technology publications in print.

How to Gain IT Efficiencies With Analytics

As mid-sized companies venture into analytics, one of the best places to start in terms of finding efficiency is right in the data center.

Turbo Charging Analytics for Faster Decisions

With the focus increasingly on real-time analytics, there are some new infrastructure technologies that are doing their part.


Re: Understanding humans
  • 11/22/2017 11:52:56 AM
NO RATINGS

Lyndon, well put! I remember a song from the 60s that part of the lyrics said, 'the words get in the way '. I don't recall the artist, but it resonated with me as so true.

Re: Understanding humans
  • 11/22/2017 8:51:53 AM
NO RATINGS

Rbaz writes that "...I fully agree that we're a long way off in achieving 100 percent accuracy, in fact I don't believe we ever will. "

Let's face it. We humans ourselves are a long way from achieving 100% accuracy in our own lingual communications. Otherwise, we wouldn't have so many misunderstandings. Like arguments with spouses, significant others, domestic partners. Or so many White House clarifications of presidential Twitter messages. 

 

Re: Understanding humans
  • 11/21/2017 12:48:55 PM
NO RATINGS

Kq4ym, I fully agree that we're a long way off in achieving 100 percent accuracy, in fact I don't believe we ever will. Not only is the science in its enfancy but it's tackling a daunting mission. Human communication even with its rules, rely heavily on individual artistic expression which is hard to measure. Some level of success will be achieved on some specific specialized bases, but not over a wide spectrum of uses.

Re: Understanding humans
  • 11/21/2017 10:13:07 AM
NO RATINGS

It does seem we have a long way to go before 100% accuracy and the costs involved in processing language will be commonplace in lots of applications that might demand accuracy. But should be an amazing breakthrough when all languages can be processed with near perfect accuracy.

Re: Understanding humans
  • 11/21/2017 8:44:30 AM
NO RATINGS

I have talked with a number of researchers and they will all agree with you.

Re: Understanding humans
  • 11/21/2017 7:36:45 AM
NO RATINGS

@Lyndon: Not to be a Luddite, but good. Here's hoping it stays that way, lest we reach Singularity.

Re: Understanding humans
  • 11/20/2017 10:26:04 PM
NO RATINGS

Tomsg writes

I find it really hard to work with Human text. There are just so many variables and things that have multiple meanings that it will always be somewhat difficult.

Adequate text analytics (specifically, understading human language) is currently one of the most daunting hurdles confronting advancement in AI development.

 

Re: Understanding humans
  • 11/20/2017 1:44:08 PM
NO RATINGS

I find it really hard to work with Human text. There are just so many variables and things that have multiple meanings that it will always be somewhat difficult.

Re: Understanding humans
  • 11/19/2017 5:04:59 PM
NO RATINGS

Speaking as someone who has done this (manually), I can tell you that this doesn't seem to be a magic fix for everything, necessarily. One big doc review project I had years ago involved reviewing advertising and packaging -- where text may not have been readily legible. (Plus, imagery and positioning were important, too.)

Moreover, there was one key document I found in that project that I suspect typical text analytics would not have discovered because we didn't know we were looking for it until we found it. The document in question demonstrated fraud and extortion -- which opened up an entirely new cause of action and theory of recovery on top of what we were already pursuing!

Understanding humans
  • 11/16/2017 3:50:39 PM
NO RATINGS

Mary thanks for the insights, do you feel that text analytics have gotten advanced enough to detect the subtleties of how we communicate, humor sarcasm etc?

INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +