How Text Analytics & Text Mining Can Be Mutually Enriching

Beth, I always get nervous when software vendors talk about everything being automatic.  There is always a human touch – it just depends where and how.  Sometimes it is afterwards like your example, and sometimes it is prior – setting up the categorization rules or statistical models or a combination of the two.  One model I've pushed for years in terms of finding information is the hybrid model of a content management software with integrated text analytics capabilities helping authors and editors to add structure (metadata) to documents as they are published – using the strengths of both human and machine.

Beth, absolutely agree on the need for an interdisciplinary approach – in most things, but certainly for anything to do with getting value from text.  Hmm, we now have knowledge management departments in organizations, what about text analytics departments?  There would be a need for interdisciplinary cooperation between the TA department and other departments like IT, BI, CI, business units, and even sales.  And also, the TA department should be interdisciplinary – NLP experts, statisticians, linguists, programmers or script writers, librarian-taxonomists, cognitive modelers, predictive analysis experts, and business analysts along-side.  (Can you tell I was just reading about Bell Labs where they had theorists and practitioners sitting next to each other?)

And the model could scale down by having some people with multiple skills so even smaller organizations could have their own TA department. 

Right now, there are new departments / titles growing out of social media (Customer Experience analysts, etc.), so who knows maybe a TA department with connections throughout the organization might become a reality?

Certainly Homeland Security has lots of people doing text analysis – they bought copies of every single vendor of TA/TM software out there – which is why companies finally stopped citing them as a customer.  And yes, all those other fields have some TA/TM capabilities that they are using – but the question is how are they using them?  Sentiment analysis became its own "field" for a lot of reasons (social media hype and promise of more sales, etc.) but also because it uses a specific set of words (the sentiment dictionary).  The methods used in sentiment analysis are really a combination of TM and TA when it's done right, so it is a difficult but interesting task to figure out which if any of these specific application areas will take off and become its own field.

I'm giving a talk on future directions in social media next week at the Social Media Analytics Summit in San Francisco in which I talk about expertise analysis and behavior prediction as two possibilities but I'm not sure is they have the potential for becoming their own field like sentiment analysis – combination of scale of interest along with specific techniques and/or terminology.  Have to give that one some more thought. 

 

It seems to me that the examples you cite are more applications areas rather than different fields (yes, it's a somewhat arbitrary line).

One immediate example is sentiment analysis which more often than not is done with mostly text mining approaches (statistical) but with a large dictionary of sentiment terms.  The vendors like to focus on the strength of their software out of the box and downplay the need for customization.  But in this case, customization is basically adding categorization to look at the context around various sentiment statements.  This context is what is needed to distinguish things like sarcasm, but also even to distinguish things like conditionals ("I'd like "this feature" if it had better design") which can come out as positive "like this feature". 

As far as ontology development, the situation is similar in that you are trying to extract "facts" or triples which can also be ambiguous in a number of ways.  First, in terms of identifying the entities (classic example is Ford the car, the person, the company, and a way to cross a stream).  Second,  in identifying the relationships between those entities where you have some of the same kinds of issues as with sentiment – conditionals, etc.

As an aside, I'd like to see ontology development have a text mining piece in it that would function like the taxonomy example I used earlier.

Beth, on silo effect – I don't see it so much as a traditional silo, but rather that TA/TM is a platform technology and most organizations are organized around specific applications.  So while, for example, Business Intelligence depends on TA/TM, they don't consider it as fundamental to what they do and the unfortunate effect of that is to downplay its importance and be satisfied with simple, one dimensional approaches. 

In some ways it is like IT as a platform for building applications, but in this case the platform involves the actual language of the business functions that want to have the application.  This sets up an unusual situation, which is that business people would not assume they could develop an IT application because it involves a specialized language, but too often think that because they know the business language, that they can develop a TA application.  However, this overlooks the fact that having an expertise in a given subject doesn't mean that the person has the expertise to categorize/organize/analyze the contents of that expertise.

 As far as the TA/TM split that does seem to be somewhat of a silo effect – but in this case a silo based on different skills such as mathematics and language or knowledge organization (I'd say librarian but that title carries such negative connotations that I hesitate to use it these days. The trick is to figure out how to enable communication between silos without losing the benefits of a silo (they do exist for a reason).