You can experiment with it yourself by visiting Sentiment140, where you can analyze and calculate sentiment scores for tweets. Just enter one of your favorite brands to see what sentiment scores come up. I tried “Red Sox” and got a 78% positive sentiment!
But sentiment alone is not the end of the story. Research has shown that most of the messages posted on Twitter are truthful, but the service is also used to spread misinformation and false rumors, often intentionally. For example, immediately after the 2010 earthquake in Chile, when information from official sources was scarce, several false rumors posted and spread on Twitter exacerbated the sense of chaos and insecurity in the local population. Further, misinformation from social networks channeled through sentiment analysis as a component of machine learning algorithms can damage the reliability of these methods. Enter credibility analysis.
Striving for credibility
Credibility analysis is a relatively new but increasingly important area of research. Its purpose is to determine the credibility of information spread through social networks, using signals embedded in the content. This technology works for Twitter now, but it can be extended to include other social networks, as well as online news in traditional media and blogs.
Some work in this field centers on detecting deceitful campaigns. The Truthy Project at the University of Indiana is a good example of this approach. Other work looks for factors that can be used to approximate perception of credibility.
A seminal research paper for credibility analysis is Credibility Ranking of Tweets during High Impact Events by A. Gupta and P. Kumaraguru. Their research shows that extraction of credible information from Twitter can be automated with a high degree of confidence.
Tuning the detector
In order to train their credibility algorithms, Gupta and Kumaraguru used supervised learning classification. They recruited a group of human evaluators and set them to work on a training set of tweets. They could slot each tweet into one of four categories: (1) almost certainly true, (2) likely to be false, (3) almost certainly false, and (4) I can’t decide. The trick was to use feature engineering to define characteristics of tweets in each category. They identified four types of features:
- Message-based. The length of a message, punctuation, URLs, user mentions, number and frequency of positive/negative sentiment words, hashtags, and retweet status.
- User-based. Registration age, longevity of the account, number of followers, number of followees, and the number of tweets the user has authored in the past.
- Topic-based. The proportion of tweets with hashtags and the proportion of positive and negative sentiments.
- Propagation-based. Depth of the retweet propagation tree, or the number of initial tweets on a topic.
The number of features generated using the above categories may yield a large vector. It is therefore necessary to perform a best-feature selection process, which yields a smaller set. The sweet spot is around 15 features.
Profile of truth
For automated credibility assessment, researchers use supervised machine learning methods such as support vector machine (SVM), decision trees, Bayes network, and so on. As it turns out, the J48 decision tree algorithm (an open-source tool for data mining) yields the best results, with an accuracy rate of 86% correctly classified instances.
The work yielded these classification rules:
- Tweets that do not include URLs tend to be related to non-credible information.
- Tweets that contain question marks or smiling emoticons tend to be related to non-credible information.
- Tweets that include negative sentiment terms are usually related to credible information.
- A low percentage of tweets with positive sentiment terms tend to be related to non-credible information.
- Non-credibility also occurs when a significant fraction of tweets mention a user.
- Low-credibility information is mostly propagated by users who have not written many messages in the past. Users with a low number of followers tend toward non-credibility, as well.
- Posts that are retweeted numerous times are related to credible information.
Members -- what do you think about the accuracy and importance of credibility analysis?