Oh Happy Day!

What do Christmas, Valentine's Day, Mother's Day, and Independence Day have in common? They're all consistently happier than average days, as proven by big-data analytics.

Back in Where the Happy People Are, we talked about an analytics project that used two weeks' worth of Twitter feeds to measure happiness in New York City, organized by geotags. Last week, however, a BBC News article about big-data and city planning brought Hedonometer to our attention.

Hedonometer is a fascinating project, conducted jointly by the University of Vermont and Mitre Corp. Hedonometer maintains a repository of approximately 10,000 words, each ranked on a scale of 1 to 9 for its relative indication of happiness. Each day, it runs a random sampling of some 50 million tweets (about 100 gigabytes of data) through an analytics engine that looks for those words. (For you cloud enthusiasts out there, Hedonometer runs on Amazon's Elastic Compute Cloud and stores its data in the AWS Simple Storage Service. The daily computation takes three hours on 1,500 processors.)

The analytics engine generates a happiness score and a "word shift" analysis, which essentially measures the number and frequency of positive and negative words that appear each day. The result is a happiness metric for each day, along with an explanation of which words contributed to the score. Hedonometer has been plotting these out since September 2008.

Hedonometer for 2013

If you want to take a break from work, check out the chart on the homepage and drill down into specific years or dates. As you can see from this snapshot of 2013, Valentine's Day, Mother's Day, and Father's Day are high points, while April 15 -- the day of the Boston Marathon attack -- is decidedly low.

It's interesting to note that April 15 in general doesn't deviate much from the center. Does that mean most American Twitter users aren't filing IRS tax returns, or that they're not particularly stressed about it?

Questions like these come from a sense of skepticism about Twitter. Aren't most Twitter users teens? How trustworthy are tweets, anyway? But they're addressed in the site's remarkably candid FAQ:

Tweets represent a non-uniform subsampling of all utterances made by a non-representative subpopulation of all people. However, there are hundreds of millions of people presently using the website to express their activities and interests, and as such it is an important social signal.

The FAQ explains that Hedonometer uses Twitter for four reasons:

  • Its happiness results correlate with traditional surveys
  • Its "garden hose" feature delivers an enormous amount of data that must be processed in real time
  • Its metadata features can help analysts drill down into specific communities and geographic locations
  • Its role as a "collective, global media voice" makes it an important indicator worthy of analysis

The study of happiness is an inexact science, at best (especially if you ask readers of All Analytics), but the Hedonometer seems to be scientifically rigorous and reasonably transparent, too. The list of keywords is freely available, and the team is working to expand its analytics to phrases, non-English languages, and other data sources.

Members, had you seen Hedonometer before reading this post? What insights jump out as you manipulate the data? Share your thoughts in the comments.

Michael Steinhart, Contributing Editor

Michael Steinhart has been covering IT and business computing for 15 years, tracking the rising popularity of virtualization, unified fabric, high-performance computing, and cloud infrastructures. He is editor of The Enterprise Cloud Site, which won the Least Imaginative Site Name award in 2012, and he managed TheITPro.com, a community of IT professionals taking their first steps into cloud computing. From 2006 to 2012, Steinhart worked as an executive editor at Ziff Davis Enterprise, writing and managing research reports, whitepapers, case studies, magazine features, e-newsletters, blog posts, online videos, and podcasts. He also moderated and presented in dozens of webinars and virtual tradeshows. He got his start in IT journalism at CMP Media back in 1998, then moved to PC Magazine, managing the popular Solutions section and then covering business technology and consumer software. He holds a Bachelor of Arts degree in communications/journalism from Ramapo College of New Jersey.

Biosensors, Robots Replacing Human Guards

In prisons and other security settings, electronic devices are predicting and detecting anomalous behavior with high accuracy and reducing the need for human personnel.

Questioning the Ethics of Learning Analytics

Schools around the country are weighing the benefits and ethical questions of learning analytics.

  • 9/18/2013 10:15:24 PM

Beth, yes I do share my experiences not via Twitter but other methods. Sometimes I have a truly amazing meal and can't wait to share it with my friends. Very rarely have I had a meal so poorly executed that I wrote a negative review, but there have been a few times I felt it was necessary to warn others.

Re: And... so what?
  • 9/12/2013 9:58:01 AM

So Maryam, what about from the other side. Would you -- or maybe you do already -- share your bad dining experiences via Twitter? The one flaw I can see this type of app is that it's pulling bad dining data from tweets, specifically those that have made diners ill. If I'm suffering the effects of a bad meal, I'm not so sure tweeting about them would be a real priority for me.

Re: And... so what?
  • 9/12/2013 12:34:34 AM

I would use the app. Sometimes even with great research you encounter a restaurant that is just terrible. Why ruin a great evening and waste money on a non memorable dining experience, if your community of friends can give you the insight beyond the reviews?

Re: And... so what?
  • 9/10/2013 1:31:16 PM

Sadly, I find I make my best points when there's nobody around to hear them. :-)

Seriously, though, we talked about one good example on the site recently in the post Of Tweets & Toilets, Data & Analytics. To recap, researchers at the University of Rochester search for tweets indicating bad dining experiences, collates them, and then ties the locations to restaurant addresses and generates a map. The thinking behind the effort, called nEmesis, is that it can help people avoid risky places to eat in real-time. It's "an inexpensive way to enhance current methods to monitor food safety (e.g., adaptive inspections) and identify potentially problematic venues," the researchers say.

Would you use the app if you were planning on dining out in NYC, which is the city the researchers are mapping for now?


Re: And... so what?
  • 9/10/2013 12:19:13 PM

That's an interesting observation, Kicheko. I'd also add that twitter encourages people to use hashtags and essentially append metadata to each tweet. Facebook realized the value of that and announced support for hashtags recently, but it's way behind the curve.

Re: And... so what?
  • 9/10/2013 12:17:53 PM

Generally speaking, Beth, I do make the best points when I don't mean to. And I agree -- if you take Twitter's firehose feed, plus (I'm just spitballing here) Facebook and Instagram, you can get a decent sense of what's happening in the 25-and-under-set. I'd add that you could throw census data into the mix, but then we're talking about potential privacy violations. What kinds of ancillary feeds are there that you think would enhance the analytics potential of Twitter?

Re: Boost your happiness
  • 9/9/2013 11:47:43 PM

Broadway I think it can be both. I think there are people who derive happiness from both not just one. I am sure that there could be a measurement for which one garners more happiness?

Re: And... so what?
  • 9/9/2013 12:45:57 PM

Michael, fair enough. I think you raise a good point, though, whether you meant to or not! That is, we might find some interest in the the tweet analysis on its own, but what'll make it really interesting or useful is when used in combination with other data.


Re: And... so what?
  • 9/9/2013 10:38:19 AM

On the points as to why they chose twitter as a platform, i might add that twitter messages are observedly more thoughtful than facebook ones. Possibly because of character limits, one has often to choose consciously what goes into that limited space.

Boost your happiness & profits?
  • 9/9/2013 10:03:17 AM

Of course all these measurement don't necessarily tell us "why" the survey came out that way. That question might be more interesting and of some financial benefit. And then there's the sort of obviousness of many of the revelations as well. Maybe happy days are days of no working. Marketers could certainly take advantage of happy days and days when there's lots of free time for example.

Page 1 / 4   >   >>