Machine-Based Sentiment Analysis is Flawed
So, it’s sad, but kind’a comical too to see how quickly the everyday use of the web for communication is eroding everyone’s grammar and syntax (cough, Twitter). What’s truly tragic, however, is the frigg’n pandemic spread of companies promising machine based linguistic and sentiment analysis services, all of them knowing oh too well that the web has damn-near its own dialect now, be it acronyms (FTW!), abbreviations (RT) or any number of adhoc classifications (#[hashtag]), and maybe more importantly, a growing appetite for unspoken gestures of expression and opinion (be it thumbs, stars, likes, or otherwise), yet, for whatever reason, these companies continue to over-promise mountains of insight and perspective into “how your customers think and feel,” based only on what a bot and an algorithm spits back!? I don’t know, it’s just, uh, flawed. Update: check out Microsyntax.org, this entire organization is diving into the ‘new’ unconventions of communication on the web.
Hayden 8:46 am on May 20, 2009 Permalink |
I know…. but its the best we currently have without getting a human to analyse it.
mike manuel 8:49 am on May 20, 2009 Permalink |
what’s wrong with using silly humans?
Hayden 8:54 am on May 20, 2009 Permalink |
Nothing, they are the best form of artificial artificial intelligence there is;
http://press20.blogspot.com/2008/09/artifical-artificial-intelligence.html
mike manuel 8:57 am on May 20, 2009 Permalink |
yeah, mechanical turk is interesting, flawed in it’s own ways, but interesting. at very least, it’s addressing the ‘human analysis can’t scale’ challenge that’s often voiced.
Hayden 9:02 am on May 20, 2009 Permalink
I’ve yet to see anyone that has automatically linked the Turk to a sentiment engine…. now that would be useful!
Although you’d have a variable cost from the Turk, you could set a maximum budget (like a Google PPC campaign).
Dave C. 8:57 am on May 20, 2009 Permalink |
I’m still going with John Henry over the jankety steam engine in this case.
mike manuel 9:02 am on May 20, 2009 Permalink |
haha, nice. me too.
Tac Anderson 9:04 am on May 20, 2009 Permalink |
It’s the classic tech biz problem of trying to provide customization at scale. The analytics will get better to incorporate these shifts in Web comm but they will always be behind and will always require a certain amount of human filtering/qualifying.
Rich Reader (WOM-buzz) 9:33 am on May 20, 2009 Permalink |
for the moment, people are still necessary (best qualified) for the scoring and classification components of sentiment analysis. If this moment should pass, then judgment day will have arrived (Terminator-style), wherein sentiment analysis will cease to be a service that is sought-after, and there will be no bail-outs.
Mike Driehorst 9:46 am on May 20, 2009 Permalink |
Of course it’s flawed — both the attempt and the way companies promote that service.
However, having a machine do it is a good starting point for us people to then dig into the results to determine their validity (how close or if the machines are accurate).
To do it in large scale by people is very time-consuming when having the machines take the first step followed by people power is a step in the right direction.
mike manuel 9:52 am on May 20, 2009 Permalink |
agreed, i think the ‘hybrid’ approach (to tac’s point above) is our best bet, but for you, me and so many others to know this and still see so many companies hucking sentiment analytics as a ‘solved problem’ is both the disturbing and flawed part of it all…
Rich Reader (WOM-buzz) 10:00 am on May 20, 2009 Permalink |
If you have a large and persistent data stream, do you then obtain stable classification error rates (type A and B)? If so, how high a pair of error rates leaves you better off than human-based coring and classification? If not, how do you place a value on the output?
Kevin Murphy 9:49 am on May 20, 2009 Permalink |
Some analyis and listening is better than none. Automated sentiment scoring is a great flagging tool for spotting key issues. But as a measurment tool, it sstill needs to eveolve.
mike manuel 9:56 am on May 20, 2009 Permalink |
yeah, i wasn’t implying it’s an all or nothing thing, kevin, just that if you’re investing solely in machine based analytic systems and services, just understand they’re over-promising and under-delivering right now.
paullyoung (Paull Young) 10:23 am on May 20, 2009 Permalink |
Twitter Comment
I agree wholeheartedly with @mmanuel “machine based sentiment analysis is flawed” [link to post]
larry levy 11:47 am on May 21, 2009 Permalink |
Mike – Most people well versed in the space recognize this as a tough problem to solve but still worth trying.
We’ve actually found that the results based on traditional vs non traditional don’t vary as much as you’d expect. This space is being somewhat tainted with companies/programmers just slapping a dictionary onto entity extraction and thinking they’ve nailed the problem.
As we improve our algorithms, we’re achieving results in the high 80%’s in terms of accuracy on negative and positive. It’s the neutrals that cause the most grief – even human inter annotator agreement on these is very low – somewhere between 55%-60%.
marlinex (Marilyn Clark) 8:55 pm on June 1, 2009 Permalink |
Twitter Comment
Machine-Based Sentiment Analysis is Flawed [link to post] [Not convinced yet that humans can be replaced in this case] – Posted using Chat Catcher