So, it’s sad, but kind’a comical too to see how quickly the everyday use of the web for communication is eroding everyone’s grammar and syntax (cough, Twitter). What’s truly tragic, however, is the frigg’n pandemic spread of companies promising machine based linguistic and sentiment analysis services, all of them knowing oh too well that the web has damn-near its own dialect now, be it acronyms (FTW!), abbreviations (RT) or any number of adhoc classifications (#[hashtag]), and maybe more importantly, a growing appetite for unspoken gestures of expression and opinion (be it thumbs, stars, likes, or otherwise), yet, for whatever reason, these companies continue to over-promise mountains of insight and perspective into “how your customers think and feel,” based only on what a bot and an algorithm spits back!? I don’t know, it’s just, uh, flawed. Update: check out Microsyntax.org, this entire organization is diving into the ‘new’ unconventions of communication on the web.
May 20, 2009 at 8:46 am
I know…. but its the best we currently have without getting a human to analyse it.
May 20, 2009 at 8:49 am
what’s wrong with using silly humans?
May 20, 2009 at 8:54 am
Nothing, they are the best form of artificial artificial intelligence there is;
http://press20.blogspot.com/2008/09/artifical-artificial-intelligence.html
May 20, 2009 at 8:57 am
yeah, mechanical turk is interesting, flawed in it’s own ways, but interesting. at very least, it’s addressing the ‘human analysis can’t scale’ challenge that’s often voiced.
May 20, 2009 at 9:02 am
I’ve yet to see anyone that has automatically linked the Turk to a sentiment engine…. now that would be useful!
Although you’d have a variable cost from the Turk, you could set a maximum budget (like a Google PPC campaign).
May 20, 2009 at 8:57 am
I’m still going with John Henry over the jankety steam engine in this case.
May 20, 2009 at 9:02 am
haha, nice. me too.
May 20, 2009 at 9:04 am
It’s the classic tech biz problem of trying to provide customization at scale. The analytics will get better to incorporate these shifts in Web comm but they will always be behind and will always require a certain amount of human filtering/qualifying.
May 20, 2009 at 9:33 am
for the moment, people are still necessary (best qualified) for the scoring and classification components of sentiment analysis. If this moment should pass, then judgment day will have arrived (Terminator-style), wherein sentiment analysis will cease to be a service that is sought-after, and there will be no bail-outs.
May 20, 2009 at 9:46 am
Of course it’s flawed — both the attempt and the way companies promote that service.
However, having a machine do it is a good starting point for us people to then dig into the results to determine their validity (how close or if the machines are accurate).
To do it in large scale by people is very time-consuming when having the machines take the first step followed by people power is a step in the right direction.
May 20, 2009 at 9:52 am
agreed, i think the ‘hybrid’ approach (to tac’s point above) is our best bet, but for you, me and so many others to know this and still see so many companies hucking sentiment analytics as a ‘solved problem’ is both the disturbing and flawed part of it all…
May 20, 2009 at 10:00 am
If you have a large and persistent data stream, do you then obtain stable classification error rates (type A and B)? If so, how high a pair of error rates leaves you better off than human-based coring and classification? If not, how do you place a value on the output?
May 20, 2009 at 9:49 am
Some analyis and listening is better than none. Automated sentiment scoring is a great flagging tool for spotting key issues. But as a measurment tool, it sstill needs to eveolve.
May 20, 2009 at 9:56 am
yeah, i wasn’t implying it’s an all or nothing thing, kevin, just that if you’re investing solely in machine based analytic systems and services, just understand they’re over-promising and under-delivering right now.
May 20, 2009 at 10:23 am
Twitter Comment
I agree wholeheartedly with @mmanuel “machine based sentiment analysis is flawed” [link to post]
May 21, 2009 at 11:47 am
Mike – Most people well versed in the space recognize this as a tough problem to solve but still worth trying.
We’ve actually found that the results based on traditional vs non traditional don’t vary as much as you’d expect. This space is being somewhat tainted with companies/programmers just slapping a dictionary onto entity extraction and thinking they’ve nailed the problem.
As we improve our algorithms, we’re achieving results in the high 80%’s in terms of accuracy on negative and positive. It’s the neutrals that cause the most grief – even human inter annotator agreement on these is very low – somewhere between 55%-60%.
June 1, 2009 at 8:55 pm
Twitter Comment
Machine-Based Sentiment Analysis is Flawed [link to post] [Not convinced yet that humans can be replaced in this case] – Posted using Chat Catcher