UK agency Freshminds found that automated sentiment is less accurate than flipping a coin.

The power and wide reaching abilities of Social Media search tools (both free and paid), make them an essential part of any social media monitoring plan, but for all their bells and whistles, these automated search tools cannot completely replace the abilities of a human.

While humans don’t come close to competing with machines in terms of search capacity and processing mass amounts of data in a short period of time, there are aspects of monitoring where machines cannot completely replace the quality of a smart, well trained, and careful human analyst, and those aspects also happen to be the most important.

At the 2011 NewMR Text Analytics event in March, the consensus view of the 12 speakers was that currently, manual or human analysis is a key element in using social media monitoring. Although humans cannot compete with the speed and consistency of automated systems, they are needed for judgment, insight, and interpretation of those results; both automated tools and humans need to work together in partnership.

Data Cleaning

Before data (mentions or comments) can be analysed and assigned sentiment, the data must first be cleaned (removal of noise). The problem that automated tools have when it come to cleaning the data (removing the false positives or noise) is that the software has significant issues deciding on what is relevant and what is not.  The main difficulties automated tools have are; Allocating an accurate location of a mention (due to API restrictions), differentiating between “Bill” (Bill Cosby) and “Bill” ($10 Bill), and processing colloquialisms and sarcasm. To ensure that accurate and useful data is the only data being analysed, data cleaning is a process that can only be done effectively by a trained human analyst.

Sentiment Analysis

The majority of search tools offer some form of automated “Sentiment Analysis”, they allow their automated systems to scan a result for language and tone to determine if the post is of a positive, negative or neutral nature. Although these systems can assign a sentiment in a matter of milliseconds, they are prone to making “inhuman” and obvious mistakes, especially when it comes to processing slang, jargon and sarcasm within some text. Automated tools also have great difficulty in deciphering back handed compliments and mixed message posts, such as “That new car looks awesome, but for $70,000, they’re kidding themselves”.

A study conducted in 2010 by UK agency Freshminds tested a range of different social media monitoring tools and found that comments were, on average, correctly categorised positive, negative or neutral only 30% of the time. With no tool achieving an accuracy rate of much higher than 50%. Automated sentiment was described as being less accurate than flipping a coin. Matt Rhodes, the director of Freshminds sister company FreshNetworks stated, “For brands, the positive and negative conversations are of most importance and it is here that automated sentiment analysis really fails”.

Do you know how accurate your social media data is?

Social Media can be a noisy pastime, with all the self appointed media mavens the world over looking for their voice to be heard over what turns out to be a cacophony.

For a brand trying to understand what is being said about them and by whom – and whether those voices are authentic, informed or influential – is a difficult challenge.

The level of noise (i.e. irrelevant mentions), while varying from client to client, does cause concern for all when trying to identify key conversations and when measuring sentiment and volume of conversation around a set of keywords.

So my question is – would you trust and make decisions off data that is 60% accurate? How about if it was 70%?

Our experiences in working with a range of clients and looking at a variety of issues indicates that of the social mentions collected, even after substantial refinement and use of negative filtering, the noise can be as high as 30-40%.

So if you don’t manually filter these results to ensure they are directly related to your product, you are working with some dodgy data to say the least.

I would hazard a guess there are many agencies and brands today working with data that is too dirty to be of any real value or provide substantial insight.

While automated monitoring tools are great, they require human filter and review to ensure that the data is valid and relevant.

How accurate is your data?

Social Media Monitoring is replacing traditional Customer Service channels.

Conversations about your brand are occurring online, even if you don’t want them too, or if you can’t hear them. Social Media has provided the “soap box” for anybody who has an opinion and wishes to share it.

With the above in mind, it should not come as a shock that the number of businesses utilising social media to engage and communicate with their customers and fans continues to grow dramatically every year, if not every month.

With so many businesses now encouraging fan and consumer engagement through Facebook Fan pages, Twitter accounts, YouTube channels and a like, it is no surprise that consumers have turned what initially was thought of as an ideal marketing tool, into an outlet to air grievances.

The following story Utilizing Social Media for Customer Service  by Jason Miller outlines how companies are now using Social Media as a key customer service tool.  He also highlights how solving issues in real time through social channels shows dedication, and transparency, to your customers.

Are you interested in seeing how a brand can be tracked online? If so, sign up for a guest account to Online1984 to see our social analytics and insights at work.

Follow us on Twitter and connect with us on LinkedIn.

The Case Study below poses the question, could Kryptonite Locks have prevented the short and long-term damage to the company and its reputation if they had been listening to Social Media and had a Risk Mitigation plan in place?

A single forum post saw a bike lock that millions of cyclists had trusted with the security of their bicycles for over 40 years rendered useless by a 10-cent plastic pen. The backlash that ensued cost the lock company $10 million in replacements, and damaged it’s reputation so much that the company still hasn’t fully recovered almost 8 years later.

The Kryptonite lock, designed in the early 70’s revolutionized biked security. Prior to the invention of the patented Kryptonite U-shaped tubular-key lock, cyclists had to rely on heavy, case hardened hexagonal linked security chains, which could easily be breached by a low cost bolt cutter to secure their bicycles.

In 2004, rumours surfaced that the state-of-the-art Kryptonite Lock, that was at the time, and to this day still owned by Ingersoll Rand  could be unlocked, or cracked by a plastic ballpoint pen.

Chris Brennan, outraged that his $50 Kryptonite lock could be compromised by a simple cylinder shaped piece of plastic was the first to post a warning of the product fault on the popular cycling forum BikeForums.net, under the thread title “Your brand new U-Lock is not safe”. Soon after instructions on how to crack the Kryptonite lock were posted online, several videos surfaced online of people putting the steps outlined in the original post into action on a range of Kryptonite locks. Some of Kryptonite’s most lauded locks that claimed to be resistant to bolt cutters, hammers and saws were found to be susceptible to the plastic pen key product fault.

News of the product fault quickly became a popular topic for blogs, and within forums around the world, till eventually the story of the impenetrable lock that could be unlocked by a plastic pen reached one of the world’s most popular blogging sites, Engadget.  Shortly after appearing on Engadget, news of the product fault was published in the New York Times, and it was at this point when the “Blogstorm” and outrage surrounding Kryptonite hit its peak.

(Source: Fortune 2005)

Even as the controversy around the company increased online, Kryptonite remained silent; it was not until a week after the original post was published on BikeForums that Kryptonite addressed the situation publicly.

With outrage escalating, the statement issued by Kryptonite did nothing to quell the situation, the company claimed that the lock would still act as a deterrent to thieves, and that they were working on an improved model that was be released for sale at a later date.

With Kryptonite not offering any remedy to the problem, or expressing any sympathy to the 100’s of 1000’s who had purchased locks susceptible to the fault, the backlash directed at the company escalated even further.  Bloggers intensified their verbal assault on Kryptonite, and angry Kryptonite owner’s bombarded forums and media sites with comments criticising Kryptonite’s lack of compassion and action on the issue.

It was not until 10 days after the original post was posted that Kryptonite announced it would replace 100,000 Bic-pickable locks, at a cost to the company of $10 million.

The way in which Kryptonite reacted to the situation saw the company awarded with Business 2.0 Magazine’s Dumbest Business Moment of the Year Award in 2004.

The incident is also credited alongside the footage of Dominos Pizza employees disgustingly tampering with customer orders, and the “United Breaks Guitar” video as online videos that forced businesses to change how they view and use social media.

Donna Tocci, the Public Relations Manager for Kryptonite recently claimed that the company was well aware of Chris Brennan’s post and the storm of criticism that ensued, but chose not to react until a reactive plan was agreed upon.

Doubts surround the claims that they were monitoring social media at the time, due to the company failing to formally acknowledge the problem until after news of the fault appeared on mainstream media, leading many to believe they were blissfully unaware of the actual size of  situation prior to the Engadget and NYT articles.

If Kryptonite had actually been aware of the situation that had developed online, they would surely have reacted, or acknowledged the issue sooner than they did.

If a Social Media Monitoring plan had been implemented correctly, Kryptonite would have identified the original post, as well as the subsequent conversations that occurred immediately following Brennan’s post. They would have realised that the situation was escalating quickly, and that they had to address the situation sooner rather than later, to limit the damage to their company and their reputation.

If Kryptonite had acknowledged there was a problem and communicated that they were working on solution sooner, they could have prevented the tide of negative sentiment that eventually caused the story to hit mainstream media.

Could this be you? Are you actively listening to the community around your products and competitors? Do you have a risk mitigation plan in place? Situations like this are not a quick fix and can drag on for years.

If you type “Kryptonite Locks” into Google right now, of the top 10 results, two will be Videos of people breaking into Kryptonite locks, and the result that would come directly after Kryptonite’s Wikipedia page will be an article that was published on Wired, called “Twist A Pen, Open a Lock”. Eight years on, Kryptonite is still struggling to recover from the “Bic Pen” fiasco.

Benchmark:
It provides a reference point at a specific time. Online1984 delivers an actual numeric score based on a number of factors including individual influence, environmental impact and the intensity of the feeling. While the automated tools provide a positive / negative / neutral view of the brand, the measurement needs to be more robust.

Competitive:
Using the sentiment score of your own product / brand / company and comparing it with your competitors will enable you the ability to track the trends and to better understand your competitive standing. The score can also show things that influence and impact each businesses sentiment.

Gain further insight by taking this data and mashing it with others such as sales, search volumes, fuel prices etc and look for correlations. From this comes learning’s of when a certain set of circumstances occur, the likely outcome will be X. This allows a reasoned approach, a response versus an ill-conceived reaction and the start of a more sophisticated risk mitigation strategy.

Senior Management:
A numerical sentiment score is easy and quick way for senior management to get a top line understanding of what is going on with a particular brand. Outlining changes in daily sentiment scores and the underlying events leads to greater understanding of the impact of your companies advertising and marketing efforts.

Sales:
This may be the over-riding reason for understanding the sentiment towards your brand. Have you ever bought a product from a brand you didn’t like, your answer may be yes, but more likely it’s no.

We gravitate towards brands that reflect our view of the world or how we wish to be seen. If you’ve bought something from a company you didn’t like or had a negative perception of, you must have had very little choice.

It is obvious that if there is a positive perception of your business and products & services, there will be less obstructions to purchase.

For some companies the sentiment will be perceived just as an available metric or even a soft piece of information, but if the right question is being asked that is relevant to the communications strategy being implemented then it can be a powerful and insightful key performance indicator.

But like all data and information that is presented, it is the correlation, the application of context and expertise, including competitive information, marketing activity, economic and political influences that give real meaning to the measurement.

Having the data is just not enough anymore.