Automated Vs. Human Analysis

Author: Michael Woldhuis

• Tweet

• Tweet

 

The power and wide reaching abilities of Social Media search tools (both free and paid), make them an essential part of any social media monitoring plan, but for all their bells and whistles, these automated search tools cannot completely replace the abilities of a human.

While humans don’t come close to competing with machines in terms of search capacity and processing mass amounts of data in a short period of time, there are aspects of monitoring where machines cannot completely replace the quality of a smart, well trained, and careful human analyst, and those aspects also happen to be the most important.

At the 2011 NewMR Text Analytics event in March, the consensus view of the 12 speakers was that currently, manual or human analysis is a key element in using social media monitoring. Although humans cannot compete with the speed and consistency of automated systems, they are needed for judgment, insight, and interpretation of those results; both automated tools and humans need to work together in partnership.

Data Cleaning

Before data (mentions or comments) can be analysed and assigned sentiment, the data must first be cleaned (removal of noise). The problem that automated tools have when it come to cleaning the data (removing the false positives or noise) is that the software has significant issues deciding on what is relevant and what is not.  The main difficulties automated tools have are; Allocating an accurate location of a mention (due to API restrictions), differentiating between “Bill” (Bill Cosby) and “Bill” ($10 Bill), and processing colloquialisms and sarcasm. To ensure that accurate and useful data is the only data being analysed, data cleaning is a process that can only be done effectively by a trained human analyst.

Sentiment Analysis

The majority of search tools offer some form of automated “Sentiment Analysis”, they allow their automated systems to scan a result for language and tone to determine if the post is of a positive, negative or neutral nature. Although these systems can assign a sentiment in a matter of milliseconds, they are prone to making “inhuman” and obvious mistakes, especially when it comes to processing slang, jargon and sarcasm within some text. Automated tools also have great difficulty in deciphering back handed compliments and mixed message posts, such as “That new car looks awesome, but for $70,000, they’re kidding themselves”.

A study conducted in 2010 by UK agency Freshminds tested a range of different social media monitoring tools and found that comments were, on average, correctly categorised positive, negative or neutral only 30% of the time. With no tool achieving an accuracy rate of much higher than 50%. Automated sentiment was described as being less accurate than flipping a coin. Matt Rhodes, the director of Freshminds sister company FreshNetworks stated, “For brands, the positive and negative conversations are of most importance and it is here that automated sentiment analysis really fails”.

Related posts:

1. How accurate is accurate?