3-1. Assessment of the Sentimental Analysis Tools

Before conducting the actual sentimental analysis on the news headlines, it is preferable to test the credibility of each sentimental analysis software which was used for this research. This time, we used three different sentimental evaluation tools; “Corenlp made by Stanford University’s staff”, “Alchemy API by IBM”, and “Text Analytics by Microsoft”. 

In order to attain the proper efficiency of the software, we ran an experiment about how accurately those applications can evaluate the sentimental elements of short texts like news headlines. Also, we tried to find a method to make the calculated results plausible enough to elicit the logical conclusion. For this preliminary survey, we collected both “50 positive news headlines” and “50 negative news headlines” from the “Newsnow” that is a news site which has contents called “Bad news” and “Good news” where we can find possible “positive” and “negative” news headlines. It goes without saying that all titles were carefully examined to check whether the sorting was completely correct.

Then we ran those software to find out how precisely they could discern the sentiments in the way of “positive headlines” as “positive” and “negative headlines” as “negative”. One thing that needs to be noted for this investigation is that those applications have different styles. "Corenlp" is a Java-based program, so it can be performed on command prompt. Likewise, the other two software are provided as web-based tools, which enable researchers to use them relatively easier. As for the classification method, three different emotions “Positive”, “Negative” and “Neutral” were adopted as indicators for the experiment. Although “Corenlp” and “IBM’s Alchemy API” exactly can clearly divide each text’s sentiment into the three category, “Microsoft’s Text Analytics” only shows those sentiments as numerical values. Therefore, the results from “Text Analytics” were converted to the three emotions manually according to the level of the value. If the value was below 40, then the text's sentiment was regarded as "negative". Similarly, if it showed 40 to 60, then the outcome was determined as "neutral". Moreover, if it became more than 60, it was set as "positive".

The results are shown below from Chart.1 to Chart.3. Firstly, it is noticeable that the low discrimination rate for “positive headlines” (Chart.1). This disappointing outcome might suggest that the three tools are not good at feeling the “positive” emotions of the short texts. In contrast, however, they showed a superb reaction to “negative headlines”. All of them were able to regard the tendency of “negative headlines” as “negative” quite well. Notably, the accuracy of IBM’s Alchemy API for “negative headlines” reached almost 100% if we ignore the values of “Neutrals” (Chart.2). In addition, the mean accuracy for the emotion detectors was all below 80%, which is relatively lower than the reported data from various theses (Table.1).

Table.1

Thesis	Classification	Accuracy
Sentiment Analysis: Capturing Favorability Using Natural Language Processing	Negative or Positive	75 – 95%
Thumbs up? Sentiment Classification using Machine Learning Techniques	Negative or Positive	72.8 - 82.9%
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank	Negative or Positive	79.4 - 85.4%