Use Cases

Sentiment Analysis of Twitter

Federico Pascual
by Federico Pascual

Sentiment Analysis of Twitter

Sentiment Analysis of Twitter data can help companies obtain qualitative insights to understand how people are talking about their brand.

With more than 321 million active users, sending a daily average of 500 million Tweets, Twitter has become one of the top social media platforms for news, information, and interaction with brands and influential figures around the world. Therefore, it is no surprise that companies consider this microblogging platform an essential channel for their marketing strategy and also, to provide customer service.

Twitter allows businesses to reach a broad audience and connect with customers without intermediaries. On the downside, it can be harm a brand’s reputation if negative content about the brand suddenly goes viral – you might end up with an unexpected PR crisis on your hands. This is one of the reasons why social listening ― monitoring conversation and feedback in social media ― has become a crucial process in social media marketing.

Monitoring Twitter allows companies to understand their audience, keep on top of what’s being said about their brand and their competitors, and discover new trends in the industry. However, when it comes to analyzing Twitter data, quantitative metrics like the number of mentions or retweets are not enough to get a full picture of a situation. What really counts is being able to grasp the nuance of those mentions. Are users talking positively or negatively about a product? And that’s exactly what sentiment analysis determines. It provides qualitative insights on what is being said about a topic or brand.

In this guide, we’ll take a closer look at sentiment analysis and how you can use it to analyze Twitter data. We’ll walk you through the entire process, from recommending some tools to gather data from Twitter, to a step-to-step tutorial for getting you up and running with Twitter sentiment analysis using MonkeyLearn. Finally, we’ll see some use cases of companies that are already using sentiment analysis with Twitter for a variety of purposes:

Let’s jump right into it!

What is Sentiment Analysis?

Sentiment analysis (a.k.a opinion mining) is the automated process of identifying and extracting the subjective information that underlies a text. This can be either an opinion, a judgment, or a feeling about a particular topic or subject. The most common type of sentiment analysis is called ‘polarity detection’ and consists in classifying a statement as ‘positive’, ‘negative’ or ‘neutral’.

For example, let’s take this sentence: “I don’t find the app useful: it’s really slow and constantly crashing”. A sentiment analysis model would automatically tag this as Negative.

A sub-field of Natural Language Processing (NLP), sentiment analysis has been getting a lot of attention in recent years due to its many exciting applications in a variety of fields, ranging from business to political studies.

Thanks to sentiment analysis, companies can understand the reputation of their brand. By analyzing social media posts, product reviews, customer feedback, or NPS responses (among other sources of unstructured business data), they can be aware of how their customers feel about their product. They can also track specific topics and get relevant insights on how people are talking about those topics.

Sentiment analysis is particularly useful for social media monitoring because it goes beyond metrics that focus on the number of likes or retweets, and provides a qualitative point of view.

Let’s say a company has just launched a new product feature and you notice a sharp increase in mentions on Twitter. However, receiving tons of mentions does not necessarily mean a good thing. Are customers tweeting more because they are expressing good things about this new product feature? Or, are customers actually complaining about the feature having lots of bugs? Performing Twitter sentiment analysis can be an excellent way to understand the tone of those mentions and obtain real-time insights on how users are perceiving your new product.

Why is Twitter Sentiment Analysis important?

Nearly 80% of the world’s digital data is unstructured, and data obtained from social media sources is no exception to that. Since the information is not organized in any predefined way, it’s difficult to sort and analyze. Fortunately, thanks to the developments in Machine Learning and NLP, it is now possible to create models that learn from examples and can be used to process and organize text data.

Twitter sentiment analysis systems allow you to sort large sets of tweets and detect the polarity of each statement automatically. And the best part, it’s fast and simple, saving teams valuable hours and allowing them to focus on tasks where they can make a bigger impact.

These are some of the main advantages of Twitter sentiment analysis:

  • Scalability: let’s say you need to analyze hundreds of tweets mentioning a brand. While you could do that manually, it would take hours and hours of manual processing and would end up being inconsistent and impossible to scale. By performing Twitter sentiment analysis you can automate this task and obtain cost-effective results in a very short time.
  • Real-Time Analysis: Twitter sentiment analysis is critical to notice sudden shifts in customer moods, detect if critics and complaints are increasing and take action before the problem escalates. You can be monitoring your brand in real-time and get valuable insights that allow you to make changes or improvements when needed.
  • Consistent Criteria: analyzing sentiment in a text is a subjective task. When done manually, the same tweet may be perceived differently by two members of the same team, and the results will probably be biased. By training a machine learning model to perform sentiment analysis on Twitter, you can set the parameters to analyze all your data and obtain more consistent and accurate results.

How to Use Sentiment Analysis with Twitter Data?

So far you’ve learned what sentiment analysis is and the benefits it delivers when using it to analyze Twitter data. Now it’s time to learn how to actually do Twitter sentiment analysis.

We can identify four main steps in this process:

  1. Data gathering
  2. Data preparing
  3. The creation of the sentiment analysis model
  4. Visualization of the results

In this section, we’ll explain each of these stages and provide tools for both coders and non-coders so you can get started with sentiment analysis of tweets right away.

1. Data Gathering: How to Get Twitter Data?

If you want to perform Twitter sentiment analysis, the first step is to gather the data. This is the data that you will use for:

  1. training the machine learning model, and
  2. running the actual sentiment analysis on Twitter data.

At this point, there’s something important to consider regarding Twitter. There are two main types of tweets you can extract from Twitter:

  • Current Tweets: this is useful to track keywords or hashtags in real-time.
  • Historical Tweets: you can use this to search past tweets during a predefined time frame.

One of the most frequent questions is how to extract data from Twitter. There are different ways to do this, some of them are free, and some require purchasing the data. Besides, some tools are oriented to developers, while others don’t need any coding skills.

Let’s take a closer look at some of the options available to obtain Twitter data:

Tools for non-developers

Zapier

Zapier is a platform that enables teams from different business areas (marketing, HR, customer support, product, etc) to connect the web apps they use on a daily basis so that they can automatically work together. It’s excellent for non-technical users, as you don’t need to write a single line of code for gathering tweets.

To set an automated workflow on Zapier (a Zap) you need to choose one app as the Trigger (this will be the app from where you’ll extract data) and another app (or apps) as the Action (where the data will be sent).

Let’s say you want to extract all the tweets that mention the brand Mailchimp in real time. You could use Zapier to connect Twitter with Google Sheets and obtain a spreadsheet that adds new tweets automatically in real-time:

Choose Twitter as a Trigger app

You could also go one step further and connect Zapier with MonkeyLearn to automatically perform Sentiment Analysis on your Twitter data. If you want to try it right away, here’s a step-by-step guide on how to create a Zap for sentiment analysis with MonkeyLearn.

IFTTT

IFTTT means ‘if this, then that’. Like Zapier, it’s a platform that allows you to connect to different apps so that they can perform a desired action when certain criteria is met. You can use it to obtain Twitter data with zero lines of code.

Export Tweet

Export Tweet allows you to track a keyword, hashtag or account in real-time, or search for historical data. However, its free version has limitations and we recommend upgrading to a paid version to take full advantage of the platform.

Tweet Download

Tweet Download enables you to download the tweets from your own account, along with the replies and mentions. This is especially useful for brands that want to keep track of which contents works better with users, what are the main things that users claim about their product, etc.

Tools for developers

Twitter API

The Twitter API (the term stands for Application Programming Interface) enables software developers to access and interact with public Twitter data. Developers can interact with this API by writing their own scripts or by using one of the open source libraries available in different programming languages.

The Twitter API has 2 APIs that are useful for extracting tweets:

  1. Twitter Streaming API: this API allows you to connect to the Twitter data stream and gather tweets in real-time. You can listen to all the Tweets matching a certain keyword, mention or hashtag, as well as collect the tweets of specific users, at the time they are being posted in the Twitter platform.

    Twitter offers a free version of its streaming API, as well as a paid version:

    • Statuses / filter (free): this API allows you to track tweets with up to 400 keywords, hashtags or mentions, monitor up to 5,000 user IDs, and up to 25 locations.

      Getting started: This Python script allows you to connect to the Twitter statuses/filter API, gather tweets in real-time that match a particular keyword, hashtag or mention, and save them into a CSV file under the following column categories:

      • Tweet content: text of the tweet
      • Matched keywords: the keywords that were a match to the search in the Twitter stream
      • Date: date and hour of the tweet
      • User: name of the author of the tweet
      • Source used to post the tweet (for example, Twitter Web Client or Buffer)
      • Tweet ID
      • Tweet URL

    • PowerTrack API (paid): this premium API enables you to receive real-time streams of Twitter data, listing to up to 250,000 keywords, hashtags, mentions, locations, and user IDs.

  2. Standard Search API: this API provides historical tweets published up to 7 days ago, that match a predefined query (the keyword, mention, hashtag, etc. that you’d like to search). Unlike real-time analysis, in this case, you are retrieving information from the past.

    While the Standard Search API is free, it has a 7-day limit. The alternatives are paid historical search APIs (like Historical PowerTrack and Full-Archive Search), that provide access to the last 30 days of tweets or even tweets from as early as 2006.

    Getting started: This Python script allows you to connect to the Twitter Standard Search API, gather historical tweets from up to 7 days ago that contain a specific keyword, hashtag or mention, and save them into a CSV file under the following column categories:

    • Tweet content: text of the tweet
    • Date: date and hour of the tweet
    • User: name of the author of the tweet
    • Source used to post the tweet (for example, Twitter Web Client or Buffer)
    • Tweet ID
    • Tweet URL

Open Source Libraries

There are many open source libraries that you can use to connect with the Twitter API. One of the most popular is Tweepy, a library with 5937 stars and 2847 forks in Github, used for interacting with Twitter’s API using Python.

This library is very easy to use and enables you to gather Tweets automatically through the Twitter API. Developers have been working on this library for over 8 years now, the community around it is very active and they keep improving the code continuously.

If you want to get started with Tweepy, here’s an excellent tutorial.

Other popular libraries you can use with the Twitter API are:

2. Data Preparing: Preprocessing and Cleaning Tweets

Once you’ve captured the tweets you need for your sentiment analysis, it’s time to prepare the data. As we mentioned earlier, social media data is unstructured. That means it’s raw, noisy and needs to be cleaned before we can start working on our sentiment analysis model. This is an important step because the quality of the data will lead to more reliable results.

Preprocessing a Twitter dataset involves a series of tasks like removing all types of irrelevant information like emojis, special characters, and extra blank spaces. It can also involve making format improvements, delete duplicate tweets, or tweets that are shorter than three characters.

3. Creating a Twitter Sentiment Analysis Model

At this point, you already know what Twitter sentiment analysis is and the most common ways to obtain Twitter data. You probably have a few ideas about types of content you’d like to analyze, so it may be the right time to get your feet wet!

MonkeyLearn is an easy-to-use machine learning platform for text analysis. With MonkeyLearn, getting started with sentiment analysis is easy and straightforward. You can sign-up for free and access several pre-trained models that will allow you to perform all kinds of automated text analysis tasks. If you want to try sentiment analysis right now, you can use our pre-trained model for sentiment analysis. Here’s what it looks like:

However, if you want to get predictions with a high level of accuracy, adapted to your criteria and domain, then the best way is to create your own customized sentiment analysis model by training it with your criteria. With MonkeyLearn, this can be done in just a few simple steps.

To help you get started, we’ve prepared a step-by-step tutorial to build your own sentiment analysis model:

1. Choose a model type

Go to the MonkeyLearn dashboard, then click on the button in the right-hand corner: Create a model’, and then choose Classifier’:

2. Decide which type of classification you’d like to do

From the list of classifier type; click on ‘Sentiment Analysis’:

3. Import your Twitter data

This data will be used as examples to train your machine learning model. You can upload data to MonkeyLearn from different sources. In this case, we’ll choose Twitter. Alternatively, if you’ve already gathered tweets using another tool, you can upload the .csv or Excel File and use it as an input for your model:

4. Search for tweets

Write a search query to obtain tweets that will be used as training data for your sentiment analysis model. It can be a keyword, hashtag or mention. As an example, we’ll use the keyword ‘zendesk’. Then, choose the column you’ll like to use (it’s often the text of the tweet). Once you are done, the tweets will be imported to your classifier:

5. Tag data to train your classifier

Now, it’s time to train your sentiment analysis model, by manually tagging each of the tweets as Positive, Negative, or Neutral, based on the polarity of the opinion. After tagging the first tweets, the model will start making its own predictions. You can correct them if the answer is not correct:

6. Test your classifier

Once you have trained your model with a few examples, you can paste your own texts to see how the sentiment analysis model classifies it:

MonkeyLearn provides different stats to measure the performance of your sentiment analysis classifier. These are accuracy, F1 score, precision, and recall. You can also find a keyword cloud featuring the most frequent terms for each sentiment.

If you are not able to see all the stats, it might mean that you need to tag more data in order to provide the model with enough examples for each category. In this case, for example, the model requires more training data for the category Negative:

Keep in mind that the more training data you tag, the more accurate your classifier becomes. Another way to improve the accuracy of your model is to check all the false positives and false negatives and re-tag the incorrect ones. Here’s how:

7. Put the model to work

Now you’ve got a sentiment analysis model that’s ready to analyze tons of tweets! The next step is to integrate the Twitter data you want to analyze with the sentiment analysis model you just created. There are three ways to do this with MonkeyLearn:

  • Batch Analysis: upload a CSV or an Excel File with new, unseen tweets. The classifier will process all the tweets and provide a new file with the results of the sentiment analysis:
  • Integrations: there are several integrations available you can use to analyze new data with your sentiment analysis model. For example, you could use Google Sheets as an input for your data or Zapier to connect a data source (like Twitter) to MonkeyLearn.
  • API: if you are a coder, you can use MonkeyLearn’s API for analyzing new tweets.

4. Data Visualization of the Results

So, you’ve finally got your Twitter sentiment analysis model up and running, and you are excited to see the first results. Depending on the tools you used to build the model, you can receive your analyzed data as a CSV or Excel file (if you don’t know how to code), or as an API response in JSON format (if you can code).

Data visualization tools are a powerful resource to take your sentiment analysis results to the next level. Raw results, in other words, long lists of text, will never get as much attention as a beautifully designed chart or graph that show insights in a visual and engaging way.

For example, the chart below shows the results of sentiment analysis of Slack reviews. The graph organizes the sentiment analysis results in an attractive way and makes it simple to understand at a glimpse:

In this section, we’ve outlined how three of the most popular data visualization tools work so that you can turn your results into valuable, easy to understand, and impactful information:

You can use this free and simple Google platform to create interactive reports. There are more than 100 sources available to import your data, including CSV, Excel, and Google Sheets. Once you’ve designed your visual report, you can share it with other teams or individuals.

This is a business data analytics platform, created to manage all sorts of data within the different areas of a company. You can connect with different databases and create charts and data tables. Learn how to get started.

Defined as a business intelligence and analytics software, Tableau allows you to work with a large number of data sources to create dynamic dashboards and compelling data visualizations. One of the best things about Tableau is that is very easy to use and doesn’t require any coding skills. However, it offers different types of products and some of them are targeted to developers.

Use Cases and Applications

Sentiment analysis of tweets provides exciting opportunities across many fields, from business to politics. Being able to analyze tweets in real-time, and determine the sentiment that underlies each message, adds a new dimension to social media monitoring.

Because, let’s be honest, even though counting the number of mentions or likes are one of the most valued KPIs when it comes to measuring brand awareness, that’s just one small part of the whole picture. Yes, a lot of people may be talking about your brand. But, are those mentions positive or negative? Sentiment analysis provides valuable insights to understand how people actually feel about the things they talk about on social media.

In this section, we’ll present some of the most frequent applications of Twitter sentiment analysis.

Social Media Monitoring

Online reputation is one of the most precious assets for brands. A bad review on social media can be costly to a company if it’s not handled properly and swiftly. After all, 80% of consumers get advice on products from social media.

Twitter sentiment analysis allows you to keep track of what’s being said about your product or service on social media and can help you detect angry customers or negative mentions before they turn into a major crisis.

At the same time, Twitter sentiment analysis can provide interesting insights to understand customer feedback. What are the things that people love about our brand? What are the aspects that bring up the most negative comments? This tweet, for example, indicates that fast shipping is one of the most valued aspects for Amazon customers:

While this one from an Uber customer complains about poor customer service:

Aspect-based sentiment analysis with Twitter can show you which aspects of your business need to be improved and what makes you stand out among your competitors.

Customer Service

Twitter has become an essential channel for customer service. In fact, a growing number of companies have specific teams in charge of delivering customer support via this social media platform. Prompt replies are key since 60% of the customers that complain on social media expect a response within one hour.

But how can you evaluate the performance of your customer support on Twitter? Twitter sentiment analysis allows you to track and analyze all the interactions between your brand and your customers. This can be very useful to analyze customer satisfaction based on the type of feedback you receive.

This tweet, for example, shows a disappointed customer after an interaction with the customer support team of Southwest Airlines.

Market Research

Twitter is a major source of consumer insight. In fact, people use it to express all sorts of feelings, observations, beliefs, and opinions about a variety of topics. By performing Twitter sentiment analysis, we can unleash the power of this data and use it as a valuable asset.

We can use Twitter sentiment analysis to track specific keywords and topics (either in real time or in the past) and this can be particularly useful for detecting customer trends and interests. Understanding what things potential customers like, what their behaviors are and how this changes over time is essential if you are planning to launch a new product.

Here’s an example: Twitter sentiment analysis of almost 4,000 tweets referring to halal food, conducted to monitor consumer’s preferences and opinions towards this type of food. This information allowed researchers to identify different motivations for halal food consumption and therefore, segment their market into different types of consumers.

Twitter sentiment analysis can also help you stay one step ahead of your competition and detect new business opportunities by identifying the pain points of other brands within your industry. Besides, doing sentiment analysis of tweets with keywords related to your industry can give you ideas on how to promote your business.

The tweet above, for example, can provide some useful insights on what type of content may work best to promote your business among people interested in machine learning.

Brand Monitoring

Whether you are launching a new feature on your platform, a site redesign, or a new marketing campaign, you may want to track the reactions to that in social media – a great opportunity for performing sentiment analysis on Twitter. After all, you will still have time to take action and make changes or improvements along the way.

Political Campaigns

A huge part of Twitter conversation revolves around news and politics. That makes it an excellent place to measure public opinion, especially during election campaigns. Twitter Sentiment Analysis can provide interesting insights on how people feel about a specific candidate (and you could even track sentiment over time to see how it evolves).

Let’s look at an example. During the US 2016 elections, we performed Twitter sentiment analysis using MonkeyLearn to analyze the polarity of Twitter mentions related to Donald Trump and Hillary Clinton. First, we were able to count the number of positive and negative mentions for each candidate during a period of time. This graph shows Trump’s tweets based on sentiment:

In contrast, the following graph shows the number of positive, negative, and neutral mentions for Hillary Clinton:

Another relevant insight consisted of analyzing the tweets on specific dates, for example on the day of the presidential debate and observing negative or positive reactions, as well as the main keywords mentioned during that day.

Wrap-up

Twitter Sentiment Analysis allows a deeper understanding of people’s feelings and opinions. It adds an extra layer to the traditional metrics used to analyze the performance of brands on social media and provides powerful opportunities for improvement.

Yes, it is something that could be done manually….until your brand starts receiving dozens of Twitter mentions every day. Text analysis with machine learning is simple, fast, and can provide consistent results with a high level of accuracy.

Many different tools allow you to easily extract data from Twitter, like the ones mentioned above, which can then be used to create and train a sentiment analysis model. Whether you have coding skills or not, you will find an option that’s suitable for you.

Machine learning models can do pretty amazing things, even when it comes to something as subjective as the human language. Ready to see Twitter sentiment analysis in action yourself? With a machine learning platform like MonkeyLearn, it’s simple to get started. Contact us today and request a personalized demo from one of our experts!

Federico Pascual

Federico Pascual

COO & Co-Founder @MonkeyLearn. Machine Learning. @500startups B14. @Galvanize SoMa. TEDxDurazno Speaker. Wannabe musician and traveler.

Notification

Have something to say?

Text Analysis with Machine Learning

Turn tweets, emails, documents, webpages and more into actionable data. Automate
business processes and save hours of manual data processing.