Twitter Sentiment Analysis with Machine Learning

Sentiment analysis is the automated process of analyzing text data and sorting it into sentiments positive, negative, or neutral. Using sentiment analysis tools to analyze opinions in Twitter data can help companies understand how people are talking about their brand.

Request a demo to learn more about how MonkeyLearn can help you perform data analysis on your Twitter data.

With more than 321 million active users, sending a daily average of 500 million Tweets, Twitter allows businesses to reach a broad audience and connect with customers without intermediaries. On the downside, it’s harder for brands to quickly detect negative content, and if it goes viral you might end up with an unexpected PR crisis on your hands.

This is one of the reasons why social listening ― monitoring conversations on social media platforms ― has become a crucial process in social media marketing. Monitoring Twitter allows companies to understand their audience, keep on top of what’s being said about their brand and their competitors, and discover new trends in the industry. 

Below, you’ll learn more about how to analyze and understand emotions on Twitter using online sentiment analysis tools that are code free, and others that can be used in Python.

Let’s jump right into it!

What Is Sentiment Analysis?

Sentiment analysis (a.k.a opinion mining) is the automated process of identifying and extracting the subjective information that underlies a text. This can be either an opinion, a judgment, or a feeling about a particular topic or subject. The most common type of sentiment analysis is called ‘polarity detection’ and involves classifying a statement as ‘positive’, ‘negative’, or ‘neutral’.

For example, let’s take this sentence: “I don’t find the app useful: it’s really slow and constantly crashing”. A sentiment analysis model would automatically tag this as Negative.

A sub-field of Natural Language Processing (NLP), sentiment analysis has been getting a lot of attention in recent years due to its many exciting business applications.

By analyzing social media posts, product reviews, customer feedback, and NPS responses (among other unstructured data), businesses can understand how their customers feel about their product or service. 

Sentiment analysis is particularly useful for social media monitoring because it goes beyond the number of likes or retweets, by providing qualitative insights.

Imagine you just launched a new product feature and notice a sharp increase in mentions on Twitter. Are customers tweeting more because they are delighted with the new feature? Or, are they actually complaining about the feature? Performing sentiment analysis on Twitter data can help you quickly understand the tone of those mentions.

Why Is Twitter Sentiment Analysis important?

Nearly 80% of the world’s digital data is unstructured, and a large portion of that includes social media data. 

Sentiment analysis tools use machine learning and natural language processing (NLP) to organize unstructured text data automatically. Sentiment analysis algorithms are able to learn from data samples to detect the polarity of Tweets in real-time. All you have to do is train sentiment analysis tools to recognize sentiment in tweets, and they’ll do the rest. Main advantages of Twitter sentiment analysis include:

These are some of the main advantages of Twitter sentiment analysis:

  • Real-Time Analysis: Twitter sentiment analysis is essential for monitoring sudden shifts in customer moods, detecting if complaints are on the rise, and for taking action before problems escalate. With sentiment analysis, you can monitor brand mentions on Twitter in real-time and gain valuable insights that tell you if you need to make updates.
  • Scalability: let’s say you need to analyze hundreds of tweets mentioning your brand. While you could do that manually, it would take hours of manual processing, and as your data grows it would be impossible to scale. By performing Twitter sentiment analysis you can automate manual tasks and gain valuable insights in a very short time.
  • Consistent Criteria: analyzing sentiment in a text is subjective. when done manually.  The same tweet may be viewed differently by two members of the same team. By training a machine learning model to perform sentiment analysis on Twitter data, you can use one set of criteria to analyze all your data, so results are consistent.

How to Analyze The Sentiment of Twitter Data

Performing sentiment analysis on Twitter data involves several steps:

  1. Gather Twitter Data 
  2. Prepare Your Data
  3. Create A  Sentiment Analysis Model
  4. Visualize Your Results

In this section, we’ll explain each of these stages and provide tools for both coders and non-coders so you can get started with sentiment analysis right away.

1. Gathering Twitter Data

It’s important that your data is representative of what you’re trying to find out because you’ll use it to: 

  • Train your sentiment analysis model
  • Test how your model performs on Twitter data

You should also consider the type of tweets you want to analyze: 

  • Current Tweets: useful to track keywords or hashtags in real-time.
  • Historical Tweets: useful to compare sentiments over different periods.

Now, you’re probably wondering how to extract data from Twitter. There are different ways to do this. Let’s take a closer look at some of the options:

Tools for non-developers

MonkeyLearn

MonkeyLearn is a machine learning platform that allows you to build your own sentiment analysis models, and connect them to Twitter data via a one-click integration. All you have to do is type your brand name into search and select the column with tweets. You’ll then be able to use this data to train your model. See further instructions on how to extract Twitter data when creating sentiment analysis models with MonkeyLearn, below.

Twitter feature that extracts tweets in a sentiment analysis tool.

Zapier

Zapier is a platform that enables teams from different business areas (marketing, HR, customer support, product, etc) to connect the web apps they use on a daily basis so that they can automatically work together. It’s excellent for non-technical users, as you don’t need to write a single line of code to gather tweets.

To set an automated workflow on Zapier (a Zap), just choose one app as the Trigger (this will be the app from where you’ll extract data) and another app (or apps) as the Action (where the data will be sent).

Let’s say you want to extract tweets that mention the brand Mailchimp in real-time. You could use Zapier to connect Twitter with Google Sheets and obtain a spreadsheet that adds new tweets automatically in real-time:

Steps to set Twitter as the trigger app when creating a Zap.
Choose Twitter as a Trigger app

You could also go one step further and connect Zapier with MonkeyLearn to automatically perform sentiment analysis on your Twitter data. If you want to try it right away, here’s a step-by-step guide on how to create a Zap for sentiment analysis with MonkeyLearn.

IFTTT

IFTTT means ‘if this, then that’. Like Zapier, it’s a platform that allows you to connect to different apps so that they can perform a desired action when certain criteria is met. You can use it to obtain Twitter data with zero lines of code.

Export Tweet

Export Tweet allows you to track a keyword, hashtag or account in real-time, or search for historical data. However, its free version has limitations and we recommend upgrading to a paid version to take full advantage of the platform.

Tweet Download

Tweet Download enables you to download the tweets from your own account, along with the replies and mentions. This is especially useful for brands that want to keep track of which contents works better with users, what are the main things that users claim about their product, etc.

Tools for developers

Twitter API

The Twitter API (the term stands for Application Programming Interface) enables software developers to access and interact with public Twitter data. Developers can interact with this API by writing their own scripts or by using one of the open source libraries available in different programming languages.

The Twitter API has 2 APIs that are useful for extracting tweets:

  1. Twitter Streaming API: this API allows you to connect to the Twitter data stream and gather tweets in real-time. You can listen to all the Tweets matching a certain keyword, mention or hashtag, as well as collect the tweets of specific users, at the time they are being posted in the Twitter platform.

    Twitter offers a free version of its streaming API, as well as a paid version:

    • Statuses / filter (free): this API allows you to track tweets with up to 400 keywords, hashtags or mentions, monitor up to 5,000 user IDs, and up to 25 locations.

      Getting started: This Python script allows you to connect to the Twitter statuses/filter API, gather tweets in real-time that match a particular keyword, hashtag or mention, and save them into a CSV file under the following column categories:

      • Tweet content: text of the tweet
      • Matched keywords: the keywords that were a match to the search in the Twitter stream
      • Date: date and hour of the tweet
      • User: name of the author of the tweet
      • Source used to post the tweet (for example, Twitter Web Client or Buffer)
      • Tweet ID
      • Tweet URL

    • PowerTrack API (paid): this premium API enables you to receive real-time streams of Twitter data, listing to up to 250,000 keywords, hashtags, mentions, locations, and user IDs.

  2. Standard Search API: this API provides historical tweets published up to 7 days ago, that match a predefined query (the keyword, mention, hashtag, etc. that you’d like to search). Unlike real-time analysis, in this case, you are retrieving information from the past.

    While the Standard Search API is free, it has a 7-day limit. The alternatives are paid historical search APIs (like Historical PowerTrack and Full-Archive Search), that provide access to the last 30 days of tweets or even tweets from as early as 2006.

    Getting started: This Python script allows you to connect to the Twitter Standard Search API, gather historical tweets from up to 7 days ago that contain a specific keyword, hashtag or mention, and save them into a CSV file under the following column categories:

    • Tweet content: text of the tweet
    • Date: date and hour of the tweet
    • User: name of the author of the tweet
    • Source used to post the tweet (for example, Twitter Web Client or Buffer)
    • Tweet ID
    • Tweet URL

Open Source Libraries

There are many open source libraries that you can use to connect with the Twitter API. One of the most popular is Tweepy, a library with 5937 stars and 2847 forks in Github, used for interacting with Twitter’s API using Python.

This library is very easy to use and enables you to gather Tweets automatically through the Twitter API. Developers have been working on this library for over 8 years now, the community around it is very active and they keep improving the code continuously.

If you want to get started with Tweepy, here’s an excellent tutorial.

Other popular libraries you can use with the Twitter API are:

2. Prepare Your Data

Once you’ve captured the tweets you need for your sentiment analysis, you’ll need to prepare your data. As we mentioned earlier, social media data is unstructured. That means it’s raw, noisy, and needs to be cleaned before we can start working on our sentiment analysis model. This is an important step because the quality of the data will lead to more reliable results.

Preprocessing a Twitter dataset involves a series of tasks like removing all types of irrelevant information like emojis, special characters, and extra blank spaces. It can also involve making format improvements, delete duplicate tweets, or tweets that are shorter than three characters. Check out this guide on how to prepare your data.

3. Create a Twitter Sentiment Analysis Model: Tutorial

MonkeyLearn is an easy-to-use machine learning platform for text analysis. With MonkeyLearn, getting started with sentiment analysis is easy and straightforward. You can sign-up for free and access several pre-trained models that will allow you to perform all kinds of automated text analysis tasks. 

However, if you want to get predictions with a high level of accuracy, adapted to your criteria and domain, then the best way is to create your own customized sentiment analysis model by training it with your industry criteria. 

With MonkeyLearn, this can be done in just a few simple steps.

  1. Choose a model type

Once you’ve signed up, go to MonkeyLearn’s dashboard and click on the button in the right-hand corner: Create a model’, then choose ‘Classifier’:

1. Choose a model type

Go to the MonkeyLearn dashboard, then click on the button in the right-hand corner: Create a model’, and then choose ‘Classifier’:

2. Decide which type of classification you’d like to do

From the list of classifier type; click on ‘Sentiment Analysis’:

3. Import your Twitter data

This data will be used as examples to train your machine learning model. You can upload data to MonkeyLearn from different sources. In this case, we’ll choose Twitter. Alternatively, if you’ve already gathered tweets using another tool, you can upload a .csv or Excel File:

4. Search for tweets

Write a search query to obtain tweets that will be used as training data for your sentiment analysis model. It can be a keyword, hashtag, or mention. As an example, we’ll use the keyword ‘zendesk’. Then, choose the column you’d like to use (it’s often the text of the tweet). Once you are done, the tweets will be imported to your classifier:

5. Tag data to train your classifier

Now, it’s time to train your sentiment analysis model, by manually tagging each of the tweets as Positive, Negative, or Neutral, based on the polarity of the opinion. After tagging the first tweets, the model will start making its own predictions. You can correct them if the answer is not correct:

6. Test your classifier

Once you have trained your model with a few examples, you can paste your own texts to see how the sentiment analysis model classifies it:

MonkeyLearn provides different stats to measure the performance of your sentiment analysis classifier. These are accuracy, F1 score, precision, and recall. You can also find a keyword cloud featuring the most frequent terms for each sentiment.

If you are not able to see all the stats, it might mean that you need to tag more data in order to provide the model with enough examples for each category. In this case, for example, the model requires more training data for the category Negative:

Keep in mind that the more training data you tag, the more accurate your classifier becomes. Another way to improve the accuracy of your model is to check all the false positives and false negatives and re-tag the incorrect ones. Here’s how:

7. Put the model to work

Now you’ve got a sentiment analysis model that’s ready to analyze tons of tweets! The next step is to integrate the Twitter data you want to analyze with the sentiment analysis model you just created. There are three ways to do this with MonkeyLearn:

  • Batch Analysis: Go to ‘Batch’ and upload a CSV or an Excel File with new, unseen tweets. The classifier will process all the tweets and provide a new file with the results of the sentiment analysis:
  • Integrations: there are several integrations available you can use to analyze new data with your sentiment analysis model. For example, you could use Google Sheets as an input for your data or Zapier to connect Twitter data  to MonkeyLearn.
  • API: if you know how to code, , you can use MonkeyLearn’s API in Python (and other programming languages) to analyze new tweets.

4. Visualize Your Results

Data visualization tools help explain sentiment analysis results in a simple and effective way. 

For example, take a look at these visual results showing sentiment analysis of Slack reviews. Using a data visualization tool, we  organized the results so that they’re easy to understand at a glance:

Below, we’ve listed some of the most popular data visualization tools that you can use to transform your sentiment analysis results.

You can use this free and simple Google platform to create interactive reports. There are more than 100 sources available to import your data, including CSV, Excel, and Google Sheets. Once you’ve designed your visual report, you can share it with other teams or individuals.

This is a business data analytics platform, created to manage all sorts of data within the different areas of a company. You can connect with different databases and create charts and data tables. Learn how to get started.

Defined as business intelligence and analytics software, Tableau allows you to work with a large number of data sources to create dynamic dashboards and compelling data visualizations. One of the best things about Tableau is that is very easy to use and doesn’t require any coding skills. However, it offers different types of products and some of them are targeted to developers.

Twitter Sentiment Analysis Use Cases 

Sentiment analysis of tweets provides exciting opportunities. Being able to analyze tweets in real-time, and determine the sentiment that underlies each message, adds a new dimension to social media monitoring.

In this section, we’ll present some of the most frequent applications of Twitter sentiment analysis.

Social Media Monitoring

Online reputation is one of the most precious assets for brands. A bad review on social media can be costly to a company if it’s not handled effectively and swiftly. 

Twitter sentiment analysis allows you to keep track of what’s being said about your product or service on social media, and can help you detect angry customers or negative mentions before they turn into a major crisis.

At the same time, Twitter sentiment analysis can provide interesting insights. What do customers love about your brand?  What aspects get the most negative mentions? This tweet, for example, indicates that fast shipping is one of the most valued aspects for this Amazon customer:

Aspect-based sentiment analysis with Twitter can show you which aspects of your business need to be improved and what makes you stand out among your competitors.

Customer Service

Twitter has become an essential channel for customer service. In fact, a growing number of companies have specific teams in charge of delivering customer support via this social media platform. Prompt replies are key since 60% of the customers that complain on social media expect a response within one hour.

But how can you evaluate the performance of your customer support on Twitter? Twitter sentiment analysis allows you to track and analyze all the interactions between your brand and your customers. This can be very useful to analyze customer satisfaction based on the type of feedback you receive.

This tweet, for example, shows a disappointed customer after an interaction with Southwest Airlines’ customer support team:

Market Research

Twitter is a major source of consumer insight. In fact, people use it to express all sorts of feelings, observations, beliefs, and opinions about a variety of topics. 

You can use Twitter sentiment analysis to track specific keywords and topics to detect customer trends and interests. Understanding what things potential customers like, what their behaviors are, and how this changes over time is essential if you are planning to launch a new product.

Here’s an example of how Twitter sentiment analysis was used to monitor 4,000 tweets that mentioned halal food. This information allowed researchers to identify different motivations for halal food consumption and segment their market into different types of consumers.

Twitter sentiment analysis can also help you stay one step ahead of your competition.  By identifying competitors’ pain points, you can focus on these areas when promoting your business.

Brand Monitoring

Whether you are launching a new feature on your platform, a site redesign, or a new marketing campaign, you may want to track customer reactions on Twitter. Taking action and making changes or improvements in real-time will help maintain customer loyalty.

Political Campaigns

A huge part of Twitter conversation revolves around news and politics. That makes it an excellent place to measure public opinion, especially during election campaigns. Twitter Sentiment Analysis can provide interesting insights on how people feel about a specific candidate (and you could even track sentiment over time to see how it evolves).

During the US 2016 elections, we performed Twitter sentiment analysis using MonkeyLearn to analyze the polarity of Twitter mentions related to Donald Trump and Hillary Clinton. First, we were able to count the number of positive and negative mentions for each candidate during a period of time. This graph shows Trump’s tweets based on sentiment:

In contrast, the following graph shows the number of positive, negative, and neutral mentions for Hillary Clinton:

Another relevant insight consisted of analyzing the tweets on specific dates, for example on the day of the presidential debate and observing negative or positive reactions, as well as the main keywords mentioned during that day.

Wrap-up

Twitter Sentiment Analysis allows a deeper understanding of how your customers feel.. It adds an extra layer to the traditional metrics used to analyze the performance of brands on social media and provides businesses with powerful opportunities.

Yes, you could sort data by sentiment manually, but what happens when your data starts to grow?. Text analysis with machine learning is simple, fast, and scalable, and can provide consistent results with a high level of accuracy.

Many different tools allow you to easily extract data from Twitter, like the ones mentioned above, which can then be used to create and train a sentiment analysis model. Whether you have coding skills or not, you will find an option that’s suitable for you.

Machine learning models can do pretty amazing things, even when it comes to something as subjective as the human language. With a machine learning platform like MonkeyLearn, it’s simple to get started with sentiment analysis of your Twitter data. Contact us today and request a personalized demo from one of our experts

Federico Pascual

Federico Pascual

COO & Co-Founder @MonkeyLearn. Machine Learning. @500startups B14. @Galvanize SoMa. TEDxDurazno Speaker. Wannabe musician and traveler.

Notification

Have something to say?

Text Analysis with Machine Learning

Turn tweets, emails, documents, webpages and more into actionable data. Automate
business processes and save hours of manual data processing.