The best way to find out what your customers are thinking is to ask them. You can do this by sending out customer surveys, or carrying out interviews and public opinion polls that are tailored around specific goals.
The kinds of questions you ask will determine the kind of responses you get from your customers. There are two main types of questions: close-ended and open-ended questions. We’ll go over the advantages and disadvantages of both of these, but first let’s give you a quick definition of each question type.
The main difference between open-ended and close-ended questions is that close-ended questions elicit quantitative responses, while open-ended questions lead to free-flowing text responses.
Before you send out surveys, you’ll need to know which type of response you need, otherwise you may end up with a bunch of responses that don’t help you achieve your goals.
So, let’s take a closer look at each type of question to help you decide which one to use:
Open-ended questions are questions that can’t simply be answered with Yes/No, True/False, multiple choice, or rated on a number or star-rating scale. Instead of fixed response choices, they require customers to provide free-form responses, in their own voice and vernacular, otherwise known as voice of customer (VoC).
Sometimes you need data that goes deeper, that delves into the opinions and thoughts of your customers, so you can find out why customers are happy or unhappy.
You can use tools like word cloud generators to quantify topics in your open-ended responses, like in this example below:
But, if you want to know why these keywords are mentioned, you’ll need to use more advanced analysis tools (more on that later).
Close-ended questions are questions that are answered with predefined multiple-choice options, like Yes/No responses, or scaled, e.g., “On a scale of 1 to 10 how happy are you with this product?” Close-ended questions, like the scaled NPS and CSAT questions, gather quantitative data, and are easy to analyze and compare on a spreadsheet.
Close-ended questions give you a quick and high-level view of what is going on in your business, so you can know if you need to take further action or not.
Let’s take a closer look at the advantages of open-ended questions and close-ended questions. Then, we’ll take a look at the disadvantages of each feedback question type.
The advantages of open-ended questions are that they gather opinions and thoughts from respondents, offering much deeper, more thorough, often subjective information.
This has many advantages for businesses:
The possible responses to open-ended questions are endless, meaning there’s no limit to your data collection possibilities.
Different respondents may approach the questions from vastly different angles, and conversational responses in the words of individual customers allows you to understand them more fully.
One of the biggest benefits of open-ended questions is the potential for wholly new information and customer insights.
Because there’s no limit to possible responses, you’re likely to receive information and real opinions you hadn’t previously even considered.
Take these open-ended questions, for example:
These questions compel the respondent to contemplate their responses more thoroughly and drive them toward more discussion rather than the hasty conclusion of a close-ended question.
Maybe a customer has discovered a new use case for your product or recommended a new feature that you hadn’t thought of. The insights can be huge.
Open-ended questions allow respondents to go into as much detail as they care to.
Open-ended responses offer more nuance, because they are written just as the respondents speak, so they can explain themselves more fluidly. Because they aren’t tied to a rated scale or multiple choice, open-ended questions lead to less ambiguous answers.
Close-ended questions offer quantitative data that’s expressed as numbers, percentages, or merely positive/negative.
This data is easy to calculate, offering quick results, but it doesn’t go deeper than uncovering what has already happened.
Open-ended questions offer qualitative customer data that can help you find out why something has happened and inform decisions. Qualitative data helps read between the lines of customer patterns to understand them as individuals, rather than numbers.
Open-ended questions allow you to understand the ideas, feelings, emotions, and opinions of your customers – because they are explaining their personal POVs.
Use it to automatically analyze huge amounts of open-ended questions in a matter of minutes, and find out which aspects of your business perform particularly well and which you need to work on.
Use your data to go beyond mere NPS and customer satisfaction CSAT surveys to more thoroughly understand the customer experience (CX) and follow the whole customer journey from the perspective of your customers.
When you’re constantly collecting qualitative customer data, you’ll need to establish a robust customer feedback loop for a regular, real-time understanding of customer satisfaction.
Close-ended responses offer a birds eye view of your business and are often the best way to start a survey. Let’s take a look at the advantages of close-ended questions:
Close-ended questions are easy and quick to reply to. Respondents don’t need to think too hard about their answer. They quickly scan the multiple choice answers, select the one that applies to them, and click send. This means that the response rate is likely to be higher and your data more reliable.
Since close-ended questions ask you to select from one of a list of predefined responses, you won’t end up with answers that you weren’t expecting. Responses will be consistent and relevant to the question, which means it will be easier for you to compare results.
Close-ended survey responses are given a number or value, which means you can easily calculate the number of respondents selecting each answer and create simple comparative charts in a spreadsheet. Since close-ended questions are easier to analyze than open-ended questions, they’re also less expensive.
It’s time to weigh the advantages and disadvantages of open-ended questions – with some tips below on how to lessen them.
With open-ended questions, respondents don’t have the option to simply select or click their choice from an online or in-app survey.
They have to write out their answers, sometimes explaining in detail. This takes considerably longer, but the data is worth it, if you can convince them to participate.
Close-ended questions, on the other hand, have a limited pre-set amount of choices and are quick and easy to answer. Take the common Net Promoter Score (NPS) survey question, for example:
“On a scale of 0 to 10, how likely are you to recommend us to a friend or colleague?”
Or multiple choice questions like, “What is your favorite aspect of our product?”
A. Usability B. Features C. Security
While they may get a higher response rate, closed-ended questions won’t deliver any wholly new information.
Because they take longer to answer, you’re likely to get lower response rates than with close-ended questions.
That means less data to analyze and fewer insights. Offering incentives to customers for completing surveys can sometimes increase your response rates.
Open-ended responses aren’t based on numbers or percentages, so they can’t be compared strictly mathematically.
They are often objective, so they’re harder to compare with consistent data points and results.
Depending on how your survey is enacted, your responses may contain a lot of “noise” – things like emojis, URLs, non-word characters, etc.
Furthermore, most people don’t write with perfect grammar – and spelling mistakes, misused words, etc., are common. Some respondents may even just ramble on, leaving you with information that is completely irrelevant or doesn’t make sense at all.
Open-ended questions are harder to analyze because they contain unstructured data.
They can’t be easily computed as numbers, and contain subjective information, so the interpretation of the data may differ from person to person.
It’s time to look at the disadvantages of close-ended questions – with some tips below on how to lessen them.
Closed-ended questions help you quantify a customer’s experience, but they don’t reveal why customers left a low or high rating. This means you may end up with a bunch of low scores without knowing exactly what’s causing them.
It’s impossible to cater for everyone in your surveys. Even if you include a long list of possible answers with your close-ended questions, it’s likely that some customers won’t agree with any of the responses. This could lead to them abandoning the survey or selecting a random response just to complete the survey. This can lead to irrelevant customer feedback data.
Close-ended options often lead the respondent to think in a completely different way because they don’t allow them to answer in their own words, from their own perspective. Respondents may see an option they weren’t expecting and may think twice about choosing their original option. Perhaps customers don’t even have an opinion but they answer anyway because they’re compelled to answer and, well, option two sounds about right.
Maybe your question is worded in such a way that it’s easily misinterpreted. It’s easily done. But it’s not easy to know whether customers have misunderstood the question or not. If a lot of customers misinterpret your question, your data will be skewed.
It’s important that you word your questions in a way that elicits honest feedback – with both close-ended and open-ended responses.
Below are some great examples of open-ended and close-ended questions that give your good feedback:
|Close-Ended Questions||Open-Ended Questions|
|From 1 to 10, how would you rate your experience?||How do you feel about your experience with us?|
|Did our product/service meet your needs?||What can we do to improve our product/service?|
|What were your favorite features? [A. B. C. D.]||How do the features help you to achieve your goals. What features are we missing?|
|Is there a product/service you prefer to ours?||How does our product/service compare to the competition?|
|Do you think you’ll use our product/service again?||What would make you want to use our product/service again?|
|Would you recommend our products/services?||What would you say to someone about our products/services?|
|Did our customer service meet your needs?||What would you say about our customer service?|
|Were you satisfied with your experience overall?||What else do we need to know?|
|Did our employees meet your needs?||What can our employees do better to meet your needs?|
Crafting survey questions that deliver good data is hard.
So we’ve put together some tips and practices that will help you get more from your surveys.
What do we mean by polar questions? Questions that imply some kind of sentiment, for example:
The problem with these types of questions is that they assume there is something the customer likes or dislikes about a new feature. They might lead the respondent to believe that they are happy or unhappy with the new feature, and don't leave room for an impartial response.
Polar questions also elicit one-word answers without context:
Here, you can’t tell if ‘Games’ is good or bad because the sentiment is in the question. Since you only analyze the survey responses, and not the questions themselves, you can see how this answer would skew your results
Many businesses ask questions like, ‘Please give examples’, in their surveys. But these types of questions either lead to long and vacuous responses, or deter respondents from answering because it’s too much hard work to fill in a reponses.
This type of question also lacks focus, which could lead to a lot of irrelevant information.
These questions clearly don’t provide any insight, but sometimes it can be hard to spot a question that leads the respondent to answer “Nothing”. Make sure you test your questions before sending them, to avoid receiving responses that reveal nothing!
These types of questions could also lead the survey taker to leave a response that reverses the sentiment in the question, like in the example below.
Analyzing this type of data is confusing.
Ask about a particular touchpoint on a customer’s journey or focus on one aspect of your employees’ role. By keeping your question focused, you’ll receive a focused response that provides valuable information about specific business operations, products, or services.
Also, use plain language, not jargon, so that there’s less chance a customer will misunderstand a question. Misunderstanding a question could lead to empty responses or irrelevant data.
A good way to format your surveys is to ask respondents a close-ended question that lets them rank something specific, and follow up with a simple, “Why?”.
It used to be that you’d have to hand-annotate and manually analyze surveys with open-ended responses, wasting employee time on hours of tedious tasks, and producing results far below the desired accuracy.
However, with advances in natural language processing (NLP), machines can now do this work for us. Custom-trained (or pre-trained) text analysis tools can automatically analyze open-ended responses for topics, themes, opinions, keywords, and more – with accuracy levels above and beyond what humans could ever do.
The aim is to quantify unstructured data, like open-ended responses, so that they can be easily interpreted in graphs and charts – in much the same way as numerical data.
MonkeyLearn is a text analysis platform with a suite of ready-to-use tools to ensure you get the most out of your open-ended survey data. They’re super easy to use and can integrate with tools you already use.
MonkeyLearn’s tools are ready to go, right out of the box, with little setup necessary. Better yet, you can train these tools, and more, (usually in just a few steps) to the language, needs, and criteria of your business, so you never have to worry about accuracy.
It’s clear that open-ended questions can offer deeper and more powerful insights than close-ended, Yes/No or multiple choice questions. They may be a bit harder to analyze, but with the help of machine learning tools, like MonkeyLearn, the extra work is minimal. And you’ll save time and money, and get much more powerful results in the long run.
For best results, combine open-ended and close-ended questions for qualitative and quantitative data. When you have the tools in place, you can analyze open-ended questions (or customer feedback from all over the internet) constantly and in real time.
January 25th, 2021