An open-ended survey asks a fixed set of questions usually via a mass communication channel such as an email or website pop-up. Answers are not constrained as in the case of multiple choice or check boxes but are free-text responses where the customer can choose the length and detail of their answer.
- Who is our customer?
- What are their pains?
- What are the jobs to be done?
- Generative market research
Time Commitment and Resources
Surveys can be quick to write and execute, often taking only 1 to 2 hours to set up. However, designing effective questions that don’t introduce biases into customers' answers can require a high degree of skill, multiple revisions, and even comprehension tests run on the survey.
Collecting results typically takes more time and depends on the communication channels available to distribute the survey. Response rates can vary from 1 percent to 20 percent on a survey sent to existing customers depending on the level of customer engagement, so large numbers of target customers and time may be required to collect data.
Analyzing the data can take 4 to 8 hours, depending on the length of the survey, number of respondents, and quality of responses. As answers are free text, a large amount of reading, transcribing, and synthesizing may be required.
Write screening questions:
- These are typically close-ended questions that help identify if the respondent is in the desired target segment. E.g “How old are you?”
- A few leading questions can be placed in a survey to identify “professional survey respondents” who will lie in order to be included in a survey or have a chance to participate in a follow-up research project for cash.
- Questions should be non-leading and non-hypothetical.
- Asking for anecdotes or historical information can generate more concrete insights.
- Conduct comprehension tests on survey questions.
- This ensures that the questions are being correctly interpreted and can often reveal leading questions.
Surveys can be sent out via any method to the target audience.
Typical distribution methods include:
- Social media
- Website pop-ups
- Regular mail
Debriefing and Interpreting Results
An open-ended survey is a generative research technique, and as such, be careful to interpret any input as simply ideas, not as a vote from the customer. The data is qualitative in nature.
Because surveys are flexible, easy to write, and easy to deploy, they are more likely to be misunderstood and misused. Surveys are often a default research method when researchers do not feel they have the time to conduct ethnography or customer interviews. They are often highly favored in a corporate setting because a large number of respondents may be considered statistically significant, even if the survey responses are qualitative.
Open-ended surveys are also sometimes combined with close-ended surveys, making it tempting to spend extended periods of time analyzing the data and looking for correlation that will draw a definitive conclusion to act upon. This tendency to use the data to drive a firm conclusion even when the data is generative in nature is the biggest argument to avoid surveys at all costs.
A typical debrief method to analyze the generative data is to read each answer and transcribe salient points on post-it notes for a sorting exercise. Patterns can then be identified.
For surveys specifically soliciting suggestions from users, the entire list of suggestions may be added to a repository for later analysis.
In the case of very large data sets, algorithmic tools such as sentiment analysis or word clouds can give additional quantitative insight but should be used to supplement the qualitative insights, not replace them.
- Selection bias: Researchers will often fixate on qualitative comments they agree with and ignore other comments.
- Sampling bias: Although the sample may not match the general population to be surveyed, if the data is taken as generative and not evaluative, this bias is less relevant. Any ideas generated still must be validated by an evaluative method.
- “You can run an open-ended survey once you know the best questions to ask. Talk to your customers to figure out the right questions.” - @TriKro
- Got a tip? Add a tweetable quote by emailing us: [email protected]
- Got a case study? Add a link by emailing us: [email protected]
- A Guide to Open-Ended Questions in Marketing Research
- FluidSurveys: Comparing Closed-Ended and Open-Ended Questions
- Midwest Political Science Association: Structural Topic Models for Open-Ended Survey Responses
- Centers for Disease Control and Prevention: A Catalog of Biases in Questionnaires
- Open-Ended Questions: Get More Context to Enrich Your Data
- Got a reference? Add a link by emailing us: [email protected]