Picture this. You run a survey to gauge your brand awareness. When the results come in, the data appears to show that your branding campaigns have broken through the noise and a large proportion of your target audience recognizes your brand.
Because of this data, a new campaign strategy is agreed upon with a focus on capitalizing on your strong awareness and converting it into paying customers. But the campaign falls flat.
What went wrong?
It could be possible that something as simple as question order in your survey has influenced how people respond, leading to certain answers being overstated. In this scenario, perhaps brand awareness wasn’t actually as high as reported.
Inaccurate data results in unreliable insights which in turn lead to ineffective strategies. While this is a worst-case scenario, it’s a useful way of demonstrating how small factors in your survey can influence respondents and how, over thousands of surveys, these inaccuracies can be massively amplified.
When it comes to conducting a survey, it is really important that it’s implemented in a way that ensures all the results are an accurate reflection of the real world. To that effect, survey design has to take into account a number of different factors so that researchers can be certain that the right people answer all the questions posed truthfully, and without being influenced by any external factors or inherent biases.
Brand randomization is one of the key techniques in removing bias from surveys and, in this article, we’re going to look at why it’s such an important step when building surveys. Plus, how it can increase the accuracy of the collected data, with an example from our own study on the topic.
What Are Biases?
There are lots of different definitions for different types of biases, but in the context of surveys, questionnaires, and even interviews, a response bias is an error that is incorporated into your data by unintentionally encouraging one answer over the rest. As we are not always aware of biases, they can often be hard to spot.
Order bias is where survey respondents often favor and thus select the first options presented to them, or conversely, skip over options that appear last. Essentially, a question’s place in a survey can unintentionally influence how respondents answer them — and over hundreds or thousands of surveys responses — this bias could skew your data.
Different factors can influence why a respondent might unconsciously change their answers as a result of the question’s order. Often, it can simply be that respondents are rushing in order to complete the survey, selecting the first answers that make sense to them and moving on quickly.
On top of that, respondents have a tendency to be influenced by the questions they have already answered, often in an attempt to maintain consistency. Specific questions about certain aspects of a particular brand, such as affordability, sustainability, or the quality of their customer service, for example, might then influence a later broader question that asks users to score their overall impression of a brand.
It may be surprising that something as simple as question order can have such a huge impact and many researchers may not have considered its effect — but through A/B testing, it has been proven that order can impact the results of your survey.
Why is Randomization Important in Surveys?
Randomization is a solution to order bias, that works simply by mixing up the order of the questions randomly. Individual respondents may still be influenced by the order of questions, but because every survey is randomized, there’s no way that this bias will skew your data and lead to unreliable insights.
How Randomization Can Make your Survey Results More Accurate
Let’s take a look at how this works in practice, using a Latana study on the effect of randomization as an example. We’ve anonymized the names of the brands from this survey to protect our partner’s data.
For this study, we ran a survey where we asked respondents to select which brands they were aware of out of a group of 10. Each brand was presented one at a time to respondents, who were split into two groups.
Group A was presented with the brands in a fixed order (reflected in the order of the brands in the table below) and Group B was presented with the same brands but in a randomized order, removing any chance of bias.
From this study, we can see a pronounced difference between Group A and Group B in reported awareness, which is most exaggerated for the brands that came at the start and at the end of the fixed order (brands 1, 2, and 10). For Brands A and B, the difference is as much as 10%.
As demonstrated by Brand 3 (which was a well-known brand), the order won’t completely skew your results, but it has the potential to over or understate certain answers and to such a degree that it may then affect the decisions you make.
While in Group A we can see the effect of order bias in practice, Group B demonstrates that if a fixed order is removed, we can get achieve more accurate results. The better-known Brand 3 still gets a strong score reflecting its higher brand awareness, while Brands 1, 2, and 10 no longer receive an advantage or disadvantage based on their place in the survey.
By implementing this simple randomization, the threat of order bias is removed and the data from your survey is more accurate. All of Latana’s surveys feature randomization as part of their methodology to ensure more accurate results, including all statement-based segmentation questions, demographic questions, and multiple-choice questions.
Final Thoughts
As demonstrated earlier, when it comes to audience research, it is easy to overlook small factors that could skew your data. Order Bias is just one of many biases that can influence respondents and lead to inaccurate answers.
Latana’s brand tracking software allows marketers and researchers to have confidence in their data. Our team of data scientists and researchers have worked to ensure that the whole data collection process is streamlined to ensure that your brand insights represent the real sentiments, opinions, and preferences of the general population or your target audience.
Not only are surveys designed to achieve the most accurate results but our MRP algorithm draws on audience characteristics to make informed predictions that are more accurate than the outcomes you can expect from quota sampling.
All of this is to ensure that our clients can easily access reliable insights into their brand health, which can then be used to inform effective strategies that power marketing campaigns and grow brands.
