If you’re conducting a survey, you’ll need to understand your results in-depth and gain a deeper understanding of your study population. In most cases, this will require you to calculate three measures namely Margin of Error, Confidence Level, and Statistical Significance.
If you’re not a statistician, these three terms may be hard to understand. Although the three terms sound difficult to grasp, the ideas behind them are very simple.
In this article, we will be discussing these terms in detail, help you know how to calculate each one of them, and explain how they differ from each other.
Statistical Significance
What is Statistical Significance?
After conducting a survey, you will always want to know whether the results are “significant”. This will determine whether the results will be used for decision-making or not.
Statistical significance is the measure of the probability of a null hypothesis being true in relation to the acceptable level of uncertainty in the true answer. Simply put, a statistically significant survey is one where the results are most likely to be true. Such a survey helps you conclude that the results obtained were as a result of work done rather than by chance.
Statistical significance determines the confidence level and risk tolerance. For example, let’s say you are running an A/B test with a significance level of 95%.
After getting the results, you’ll be 95% confident that they are real and not an error resulting from randomness. At the same time, it means that there is a 5% chance that your results are wrong.
Does Statistical Significance Matter?
Statistical significance is used to prove whether a particular statistic is reliable or not.
Before making a decision based on the results of a survey, you should be sure that there is an actual relationship. Digital advertisers are nowadays conducting A/B tests to get statistical significance before making conclusions.
Statistical significance is also a good tool for hypothesis testing. For example, you want to investigate whether changing the position of a button on a web page will result in more clicks. If the button is at the top, that is known as the “null hypothesis”. Moving the button to the middle is called the “alternative hypothesis”.
To know the difference in the significance test, you should consider two outputs namely the confidence interval (MoE) and the p-value. The confidence interval will be discussed later in this article. The p-value is the probability of getting an effect from a sample population.
Although you should care about statistical significance, it is not always necessary. There are situations where you can still draw meaningful conclusions from your survey results without using statistical significance.
At the same time, it’s worth noting that statistical significance doesn’t give you a guarantee that your survey results are useful. This means that it doesn’t add any importance to your survey. If the survey has a good statistical significance, but it’s a bad survey, it will remain a bad survey. If you’re conducting research to get valuable data about brand performance, that data must not be statistically significant. Hence, a surveyor should focus on a statistical measure like Margin of Error (MoE) than statistical significance.
Margin of Error
What is Margin of Error?
Your survey results will not exactly match your study population. However, you can know how close you are using the margin of error (MoE), also known as the confidence interval. It tells you how your survey results reflect the views of the entire population.
Remember that in surveying, a smaller group (survey respondents) is used to represent a much larger population (target or total population). See the MoE as a way of determining the effectiveness of the survey. A small value for the MoE means that you’re more confident with the survey results. The vice versa is true, a large value for the MoE means that the survey results stray far from the views of the entire population.
The MoE is a range of values below and above the actual survey results. For example, if “65%” of the individuals taking your survey say “yes” with an MoE of 5%, it means that 60-70% of the total population would say “yes”. If you reduce the value for the MoE, say to 3%, the results will even be more accurate. The spread will be tighter, that is, 63-68%.
Most industries use an MoE of 5%. However, you can change this value depending on the confidence that you have in your survey results.
How Do You Find the Margin of Error
MoE is calculated using a simple formula as shown below:
MoE=z × √n Where: z = z-score σ = standard deviation of the population n = sample size
Steps: Determine the sample size (n). Calculate the standard deviation of the population (σ). Get the square root of the sample size (√n). Divide the standard deviation by the square root of sample size. Multiply the result from step 4 with the z-core. Note: The z-score should be consistent with the confidence interval that you need to achieve as summarized in the following table:
Why Margin of Error (MoE) Matters?
All statistical results should include the MoE.
MoE is important because most statistics only use a part of a population to estimate numbers regarding the whole population. There is no single-number result that is an accurate estimation of a whole population unless the researcher collected data on every single member of the study population.
What happens is that data is collected from a sample of a population, the sample results are analyzed, and conclusions regarding the whole population are made based on the results from the sample population. The fact is that sample results will vary from one sample to another, and this variation should be reported.
MoE is the statistic used to measure the level of precision in the sample results of a study. The error in the margin of error doesn’t necessarily mean that a mistake was made and it needs to be reported. It only means that since your results are based on a sample of the entire population, there is a potential gap between your results and the real value that you need to estimate for the population.
Confidence Level
For your survey to be statistically significant, you must calculate the confidence level.
In our previous example, where we had an MoE value of 5% and 65% of the sample population saying “yes”, 60-70% of the target population would say. But how can we be certain that 60-70% of the entire population will say “yes”? That’s why we need the confidence level, which is the level of certainty that you have in a particular result.
So, confidence level measures the probability by which an estimation parameter in a sample population is also true for the whole population. Note that confidence level should not be confused with confidence interval (also called the margin of error). After conducting your survey on a sample population, it’s difficult to have a 100% certainty that the obtained results reflect the views of the whole population.
The reason is that some niche views in the target population may not be represented in the sample population used in the survey. So, the confidence level will tell you whether your results are reproducible or outliers.
Most industries use a confidence level of 95%, which means that if you conduct the same survey on the total population repeatedly, the results would match those of the sample population 95% of the time. For example: In a survey involving 2,000 Americans on whether they supported a ban on smoking in restaurants, 75% of the respondents said “yes”. The confidence level for the survey was 95% while the margin of error was 2%. Using the MoE, we can tell that the actual supporters of the ban range between 73% and 77%.
If the survey is conducted 100 times, with 2,000 participants each time, the number of people supporting the ban fall between 73% and 77%, 95 out of 100 times. However, the number of Americans supporting the ban would fall below or above 73-77%, 5 out of 100 times.
Some industries use confidence levels of 98% and 99%, increasing the chances of getting the correct results. However, the use of such confidence levels also means that larger sample population sizes should be used. To calculate the appropriate sample size that can give you statistically significant results, you’ve to combine different values for confidence level and margin of error.
However, the good news is that there are several sample size calculators that you can find on the internet.
Conclusion
Margin of error, confidence level, and statistical significance is not something you can grasp on the first go. If there is anything you learn from this article, remember:
Statistical significance determines the reliability of survey results.
A statistically significant survey is a survey whose results are most likely to be true.
Although statistical significance is a popular statistic, it is not always necessary. There are situations under which you can still draw meaningful conclusions from your survey without considering the statistical significance.
The Margin of Error (MoE), also called the confidence interval, tells us how the survey results reflect the views of the entire population.
Since only a sample of the entire population is used to conduct a survey, there will be a potential gap between your survey results and the real value that you need to estimate for the population. This gap can only be accounted for by the MoE, making it an important statistic in every survey.
The confidence level tells us the level of certainty in a particular result.
The confidence level is not as important as the confidence interval (MoE) when analyzing survey results.
If you would like to learn how margin of error fits into brand tracking, check out this whitepaper.
