In this article, we will go through a sweet statistical concept: the so called "p-value". There are a few statistical estimates that provide tremoundous joy or disappointment to marketers, the p-value definitely belongs to this category.
Measure-geeks, I am super excited to talk about the p-value and I hope my thoughts will be very easy to digest. In a previous article, we already discussed the importance of statistical significance when running tests for your conversion rate optimization strategy.
When researchers / scientists / marketers are calculating the p-value, they are putting on the test what is called the null hypothesis (which I really hate 🙄). The p-value includes two hypothesis: the null hyphotesis and the alternative hyphotesis:
The p-value is that sweet number which helps you determine the statistical significance of your experiments.
Let me break down the concept with a more concrete example.
You start an A/B test on your landing page. Control version = landing page with 3 CTAs, Variant version: landing page with 5 CTAs. You split the traffic 50/50. You want to check if whetever increasing the number of CTAs might improve conversion rates or not. The null hypothesis is the devil’s advocate argument, it states that the difference in conversion rates can't be explained by the different number of CTAs in your landing pages. It basically tells you that results are due to randomness.
The level of statistical significance is often measured as a p-value between 0 and 1. The smaller the p-value, the stronger is the evidence that you can easily reject the null hyphotesis. Usually when the p-value is less than 0.05, you can assume that your test got statistically significant. In other words, if p-value is less than 0.05, it is very unlikely that the null hypothesis is true (less than 5%).
Why do we consider this 5%?
There are two main reasons: the first one for simple mathematics, the second one for more "serious" mathematics.
Simple mathematics says that the probability that a flipped coin will show the same face 5 times in a row is 0.5 × 0.5 × 0.5 × 0.5; that is = 0.0625. This p-value, 0.0625, is rather close to the value 0.05 that is by general convention set as the cut-off for “statistical significance.”
A more advanced scientific reason for choosing 5% as the cut-off is that approximately 5% (4.5%, to be more precise) of the normal distribution comprises outlying or “significantly different” values, that is, values that are more than two standard deviations distant from the mean. The credit for setting this 5% goes to Ronald A. Fischer, one of the fathers of the modern statistical analysis. (Don't worry if those concepts are unfamiliar! In the upcoming weeks, we will go together through same basic statistical concepts like mean, standard deviation, normal distribution etc...)
The question you might ask now is: how can I calculate the p-value? Luckly machines and softwares do it for us. I usually use this website to calculate the p-value of my experiments.
Happy reading you all ❤️
Comments