Null Hypothesis
The default assumption in a statistical test that there is no meaningful difference between the control and test groups - any observed difference is due to random chance rather than a real effect.
Also known as: H0, no-effect hypothesis
Why It Matters
The null hypothesis is the skeptic in every experiment. It assumes your change had no real effect until the data proves otherwise. This is a feature, not a limitation - it sets a high bar for declaring victory and protects you from acting on random noise.
Understanding the null hypothesis clarifies what your test actually measures. You are not proving that your variant is better. You are determining whether the data is inconsistent enough with "no effect" that you can reject that assumption with confidence. This subtle distinction matters because it keeps teams honest about what experiments can and cannot prove.
When a test "fails to reject the null hypothesis," it does not mean your change definitely had no effect. It means the data did not provide strong enough evidence to conclude there was one. This distinction is important for deciding whether to re-test with a larger sample, iterate on the variant, or move on to a different opportunity.
Industry Applications
A grocery delivery service tests a redesigned product search. The null hypothesis states there is no difference in search-to-cart conversion. After 50,000 searches per variant, they fail to reject the null - the redesign looks better but performs identically. They avoid a costly migration.
A CRM company tests a simplified onboarding flow. The null hypothesis (no effect on 7-day activation) is rejected at 95% confidence, with the new flow showing 12% higher activation. The clear rejection of the null gives leadership confidence to roll out the change company-wide.
How to Track in KISSmetrics
Every experiment in KISSmetrics implicitly tests against the null hypothesis. When viewing test results, the significance indicator tells you whether the null hypothesis has been rejected. If the result is marked as "not significant," the null hypothesis stands - the observed difference could plausibly be random variation.
Common Mistakes
- -Interpreting a failure to reject the null as proof that the variant had zero effect
- -Designing experiments that are too small to reject the null even when a real effect exists (underpowered tests)
- -Ignoring the null hypothesis and making decisions based on directional trends regardless of statistical significance
- -Not understanding that the null hypothesis applies to the specific metric you defined, not to all possible effects of your change
Pro Tips
- +Frame your null hypothesis explicitly before every test: "We expect no difference in conversion rate between control and variant"
- +Use the null hypothesis as a communication tool - it helps stakeholders understand that maintaining the status quo is a valid outcome
- +When the null is not rejected, investigate whether the test was underpowered before concluding the change had no effect
- +Consider one-tailed versus two-tailed null hypotheses depending on whether you only care about improvement or also want to detect harm
Related Terms
Hypothesis Testing
A statistical method used to determine whether observed differences in data - such as a higher conversion rate in a test variant - are likely real or could have occurred by random chance.
P-Value
The probability of observing a result as extreme as the one measured, assuming the null hypothesis is true. A small p-value (typically below 0.05) suggests the observed difference is unlikely due to chance alone.
Type I Error
A false positive in hypothesis testing - incorrectly rejecting the null hypothesis and concluding that a change had a real effect when the observed difference was actually due to random chance.
Type II Error
A false negative in hypothesis testing - failing to reject the null hypothesis and concluding that a change had no effect when it actually did produce a real improvement.
Statistical Power
The probability that a test will correctly detect a real effect when one exists, typically set at 80% as a minimum standard. Higher power means a lower chance of missing genuine improvements.
See Null Hypothesis in action
KISSmetrics tracks every user across sessions and devices so you can measure what matters. Start free - no credit card required.