Control Group
A control group is the subset of users in an experiment who receive the existing or unchanged experience, serving as the baseline against which the performance of test variants is measured.
Also known as: baseline group, control variant, holdout group
Why It Matters
The control group is what makes an experiment an experiment rather than a before-and-after comparison. By simultaneously showing some users the original experience (control) and others the new experience (variant), you eliminate the confounding factors that plague sequential comparisons - seasonal changes, marketing campaigns, press coverage, and other external factors that affect metrics over time.
Without a control group, you cannot know whether a change in your metrics was caused by your test or by something else happening simultaneously. If you launch a new checkout flow on the same day a competitor runs a major sale, your conversion rate change could be entirely explained by the competitive environment. A properly randomized control group experiences the same external factors, isolating the effect of your change.
Control groups are also used outside of A/B testing as holdout groups for marketing campaigns. By holding back a small percentage of your audience from receiving a promotion, you can measure the true incremental impact of the campaign rather than conflating it with organic behavior.
Industry Applications
An ecommerce brand holds back 10% of their email list from receiving a promotional campaign. The control group purchases at a baseline rate of 2.1%, while the promotional group purchases at 3.8%. The true incremental lift of the campaign is 1.7 percentage points, not the full 3.8%.
A SaaS company maintains a 5% holdout group for their new onboarding flow for three months. They discover that while the new flow shows 15% higher week-1 activation, the 90-day retention difference between control and variant is only 3%, indicating the new flow accelerates activation but has a smaller long-term impact.
How to Track in KISSmetrics
Your A/B testing tool manages control group assignment automatically. In KISSmetrics, tag users with their experiment group assignment as a user property so you can analyze control vs variant behavior across any metric over time, not just during the test window.
Common Mistakes
- -Not verifying that control and variant groups are properly randomized and balanced on key characteristics.
- -Contaminating the control group by exposing them to elements of the test variant through shared sessions or linked accounts.
- -Making the control group too small relative to the variant, which reduces statistical power and makes results less reliable.
- -Not maintaining holdout groups for rolled-out features, preventing long-term impact measurement.
Pro Tips
- +Run A/A tests periodically (both groups see the same experience) to verify that your experimentation infrastructure is working correctly and not introducing bias.
- +Use stratified randomization for small tests to ensure that control and variant groups are balanced on important dimensions like device type and traffic source.
- +Maintain a small permanent holdout group (1-5% of users) for major features to measure cumulative long-term impact.
- +Verify control group health by checking that key metrics (traffic volume, session duration, demographics) are statistically equivalent between groups before analyzing test results.
Related Terms
Variant
A variant (also called a treatment or challenger) is an alternative version of a page, feature, or experience being tested against the control in an experiment, incorporating the specific changes hypothesized to improve performance.
A/B Testing
A/B testing is a controlled experiment that compares two versions of a web page, email, ad, or feature by randomly splitting traffic between them and measuring which version performs better on a defined success metric.
Statistical Significance
Statistical significance is a measure of confidence that the difference observed between test variants is real and not due to random chance, typically expressed as a percentage (e.g., 95% confidence) or a p-value threshold.
Sample Size
Sample size is the number of users or observations included in each variant of an experiment, determining the statistical power of the test and how confidently you can detect real differences between variants.
Conversion Rate
Conversion rate is the percentage of users who complete a desired action out of the total number of users who had the opportunity to do so, serving as the primary measure of how effectively a page, campaign, or experience turns visitors into customers.
See Control Group in action
KISSmetrics tracks every user across sessions and devices so you can measure what matters. Start free - no credit card required.