Ab Sample Size Calculator

Estimate how many users you need in your control and variant before launching an experiment. This calculator uses a standard two-sample proportion power analysis for conversion-rate testing, then visualizes how your required sample changes as your minimum detectable effect shifts.

Expert Guide to Using an A/B Sample Size Calculator Correctly

An A/B sample size calculator helps you answer one of the most important questions in experimentation: how much traffic do we need before we can trust the test? Teams often obsess over button colors, pricing layouts, copy variations, and funnel changes, yet many experiments fail long before the design decision matters. The real issue is statistical planning. If your sample is too small, a promising change can look like noise. If it is unnecessarily large, you waste time, traffic, and product momentum. A reliable calculator creates discipline before the test begins.

For conversion-focused experiments, sample size planning usually comes down to five inputs: your baseline conversion rate, your minimum detectable effect, your significance threshold, your desired power, and the test shape itself, usually one-sided or two-sided. Once you enter these assumptions, the calculator estimates the number of visitors required in each group. In practical terms, that tells you whether your experiment is realistic next week, next month, or only after a longer data collection period.

Core idea: smaller expected improvements require much larger samples. Detecting a tiny uplift is expensive. Detecting a large uplift is faster. That trade-off is the central planning problem in experimentation.

What this calculator is actually measuring

This page uses a standard power analysis for a two-sample test of proportions. In A/B testing, your primary metric is often binary at the user level: converted or not converted, subscribed or not subscribed, clicked or not clicked. The calculator estimates the sample needed to separate two proportions, where the control has a known or assumed baseline conversion rate and the variant is expected to perform better or worse by a chosen amount.

Suppose your current conversion rate is 5%. If you want to detect a 15% relative uplift, you are not testing for a new rate of 20%. You are testing whether the variant can raise 5.0% to 5.75%. That sounds small, and statistically it is. The closer the two rates are, the more observations you need to be confident that the difference is not just random fluctuation.

Why baseline conversion rate matters so much

Many teams underestimate the importance of the baseline rate. If your starting conversion rate is very low, such as 1%, even modest-looking uplifts can be difficult to detect. If your baseline is higher, the absolute difference generated by the same relative uplift can be larger, which changes the required sample. You should always use a realistic baseline based on recent, stable traffic rather than an old quarterly average or a best-case month.

• Use recent data from a period with similar traffic sources, device mix, and seasonality.
• Avoid inflated baselines caused by campaign spikes or temporary pricing promotions.
• Align the baseline with the metric definition you will use during the experiment, such as session conversion versus user conversion.

Understanding minimum detectable effect

The minimum detectable effect, often shortened to MDE, is the smallest relative improvement worth detecting. This is not simply a statistical preference. It is a business decision. If a 2% relative gain does not materially affect revenue, retention, or contribution margin, designing a test to detect it may not be worth the traffic cost. On the other hand, if your site gets millions of visitors and a 2% change in checkout completion is financially important, then a smaller MDE may be exactly right.

A practical way to choose MDE is to tie it to annualized business impact. Estimate how many extra conversions a given uplift would create, then compare that lift with engineering effort, opportunity cost, and implementation risk. A sample size calculator becomes much more useful when it supports strategic planning rather than purely academic significance.

Scenario	Baseline Rate	Relative Uplift	Variant Rate	Approx. Sample per Group
Ecommerce checkout optimization	5.00%	10%	5.50%	31,208
Ecommerce checkout optimization	5.00%	15%	5.75%	14,174
Ecommerce checkout optimization	5.00%	20%	6.00%	8,147
Ecommerce checkout optimization	5.00%	30%	6.50%	3,775

The values above are computed with a two-sided test, 95% confidence, and 80% power. They illustrate the steep relationship between effect size and sample size. A small change in MDE can dramatically affect the feasibility of a test.

Confidence level and power: what they mean in plain English

Confidence level reflects how strict you want to be about false positives. A 95% confidence level corresponds to a 5% significance threshold, commonly written as alpha = 0.05. If no real difference exists, you would expect to falsely declare significance about 5 times in 100 similar tests. Power measures the chance that your test will correctly detect a real effect of the size you planned for. An 80% power level means that if the true uplift equals your selected MDE, the test should detect it 80% of the time.

Choosing higher power or a stricter confidence level increases the required sample. That is not a flaw. It is the mathematical cost of being more demanding.

Setting	Common Value	Interpretation	Typical Use
Confidence level	95%	Controls false positive risk at 5%	Default for most product experiments
Power	80%	Detects the planned effect 4 times out of 5	Balanced choice for growth teams
Power	90%	Lower false negative risk	High-stakes pricing or funnel decisions
Hypothesis type	Two-sided	Checks for uplift or decline	Recommended when any directional change matters

One-sided versus two-sided tests

Some experimentation platforms allow one-sided tests because they require a smaller sample for the same assumptions. The trade-off is interpretive. A one-sided test assumes you only care about improvement in one direction. That can be justified in narrow cases, but many product teams regret this choice when a variant performs worse. A two-sided setup is usually more defensible because it protects against surprise regressions and aligns better with careful decision-making.

How to estimate test duration from sample size

Once you know the sample required per group, you can estimate duration using your average daily eligible traffic. In a simple 50/50 split, divide the per-group requirement by half of your daily visitors. If you need 14,000 users per variant and you receive 10,000 eligible users per day, then each group gets around 5,000 users daily, implying a runtime of roughly 2.8 days. In practice, most teams still let experiments run through complete business cycles to absorb weekday versus weekend behavior, promotion timing, and other temporal variation.

1. Calculate the required sample per group.
2. Split daily eligible users evenly across control and variant.
3. Estimate raw runtime in days.
4. Add calendar realism for traffic volatility, exclusions, and seasonality.

Common mistakes that make sample size calculations unreliable

Even a mathematically sound calculator can produce poor recommendations if the assumptions are weak. The most common failure modes are not coding issues. They are planning issues.

• Using an unrealistic MDE. Teams often choose a tiny uplift because it sounds precise, not because it is commercially meaningful.
• Ignoring metric instability. If your baseline is changing due to campaigns, outages, or audience shifts, the input is not trustworthy.
• Peeking too early. Stopping as soon as the chart looks good inflates false positive risk.
• Underestimating segmentation. If you plan to analyze mobile, desktop, new users, and returning users separately, each segment needs enough sample.
• Confusing visitors with conversions. Sample size is based on users or sessions exposed, not just on completed conversions.

When your calculator says the test is too expensive

This is often the most valuable outcome. If the estimated sample is unrealistic for your traffic level, the experiment may need to be redesigned. You might broaden the change to target a larger effect, focus on a more sensitive funnel step, reduce metric noise, improve instrumentation, or test on a higher-traffic audience first. Sample size calculators are not just go or no-go tools. They reveal where your experimentation strategy needs adjustment.

Strategic rule: if your expected runtime is so long that seasonality, promotions, roadmap changes, or user behavior shifts will distort the result, the experiment design should be revisited before launch.

How professionals choose the right assumptions

Experienced analysts do not pick defaults blindly. They inspect historical conversion data, understand the economics of change, and align significance thresholds with the cost of error. A pricing experiment may justify more statistical rigor than a low-risk homepage copy test. Conversely, a rapid exploration program may accept 80% power and only pursue changes with larger practical effects. Good sample size planning is not rigid. It is context-aware.

For executive communication, it also helps to translate technical settings into decision language. Instead of saying, “we used 95% confidence and 80% power,” explain that you are balancing the risk of shipping a false winner against the risk of missing a meaningful improvement. Stakeholders respond better when the statistical design is connected to business outcomes.

Authoritative sources for deeper study

If you want more background on hypothesis testing, power, and sample size for proportions, these resources are excellent starting points:

• NIST Engineering Statistics Handbook
• Penn State Online Statistics Programs
• UC Berkeley Department of Statistics

Final takeaway

An A/B sample size calculator is one of the highest-leverage tools in experimentation because it forces clarity before exposure begins. It tells you whether the expected gain is measurable, whether the timeline is realistic, and whether the test design is proportional to the business question. Use it before every serious experiment, revisit your assumptions when traffic changes, and remember that a statistically elegant test still needs a meaningful business objective. The best experiments are not just powered well. They are worth powering in the first place.

This calculator is intended for planning two-group conversion experiments with a binary outcome. For highly complex designs, sequential testing frameworks, or multiple-comparison programs, a professional statistician should review the experimental plan.