Test whether your A/B test results are statistically significant using a two-proportion Z-test. See p-value, confidence interval, and effect size.
After running an A/B test, you need to determine whether the observed difference between your control and variant is real or simply due to random chance. Statistical significance testing answers this question by calculating the probability that the observed difference (or a larger one) would occur if there were actually no difference between the two versions.
The standard approach for comparing two conversion rates is the two-proportion Z-test. It computes a Z-statistic from the observed rates and sample sizes, then converts it to a p-value — the probability of seeing such a result under the null hypothesis of no difference. A p-value below your significance threshold (typically 0.05) means the result is statistically significant and unlikely to be due to chance alone.
This calculator takes your A/B test results (visitors and conversions for each variant), performs the two-proportion Z-test, and reports the Z-statistic, p-value, confidence interval for the difference, and a clear verdict on significance. It helps you make confident decisions about whether to implement the winning variant.
Declaring A/B test winners without proper significance testing is a recipe for implementing changes that don't actually work. This calculator gives you the statistical rigor to separate real effects from noise. It produces the p-value, confidence interval, and relative lift with clear pass/fail verdicts, so you can make data-driven decisions with confidence.
p̅ = (x₁ + x₂) ÷ (n₁ + n₂) [pooled proportion] Z = (p̂₁ − p̂₂) ÷ √(p̅(1−p̅)(1/n₁ + 1/n₂)) Where p̂₁ = x₁/n₁, p̂₂ = x₂/n₂ CI = (p̂₂ − p̂₁) ± Zα/2 × √(p̂₁(1−p̂₁)/n₁ + p̂₂(1−p̂₂)/n₂)
Result: p-value = 0.023, Significant at 95%
Control: 500/10,000 = 5.00%. Variant: 580/10,000 = 5.80%. The difference of 0.80 percentage points (16.0% relative lift) gives Z = 2.28 and p-value = 0.023. Since p < 0.05, this result is statistically significant at the 95% confidence level. The 95% CI for the difference is [0.11%, 1.49%], confirming the variant outperforms the control.
The p-value is the probability of observing results as extreme as yours if the null hypothesis (no difference) were true. It is NOT the probability that the null hypothesis is true. A p-value of 0.03 doesn't mean there's a 97% chance the variant is better; it means the data would be unlikely (3% chance) if there were no real difference. This distinction is crucial for proper interpretation.
Peeking at results during the test and stopping early when significance is reached inflates false positive rates dramatically. Multiple comparison problems occur when testing many metrics without correction. Simpson's paradox can make overall results misleading when subgroups have different patterns. Always pre-register your hypothesis, primary metric, sample size, and analysis plan.
Report effect sizes (relative lift, absolute difference) alongside p-values. A 20% relative lift that's significant is more actionable than a 2% lift that's also significant. Combine statistical results with business context: implementation cost, opportunity cost, and long-term strategic value should all factor into the decision of whether to ship the winning variant.
Statistical significance means the observed result is unlikely to have occurred by random chance alone. In A/B testing, it means the conversion rate difference between control and variant is probably real, not noise. The p-value quantifies this: a p-value of 0.03 means there's only a 3% chance of seeing this large a difference if the variants were identical.
The standard threshold is 0.05 (5%), meaning you accept a 5% chance of false positives. For high-stakes decisions (pricing changes, major redesigns), some teams use 0.01 (1%). For exploratory tests, 0.10 may be acceptable. The key is to set the threshold before running the test and stick to it.
A confidence interval gives a range of plausible values for the true difference between variants. A 95% CI of [0.5%, 1.5%] means you're 95% confident the true difference lies between 0.5 and 1.5 percentage points. If the CI includes zero, the result is not significant at that confidence level.
Absolutely. With very large sample sizes, even tiny differences can be statistically significant. A 0.01% conversion lift might be significant with millions of users but isn't worth implementing. Always consider practical significance (is the effect large enough to matter?) alongside statistical significance.
A non-significant result means you failed to detect a meaningful difference, not that there is no difference. The test may have been underpowered. Check whether you reached the planned sample size. If yes, the change likely has no practical impact. If no, consider running longer or accepting a larger MDE.
Two-tailed tests are the standard because they detect both improvements and degradations. Use them unless you have a strong prior that the variant can only improve the metric (rare in practice). This calculator uses a two-tailed test, which is more conservative and appropriate for most A/B testing scenarios.