Using the Humblytics A/B Sample‑Size Calculator
A/B (split) testing compares a control (Variant A) with a variation (Variant B) so you can make evidence‑based improvements to your website or app. The Humblytics Sample‑Size Calculator tells you exactly how many visitors each variant needs before you can trust the result.
1. Why Sample Size Matters
Too Small
Just Right
Too Large
Results look erratic; you risk acting on noise.
Detects real differences with high confidence.
Wastes time and traffic without adding precision.
Choosing the correct sample size balances statistical rigour with business velocity.
2. Input Definitions
Field
What It Means
Example
Baseline Conversion Rate
Your current conversion rate.
5 %
(5 of every 100 visitors convert)
Minimum Detectable Effect (MDE)
The smallest lift you care about.
+1 %
absolute (from 5 % → 6 %)
Statistical Significance
Confidence level that the observed lift is real, not random.
95 %
(industry default)
Statistical Power
Probability of detecting an effect if it exists.
80 %
(common default)
Tip: Lower MDE or higher confidence / power settings will increase the required sample size.
3. Step‑by‑Step
Open the Humblytics Sample‑Size Calculator.
Enter each value defined above.
Click Calculate.
Record the required visitors per variant shown in the results panel.
Plan your test window so you can realistically hit those numbers.
4. Worked Example
Baseline Conversion Rate:
5
MDE:
1
Significance:
95
Power:
80
▶︎ Result: ≈ 4,000 visitors per variant (total ≈ 8,000 sessions). Run the experiment until both A and B have reached these counts before analysing.
5. Best‑Practice Reminders
One Variable at a Time — isolate the element you’re testing.
Run Full Business Cycles — capture weekday/weekend traffic differences.
Don’t Peek Early — premature stops inflate false‑positive risk.
Look Beyond Win/Loss — examine bounce rate, engagement, revenue per visitor.
Iterate — document learnings and queue up the next hypothesis.
Following this workflow ensures every Humblytics test is powered correctly, statistically sound, and focused on meaningful business impact.
Last updated
Was this helpful?