How to Analyze Split Test Data
Read a test's results page, understand the verdict, and decide what to ship.
Go to Optimize → Experiments and click a test. Running tests are in the Running column (Board view) or under Active (List view).
ADD GIF HERE
The verdict
The box at the top tells you, in plain words, where the test stands:
Statistically significant - safe to implement winner
Confidence reached your target and at least 80% of the needed sample is in.
Ship the winner.
Trending positive but not yet significant
The variant looks better and confidence is close to your target.
Keep it running.
Inconclusive - sample size too small
Less than half the needed visitors so far.
Keep it running.
No significant difference detected
Enough data, but the versions perform about the same.
Stop it and test a bigger change.
Next to the verdict you'll see Confidence, Sample (how much of the needed sample is in), Days (days running / recommended minimum), and Lift. A Low traffic warning means the test may take months. Consider testing a higher-traffic page or a bolder change.
Headline numbers
Unique Visitors
People included in the test, across all versions.
Conversions
Times your goal happened, across all versions.
Improvement
The variant's lift over the control, once there's a clear result.
Confidence
How sure you can be that the difference is real. Aim for your target, usually 95%.
Results tab
Performance Comparison: visitors, conversions, rate, and vs Control for each version. For revenue tests you'll also see Total Revenue, Rev/Visitor, and AOV (average order value).
Statistical Analysis: confidence for each version against the 95% Confidence Threshold, sample progress, days running, daily visitors, and confidence intervals (the likely range of each version's true rate).
Test Health: readiness checks and recommended next steps. Also available from Test Health at the top of the page.
Trends: conversion rates side by side, and daily conversion rate over time. Lines that keep crossing mean the result isn't settled yet.
Click-Through Details: every element clicked in each version, with its destination. Useful for spotting what people clicked instead of your CTA.
Settings tab
The test's configuration: goal, tracking level (per visit or sticky), significance target, MDE, baseline rate, traffic split, start and end dates, and how it was stopped.
Deciding what to do
When the test is Ready to Call, you'll see:
Call this test: record the result and end the test.
Ship winner to 100%: send all visitors to the winning version. For no-code tests, Humblytics applies the winning changes for everyone through the tracking script, so you don't need to rebuild anything right away.
Keep running: collect more data first.
Before you ship, sanity-check the other numbers. A variant that gets more clicks but fewer purchases isn't a real win.
To end a test without a winner, click Stop at the top of the page. From the list, the ⋯ menu also has Pause Test and Archive. Data collected so far is kept.
After the test
Make it permanent. For page-level tests, publish the winning page in your site builder. For no-code tests, apply the winning change to your site when you can.
Write down what you learned. Use Capture in the Ideas column of the Board to save the result and the next test idea.
Last updated