How Humblytics Split Testing Works - Under the Hood

How visitors are assigned to a version, why they don't see a flicker, and how Humblytics decides a winner.

Showing each visitor a version

Everything runs through the tracking script you already installed. When someone lands on a page with a live test:

1. The page is briefly hidden. The first part of your tracking snippet hides the page for a moment, 2 seconds at most, so the visitor never sees the original flash before their version appears.

2. The visitor is assigned a version based on your traffic split. With Per visit, each visit is assigned again. With Sticky visitor, the same person keeps seeing the same version.

3. The version is shown.

   • No-code tests: the script applies your editor changes (text, styles, images, and so on) to the page.

   • Page-level tests: the script sends the visitor to the variant URL. The redirect on the original page isn't counted as a visit, so nobody is counted twice.

4. The page appears, and tracking carries on as normal.

Single-page apps work too: the script also runs tests when the URL changes without a full page load.

Measuring each version

For every version, Humblytics counts visitors and conversions for your goal, then calculates:

Number	What it means	Example
Conversion rate	Conversions ÷ visitors	60 signups ÷ 1,000 visitors = 6%
Lift	How much better or worse the variant is than the control, as a percentage of the control	6% vs. 5% = +20% lift
Confidence	How sure you can be that the difference is real, not luck	95%

The name of the "conversion" depends on your goal: Submissions for forms, Clicks for click-through rate, Orders for revenue, and so on. See each goal page for details.

How confidence is calculated

Humblytics compares the control and each variant with a two-proportion z-test, a standard statistics test for comparing two rates.

1. Conversion rate of each version: p₁ = conversions_A ÷ visitors_A, p₂ = conversions_B ÷ visitors_B.

2. Combined rate across both: p = (conversions_A + conversions_B) ÷ (visitors_A + visitors_B).

3. Standard error: SE = √[p × (1 − p) × (1 ÷ visitors_A + 1 ÷ visitors_B)].

4. Z-score: Z = (p₂ − p₁) ÷ SE.

5. The z-score converts to a p-value (the chance of seeing a difference this big by luck alone) and a confidence level. A z-score of about ±1.96 equals 95% confidence.

The results page also shows confidence intervals: the likely range of each version's true rate, and of the lift. If the ranges don't overlap, the difference is probably real.

When a test is ready to call

A test is Ready to Call only when all three are true:

1. Confidence has reached your target (95% by default).

2. At least 80% of the needed sample has been collected.

3. The test has run for the recommended minimum number of days for its goal.

Until then, you'll see Keep Running. Checking early and stopping as soon as something looks good is the most common way to pick a false winner. How to read the verdicts.