A/B testing done properly: hypothesis, enough traffic and patience

The idea is simple: you show half of your visitors the current page and the other half a changed version, and see which brings more orders or enquiries. In practice, A/B testing is mostly about discipline: a test run without method readily produces a winner that does not exist, and the company changes its site on the strength of noise.

For a high-traffic site, tests are the most honest way to decide between two versions. We handle the whole chain, from choosing what is worth testing to reading the result, and we say from the outset when a test is pointless.

The hypothesis: what you change, for whom and why you think it helps

A good test does not begin with “let’s try a green button”. It begins with an observation from the data and an explanation. For example: “on the product page, visitors on phones scroll a long way to reach the delivery cost; if we show it next to the price, more of them will add to the basket”. The hypothesis states what we change, for whom and which metric should move.

We choose a single primary metric (orders, enquiries, items added to the basket) and a few guardrail metrics that must not get worse: average order value, returns, page speed. A version that brings more orders, but smaller ones, can be a loss.

• The observation: what the data or the session recordings show.
• The change: an exact description of the variant.
• The audience: all visitors or just one segment (mobile, new customers).
• The decision: what we do if the variant wins, loses or shows no difference.

One change per test, and how it reaches the visitor

If the new version has a different headline, a different photo and a different form, and the result is better, you have no way of knowing what made the difference. We test one change at a time or, when comparing two entirely different approaches to a page, we accept that we will learn only which works better, not why.

Technically, the variant is served either from the browser, by a script that modifies the page after it loads, or from the server, where the visitor receives their version directly. The first route is quick for copy and visual elements; the second suits prices and checkout steps. Testing tools use cookies, so we tie them to the consent given in the banner, as determined by the data protection officer.

How much traffic you need and how long a test runs

Before starting we calculate the sample size: how many visitors need to see each version for the difference we are looking for to be distinguishable from chance. The calculation starts from the current conversion rate and from the smallest difference that would be worth the effort. The smaller the difference you are looking for, the more visitors you need.

The duration follows from this calculation and from the page’s traffic: the test runs until it has enough data, in whole weeks, because people buy differently on a Monday than on a Saturday. We avoid unusual periods, such as big sales or holidays.

Why you do not stop the test at the first good result

In the first few days, the difference between the versions jumps up and down. If you look every day and stop the test when the tool displays a winner, you easily catch a lucky moment: every check made with a view to stopping is another chance for randomness to look like an effect. The duration and the sample are set in advance and kept to.

A test with no clear difference is not a failure: it shows that the change does not matter as much as you thought. And before believing a good result, we check a few more things.

• The traffic split: if you asked for half and half and the groups come out visibly unequal, there is a technical problem and the result is not used.
• The novelty effect: existing customers click on what is new simply because it is new, and the difference can fade.
• Flicker: the old page glimpsed for an instant before the new one spoils the experience of the variant.
• Segments hunted for after the test has ended: “it only won on tablets” is an idea for a future test, not a conclusion.

When you do not have the traffic for A/B testing

Many sites do not collect enough conversions for a test to finish in a reasonable time. A test stretched over half a year no longer measures anything cleanly: in the meantime the season, the prices and the campaigns all change.

The options below do not give the certainty of a test, but they are more honest than a result drawn from too few visitors.

• User testing: a few people from your audience are given a task on the site, and you watch how they get on.
• Fixing obvious problems directly, without a test.
• Big changes instead of tweaks: a large difference shows up even with less traffic.
• Tests in ads and in email, where the platforms split the audience between versions by themselves.

This is a typical project description: it shows how we usually approach this kind of work and does not present a project carried out for a particular client. Every real project starts from your company’s situation, and the stages, timescales and price are agreed after the initial discussion.