A/B test a landing page: one variable, and the patience to leave it alone.

A test answers one question, and it is narrower than you want it to be.

An A/B test is the same traffic, split between two versions of a page that differ in exactly one thing, judged on one number chosen before the test began. Every part of that sentence is doing work, and dropping any of them turns a test into an anecdote.

What a test buys you is a decision you would otherwise make by argument. Two people who both have opinions about a headline can spend an hour disagreeing; a test ends the disagreement with a number. That is a real and valuable thing, and it is the whole product. A test does not tell you why the winner won, whether it will still win next quarter, or whether the same change works on a different page.

Three things get called tests and are not. Changing the page and comparing this month to last month is a before-and-after: the world moved as well as the page, and you cannot separate the two afterwards. Running version A this week and version B next week is the same problem wearing a lab coat. Changing six things at once and keeping the result is a redesign — which is fine, as long as you do not tell yourself you learned something transferable.

The four things you decide in advance

The page, the single variable, the primary metric and the stopping rule. All four before the first visitor arrives. The reason is not ceremony: every one of them can be chosen after the fact in a way that flatters the result, and the temptation to do so is strongest exactly when the numbers are closest.

Most small pages cannot run a useful test, and it is better to know that now.

This is the section that testing articles skip, because it disqualifies most of the people reading them. It is also the only section that can save you six weeks of running something that was never going to produce an answer.

The problem is not that small numbers are imprecise. It is that small numbers swing on their own, with nothing changed at all. Flip a fair coin twenty times and you will rarely get exactly ten heads; run the same unchanged page for two consecutive weeks and the conversion counts will differ. That ordinary swing does not disappear because you have labelled the two versions A and B. It sits on top of whatever difference your change actually made, and a test can only see a difference that is bigger than the swing.

Splitting the traffic makes it worse before it makes it better: each version now sees half of what the page used to get, so the counts you are comparing are half the size and the swing is proportionally larger. That is the price of a clean comparison, and it is worth paying — but it has to be budgeted for.

The check to run instead of quoting a threshold

You will find articles naming a magic number of visitors or conversions per variant. Do not use them: the required number depends entirely on your current conversion rate and on how small an improvement you want to be able to detect, neither of which the author knows. Do this instead, in about five minutes.

1. 01Write down your current conversion rate for this page. Not an industry rate, not last year — this page, this quarter, from your own analytics.
2. 02Decide the smallest improvement you would actually act on. Be honest. If a two per cent relative gain would not change a single decision you make, do not ask a test to detect one.
3. 03Put both into the sample-size or significance calculator your testing tool provides. Almost every testing tool ships one, and free ones exist. It will return a number of visitors per variant.
4. 04Convert that number into weeks of your real traffic. Divide by the visitors this page genuinely receives in a week, then double it, because the traffic is split between two versions.
5. 05Read the answer as a duration, and decide. Weeks of your traffic is the number that matters — not someone else’s rule of thumb. If the answer is three weeks, run it. If it is eight months, you have your answer about testing this page.

When the duration comes back longer than the offer will exist, you have three honest options and no dishonest ones. Make the change on judgment and move on. Go and get more traffic first, which is usually the higher-value work anyway. Or test something larger — a genuinely different page rather than a genuinely different sentence — because only a big difference is visible at small volume.

Test the things that could plausibly produce a large difference.

On modest traffic only large differences are visible, so the testing queue should contain only changes capable of producing one. That single constraint sorts the list for you, and it puts button colour where it belongs.

Why colour is almost never the answer

Button-colour tests are famous for two reasons: they are trivial to run, and they make a good story. Neither reason is about your page. A colour rarely changes whether a stranger believes the page is for them, which is the decision the page exists to influence. If your button is genuinely hard to see, that is not a test — it is a repair, and you should just do it.

The same reasoning promotes the headline to the top. It is the only element every visitor reads, and it is the one that decides whether the rest of the page gets read at all. Where the elements sit and what each is doing is covered in the landing page anatomy guide; how to write two variants that are genuinely different rather than reworded is covered in landing page copywriting.

One variable, or one whole page?

The one-variable rule exists so that a result means something: if three things changed and the variant won, you have to keep all three forever, because you do not know which one carried it. That is the argument for discipline.

The counter-argument is arithmetic. On a small page, a single sentence rarely moves the number far enough to be detected, while a genuinely different page sometimes does. Both positions are defensible; what is not defensible is running a six-change test and then reporting it as though you learned which change mattered. Name the trade in the log: this test buys a decision, not a lesson.

Seven rules, and the discipline is all in the last two.

Setting a test up is easy and takes an hour. Running it honestly takes several weeks of not touching it, which is the part that actually fails.

1. 01Write the question before you write the variant. One sentence: does leading with the outcome instead of the format increase registrations? A test without a written question turns into a hunt for any number that moved.
2. 02Pick the primary metric first, and pick the furthest one you can measure honestly. Registrations beat clicks. Attendees beat registrations. Customers beat attendees. Choose the furthest number you can actually attribute to the variant, then hold to it — a secondary number that looks better later is not the result.
3. 03Set the stopping rule in advance. A date, a visitor count, or both, with the later of the two winning. Written down where you will see it. A stopping rule invented after you have looked at the numbers is not a stopping rule.
4. 04Split the traffic; do not alternate the weeks. Running A this week and B next week compares two weeks, not two pages. Any difference includes the payday, the holiday, the partner send and the weather. If your tool cannot split, say so in the log rather than pretending it did.
5. 05Point the same traffic mix at both versions. If one variant gets the newsletter and the other gets cold ads, you tested the audience. Where a source is very different from the rest, run the test inside that source alone.
6. 06Freeze everything else for the duration. No price change, no new promotion angle, no tidying the page while it runs. Every simultaneous change is a second variable you did not agree to.
7. 07Leave it alone until the rule is met. Looking is fine. Acting on a look is the single most common way a small-traffic test produces a confident wrong answer, because early numbers swing hardest.

Two capabilities to confirm before you plan anything

Can your tool split traffic between two versions of the page, and can it report the outcome by variant? They are separate features and a stack can easily have the first without the second, which leaves you with a running test and no way to read it. Confirm both in your own account rather than from a features page, and confirm them before you write a variant.

If you cannot split traffic at all, you can still learn something — but call it what it is. Run the change, hold everything else steady, compare like periods, and write the caveat into the log: anything that changed in the world during those weeks is inside the result and cannot be extracted later. Decisions made this way are ordinary business decisions. They are just not evidence.

Three ways this ends, and only one of them is the one people plan for.

Everyone sets up a test imagining a clear winner. The most common outcome by some distance is that the test cannot tell the two versions apart, and knowing how to handle that outcome well is most of the skill.

1. 01B wins, and it is well clear of the noise. Ship it, record the numbers you actually saw, and expect the real-world gain to be smaller than the test showed. Part of why the winner won is luck in that particular sample, and the luck does not ship with the page. Treat the measured lift as the optimistic end of a range.
2. 02No detectable difference — the most common result. This does not mean the two versions are the same. It means this test, at this traffic, could not tell them apart. Two honest responses: keep whichever version you prefer for reasons that have nothing to do with the test — simpler, truer, easier to maintain — and move on; or conclude the change was too small to be visible and test something bigger next time. What you must not do is call a two-conversion gap a win.
3. 03B loses. Also a result, and a cheap one. Keep A, and write down what it implies about the reader rather than about the page — that the audience wanted the format named, that the shorter form read as evasive. That sentence is the thing you will reuse; the variant itself you will never look at again.

The number you judge it on

A variant that lifts form submissions and lowers the quality of who submits has not won anything. Remove two form fields and more people will fill in the form; whether more of them turn up, buy or stay is a different question, and it is the one you are actually asking. Judge on the furthest number you can honestly attribute — for a registration page that is attendance, for a sales page it is purchases, for a lead magnet it is whatever the magnet is supposed to lead to.

This is where most small-team testing quietly gives up, because the further down the funnel the number sits, the more systems it lives across. If registrations are in the page tool, attendance is in the webinar tool and purchases are in the payment processor, then judging a variant on customers means reconciling three exports — and nobody does that twice.

One more honest caveat before you generalise: a result belongs to the page, the audience and the moment it was measured in. A headline that beat its control against cold ad traffic may lose against your own list, who already know the format and do not need it named. Carry the reasoning to the next page. Do not carry the sentence.

Before you have traffic, testing is a way of not deciding.

A new page with no audience does not have a measurement problem. It has an unknown-audience problem, and no amount of splitting traffic you do not have will solve it.

Ship the version you can defend out loud — the one where you can say who it is for, what they get and why the trade is fair — and then go and find out whether you were right by talking to people. Five conversations in a week will teach you more than five weeks of a test that cannot reach a conclusion, and they produce something a test never does: the objection in the reader’s own words.

What to ask instead of what to test

• →What were you trying to do when you landed on this page? You are looking for the job they arrived with, which is often not the one the page is written for.
• →What nearly stopped you? The hesitation, named. This is the sentence that becomes the section you are missing.
• →What did you expect this to cost you — in money, time or email? Mismatched expectations show up here long before they show up in a conversion rate.
• →What would you have called this, if you were describing it to a colleague? Their words, not your product name. Half of all headline rewrites come out of this question.

Do this with five people who resemble the audience and you will have more usable material than a quarter of testing would produce, plus a set of hypotheses worth testing later, when you have the traffic to test them with.

One page per test, or the third test will repeat the first.

The log is not admin. It is the only thing that turns a series of isolated tests into knowledge about your audience — and the only defence against re-running a test you already lost eight months ago.

Fill in the top half before the test starts and the bottom half when the stopping rule is met. The field that earns its place is the second-to-last one: what you learned about the reader, stated in one sentence with no reference to the specific page. That sentence is the part that transfers.

Test the page after you have checked it is the page.

A conversion problem is often not a copy problem, and a test is the slowest available way to discover that. Run the cheap checks in order first; each one takes an afternoon and can retire the test entirely.

1. 01Is the traffic the right traffic? A page converting badly on one source and fine on another has a source problem wearing a page problem’s clothes. Segment before you test.
2. 02Does the page load, on a real phone, on a real connection? Some of what reads as a copy problem is a load problem, and it hits mobile hardest — which is where most of the traffic is.
3. 03Does the promise match what the visitor was promised before they clicked? The ad, the email and the page have to use the same words. Broken handoffs look exactly like weak copy in a report.
4. 04Is the failure actually further down? People register and never attend, or buy and refund. That is not the page, and no variant will fix it.
5. 05Only then: is it worth a test? By this point you usually know what to change, and half the time you will change it on judgment and be right.

Working the list in order is the same discipline as funnel diagnostics: find the stage that is failing before you optimise a stage that is not. Once you know it is the page, this guide tells you what to change and whether you can measure it.