The idea in one paragraph
An A/B test shows two versions of something, say a headline or a button label, to different visitors at the same time, then compares what each group did. Version A is usually the current page; version B has a single change. Because both run in parallel, things like the day of the week or a seasonal rush affect both groups roughly equally, which is the main advantage over making a change and comparing this week with last week.
Split testing is a way to reduce guessing, not to eliminate it. It can say that one version did better in your test. It cannot say why, and it cannot promise the result will hold next month.
What you need before you start
Three things make a test worth running.
A clear question. “Does a shorter form get more people to finish it?” is testable. “Is the new design better?” is not, because “better” is undefined.
One measurable outcome. Pick the action you care about, such as signing up, clicking a link or completing a form, and decide it in advance. If you track many outcomes after the fact, one of them will probably look impressive by chance.
Enough traffic to learn something. This is where small sites struggle most, and it deserves its own section.
Why small samples mislead
Chance alone produces differences. If you flip a coin ten times, getting seven heads is not surprising, and it says nothing about the coin. Visitors behave the same way: whether a given person acts depends on mood, device, timing and a hundred other things unrelated to your headline.
Here is a hypothetical example, with made-up numbers. Suppose 50 visitors see version A and 50 see version B. Four people sign up from A and six from B. That is 8 percent against 12 percent, which looks like a 50 percent improvement (6 is 1.5 times 4). Yet the actual gap is two people. If two different visitors had behaved differently, the result could have reversed. With groups this small, such a gap is well within the range that chance can produce.
Larger samples smooth out the noise. Roughly speaking, the smaller the real difference between versions, the more visitors you need to detect it reliably. Sites with a few hundred visits a month often cannot detect modest differences in a reasonable time. That does not make testing pointless, but it changes what is realistic:
- Test bigger, bolder changes, since large differences show up sooner than subtle ones.
- Test on your busiest pages, where data accumulates faster.
- Choose an outcome that happens often, like clicks, rather than a rare one, like purchases.
- Accept that some tests will end inconclusive. “No clear difference” is a legitimate result, and it tells you the change probably does not matter much.
Many online calculators estimate the sample size a test needs. Use them as rough guides, and be wary of any tool that declares a winner within hours.
Keeping the test honest
Honesty here mostly means not fooling yourself. These habits help.
Change one thing. If you alter the headline, image and button together, you will not know which change mattered.
Decide the plan in advance. Write down the question, the outcome and how long the test will run before you look at any numbers.
Do not stop early because you like the result. Checking daily and stopping when B pulls ahead is a well-known way to get false winners, because early gaps often shrink or vanish as more visitors arrive. Run for the planned period, ideally covering whole weeks so every day is represented equally.
Split visitors randomly. Do not send A to one traffic source and B to another. A page seen by newsletter subscribers will probably behave differently from one seen by strangers from a search engine, and that difference would hide inside your test.
Keep each visitor in one group. Someone who sees A on Monday should not see B on Tuesday, or their behavior is muddied.
Test one idea at a time. Running several overlapping tests on the same page makes results hard to interpret.
Be fair to your visitors. Test things that are truthful. Do not test misleading claims, fake scarcity or hidden costs, because a “win” built on confusion tends to cost you trust later. Where your testing tool uses cookies, your cookie consent setup and privacy policy should reflect that.
Reading the result
When the test ends, look at the numbers for each group and the size of the gap, not only which is ahead. Ask how much a difference of that size is worth and whether it is large enough relative to the sample to trust. If the result is close, treat it as a tie and choose on other grounds, such as which version is simpler or fits your brand.
Be careful about generalizing. A result for one page and one audience may not carry over to other pages. A “winner” may also fade as visitors become familiar with it. Re-testing an important finding is reasonable.
Ideas worth testing on a small site
Good first experiments are simple and tied to a real hesitation visitors may have:
- Two versions of a headline that state the benefit differently.
- A button label that is more specific, for example “Get the checklist” versus “Submit”.
- A shorter versus longer form.
- The position of a key piece of reassurance, such as a line about what happens after someone signs up.
Our notes on landing pages cover what such a page needs in the first place, since testing a weak page just helps you choose between two weak versions.
When not to test
If your traffic is very low, you may learn more by watching a handful of real people use the page, reading the feedback they send, and checking your website analytics for where visitors drop out. Fix obvious problems directly, like a broken form or text that is unreadable on a phone, without a test. Testing is useful for close calls, not for repairs. And if the choice is genuinely hard to measure, sometimes the honest answer is to pick one version, watch what happens, and adjust.