A/B Testing

What is A/B Testing?

A/B testing (also called split testing) means showing two versions of the same page, email or screen to two random groups of people at the same time and measuring which version gets more of the action you want. Version A is your current one, the control. Version B carries ONE change: a new headline, a shorter form, a different button.

Because both groups see their version in the same week, from the same traffic sources, the difference in results comes from your change. A holiday, a viral post or a slow Monday hits both groups equally. That is the whole trick, and it only works with enough people. Most early startups don’t have them yet, which is why half of this page is about knowing when to wait.

When should you use it?

Use it when a single page gets at least 1,000 visitors a week and you have a specific change you believe will move one number. That is our rule of thumb; the sample size step below gives you the real figure for your case.

• Fits: sign-up pages, pricing pages, checkout flows, email subject lines on a list of several thousand, app screens with steady daily traffic.
• Fits B2C better than B2B: a B2B tool with 40 demo requests a month will wait a year for a clean answer.
• Skip for now: below a few hundred visitors a week. Watch session recordings, talk to users and ship obvious fixes directly.

How do you do it, step by step?

1. Write the hypothesis in one sentence. Use this template: “Because [evidence], we believe [change] will raise [metric] from [X%] to [Y%].” Say: “Because 12 of 20 recordings show people hesitating at the 8-field form, we believe cutting it to 3 fields will raise sign-ups from 4% to 6%.” No evidence, no test.

2. Check your sample size BEFORE building anything. Open Evan Miller’s sample size calculator, enter your current conversion rate and the smallest change worth detecting. With its own default inputs (a 10.2% baseline, detecting a move to 13.2%) it asks for 2,545 visitors per version. Divide the total by your weekly traffic. If the answer is more than 6 weeks, test a bolder change or skip the test.

3. Change exactly one thing. If version B has a new headline AND a new image AND a new price, a win tells you nothing about which change mattered. Bold changes are fine, as long as there is only one of them.

4. Split by person, 50/50. Each visitor must see the same version on every visit, or you count the same person in both groups. Tools like PostHog handle this through a feature flag; its experiments documentation explains how each experiment is backed by a flag that assigns users and records who saw which version.

5. Fix the stopping rule in advance. Write down the sample size and end date before launch, and run for whole weeks (at least two) so weekday and weekend visitors are both in. Peeking daily and stopping the moment B looks ahead is the most common way to “prove” a change that does nothing.

6. Read the result honestly. When the planned sample is reached, look at the difference and the confidence your tool reports. A win: ship B to everyone. No clear difference: keep the simpler version and log what you learned. A loss is useful too; it tells you what your customers value.

7. Keep a test log. One row per test in a sheet: date, hypothesis, sample per version, result, decision. After ten tests this log is worth more than any blog post on conversion tips, because it describes YOUR customers.

What does a worked example look like?

Say Hana sells scheduling software to physiotherapy clinics. Her demo-request page gets 1,200 visitors a month and converts at 4%. She wants to test a new headline and expects it to lift requests to 5%.

She runs the numbers first. By the standard two-group formula at 95% confidence and 80% power, detecting a move from 4% to 5% needs roughly 6,700 visitors per version, about 13,400 in total. At 1,200 a month, that is 11 months. Even a bold change from 4% to 6% needs roughly 1,900 per version, more than 3 months of her traffic.

So Hana skips the split test for now. She does three things instead:

• She shows both headlines to 6 clinic owners in short calls and asks them to explain, in their own words, what the product does after reading each one.
• She ships the headline that 5 of 6 understood and compares the next 4 weeks against the previous 4, noting that a holiday or a newsletter mention can blur such a before-and-after view.
• She saves real A/B tests for her email list of 9,000 clinic staff, where a subject line test reaches its sample in a single send.

Six months later, with traffic at 5,000 visitors a month from her content, the same page qualifies for a proper split test.

Which free tools help?

What mistakes should you avoid?

• Testing with too little traffic. A test with 150 visitors per version will “find” winners that vanish next month. Run the sample size check first, every time.
• Stopping when it looks good. Early leads flip all the time in small samples. Decide the end point before you start and honour it.
• Testing tiny things. Button colour tests need huge samples to detect tiny effects. Early on, test big differences: a new offer, a shorter form, a different promise.
• Measuring the wrong number. A headline that raises sign-ups and lowers paid conversions is a loss. Track the step after your target too.
• Running overlapping tests on the same page. Two tests on one page at once contaminate each other. One test per page at a time.

How do you know it worked?

A test worked when it reached its planned sample, ran for whole weeks, and gave you a decision you can write in one line in your log, win, loss or no difference.

Our rule of thumb: ship a winner only if it beat the control at your tool’s 95% confidence level AND the step after it held steady. If 3 tests in a row show no difference, your changes are too small or your traffic too low; switch to qualitative methods until one of those changes.

What should you do next?

Use your conversion funnel to choose which step deserves a test, and use price testing when the question is what to charge. For ideas that don’t exist yet, a fake door test measures demand with no split-test tool at all. If you want a sparring partner to sharpen hypotheses before you test, the F/MS startup game gives you an AI co-founder that challenges each assumption.

FAQ

How much traffic do I need for an A/B test?

It depends on your current conversion rate and the size of the change you want to detect. Put both into a sample size calculator before you start. For a page converting around 3 to 5%, detecting a modest lift needs thousands of visitors per version.

How long should an A/B test run?

Until it reaches the sample size you calculated, and for at least two full weeks so every weekday is covered twice. Stopping early because one version looks ahead is the most common cause of false winners.

Can I run an A/B test with low traffic?

Not a reliable one. With a few hundred visitors a week, use customer conversations, session recordings and before-and-after comparisons instead, and treat what they show as direction without proof. Email lists and paid ads reach large samples faster, so test messages there first.

Updated 26 October 2026 · Written by the Fe/male Switch team

Mark this method as done