How long should an A/B test run?

Long enough to collect enough data and cover normal variation, such as full weeks. Stopping too early can produce misleading results.

Can I test more than one change at a time?

Testing one change at a time shows what caused the difference. Testing several together is multivariate testing and needs more traffic and careful design.

What counts as a winning result?

A difference that is large enough to matter and unlikely to be due to chance, judged with an appropriate statistical test and sample size.

What Is A/B Testing? Definition and How to Run One

A/B testing, also called split testing, is a method of comparing two versions of something to find out which one performs better. Version A is usually the current design, and version B is a variation with one change. Visitors or recipients are randomly divided, each group sees one version and the results are compared on a measure such as clicks, sign-ups or purchases.

What can be tested

Web pages: headlines, images, button text, colors, layouts and forms.
Emails: subject lines, sender names, send times and content.
Ads: copy, images and targeting.
Apps and products: features and flows.
Prices and offers.

How to run a test

Set a goal and metric, such as click-through rate or conversion rate.
Form a hypothesis: "A shorter form will increase sign-ups."
Create the variation, changing one thing.
Split the audience randomly and evenly.
Decide the sample size and duration in advance.
Run the test without changing it midway.
Analyze the results, using a significance test or tool.
Act on the outcome, adopting the winner or learning from a neutral result.
Document and repeat.

Why random assignment matters

Random assignment makes the groups comparable, so differences are likely to be caused by the change and not by who saw which version. Running version A in one week and B in another can be misleading because other things change.

Statistical significance

Because results vary by chance, tests use statistics to judge whether a difference is real. Too small a sample can produce false conclusions. Many tools calculate this, but understanding the principles prevents mistakes. See statistical significance.

Common mistakes

Stopping early when one version seems ahead.
Testing many things at once without enough traffic.
Ignoring seasonality or events that affect results.
Using too small a sample.
Choosing the wrong metric, such as clicks when you want sales.
Overlooking user experience and long-term effects.
Running tests that raise ethical concerns, such as deceptive designs.

Ethics and privacy

Be transparent with how you use data, follow privacy laws and avoid manipulating users in harmful ways. In research settings, some tests may require ethics review.

For students

A/B testing is used in marketing, product design, data science and psychology. You can try it with a club email list or a small website, learning how to design and interpret experiments.

Related terms

See conversion rate, click-through rate, statistical significance, landing page and call to action.