A/B test hypothesis generator and sample size calculator
Write the hypothesis before you build the test. Say what you saw and what you want to change, and get a hypothesis, a null hypothesis, the metrics to watch and a check for weak spots. Then work out how many visitors the test needs and keep your ideas in a ranked backlog. Everything runs in your browser.
How to write a hypothesis
A good hypothesis is written before the test and has five parts. The builder above puts them in one sentence.
- Observation. What you saw, and where you saw it: a step in the funnel with a high drop-off, a section visitors skip in recordings, a question that keeps arriving at support.
- Change. The one thing that will be different in the variant. Name the element and say what changes: its wording, its position or whether it is there at all.
- Audience. Who the test covers, if it is not everyone. This part is optional.
- Metric. The one number that decides the test, and whether you expect it to go up or down.
- Reason. Why the change should work. The reason is what you learn from, whether the variant wins or loses.
A worked example
The inputs below are made up to show how the parts fit together. They are not real data from any site.
- Page or asset
- Pricing page
- Goal
- Sign-ups
- Audience
- visitors on mobile
- Observation
- most visitors on mobile leave the pricing page without opening a plan
- Evidence
- Heatmap or session recordings
- Proposed change
- Adding a short comparison of what each plan includes
- Expected direction
- Increase
- Reason
- Clarity
- Reason detail
- visitors can see the differences between plans without scrolling
Hypothesis: Because we saw most visitors on mobile leave the pricing page without opening a plan, we believe that adding a short comparison of what each plan includes for visitors on mobile will increase sign-up conversion rate, because it makes the offer and the next step clearer (visitors can see the differences between plans without scrolling).
Null hypothesis: Adding a short comparison of what each plan includes makes no difference to sign-up conversion rate for visitors on mobile. Any gap between the original and the variant is down to chance.
Metrics: primary: sign-up conversion rate. Guardrails: Share of new sign-ups that go on to use the product; Form error rate; Bounce rate.
Sample size. The sample size numbers are made up too. This is what the calculator gives for them: 10,317 visitors per variant, 20,634 in total (baseline 4%, minimum detectable effect 20% relative (4% to 4.8%), 95% significance, 80% power, 2 variants). At 800 visitors a day: about 26 days, so run it for 4 full weeks.
What is the null hypothesis?
The null hypothesis says the change makes no difference. A test does not try to prove your idea. It asks whether the data would be hard to explain if the change did nothing.
If the gap between the original and the variant is larger than chance alone would usually produce, you reject the null hypothesis. If it is not, you have not shown a difference. That is not the same as showing there is none: a small test can miss a real effect.
Why sample size and full weeks matter
Decide how many visitors you need before the test starts. With too few, the test cannot tell a real change from chance. The number depends on your baseline rate, the smallest change you care about (the minimum detectable effect), the significance level and the power. Halving the effect you want to detect roughly quadruples the visitors you need.
A 95% significance level means that if the change does nothing, the test will wrongly report a difference about 1 time in 20. It is not the chance that the variant is better. A power of 80% means that if the real change is as large as your minimum detectable effect, the test will find it about 8 times in 10.
With more than two variants, each one is compared with the control at a stricter threshold: the significance level divided by the number of comparisons (a Bonferroni correction).
People behave differently on weekdays and at weekends. Run the test for whole weeks, so each day of the week counts equally. Do not stop early because the result looks good. Checking the result again and again, and stopping the first time it looks significant, raises the chance of a false positive.
What is a guardrail metric?
A guardrail metric is a number you watch to make sure a win on the primary metric does not cause damage elsewhere. A button that gets more clicks but leads to more refunds is not a win.
Choose two or three before the test starts, and decide in advance what movement you would not accept. Guardrails are watched, not tested for a win: the primary metric decides the test.
What to do with an inconclusive result
An inconclusive result means the data do not show a difference you can tell from chance. It does not mean the change has no effect. It means the test could not show one at the size you planned for.
Check that the test ran to the planned sample size and for full weeks, and look at the guardrail metrics. Ask whether the result points to a smaller effect than you planned for, and whether that smaller effect would still be worth having. Then decide: keep the original, try a bolder version of the same idea, or move to the next idea in the backlog.
Write down what you learned either way. Avoid cutting the result into many segments until one looks significant: every extra cut adds another chance of a false positive.
Why test one change at a time
If you change the headline, the image and the button together and the variant wins, you do not know which change did it. If it loses, one change may have helped while another hurt.
A single change keeps the result readable. It can still be a big change: a rewritten section is one change if it is one idea. If you must test several changes together, treat them as a package, say so in the hypothesis and do not claim to know which part worked.
Questions and answers
What is an A/B test hypothesis?
A statement, written before the test, of what you will change, who it is for, which metric you expect to move, in which direction and why. It turns an idea into something a test can support or contradict.
Does a 95% significance level mean the variant is 95% likely to be better?
No. It means that if the change did nothing, a test run this way would wrongly report a difference about 5 times in 100. It says how often the method gives a false alarm, not how likely it is that your variant is better.
Which baseline conversion rate should I use?
Your own recent rate for the same page and the same kind of traffic, measured over at least a few full weeks. This tool gives no benchmark rates, because rates differ widely between sites, audiences and traffic sources.
What is the minimum detectable effect?
The smallest change worth detecting. Choose it from what would justify the work, not from what you hope to see. A relative effect of 10% on a 10% baseline takes the rate from 10% to 11%. An absolute effect of 1 percentage point on the same baseline is the same change.
Can I use the sample size for average order value or revenue?
No. The calculator is for rates: the share of visitors who do something. An average such as order value also depends on how much the values vary, which needs a different calculation.
How is the ICE score worked out?
It is the average of the Impact, Confidence and Ease scores you give, each from 1 to 10. A higher score means the idea is worth doing sooner. The scores are your own judgement, so the ICE score helps you rank ideas and does not predict results.
Is anything sent to a server?
No. The tool runs in your browser. The backlog is saved in this browser only. A copied link holds your inputs in its address, so anyone you send it to can read them.
Related calculators
Conversion rate calculatorCTR calculatorBreak-even ROAS calculator