
An A/B test compares alternatives under a defined experimental design. Before changing a page, decide what you want to learn and what evidence would justify a decision.
Describe the problem and why the proposed change might help. Choose a primary outcome that reflects the task, such as a completed qualified inquiry, and define any guardrails that would make the change unacceptable.
Record the unit being assigned, the eligible audience and how you will avoid exposing the same participant inconsistently. Confirm that the experiment tool can implement the intended design.
Check event recording and assignment with test traffic. Use the tool’s documented statistical method to plan the sample and decision rules. Required sample size depends on the outcome, baseline and effect you want to detect; a fixed number of days does not guarantee a reliable result.
Do not repeatedly stop and restart a test because the current result looks favorable. Follow the chosen method’s rules, including any supported sequential analysis.
Review the primary outcome and guardrails together. Note tracking failures, unusual traffic and changes in the business during the experiment. An inconclusive result is not proof that the variants are equivalent.
Keep the hypothesis, configuration, observations and decision in a record another reviewer can inspect. For low-traffic sites, consider whether usability research or a simpler observation would answer the immediate question more effectively. This guide supplies a planning framework, not a claim of statistical significance or guaranteed conversion improvement.
Microsoft Research describes pre-experiment checks and their role in trustworthy experimentation.
Start with a website URL, then hand off to RoboWrite when you are ready to write.
Free scan • Results in about a minute • Continue into RoboWrite