A/B Testing: Experimental Design, Analysis and QA
A/B testing is a controlled experiment that randomly assigns eligible units to a control and one alternative so a predeclared outcome can be compared under the same measurement and maturity rules.
What is A/B testing?
A/B testing is a controlled experiment that assigns eligible units to a baseline or a deliberately changed treatment, then compares a predeclared outcome under the same measurement rules. Its purpose is to reduce uncertainty about a causal decision. It is not a contest between two screenshots, a reason to stop at the first favourable dashboard reading or proof that one result will transfer to every audience.
This page explains the general method: question, unit, assignment, treatment, outcome, sample, analysis, validity and decision record. The A/B testing ad campaigns page handles auction and media-delivery conditions. The A/B testing tools page evaluates whether software can preserve allocation and evidence. Those are related but separate information jobs.
Method references checked 2026-08-10: NIST experimental-design material, FTC advertising principles, NIST and ICO privacy guidance, WCAG 2.2 and Google people-first content guidance inform the controls below.
1. Turn the business uncertainty into one decision
Begin with an action the result can change: keep the baseline, release the treatment, revise the hypothesis, collect more evidence or stop. Name the accountable owner and the date after which the answer loses practical value. A broad request to improve conversion invites dozens of unplanned comparisons and makes a convenient winner easy to manufacture.
State why an experiment is appropriate. Random assignment can estimate the effect of a controlled change when eligible units, exposure and outcomes can be observed. It cannot repair an unclear product decision, supply missing demand or prove a claim that was never measured.
2. Define the eligible population before assignment
Describe who or what can enter, when eligibility begins and which exclusions apply. The population might be new sessions reaching a page, signed-in accounts meeting a product condition or geographic areas available for a service. Keep eligibility independent of the outcome so later behaviour does not decide who counted in the test.
Record sampling and coverage limits. A result observed among consenting desktop visitors during a short period may not represent returning mobile users, assisted buyers or another market. Generalisation needs evidence; it is not supplied automatically by statistical significance.
3. Choose the correct unit of assignment
Assign at the level where contamination can be controlled. A visitor-level treatment needs a stable identifier; an account-level price or workflow change may need every user in that account to receive the same state; a store or region intervention may require cluster assignment. The unit of analysis should respect that design.
Do not count repeated events as independent people. If one person creates many sessions, a session-level comparison can overstate precision and expose the same person to conflicting experiences. Document identity resets, cross-device gaps, shared accounts and other ways the assigned unit can be lost.
4. Write a falsifiable hypothesis and estimand
A useful hypothesis names the treatment, baseline, eligible population, expected outcome and reason. The estimand states the effect to be estimated, such as the average difference in completed eligible purchases per assigned account during fourteen mature days. This prevents a later switch to whichever metric looks strongest.
Separate the predicted mechanism from the decision rule. A clearer form may reduce errors and increase completed applications, but comprehension, error rate and completion are distinct observations. If the mechanism fails while the commercial outcome rises, the team should investigate rather than rewrite the original explanation.
5. Keep the treatment singular and reproducible
Specify exactly what changes and what remains fixed. Preserve copy, component version, rules, destination, eligibility and release time for both arms. A package of changes can estimate the package effect, but it cannot identify which ingredient caused the difference.
Store the baseline and treatment as released artefacts rather than descriptions in a slide. Record dependencies, feature flags and fallback behaviour. Another reviewer should be able to reproduce both experiences without relying on the implementer's memory.
6. Randomise and verify allocation integrity
Use an assignment mechanism that does not depend on predicted outcome or operator preference. Check allocation before interpreting results: expected split, stable persistence, no systematic differences in important pre-treatment characteristics and no selective failures in one arm.
A sample-ratio mismatch is an investigation signal, not a metric to hide. Possible causes include broken bucketing, bot filtering, delayed logging, redirect loss, consent behaviour or a treatment that changes whether an exposure event fires. Resolve the cause or qualify the experiment before using its result.
7. Define one primary outcome and guardrails
Choose the measure closest to the decision that can mature within the test. Define numerator, denominator, eligibility, source, exclusions, time zone, deduplication, attribution and maturity. A label such as conversion rate is incomplete until every part of that contract is explicit.
Add a small set of guardrails for harm that the primary outcome could conceal: error, cancellation, accessibility failure, support demand, latency, complaint or accepted-value quality. Diagnostic measures explain the path; they should not silently replace a failed primary outcome.
A/B test design record
Complete the design before observing decision outcomes.
| Field | Required definition | Why it matters | Failure signal |
|---|---|---|---|
| Decision | Owner, action and deadline | Makes evidence usable | No result can change the plan |
| Population | Eligibility and exclusions | Defines who the result represents | Outcome determines inclusion |
| Assignment | Unit, mechanism and persistence | Supports a fair comparison | Crossover or split mismatch |
| Treatment | Exact baseline and change | Makes the contrast reproducible | Several undocumented changes |
| Outcome | Formula, source and maturity | Prevents metric switching | Definition changes after launch |
8. Plan sample and duration around useful precision
Set the smallest effect worth acting on, baseline variation, desired precision and decision risk before launch. A large sample can make a trivial difference look certain, while a small sample may leave an important effect unresolved. The required evidence depends on the design and business consequence, not a universal visitor count.
Run across a representative operating window and let the outcome mature. Weekday mix, pay cycles, inventory, onboarding delay and repeat behaviour can matter. Do not extend or stop solely because the latest line crosses a dashboard threshold unless that sequential rule was planned.
9. Validate instrumentation without contaminating the result
Test assignment, exposure and outcome events before opening the decision window. Confirm identifiers, time zones, deduplication, consent states, server and client records, error paths and the join between treatment and outcome. Use synthetic or excluded traffic for QA where possible.
Freeze material definitions during the run. If a tracking repair is unavoidable, timestamp it, identify affected records and decide whether to restart, segment or invalidate the comparison. Quietly backfilling one arm can create a cleaner chart and a less truthful experiment.
10. Analyse the effect with uncertainty visible
Report the baseline, treatment, absolute difference, relative difference where useful, interval estimate, sample size and missingness. Explain the analytical method and whether it matches individual or clustered assignment. Avoid presenting a probability or confidence label without the effect magnitude the decision actually concerns.
Inspect predeclared segments and guardrails, but control the number of comparisons. Exploratory patterns can create the next hypothesis; they should not be promoted to confirmed findings. Preserve inconclusive results because they bound current knowledge and prevent repeated hopeful testing.
11. Check validity before generalising
Internal validity asks whether the observed difference can reasonably be attributed to the treatment within this test. Review assignment, attrition, crossover, instrumentation, concurrent changes and interference between units. External validity asks where the finding could apply beyond the tested population and period.
Document novelty, learning and carryover. A prominent new interface may attract early attention that fades, while a workflow change may improve only after familiarity. A short exposure result should not be described as durable behaviour without a follow-up window.
12. Protect people, privacy and accessibility
Minimise collected data, define purpose and retention, restrict access and follow the applicable consent or other lawful process. Do not use an experiment to disguise a material reduction in privacy, choice or service quality. Sensitive populations and consequential decisions require specialist review beyond an optimisation checklist.
Both variants must remain usable. Test keyboard access, focus, reflow, contrast, labels, errors and assistive-technology semantics where relevant. A treatment that excludes users is a product defect even when the remaining sample converts at a higher rate.
13. Apply, ramp or revert through a written rule
Use the predeclared primary result, guardrails, uncertainty and operational readiness. A favourable average does not require immediate full release. Ramp gradually when scale could change latency, inventory, support load or audience composition, and keep the prior state available until the new one is stable.
Revert when the harm boundary is crossed or implementation integrity fails. Revise when the evidence exposes a mistaken mechanism or incomplete treatment. Continue collecting evidence only when the additional window can meaningfully change the decision, not because the current answer is disappointing.
Result interpretation matrix
The primary effect and guardrails lead to different actions.
| Primary evidence | Guardrails | Interpretation | Responsible action |
|---|---|---|---|
| Useful improvement | Within boundary | Treatment supports the decision | Controlled ramp and verify |
| Useful improvement | Material harm | Average benefit hides damage | Do not release; diagnose |
| No useful difference | Within boundary | Treatment has not earned change | Keep baseline or redesign |
| Inconclusive | Within boundary | Precision is insufficient | Continue only under planned rule |
| Any result | Integrity failure | Causal interpretation is unsafe | Repair and restart or invalidate |
14. Preserve the experiment as reusable evidence
Archive the question, population, assignment unit, variants, source files, dates, metric contract, sample plan, analysis, deviations, result and decision. Include negative and inconclusive tests. A searchable record prevents duplicate work and makes later changes comparable with the state that actually ran.
Write the conclusion at the scope supported by evidence: what changed, for whom, under which conditions and with what remaining uncertainty. Do not reduce a bounded finding to a slogan such as variation B always wins. The record should help a future owner decide whether retesting is justified.
15. Distinguish A/B tests from adjacent methods
Use usability research to observe comprehension and task barriers, surveys to collect reported attitudes, analytics to describe behaviour, multivariate designs to study several factors and phased releases to manage operational risk. These methods can inform an experiment, but they answer different questions and use different evidence.
An A/A test can check allocation and measurement noise before a costly launch. A holdout can estimate the incremental effect of an intervention against no exposure. Choose the method from the uncertainty and feasible control, not because one label sounds more scientific.
16. Connect a general experiment to paid distribution
FroggyAds is a self-serve DSP and global ad network for advertisers and media buyers. It can provide bounded distribution for push, native, display and pop campaigns across available inventory after the experiment's audience approximation, creative, destination, budget, outcome and pause conditions are approved.
Platform delivery evidence does not replace the general design. Preserve the assigned comparison, reconcile accepted outcomes in the authoritative business system and report auction or inventory changes as limitations. Use the dedicated campaign-testing guide when delivery itself is part of the treatment.
Questions about controlled A/B experiments
What is an A/B test?
It is a controlled experiment that assigns eligible units to a baseline or treatment and compares a predeclared outcome under common measurement rules.
What is the difference between A/B testing and split testing?
The terms are often used interchangeably; the important details are the assignment unit, controlled contrast, outcome definition and analysis.
How many variables should an A/B test change?
Change one interpretable treatment when you need to attribute the effect; a defined package can be tested when only the package decision matters.
How large should the sample be?
Plan it from baseline variation, the smallest useful effect, desired precision, assignment design and consequence of a wrong decision.
How long should an A/B test run?
Run through a representative operating window and until the predeclared outcome matures; calendar duration alone is not a stopping rule.
What is a primary outcome?
It is the predeclared measure that determines the main decision, with an explicit population, formula, source, exclusions and maturity window.
Can statistical significance prove business value?
No. The effect size, uncertainty, total economics, guardrails and practical consequence must also support the decision.
When should an experiment be stopped early?
Stop for a planned harm boundary, broken assignment, invalid measurement, policy failure or another predeclared safety condition.
Can an inconclusive test be useful?
Yes. It can rule out large effects, expose measurement limitations and prevent an unsupported rollout when the uncertainty is reported honestly.
What should an experiment archive contain?
Keep the question, population, assignment, variants, metric contract, dates, deviations, analysis, uncertainty, decision and released artefacts.
Primary references for experimental design and responsible implementation
Run a bounded media experiment after the design is approved
Use FroggyAds to distribute approved variants when audience, source, budget, destination, outcome and rollback controls are documented.
Create My Free Account