Digital marketing, privacy, experimentation and measurement

A/B Testing: Experimental Design, Analysis and QA

By FroggyAds, the self-serve advertising platform. Updated: 2026-10-24.

A/B testing is a controlled experiment that randomly assigns eligible visitors to a control and one alternative, so a predeclared outcome can be compared under the same measurement rules. On paid traffic it works when you hold the media purchase steady and change one thing that you can randomize.

A/B testingSample sizePaid traffic
A/B Testing framework for planning, production, measurement and controlled improvement

How do you run a valid A/B test with paid traffic?

Quick answer: Run one campaign with fixed format, targeting, bid and source settings. Send every click to a single entry URL, and let your tracker or page-testing tool randomly split visitors between the control and one alternative. Fix the primary outcome, sample size and stopping rule before the first click. We supply the traffic and the buying controls; your experiment system supplies randomization, variant delivery and analysis.

For experimental-design principles and statistical methods, consult the NIST Engineering Statistics Handbook. Choose the method around your question and data rather than treating one result threshold as universal.

Five decisions to settle before you buy test traffic

  • The decision rule. Write down what you will do if the alternative wins, loses or stays inconclusive. A test without an action attached is only a report.
  • The assignment unit. Choose visitor, device or account, and use one stable identifier so a returning visitor never switches variants.
  • The place of randomization. Randomize after the click, on your side. The ad auction decides who sees an ad, so it cannot give you a fair split between two campaigns.
  • The sample size. Calculate visitors and budget before funding the test. The calculation below shows why a small deposit rarely detects a small lift.
  • The rollout path. Plan a stepwise release and a reversible fallback so the winner can be scaled without risking the account.

What does random assignment let you compare?

Random assignment supports a fair comparison between two experiences. It does not remove measurement faults or uncertainty.

Assign each eligible visitor at random to the control or the alternative, without using the outcome you hope to improve. This reduces systematic selection differences at the moment of assignment. It does not guarantee identical group counts, remove chance variation or repair a broken event pipeline. Those limits belong in the interpretation of every result.

Define the unit before choosing the split. Repeated sessions from one person are not independent participants in a visitor-level test. An account-level change may need assignment by account so colleagues see the same experience. Decide which identity your implementation can actually maintain, including what happens when people return or switch devices.

A before-and-after comparison asks a different question. It may reflect a changed page, but also a different audience, season, bid level or offer. Two separately delivered campaigns can differ for similar reasons. Describe those comparisons honestly, and establish and verify the assignment process before claiming a controlled treatment effect.

Assignment unit
The visitor, account or other entity allocated to a treatment; its identity determines which observations belong together.
Control
The reference experience against which the planned alternative is evaluated.
Percentage-point difference
The subtraction of one percentage rate from another; distinct from relative percentage change.
Statistical power
The probability that a planned test detects an effect of a stated size if that effect truly exists.
Sample ratio mismatch
Observed group counts that differ from the planned split by more than chance would explain, which usually points to an assignment or logging fault.
Stopping rule
The planned conditions under which outcome analysis ends or an intervention occurs.

Which kind of test fits your question on FroggyAds traffic?

Not every comparison you can run in an ad platform is an A/B test. The table separates comparisons where you can randomize from comparisons where the ad auction decides who is reached. Choose the row that matches the claim you want to make.

QuestionWhere to randomizeWhat you hold constantClaim you can support
Does landing page B convert better than page A?After the click, in your tracker or page-testing tool, behind one entry URL.One FroggyAds campaign: format, targeting, bid, budget and sources.A causal page effect among the eligible arrivals in the test.
Does creative B attract better visitors than creative A?Not randomized when you run separate campaigns; delivery decides who sees each ad.Targeting, bid, budget, format and schedule across both campaigns.Buying evidence with stated limits, not a clean causal result.
Which sources return the best outcome per dollar?Not randomized; sources have different audiences.Creative and landing page, with source identifiers recorded.A ranking of sources for follow-up and filtering decisions.
Which format fits my offer: Push, Native, Display or Pop?Not randomized; each format brings different intent and placement.Creative message, page, geo and a comparable budget.Channel economics for your offer, not an assignment-based effect.
What does a higher bid buy me?Sequential or parallel campaigns; neither is randomized.Everything except the bid, with timing noted.A cost-versus-volume tradeoff to guide your next budget.

Our recommendation is to start with the first row. The landing page is the one place where you control assignment completely, and an improvement there lifts every campaign you send to it. Use the other rows to decide where to buy, then confirm the winners with a page-level test.

Ready for paid acquisition

Bring suitable visitors to a ready experiment

Bring a defined paid-acquisition audience to your experiment with FroggyAds and control the purchase through campaign settings. We let you select available targeting, bids, budgets and sources. Connect the campaign to a validated experiment destination, and keep the traffic configuration stable enough to interpret what the page treatment changes.

Create My Free Account
Illustration of audience targeting controls for an A/B testing campaign

How many visitors does an A/B test need? A transparent calculation

Plan traffic from three inputs: your baseline outcome rate, the smallest improvement worth acting on, and the error rates you accept. For a comparison of two proportions with an equal split, a common planning approximation is:

n per variant ≈ (1.96 + 0.84)² × [p1(1 − p1) + p2(1 − p2)] ÷ (p2 − p1)²

Here p1 is the control rate and p2 is the rate you hope to detect. The value 1.96 corresponds to a two-sided 5% significance level and 0.84 to 80% power. This is a fixed-horizon, normal-approximation estimate. Use a statistician or your experiment tool for complex designs, multiple variants or sequential monitoring.

Hypothetical scenarioBaseline to targetAbsolute / relative changeApprox. visitors per variantApprox. total visitors
Large change2.0% to 3.0%1.0 point / 50%3,8007,600
Moderate change2.0% to 2.4%0.4 points / 20%21,10042,200
Small change2.0% to 2.2%0.2 points / 10%78,700157,400

These figures are illustrative arithmetic, not FroggyAds results or forecasts. They show a pattern worth remembering: halving the lift you want to detect roughly quadruples the traffic you need.

Turn visitors into a budget

  1. Take the total visitors from the table or your own calculation.
  2. Multiply by your effective cost per landing-page visit. This depends on your bid, your sources and the share of clicks that actually load your page, so count page views rather than clicks.
  3. Add time for implementation, quality checks and analysis.

Hypothetical example: at an all-in cost of $0.01 per landing-page visit, the 42,200-visit test costs about $422. Our starting prices, from $0.003 per click for Push and Native and $0.10 per 1,000 impressions for Display, show where bidding can begin. They are not a forecast of what your traffic will cost, so check current figures and your own delivery data in the platform.

The $50 minimum deposit is not a test budget. It is the minimum you deposit to start a campaign. At the same hypothetical $0.01 per visit, $50 buys about 5,000 visits, or 2,500 per variant. With a 2% baseline, that can only detect a lift of roughly one percentage point, which is very large for a page change. Fund the sample you calculated, or test a bolder change.

If the required traffic exceeds your budget, you have three honest options. Test a larger change, narrow the decision, or use a more frequent outcome as the primary measure only if it genuinely reflects the business decision. Buying visitors without a feasible analysis plan turns a small uncertainty into a large bill.

From brief to decision: an eight-step A/B test workflow

1. Write the decision rule

State the action that follows each outcome. For example: if B lifts confirmed orders by at least the effect you planned for, with refunds and invalid leads inside tolerance, B replaces A. Otherwise A stays. Agreeing this before data arrives stops the team from arguing about what counts as a win.

2. Fix the media purchase

Create a single FroggyAds campaign and settle format, geo, device targeting, bid, budget and sources before launch. Keep it unchanged for the run. If you must change a setting, log the date and time so you can tell a buying change from a treatment effect later.

3. Randomize after the click

Point the campaign at one entry URL and let your tracker or page-testing tool split visitors between A and B. Use a persistent identifier so a returning visitor keeps the same variant. Test the split from several devices and browsers before you spend real budget.

4. Define the primary outcome

Write the outcome in plain language with its denominator and timing rule, such as confirmed purchases per assigned visitor within seven days. Name supporting guardrails separately, such as refund rate or lead validity. One primary outcome keeps the analysis honest.

5. Rehearse with a dry run

Send a small batch of internal or low-budget visits and trace each one from assignment through exposure to the recorded outcome. Check duplicate, missing and delayed events. Exclude rehearsal records from the analysis and keep them out of the customer-response counts.

6. Monitor safety, not winners

While the test runs, check delivery, page errors, postback firing and allocation balance. These are operational checks. They are separate from deciding which variant leads, and a harmful or broken variant can be stopped with the reason recorded.

7. Analyse at the planned point

Begin with data integrity, then read the primary outcome with its uncertainty under the method you chose in advance. Check guardrails and traffic quality next. Show assigned, exposed and converted counts together so missing records cannot hide behind a tidy percentage.

8. Decide and scale in steps

If the alternative earns release, move traffic in stages while you keep the control available. Raise FroggyAds budgets gradually and watch that cost per outcome holds as exposure grows. File the result, including rejected or inconclusive tests, so nobody pays to answer the same question twice.

Keep traffic comparable

Keep the media purchase aligned with the experiment

Use the FroggyAds format that fits the arrival experience: Push, Native, Display or Pop where appropriate for your offer. Keep format changes out of a page-only comparison. If you want to compare formats themselves, make that a separate buying decision with its own delivery assumptions, sample plan and outcome measurement.

Create My Free Account
Illustration comparing advertising formats for A/B testing execution

What should an A/B test report contain?

Report the primary outcome as the difference between control and alternative on the basis fixed in your analysis plan. Keep exposure health and customer guardrails alongside it. This prevents a precise-looking percentage from hiding missing participants, delayed outcomes or a commercially poor tradeoff.

MeasureDefinition to useWhen to inspect it
Accepted outcome rateQualified outcomes divided by eligible assigned units under the analysis plan. Show both group counts and rates, and identify any different denominator explicitly.At the planned analysis point after required outcome follow-up.
Allocation integrityObserved assigned or exposed group counts compared with the configured ratio. State which counts the check uses and investigate unexplained mismatches.During launch checks and monitoring, before interpreting treatment effects.
Outcome maturityThe extent to which each cohort has completed the agreed observation window and qualification process.Before each authorized result review; show pending outcomes separately.
Absolute and relative effectPercentage-point change alongside relative change. A move from 2% to 2.2% is 0.2 percentage points and 10% relative.With uncertainty at the prespecified decision point, not as a stand-alone headline.
Customer and business guardrailsMeasures that could make a nominal win unacceptable, such as invalid leads, cancellations, refunds or workload. Define tolerated harm before launch.Continuously for serious harm; with the final commercial assessment.
Traffic quality and version fidelityWhether bot filtering, source mix and the delivered treatment stayed the same for both variants throughout the comparison.After deployment and every release or campaign change; annotate deviations immediately.

Worked example: checking for a sample ratio mismatch

Hypothetical example: you planned a 50/50 split and expected 21,100 visitors in each group. The tracker shows 21,400 in A and 20,800 in B, a total of 42,200. The chi-square statistic is 2 × 300² ÷ 21,100, about 8.5. That exceeds the 3.84 value for a 5% threshold with one degree of freedom, and the p-value is about 0.003. A 1.4% gap looks small, but it is unlikely to be chance. Investigate redirects, caching, consent behaviour and bot filtering before you read any conversion rate. Do not delete records until the ratio looks right.

Reconcile the experiment report with the business system that accepts a lead or order. Avoid joining only successfully converted records, which loses the rest of the assigned population. Keep missing data explicit. A technically valid calculation cannot answer your original question if its input population silently changed during processing.

Four setups with different decision boundaries

A landing-page test on a Native campaign

Hypothetical walk-through: you sell a subscription and suspect that explaining the trial terms above the form will cut unsuitable sign-ups. You run one Native campaign with fixed targeting and budget, and your tracker splits arrivals between the current page and the revised page. You record the variant with each sign-up and compare accepted trial starts, plus cancellations in week one.

This supports a claim about the page among those arrivals. It does not prove the same effect on Push or Pop traffic. Test those sources separately if the decision matters there.

Compare advertising creatives under real delivery

A creative comparison is useful, but equal budgets do not prove random assignment. Auction conditions, optimization and available audiences can make two campaigns reach different people. Match targeting, bid, format and schedule, then inspect source and device mix before trusting a gap. Treat the result as buying evidence. If you need a causal message comparison, use a system that randomizes exposure.

Compare sources and filter with evidence

Record the source identifier with every visit and compare outcomes per spend across sources that have enough volume to judge. Use whitelists for sources that clear your threshold and blacklists for those that clearly fail it. A source ranking is observational, so confirm a promising source with a longer run and a page-level test before you scale it.

Use heatmaps to propose a testable change

Heatmap patterns can suggest that visitors miss an explanation or mistake a visual for a button. Inspect the page directly and propose a specific change, then evaluate it with an experiment if uncertainty remains. Microsoft Clarity describes heatmaps as aggregated interaction views. A pattern alone does not establish a treatment effect.

Why does an A/B test result look wrong, and what should you check?

Most surprising results come from experimental plumbing, not customer psychology. Work down this table before you build a story around the number.

SymptomLikely causeWhat to check
Group counts differ from the planned splitRedirect drops, caching, consent blocking or uneven bot filtering.Compare assigned and exposed counts by device and browser, then run the ratio check shown above.
Variant B leads on sign-ups but not on accepted ordersTracking is broken on one variant, or the change attracts lower-quality submissions.Postback or event firing rate per variant, plus invalid-lead and refund guardrails.
A large early lead shrinks over timeChance variation, repeated peeking or a novelty effect.Wait for the planned sample and maturity window; do not stop on a favourable chart.
Dashboard clicks exceed landing-page visitsSlow loading, redirect chains, early exits or non-human clicks.Click-to-visit ratio, page speed, and traffic-quality signals including Adscore where connected.
One source behaves very differentlyA different audience mix rather than a different treatment effect.Treat the segment as a follow-up hypothesis and confirm it with its own planned test.
Conversions keep arriving after the reportAn outcome window that has not matured.Delay analysis until follow-up is complete and show pending outcomes separately.

An experiment that rules out a risky change can save future spend. An inconclusive one can reveal that assignment or outcome collection needs work. State that learning directly rather than giving every run a winning label.

Fund a validated route

Buy traffic after the test route is validated

We recommend confirming assignment and measurement before increasing paid exposure. With FroggyAds, you can set a campaign budget that matches your calculated sample and review source-level delivery while your experiment collects outcomes. Keep the campaign history with the test record so the team can distinguish a treatment change from a buying change.

Create My Free Account
Illustration of a campaign launch checklist for A/B testing

Why run your A/B test traffic through FroggyAds?

We built FroggyAds for advertisers, affiliate marketers and media buyers who want to control the purchase themselves. A clean experiment needs stable inputs, and our self-serve controls let you hold the media side steady while your tracker handles the split.

  • Formats for your offer: Push, Native, Display and Pop, so you can match the arrival experience to the page you are testing.
  • Control over the buy: targeting, bids, budgets and sources that you set and can leave unchanged for the full run.
  • Quality controls: Adscore and internal quality checks to help filter bot traffic, plus whitelists and blacklists for sources.
  • Reach for sample size: access to a global network with 750+ SSP integrations, which helps when your test needs volume.
  • Measurement hooks: source identifiers and tracking macros to connect delivery to later activity, with supported postbacks to report conversions to your tracker.
  • A low entry point: a $50 minimum deposit to start, which you then size up to the sample your test needs.

We do not present FroggyAds as your randomization engine. Your website or experimentation service distributes eligible arrivals between experiences and retains assignment. A campaign identifier alone does not tell you which page version a returning visitor received, so keep the variant in your own measurement route.

Choose us when you want practical control over the media inputs to a test. You can refine your acquisition setup as evidence arrives, then run the next comparison from a documented starting point. We cannot promise that buying traffic produces a winner. What we offer is the ability to fund, inspect and adjust the purchase.

Start your next campaign with FroggyAds

What should your experiment tool be able to show?

  • How is the participant identified, and what happens to assignment when someone returns, signs in or changes devices?
  • Does the tool record allocation separately from the measured outcome, so missing conversions do not hide missing participants?
  • Which analysis and stopping method does the report use, and how does it handle multiple variants or metrics?
  • Can the team inspect the exact treatment, event definitions and changes that were active during each part of the run?
  • How quickly can you stop a harmful variant and restore the reviewed control without losing the experiment record?

Ask these questions against your actual setup. A landing-page editor, a product feature system and an advertising-platform experiment operate at different units and expose different controls. Google Ads, for example, documents its own campaign experiment settings. Those are Google product capabilities, not evidence of identical experiment features in FroggyAds.

Frequently asked questions

What is A/B testing in a paid traffic campaign?

It is a controlled comparison in which eligible visitors from the same campaign are randomly sent to a control or one alternative, such as two landing pages, and a primary outcome like sales, leads or sign-ups is compared under identical tracking. With FroggyAds you fix the format, targeting, bid, budget and source settings, while your tracker or page-testing tool does the random split. Comparing two separate ad campaigns is a useful buying comparison, but it is not a randomized experiment.

How many visitors does an A/B test need?

It depends on your baseline rate and the smallest lift worth detecting. With a two-sided 5% significance level, 80% power and an equal split, a 2.0% baseline needs about 3,800 visitors per variant to detect 3.0%, about 21,100 per variant to detect 2.4%, and about 78,700 per variant to detect 2.2%. These are illustrative calculations, not results, so run your own numbers before you fund the test.

Is the $50 minimum deposit enough for an A/B test?

The $50 minimum deposit lets you fund your account and start a campaign, but it is not a recommended test budget. Your budget is the visitors you need multiplied by your effective cost per landing-page visit. Hypothetically, at $0.01 per visit, $50 buys about 5,000 visits, which only detects large effects of roughly one percentage point on a 2% baseline. Our starting prices, from $0.003 per click for Push and Native and $0.10 per 1,000 impressions for Display, are starting points; your actual cost depends on your bids, targeting and sources.

Can I split test ad creatives or landing pages in FroggyAds?

Test landing pages by sending one campaign to a single entry URL and letting your tracker or page-testing tool randomize visitors. You can compare creatives by running separate campaigns with matching targeting, bids, budgets and formats, but the auction decides who sees each ad, so treat the result as buying evidence rather than a randomized causal result. We do not describe FroggyAds as a built-in randomization engine, so pair it with the tracker you already use.

Which ad format should I use for my first test?

Pick the single format that best fits your offer, Push, Native, Display or Pop, and hold it constant while you test the page. Changing format alters intent, device mix and audience at the same time, so a format comparison is a separate buying experiment with its own budget and sample-size plan. Choose the format whose traffic suits your offer and lets you reach the required sample affordably.

How do I keep low-quality traffic from distorting an A/B test?

Set quality rules before launch and apply them equally to both variants. FroggyAds works with Adscore and internal quality controls to help filter bot traffic, and you can use whitelists and blacklists to manage sources. Never remove visitors from only one variant after seeing results. If you exclude a source, document the reason and rerun the analysis from the original assignment records.

How do I record which variant produced each conversion?

Store the variant ID with the visitor at assignment and pass it through to the conversion record in your tracker. FroggyAds source identifiers and tracking macros can help link clicks to later activity, and supported postbacks can report conversions, but the campaign ID alone does not tell you which page version a returning visitor saw. Keep assigned, exposed and converted counts separate so missing events cannot hide missing participants.

How long should an A/B test run?

Run until the planned sample is reached and the outcome window has matured, covering at least one full weekly cycle so weekday and weekend behaviour are both included. If purchases or lead approvals arrive days later, include that delay before analysing. Checking daily and stopping at the first favourable reading inflates false positives, so use a method designed for sequential looks if you want the option to stop early.

What is a sample ratio mismatch and why does it matter?

It is a gap between the actual group counts and the planned split that is larger than chance explains. For example, a hypothetical 50/50 test with 21,400 and 20,800 visitors gives a chi-square of about 8.5 and a p-value near 0.003, which signals an assignment, redirect or logging fault. Investigate the cause before you read any conversion rate, and do not delete records to make the ratio look right.

When should I roll out the winner and raise the budget?

Roll out after the primary outcome meets your pre-set analysis plan, guardrails such as lead validity or refunds stay acceptable, and the effect is large enough to matter after costs. Move traffic in steps, keep the control as a fallback, and raise FroggyAds budgets gradually while checking that your cost per outcome holds. A winning test reduces risk, but it does not guarantee future sales or profit.

Primary references and their scope

The NIST handbook provides experimental-design and statistical context. Google documents experiments within Google Ads, and Microsoft documents its Clarity interaction views. Those tools have their own capabilities and requirements. The accessibility, privacy and content references below support implementation review rather than certifying this page or any proposed experiment as compliant.

Write the result so the next buyer can use it

Begin with the decision, the population and the tested difference. Show participant and accepted-outcome counts before any percentage, then state the analysis method, the observation window and any deviations. Give the effect with appropriate uncertainty and explain whether it is large enough to change the business decision.

Separate the observation from the recommendation. A small positive estimate may remain too uncertain to justify rollout, and a persuasive increase may still fail a customer guardrail. If you recommend another test, say what it would resolve and why the extra cost is justified.

Close the record with the exact version selected, the rollout conditions and a named owner for follow-up. Keep rejected and inconclusive versions findable as well. That history stops a team from buying traffic to answer a question it has already investigated.

Start with a traffic plan your experiment can explain

Create your FroggyAds account when the audience, destination and measurement route are ready. We provide the campaign controls to purchase relevant traffic within your budget. Start with a manageable setup, retain its source context and let the planned experiment analysis guide the next change.

Create My Free Account
Interpret the result

What does a reported lift actually mean?

A percentage-point change and a relative increase describe the same observation on different scales. Both need the underlying counts and a clearly defined population. Neither, by itself, shows that chance variation, measurement loss or selection effects have been ruled out. Treat the arithmetic as the start of interpretation.

Read the comparison in order: Check who was assigned, whether the intended experience was delivered, which outcomes matured and how the estimate was calculated. Only then decide whether the difference is practically useful. This order stops an attractive headline from distracting the team from a broken comparison.

Before adopting the alternative: Confirm the uncertainty under the chosen method, the customer guardrails and the tested version. Explain any departure from the original plan. If those conditions do not support a decision, retain the control and name the specific gap that further investigation would need to close.

Decision gateQuestion to answerEvidence to retain
PopulationDid eligible participants enter the planned groups without treatment-dependent exclusions?Assignment counts, eligibility rules and the outcome of the allocation-integrity check.
OutcomeDo the accepted-action counts represent the same follow-up window and qualification rules?Group denominators, mature outcomes, missing-record handling and the agreed analysis method.
ActionDoes the effect and its uncertainty justify rollout without unacceptable customer or operating harm?The effect estimate, guardrail findings, selected version and reversible rollout decision.

Worked example: 0.2 percentage points is not 10 points, and not yet a result

Hypothetical example: Control A has 200 accepted purchases among 10,000 eligible assigned visitors; alternative B has 220 among 10,000. The observed rates are 2% and 2.2%. B is ahead by 0.2 percentage points, or 10% relative to A. A standard two-proportion z-test on these counts gives z of about 0.99 and a two-sided p-value near 0.32, so a gap this size is easily explained by chance at this sample. The sample-size table above shows why: detecting a 0.2-point lift at a 2% baseline needs roughly 78,700 visitors per variant. These counts are not a FroggyAds result.

Choose FroggyAds for the acquisition side of your test

Use our targeting, budget and source controls to buy the audience your experiment is ready to receive. Preserve the assignment and outcome analysis in your experiment system, then use what you learn to plan the next purchase. Create your free FroggyAds account

Direct answer

A useful experiment ends with an honest decision

An A/B test is useful when a fair comparison helps you choose what to do next. Keep the control when the alternative disappoints, release the alternative when the evidence supports the change, and acknowledge uncertainty when it remains. FroggyAds gives you the paid-traffic controls to fund that learning at a sample size your plan can justify.