Run an experiment

Previous Next

Run an experiment

An experiment runs a panel of synthetic shoppers against two versions of one page — your current page and a new version you want to test — and files a verdict on which converts better. You supply both versions. Squoosh does not write the new version for you; it tests the real change you made.

This page walks through the wizard from start to launch.

Before you start

Two things must be true before Squoosh will launch a run:

  • Your synthetic shoppers are calibrated. Calibration builds the panel of shoppers that runs your test. Until it's done, the launch button is disabled and the wizard shows a Set up shoppers link. See Calibrate synthetic shoppers.
  • Your workspace has enough credits. Each run costs credits, and the wizard shows the exact cost before you launch. If your balance is short, the wizard says so, and you can pick a smaller audience or top up. See Credits and cost.

You also need the URL of version B, the new page you want to test. Squoosh tests the page you point it to; it does not generate a variant.

Open the wizard

  1. In the sidebar, click Experiments.
  2. Click New experiment.

The wizard opens as a single sheet with numbered sections. Fill them top to bottom.

Target

Section 01 Target is where you name the experiment and enter the page you're testing against.

  1. In Control page (A), enter the URL of your current page. A valid URL shows Looks valid.
  2. In Experiment name (optional), type a name that describes what you're testing, for example PDP layout v2 vs current. Leave it blank and Squoosh names the experiment for you from the page and goal.

Control page (A) is your current, live version, the baseline every result is measured against.

Variant

Section 02 Variant is where you tell Squoosh how to reach version B. Squoosh does not generate the variant. You bring it, and Squoosh tests your real change.

Under How do you ship variants?, choose one mode:

Mode Use it when
Two URLs Version B lives at its own web address. This is the default and the verified path. Use it unless you have a reason not to.
VWO / Optimizely / AB Tasty Your variant shows on the same URL through a client-side testing tool.
Feature flag Your variant is selected on your server by a flag value (LaunchDarkly, Statsig, Split, or a custom cookie, header, or query parameter).

For Two URLs:

  1. Confirm the A · Control card shows the control URL from the Target section.
  2. In Variant page (B), paste the URL of the new version. It must be a valid URL and must differ from the control. When it's accepted you'll see Looks valid · differs from control.

Note

The VWO / Optimizely / AB Tasty and Feature flag modes are early and still in validation. If you run variants through one of those tools, contact support@squoosh.ai to set up your first run. For everything else, use Two URLs.

Audience

Section 03 Audience sets how many synthetic shoppers run.

The default and recommended size is Standard, a panel of 1,000 shoppers. The card shows the credit cost and an estimated run time.

A larger sample takes longer, costs more, and calls closer results. It is useful when an earlier run landed too close to call. A smaller sample is faster and cheaper but only directional. If your calibrated panel is too small to staff a size, the wizard surfaces a size that fits.

Goal and traffic

Section 04 Goal & traffic sets what you're optimizing and how shoppers arrive.

  1. Under Primary goal, choose Add to Cart (default) or Checkout, the conversion the experiment measures.
  2. Under Traffic profile, choose Organic / Direct (default) or Paid Ad (PDP), where shoppers enter from.

Traffic profile sets the source and intent of your shoppers. It is not a split between the two versions; there is no traffic-split control.

Review and launch

Section 05 Review & launch summarizes the run and lets you edit the shopper tasks.

  1. Check the summary grid: Type, Variant via, Goal, and Sample.
  2. Review the Shopper tasks, the shopping objectives each shopper attempts, drafted for you while you filled the form. Edit any task inline, or remove ones you don't want. At least one task is required. The list you see is the list that runs.
  3. Note the Auto-stop at 80% confidence row. Auto-stop is always on: the run ends early once a clear winner holds, but never before 40% of the panel finishes.
  4. Confirm the cost line — Shoppers, Cost in credits, and the balance left after — then click Launch experiment.

If the launch button is disabled, one of the gates in Before you start isn't met: shoppers aren't calibrated, the variant URL is missing or matches the control, or your credit balance is short.

After you launch

Squoosh reserves the panel, warms up both versions, and moves you to the live experiment page. The results fill in as the first shoppers enter the funnel, so you can watch the run build.

When the run finishes, or auto-stops once a winner holds, read the verdict at the top of the experiment page: the outcome, the lift, and the confidence. "Keep the original" is a real outcome; a change that costs conversions is worth knowing before you ship it. There is no ship or deploy button. Rolling out the variant is your call.

See How Squoosh decides a winner and Reading lift and confidence for how to read the result.

Credits and cost

Credits are what you spend to run tests, and your workspace shares one balance. A test is one experiment run, an A/B test or a Conversion Report. The sidebar shows N tests left: how many Standard tests your current balance covers. The wizard shows each run's exact credit cost before you launch and the balance you'll have after; the Standard 1,000-shopper sample costs 200 credits.

Your plan caps how many tests you can run per month and how many properties you can have. Hitting a cap prompts you to upgrade. To add credits or change your plan, contact the Squoosh team at support@squoosh.ai; self-serve billing isn't available yet.

Calibrate synthetic shoppers Synthetic shoppers