How does a Shopify product page A/B test work?
A Shopify A/B test randomly shows comparable shopper groups different product-page variants, then compares a predefined result after enough eligible observations.
Stores with limited traffic can use usability research, customer interviews, or a staged release. These methods answer different questions and should not be presented as statistical A/B proof.
Write one Shopify buying question
Name the page decision, the approved change, the target devices, and the commerce result. Avoid broad questions such as whether a fresh design performs better.
A clothing store might compare approved size-guide layouts on mobile. It can choose add-to-cart rate as the KPI while monitoring purchases, returns, accuracy, and page errors.
- One main question
- At least two experiment-ready approved variants
- One all-traffic eligibility rule
- One KPI and useful safety checks
- One measured-session threshold per breakpoint
Choose a clear commerce result
Choose add-to-cart rate for an early buying action or purchase conversion rate for completed orders. Pick one KPI before launch.
Only measurement-eligible sessions count toward the measured-session threshold for each desktop, tablet, and mobile breakpoint.
Reaching a threshold triggers an operational choice. It does not establish a p-value, statistical significance, or causal certainty.
| Decision | Set before launch |
|---|---|
| Variants | Every experiment-ready approved version |
| Scope | All measurement-eligible traffic |
| Weights | An equal split across included variants |
| KPI | Add-to-cart or purchase conversion rate |
| Thresholds | Measured sessions for each breakpoint |
Set up the test in 100 Rocks
100 Rocks allows one active test per store. Its current test uses an all-traffic scope and equal weights across every experiment-ready approved variant.
Measurement-eligible sessions advance separate thresholds for desktop, tablet, and mobile. The test does not target a behavior audience or let merchants set unequal weights.
- Approve at least two variants.
- Confirm every experiment-ready variant is included.
- Select the add-to-cart or purchase KPI.
- Set measured-session thresholds by breakpoint.
- Launch, monitor, pause, or resume the test.
Read next: Keep Shopify product page variants consistent with your brand
Read Shopify test progress carefully
At each breakpoint threshold, 100 Rocks automatically promotes the variant with the highest observed KPI rate for that breakpoint.
That promotion does not prove significance or a true winner. Review sample quality, effect size, operating time, customer safety, and other evidence before broader decisions.
