How long does an Amazon A/B test run for?
The duration for an Amazon A/B test depends on a variety of factors. Namely:
- The sales volume of your product
- The number of variants in a test
- The size of the difference between variants
- The minimum significance of your test settings
The minimum test duration is 7x(the number of variants), so for a 2 variant A/B test the minimum test duration is 14 days. This is because each variant must have a try on each day of the week, to counteract any possible discrepancies caused by day of the week.
After that, we will analyse the statistical significance of the results so far. If you have a large sales volume, it will be easy to hit the minimum significance threshold very quickly, oftentimes tests will end on day 14. This is because we have a larger data set which means more accurate results.
However, products with smaller sales volumes can still reach the minimum significance threshold quickly, so long as the difference between the variants is quite large, and gives very obvious results.
If the sales volume is small and there is no clear winner after 7x(the number of variants) days, Splitly will likely recommend you abort the test and try something else.
If you are very close to achieving the minimum significance threshold we will display an approximate number of days remaining until a winner is successfully found.
Allow 7 x (the number of variants) days before we run any analysis. After this, we will give you recommendations of whether to pause/abort the test or continue, with an approximate number of days remaining until we can find a winner.
For more information about statistical significance in A/B testing, please read here.