How do I interpret Splitly test results?
Analysing your test results to see how your pricing, images or copy performed is possibly the most exciting moment of an A/B test, but also, the most important. The decisions you make from A/B tests have vast impacts to your business, boosting your sales, profit and/or conversion rate; whatever is your goal of the test is.
When a test completes, your Splitly account will show four charts like the following:
The charts are:
- Average daily sessions
- Average daily sales
- Average conversion rate
- Average daily profit
The results are displayed as average daily values, because this is how Amazon groups its data for each of your variants.
In the example above the goal was to boost the average daily profit, this is because the chart for average daily profit is highlighted green. Depending on what your goal of the test is, the green highlight will move to a different chart. This chart gives you the overall result for your specific goal.
However, it is still important to check the other charts too. A sudden increase in sessions does not always lead to an increase in sales, so, it is always good to run these charts through your "common sense check" to make sure everything is in order.
The statistical significance values are displayed in the variants. The definition for statistical significance is the probability of a variant over or under-performing the original. Usually, we recommend running a test until it reaches >90% statistical significance. The additional 10% buffer allows for some degree of uncertainty, but also means we can end a test more quickly.
Finding a winner
In the example above the goal was to increase profit. Variant 21.88 was the clear winner in this test, almost doubling the amount of profit for this product! We saw better metrics across the board: more sessions, better conversions, more sales and more profit. In this case the product was underpriced, which is a common mistake sellers make. Many sellers have the belief that in order to get more sessions and sales, one simply drops the price so more customers buy. However, sometimes customers want to buy a more expensive product because the perceived quality and value increases with price, leading to more sales. This is probably what happened in this case. The only way to truly know for sure is by running an Amazon A/B test.
When a test is unsuccessful
Your results might not always be so dazzling though. Sometimes your original might perform better than any variant. You might think you have wasted your time, but this is far from the case. You have learned what does not work.
I have not failed. I've just found 10,000 ways that won't work. - Thomas A. Edison (he invented the lightbulb)
Knowledge on what didn't work is very valuable, because you can apply that to other products and tests. After running several tests and finding winners you can start to spot patterns for different kinds of variants that increase or decrease your profits, you will become a better seller by knowing this valuable information.
When a test takes too long
Sometimes an A/B test could drag on for several weeks or months before the test becomes significant to find a winner. This can happen for two reasons:
- The variants are too similar to each other
- Your sales volume is too small
If you are only making a handful of sales each month (about less than 20) then it probably isn't a good idea to focus on A/B testing at this moment in time. A/B testing works best once your product has gained some initial traction and you have enough volume of sales to determine how different variants perform.
In most cases the reason why a test drags on for a long time is because the variants are too similar to each other. If that is the case then we'd recommend you abort the test, our recommendation will be shown in the test results page. You should abort the test, re-run the test on the same product but ensuring the differences between variances are big enough that has an effect on how customers behave when on Amazon. An example of this would be to raise or lower the price even further.