Your first Amazon A/B test
Are you excited to get started with Amazon optimization and see how much you can improve your profits? Great! We are too!
It's important to us that your first A/B test goes as smoothly as possible, which is why we have written this guide as a starting point for your first A/B test.
First, we recommend either a pricing or primary image test to start with. Why? Because usually these kinds of tests are very sensitive to either sessions or sales on Amazon, and as such we're usually able to find a clear winner in a short amount of time (ideally, within a couple of weeks).
Please Note: We don't recommend a title test for your first test. This is because Amazon often places restrictions on product titles, and you'll have to check with Amazon to see if you have listing control for the product title first, before starting the test. Since Amazon usually doesn't place restrictions on pricing and images, these are good tests to start with.
Now that you've chosen what you're going to test, you then need to setup your Splitly account. This only needs to be done once, and you can do it by following these directions:
- To start, please login to your Splitly account and go to the MWS setup page and click on the green 'setup' button to add your Amazon seller accounts to Splitly.
- Next, please go to the products page and sync your product feed by clicking on the green 'sync product feed' button. More information on what this is can be found here, and the page should look like this:
- Now, once your product feed has synced with Splitly (this will; take some time), please click the green 'Create Test' button to the far right of the product you wish to optimize. That can be found on the products page.
- When you're, please give the test a name, such as "Jungle Stix pricing test". You will also need to enter in the unit cost of your product. The other fields in the 'Fees & Costs' domain should populate automatically:
The unit cost is the amount of money you have spent through manufacturing, shipping to Amazon and anything else you can think of, excluding Amazon fees! We already know what Amazon charges you on a per product basis.
For tests other than pricing, choosing the conversion rate is usually the best one to choose. But for pricing, we recommend "Profit" as your goal because it's important to consider the differences in pricing and what they will mean to your bottom line. A higher conversion rate does not always equate to higher profits; there is always a sweet spot for product pricing and split testing price is the way to determine that.
As for the minimum statistical significance, that is basically the accuracy setting of the test. It's important Splitly provides sellers with accurate results and, to accomplish this, we run tests until they reach statistical significance. We recommend reading how statistical significance benefits you in the long run.
For most sellers, 90% minimum significance is adequate. However, if you are a high volume seller (more than 100 sales per day) then we recommend a statistical significance of 95% or higher. But so you know, the higher the minimum significance, the longer the test will need to run before an accurate winning variant is found.
Here's an example of how to run a pricing test on a product that also has a sale price...
- First, we'll create two variants: one with a high price and one with a low price.
- Next, we'll name the variants accordingly, so we don't get confused.
- After that, we'll edit the sale price for the high price and low price variants, changing them by about +10% and -10%. By doing this, we can find the direction (higher or lower prices) we need to go in order to optimize our profits. After this test, we can perform localized tests around the winning variant, to hone in on the optimum price for our product.
- Now we need to make sure we check that the profit per unit makes sense. If it doesn't make sense, we either entered in incorrect values for the unit cost, or the pricing is wrong. Splitly does allow for negative profits, so you will need to check this manually.
- Finally, we need to save the test at the bottom and click the green "Run Test" button. Splitly will then begin the test and start making changes to your product listing, rotating through your variants as and when necessary.
Tips for further tests