Your first Amazon A/B test

Are you excited to get started with Amazon optimization and see how much you can improve your profits? Great! We are too! 

It's important to us that your first A/B test goes as smoothly as possible, which is why we have written this guide as a starting point for your first A/B test.

First, we recommend either a pricing or primary image test to start with. Why? Because usually these kinds of tests are very sensitive to either sessions or sales on Amazon, and as such we're usually able to find a clear winner in a short amount of time (ideally, within a couple of weeks).

Please Note: We don't recommend a title test for your first test. This is because Amazon often places restrictions on product titles, and you'll have to check with Amazon to see if you have listing control for the product title first, before starting the test. Since Amazon usually doesn't place restrictions on pricing and images, these are good tests to start with.

Now that you've chosen what you're going to test, you then need to setup your Splitly account. This only needs to be done once, and you can do it by following these directions:

  • To start, please login to your Splitly account and go to the MWS setup page and click on the green 'setup' button to add your Amazon seller accounts to Splitly.

  • Next, please go to the products page and sync your product feed by clicking on the green 'sync product feed' button. More information on what this is can be found here, and the page should look like this:

  • Now, once your product feed has synced with Splitly (this will; take some time), please click the green 'Create Test' button to the far right of the product you wish to optimize. That can be found on the  products page.

  • When you're, please give the test a name, such as "Jungle Stix pricing test". You will also need to enter in the unit cost of your product. The other fields in the 'Fees & Costs' domain should populate automatically:

The unit cost is the amount of money you have spent through manufacturing, shipping to Amazon and anything else you can think of, excluding Amazon fees! We already know what Amazon charges you on a per product basis.

For tests other than pricing, choosing the conversion rate is usually the best one to choose. But for pricing, we recommend "Profit" as your goal because it's important to consider the differences in pricing and what they will mean to your bottom line. A higher conversion rate does not always equate to higher profits; there is always a sweet spot for product pricing and split testing price is the way to determine that.

As for the minimum statistical significance, that is basically the accuracy setting of the test. It's important Splitly provides sellers with accurate results and, to accomplish this, we run tests until they reach statistical significance. We recommend reading how statistical significance benefits you in the long run.

For most sellers, 90% minimum significance is adequate. However, if you are a high volume seller (more than 100 sales per day) then we recommend a statistical significance of 95% or higher. But so you know, the higher the minimum significance, the longer the test will need to run before an accurate winning variant is found.

Here's an example of how to run a pricing test on a product that also has a sale price...

  • First, we'll create two variants: one with a high price and one with a low price. 
  • Next, we'll name the variants accordingly, so we don't get confused.

  • After that, we'll edit the sale price for the high price and low price variants, changing them by about +10% and -10%. By doing this, we can find the direction (higher or lower prices) we need to go in order to optimize our profits. After this test, we can perform localized tests around the winning variant, to hone in on the optimum price for our product.

  • Now we need to make sure we check that the profit per unit makes sense. If it doesn't make sense, we either entered in incorrect values for the unit cost, or the pricing is wrong. Splitly does allow for negative profits, so you will need to check this manually.
  • Finally, we need to save the test at the bottom and click the green "Run Test" button. Splitly will then begin the test and start making changes to your product listing, rotating through your variants as and when necessary. 

Tips for further tests

✓ Experiment using 2 variants only. Tests take a longer to complete when you add more variants. Consider running a test using only 2 variants, then run further localized tests around that original winning variant. The minimum test duration is (the number of variants) multiplied by 7. This is because every variant will be run on each day of the week at least once. So a test with two variants will run for a minimum of 14 days, a test with three variants would run for at least 21 days, and so on.
✓ Create significant, obvious differences between your variants. If your variants are too similar then it will be too difficult to determine a winner. Winners are found faster when variants are noticeably different from each other.
✓ Only vary one field in each test. For example, vary your pricing or your primary image, but not both at the same time.
✓ Enter in your fixed cost correctly. The fixed cost includes manufacturing, shipping, FBA fees and anything else you can think of.
✓ Don't swap your bullets. The bullet order doesn't make much difference.
✓ The best fields to test are: PRICING and PRIMARY IMAGE. These are the most sensitive fields. Other fields can give decent results, but tests can take a long time to complete.
✓ Pick the right goal. Choose profit for testing your prices, choose sessions or conversion rate for anything else. The test will end when we find a winner using your chosen goal.
✓ Don't abort the test prematurely. Let the test run its full course. We'll advise you what to do as the test continues. We will automatically end the test when we find a winner and will update your listing with the winning variant for you.
✓ Ask for help when you need it. Not sure what to do? Contact us!

Next steps

Still need help? Contact Us Contact Us