What is statistical significance?

Statistical significance is a method to produce accurate results. Splitly automatically ends a test when it reaches the minimum statistical significance threshold, or when a certain level of accuracy has been reached.

Broadly speaking, statistical significance is the probability of one variant out or under-performing another. When we have enough data we show the statistical significance values in test results.

We always recommend a minimum significance of 90% - that gives a 10% margin of error. If you decide to abort the test before the significance is satisfied, you should not trust the results.

If you are seeing a small statistical significance or roughly a 50/50 result - this means your variants are too similar and the test will take a very long time to complete before considered accurate. It would be best to end the test there and try something else, such as a bigger change in your variants.

Learn more about statistical significance in Amazon A/B tests here.

Still need help? Contact Us Contact Us