If you are just starting out, or even if you have completed some A/B testing before, you probably have a number of questions about what to do with A/B tests.
How long should the test last?
What variable should you test?
What makes a test significant?
We’ve put together a few guidelines to consider before doing any future A/B test that will make it easier to measure your results and find out which variation performed better.
Number 1 – Only conduct one test at a time
So you’ve got a new product launch. You want to test which audience works best, which ad copy resonates more and which landing page gets the best conversion. But doing too much muddles results and you’ll never really know what is truly contributing to improvements, or god forbid, declines in your performance.
Be patient, test one hypothesis at a time, and you’ll have results that will lead to stronger, more statistically relevant conclusions.
Number 2 – Test one variable at a time
So following on from the above, you may decide to go with a new landing page test, but changing multiple things on the landing page again can lead to messy data.
Instead try one variable at a time too. Test changing the CTA one time and the colour of an image for example in another test to confirm the effectiveness of the change.
Number 3 – Think little
Although we all want to think that big, sweeping changes can be the answer to all our prayers, it is often the small changes that can have the biggest effects.
Think of simple changes like switching the colour or repositioning your call-to-action button.
Number 4 – Measure more than just the immediate KPI
So your A/B test might have a positive impact on your landing page conversion rate, but how did it impact your sales conversion rates?
A/B testing can have a significant impact on your bottom line. It could be that an uplift in one area has the opposite effect elsewhere – so keep a close eye on the whole picture.
Number 5 – Decide what you want to test
With so many possibilities, it is important to drill down into exactly what tests you think will have true impact.
Get all the resources you have available (marketing, devs, sales) and brainstorm on areas to test. Come away with a plan and gantt chart, depending on resource an availability, and start scheduling in your future game changing tests.
Number 6 – Finally, ensure statistical significance before finalising results!
Rope in your data team, utilise built in platforms such as Google optimise or use an online calculator.
Don’t put in all that work just to fall at the last hurdle and not have statistically significant results to show.
It takes as long as it takes. Wait and get the true results.