How to A/B test a low-traffic website?

In the year 2017 literally everyone is talking about A/B testing. Each new feature of your website needs to be validated before it is ready to go live. For those who are not yet part of the A/B movement, here is a short summary.

A/B testing is a website optimisation method where randomly one out of two or more different versions of the same product (app or website view) will be shown to the user. The users don’t know that they are part of an experiment and that their interactions with the website are being tracked. The test is successful if you get a winner variation after a certain time period. That means that one variation performs better and reaches a better conversion rate. We talk about a significant winner if the confidence interval of your test is above the magical 95%. This number is a good indicator how well the the winner variation can be applied to the product to reach the same results over a longer period.

How A/B testing works.

The simple rule of thumb for getting significant data from the analysis is the more users are part of your test the more meaningful and reliable are your findings. This is pretty straight forward but what if your website is not one of the lucky fellows like Google or Facebook with millions of visitors every day? No worries, you are still able to get great results by running A/B tests on your “low-traffic” website (low-traffic means <100 users a day) if you keep the following in mind:

  1. Expand your testing period if necessary
    Don’t stop your tests just after a few days. Especially with a few users your test needs a certain time to find its balance. If you have good and convincing results after a short time period thats mostly related to the premature testing dilemma. This phenomenon shows that early conversion peaks level off to real and reliable trends. To get better insights of how your product is really performing let your test run another few days or even weeks if necessary.

If you keep these tips and tricks in mind you should be able to get better and more reliable results, even if you have only a few users in your test. By implementing new features to your product which you base on better and more reliable A/B tests you can improve the user experience of your product and this might also result in a growing user base.

Founder of Beyond, PM Library & Riptide, Ex-N26, Co-Organizer of Product Tank Hamburg & Barcelona

Founder of Beyond, PM Library & Riptide, Ex-N26, Co-Organizer of Product Tank Hamburg & Barcelona