Detailed Guide to A/B Testing: Experience, Tips, and Planning

When our clients reach us they all seek different software solutions. However, some requests tend to be more popular than others, for example, to increase the effectiveness of the application, website, landing page, etc.

Let’s talk about the ideal solution to reach the mentioned goal - A/B testing. 

I will refresh your memory and quickly explain what A/B testing actually means.

Imagine, you have an idea or a hypothesis that adding a new widget to your website or changing the CTAs’ color will increase the number of visitors who make the desired and targeted action. To test this hypothesis you need to create a copy of the website and stream some of your traffic to a copy. Therefore, you will be able to compare the results and the effect of the applied changes. Sounds simple, right? However, I still have several tips for optimizing the process and using A/B testing in the most effective and cost-efficient way.

1. Make sure you have enough data and users

In case you don’t have a clear image of what common or regular user behavior is like, or if there is a low number of users, A/B testing may not be useful. It is hard to track any change in users’ behavior without comparing it to some kind of default scenario.

2. Track Your Data

It seems obvious, but I’ll still remind you - there is no point in conducting an A/B test if you don’t track the A and B metrics. You will not succeed in case you choose to just memorize this data or write down the numbers approximately.

3. Free Up Your Time

It is crucially important to spare a time gap to successfully perform A/B testing and analyze the results. Therefore, if you have strict deadlines for the product or if you don’t have enough resources, it may become an obstacle. In addition, it is obligated to spend some money on traffic. My advice is to plan any testing stage in advance.

If you are all set up and ready to start, it’s time to think about the internal and external factors that may impact the results of your testing practices.

We seek statistically correct data, therefore:

— when you divide your audience make sure that no users appear in both groups;

—  track both A and B data simultaneously and without any connection to one another. In this way you can exclude the impact made by ad campaigns, season trends, time, etc;

— Filter your internal traffic with help of Google Analytics.

Which metrics you could increase with the help of A/B testing? Let’s look at the top 3:

1. Sales Metrics

This one is most accurate for e-commerce projects. For example, you could increase the average paycheck in terms of online shopping.

2. Behavioural Factors

The most popular metrics to look at are pageview depth, average session duration, bounce rate, and retention rate.

3. Conversion 

The most common request is to increase the conversion, which means increase the number of visitors who made targeted action.

In Vilmate we prefer to use the following A/B testing algorithm:

— Set a goal

— Choose the key metric

— Define the hypothesis

— Define the selection ( what percentage of your audience will be redirected to the B variant and what are the characteristics of this audience)

— Contact the testing

— Analyze the results

Be aware of 2 common but complete opposite mistakes while conducting the testing:

• You will see the connection between data sets when there isn’t one 
• You won’t see the connection between the data sets when there is one.

Of course, a lot of online testing tools designed to help you are on the market and you can try to use them for your benefit, However, you risk wasting your time and resources and end up with zero results, totally unmotivated to initiate change next time.

To avoid this scenario we recommend hiring experienced in working with a huge amount of data analytics specialists. Vilmate team members are ready to apply their technical expertise to your project and help you test your most controversial and brave ideas. Just send us a welcoming note and we will come back to you with a solution.

This is the sum up of all A/B testing key points and its’ benefits for your business. I hope you find my experience worth noticing! I will be glad to receive any kind of feedback, hear your opinion, or answer any questions.

Share your experience with A/B testing - whether it was positive or negative. Did you confirm your hypothesis?