How to improve a product with A/B testing

How to improve a product with A/B testing

Alexey Vasenin

Alexey Vasenin

Senior System Analyst Croc Code

If you want to create a truly high-quality product, then you need to conduct UX research. One of the key UX studies is A/B testing. In this article, we will look at its life cycle using specific examples, discuss in which cases it should be used, and analyze the errors.

Let’s start with the case

In 2019, the popular social network changed the menu item “Groups” to “Communities”.

I think that from the point of view of UX, this is a disaster: the words “Messages” and “Communities” are visually similar, which means that people will start to get confused and move to an inappropriate section (the situation is further complicated by the fact that “Messages” and “Communities” are the most popular sections of the network).

Old and new design. Here and further source: report “A/B testing as a product improvement mechanism”

After going through the discussions of the change, I noticed a lot of negativity about it, which obviously had a bad effect on impressions about the product.

Comments left by users after the design change

This problem could have been avoided with A/B testing.

What is A/B testing?

We have several versions of the same project. For example, a page with a banner: in the first case, the banner is at the top of the page, and in the second — at the bottom (there may be more options — it all depends on the number of hypotheses and the size of the audience).

Example of A/B testing of a page

We take the audience, the visitors of the page, and divide it into several groups — according to the number of product versions. Next, each group uses its own version of the site, and we monitor the key metric: here it is the number of clicks on the banner.

In the first case, they are 15%, and in the second — 21%. We conclude that the second option is better, and send it to production.

Stages of research:

  • defining goals and metrics;
  • definition of hypotheses;
  • design definition;
  • conducting testing;
  • analysis of the results.

Let’s look at each stage in more detail using an example from my practice. I analyzed this case in a report at the Analyst Days-13 conference.

Once, my team and I implemented a gamification system in a corporate portal. Its essence was that the company’s employees answered questions from colleagues, offered ideas for development, helped newcomers, replenished the knowledge base, participated in conferences. For this, they received bonuses that they could spend on prizes, for example, a laptop or tablet.

Gamification system

The system turned out to be good, but after a while users began to visit the service less often. We decided that this was a reason to use A/B testing and make changes to the product.

First we identified the goal, the global result we want to achieve. In our case, this is to increase engagement. Then we chose a metric — an indicator by which we can judge progress. The number of points earned by users per day was important to us.

At the next stage, we conducted a study and we found out that the service is used in three time intervals: from 9 to 10 in the morning, from 13 to 14 in the afternoon, and from 17 to 18 in the evening. We decided to test all three periods and determine when it is better to shift the focus inside the portal to the gamification system in order to get the best result.

Next, we developed a new design:

This is a prototype, but it shows the essence

We have a home page with widgets. Users can follow them to the necessary sections or see important information on them. We have increased the game widget at certain hours, and also added a bright button and a rating, which, in theory, motivates users to go to the section and earn points.

Then the testing itself began. I have divided it into four sub-stages:

  • Setting up parameters: we have determined how long the A/B testing will last and how we will segment the audience. We set up the work and the user tracking system.
  • Launch.
  • Monitoring: we made sure that testing goes as it should, features appear when they should, data arrives, and so on.
  • Completion. We finished testing two weeks later — when we had collected enough data.

As a result, we got the following results:

The best result was in the middle of the day — the indicator increased by 30%

Blue marks the indicators before the A/B testing — when the banners were of standard size. Red is what happened when we doubled the banner and added a button and a rating.

Thanks to our changes, we saw an increase in the average number of points. Engagement continued to decline, but not as fast as at the beginning of the work.

When to conduct A/B testing

In conclusion, I want to tell you what conditions are needed for the study:

  1. We want to get objective data that does not depend on the opinion of the designer or the designer.
  2. We have a large sample — with a small one we will not be able to get objective results. In my opinion, the study can already be carried out if there are more than 500 users.
  3. We have enough resources. A/B testing can be very expensive, especially if you need to do it several times. At the company, we put resources into three iterations: if nothing works out the first time, it will work out the second time, but not like that, and only the third time everything will go smoothly.

That’s it. Ask questions on the topic and share your experience in the comments.

Tech Outsourcing | Dedicated Software Team

Ready to see us in action:

More To Explore
Enable registration in settings - general
Have any project in mind?

Contact us: