A More Calculated Approach to an A/B Test

What are some of the reasons why you do an A / B test?

When I think of the benefits of A / B testing, I think of one of the most popular and concrete ways to experiment with ad designs that are effective for audiences. I think about how changing a simple element can be the deciding factor for customers and that doing a test helps me figure out the preferred design.

Until recently, I thought there was only one type of A / B test. After all, the definition itself is pretty simple.

Then I came across another type of A / B test. This method still involves testing variants to determine audience preference, but it requires more computation and trial and error.

This method is called Bayesian A / B testing. If you want to take a more specific tactical approach to testing ads, this may be the answer.

But first let’s talk about how Bayesian A / B tests differ from traditional A / B tests.

Bayesian A / B tests

There are two types of A / B tests: Frequentist and Bayesian.

Every A / B test has the same few components. They use data that are based on a metric and determine variants A and B. For example, a metric can be the number of times an ad is clicked. To determine the winner, this metric is measured statistically.

Let’s apply this to an example of using the frequentistic or traditional approach. In this scenario, you would design two displays and change one variable, e.g. B. the copy of the advertisement. Then select the metric of how often an ad is clicked.

The winner of the common A / B test in this example would be which ad your target audience clicked on most simply based on the results of this experiment.

If you illustrated these components in a Bayesian A / B test, you would approach the test with different data.

This definition may sound a little difficult to visualize without an example. So let’s go through one.

If your previous ad on Facebook attracted 867 unique visitors and had 360 conversions, which is a conversion rate of 41%, you would use this data to inform an expectation. If you found out that your next Facebook ad reached 5,000 unique visitors, you can conclude that based on that previous experience, you would get 2,050 conversions. This would be the variant “A.”

Suppose you look at the performance of a similar Facebook ad and you end up with a conversion rate of 52%. This is the variant “B.” By collecting the data from the two variants, you have calculated the posterior distribution. The tests previously performed are now the basis for your Bayesian test.

If you had any conclusions about the conversion rates achieved with each variable before calculating the posterior distribution, you can now update them based on the data you have collected. You can ask hypothetical questions about your test, e.g. “How likely is it that ‘B’ is bigger than variant ‘A’?” In this case, you can conclude that the answer is 9%.

Then the trial and error part begins.

Bayesian methodology makes decisions by drawing conclusions. You can calculate the expected loss based on the rate at which your metric decreases when you select one of the variables. Set a limit, e.g. B. 2%, under which the metric should fall. As soon as you have collected enough data to support that a variant has fallen below 2%, you have your test winner.

Since your derived loss for a variant is the average amount by which your metric would decrease if you chose that variant, your limit should be small enough to conveniently indicate that an error is so large.

The method suggests that you’re more willing to make a mistake in a certain amount and then move on to a more refined experiment, rather than wasting time on an error that has dropped below this threshold.

If you did two experiments, they would stop if the expected loss was below this 4% limit. You would use your variant values ​​to calculate your average loss. Then you would start the test again and use these values ​​as your wealth distribution.

Bayesian A / B testing proves that you can make a business decision that doesn’t go below the limit you set. You can use the collected data to run tests continuously until you find that the metrics increase with each experiment.

If you use Bayesian tests, you can change the test regularly and improve the results during the test run. Bayesian A / B testing uses constant innovation to achieve tangible results by making small incremental improvements. You do not have to use inference, but use it as a variant.

If you run A / B tests on software or different channels, you do not need to change them to run a Bayesian A / B test. Instead, you can look at the tools that are available in this software to get calculated results. You can then run and analyze these tests continuously to select your winners.

You can use a Bayesian A / B test instead of a traditional A / B test if you want to include more metrics in your results. This is a really good test to calculate a more specific ROI for ads. Of course, if you have less time available, you can always use a frequentistic approach to draw a better conclusion.

Whichever method you choose, A / B testing is popular because it gives you a conclusion that you may find useful in future campaigns.