What is AB Testing Vs Pre-Post testing? Pros & Cons?

As a publisher, what if you could set different floor prices in parallel (Google AdX Floor Prices) for the same inventory at the same time and know the best working floor price within couple of hours instead of wasting a day or two? Sounds great right? Well, we are trying to do exactly same with Tercept Dynamic Pricing Solution. So before going to dynamic pricing solution, let us have a look at the conventional way of setting and measuring performance that most of the publishers follow – Pre-Post Testing, where they measure the pre-post performance of the floors they set and compare it.

So what is Pre-Post Testing?
For Publishers who are using Google Ad Exchange, setting up of pricing rules and optimum floor price would be challenging. Many times, publishers would not know if the floor set was the best possible floor or not. So arriving at the right floor involves a lot of trial and error iterations. For examples, let us assume a publisher does a lot of analysis on historic data and arrives at a floor price to set. But due to change in market behavior and seasonality, they might not get the result they have expected, so they will make a correction in their floor and after waiting for a day or two, they set a slightly different floor and wait for two more days. Then they’ll observe how the new floor is working in these two days and if they feel there is still potential upside, they have to change the floor price again and wait for 1 or 2 days to get a picture of how the latest floor price is working. Still publishers might not be sure if they have arrived at the right floor price and if they want to test some more floors. So more the number of floors they want to test, more the number of days it is going to take!
The above process can be time consuming and may cause loss of potential incremental revenue for publishers.

What is A/B Testing?
With A/B Testing framework, publishers can set multiple floors at the same time for a particular inventory and know what the best suitable floor is in couple of hours. And if none of the floors are working as expected, they can set another set of floor prices. So publishers can quickly know what the best working floor for their inventory is without wasting weeks together to test it out and analyzing it.

Since programmatic media buying is so dynamic, ideal floor price affected by weekends, holidays, festivals, month-ends etc. The right price floor depends on multiple variables including contextual information as well as market behavior. These variables change constantly over time. Price floors need to be tested & optimized on a regular basis to maximize revenue and one may not be able to change floors efficiently and as frequently capturing all these situations. There is no easy way to test price floors for your inventory. The only option available is above mentioned pre-post testing. But pre-post testing has obvious flaws – it is extremely time consuming & market behavior constantly changes over time and hence price floor optimization using pre-post testing seldom works!
A/B Testing helps putting suitable floors capturing all market behaviors and market fluctuations. Tercept Dynamic Pricing Solution uses A/B Testing framework that automates the above process by running multiple price floors in parallel on Google Adx so that publishers won’t have to break their heads setting up the floors all the time.

But reporting may not be as straight forward as before as publishers will be having many chunks of data for the same inventory. But that should not stop a publishers from experimenting with A/B testing as it may give anywhere between 20 – 60% upside derived from our past experience.

You can see in the below snapshot, where we are running A/B testing for one of the ad units named mweb_mrec_300x250, and we have set 7 different floor prices for the same ad unit. We have used naming convention in such a way where A10 means Anonymous floor was set to $0.10 and B100 where Branded floor was set to $1.00. Here we can observe that different pricing rules (for the same inventory) performing differently and also, the best floor (highest ad request CPM) for June 1st is not the best floor for June 2nd!

Due to second price auction, multiple floors with A/B testing would add immense value for publishers as they can continuously measure data and making changes accordingly results in significant improvements in CPM and revenue.

Author
Vinay B Rao
Senior Business Analyst, Tercept

  • Share:

Leave a Comment