Your Ad Copy Test Results Can’t Be Trusted: A Need-to-Read Article for Search Engine Marketers
If you are like us, you’re constantly running A/B ad copy tests in your AdWords campaigns. It’s possible that over the last several years you’ve been making the wrong decisions based on very misleading data.
Many of us use metrics such as conversion rate, average order value (AOV) and revenue per impression to choose a winner in an A/B ad copy test. Based on the statistically significant ad copy test results below, which ad would you choose to run?
|
Ad Iteration |
AOV |
Conversation Rate |
ROI |
Revenue/Impression |
|
Ad A (control) |
$225 |
3.15% |
$7.79 |
$0.42 |
|
Ad B (test) |
$200 | 2.65% | $6.79 |
$0.37 |
The answer couldn’t be clearer. You should run ad copy A, right? After all, it does have a higher AOV, a higher conversion rate, a higher ROI and it produces more revenue per impression than Ad B. What on earth could possibly convince you otherwise? The metrics above tell a very clear story. But are these the right metrics to look at?
Measuring A/B Tests: What Metrics Should You Consider?
Conventional wisdom tells us that if we’re running a true A/B test, then impressions will be split 50/50 between the two ad iterations. If this assumption holds true, then the metric we really should be focused on is revenue per impression. This metric tells us how much revenue we’ll generate for every impression served, which accounts for differences in CTR, AOV and conversion rate. If your business is focused on maximizing growth, then this may be the only metric to consider. If you also are focused on efficiency, then you will consider ROI and choose the ad that you believe provides the optimal combination of revenue and efficiency. While this approach is common, it is also fatally flawed. Here’s why…
Why Google Can’t Guarantee True A/B Ad Copy Tests
Earlier, we made the assumption that impressions are split 50/50 in an A/B test. However, when running our own A/B tests we noticed that certain ads were receiving well over 50% of the impressions, and in some cases, upwards of 70-90% of the impressions. We experienced these results when selecting the ‘rotate indefinitely’ ad setting, as well as in AdWords Campaign Experiments (ACE) tests. So why were we seeing an uneven impression split? Did we do something wrong? Well, yes: we made the mistake of assuming that impressions would be split 50/50.
How Google Serves Ads – And Why Quality Score Is Not a Keyword-Exclusive Metric
When you set up an A/B ad copy test in AdWords, Google will split eligible impressions 50/50, but served impressions are not guaranteed to be split 50/50, or even close to 50/50. Eligible impressions will differ from served impressions when one ad produces a higher CTR than the other. Since CTR is the primary determinant of Quality Score (and thus, Ad Rank), the AdWords system may actually serve a higher CTR ad more often than a lower CTR ad. This happens because your keywords’ Quality Scores will change for each impression depending on which ad is eligible to show for that impression. In other words, each time the lower CTR ad is eligible to show, the keyword that triggered the ad will have a lower Quality Score for that impression, and thus, a lower Ad Rank (because the expected CTR is lower with that ad), so the lower CTR ad will win the auction less often than the higher CTR ad. Naturally, this results in more impressions for the higher CTR ad, even though the two ads each receive roughly 50% of eligible impressions. If you use revenue per impression, one of the metrics we suggested earlier, then you will have failed to account for the discrepancy in impressions caused by varying CTRs. So, does this mean that your A/B ad copy test results are now meaningless? Not so fast.
Evaluating Test Results Are Easier Than You Think – Just Look at Revenue (or Revenue per Eligible Impression)
Let’s assume that your goal is to maximize revenue. The simplest metric to look at in an A/B ad copy test is revenue, but you can also look at revenue per eligible impression. Both metrics allow you to account for the variations in impressions due to different CTRs. To calculate revenue per eligible impression for each ad, divide the revenue from that ad by the impressions from whichever ad produced the higher number of impressions. Here’s an example: let’s assume Ad A generated a CTR of 6% and received 50,000 impressions and Ad B generated a 4.5% CTR and received 30,000 impressions. Between the two ads, Ad A received more impressions, so we can conclude that there were 100,000 total eligible impressions (twice the number of impressions generated by Ad A). Ad B was not served for 20,000 of the eligible 50,000 impressions due to a lower CTR (which impacted the keywords’ Quality Scores and Ad Rank for those impressions). If the revenue per impression metric is confusing, just focus on revenue: it will give you the same outcome. Let’s revisit the test results we showed earlier, which now include additional data.
|
Ad Iteration |
Impressions | CTR | Revenue | Transactions | AOV | Conv. Rate | ROI | Revenue / Impression | Revenue / Eligible Impression |
|
Ad A |
114,048 | 5.95% | $48,095 | 214 | $225 | 3.15% | $7.79 | $0.42 |
$0.36 |
|
Ad B |
135,000 | 7.00% | $50,085 | 250 | $200 | 2.65% | $6.79 | $0.37 |
$0.37 |
While Ad A outperformed Ad B based on its revenue per impression, it actually generated less revenue and less revenue per eligible impression than Ad A. Ad A did generate a higher ROI, however, so the tradeoff between efficiency and revenue should also be taken into account.
Interestingly, Ad A’s 19% higher conversion rate and 13% higher AOV still couldn’t make up for Ad B’s 18% higher CTR. This is because Ad A also received 16% fewer impressions than Ad B. Remember, a lower CTR will lead to fewer clicks AND fewer impressions – the double whammy.
The Conclusion – Focus Less on Revenue/Impression and More on CTR
Historically we have treated CTR as a secondary metric when evaluating ad copy performance. It’s easy to manipulate CTR with Keyword Insertion or misleading offers, but it’s quite difficult to generate more revenue and/or improve efficiency with new ad messaging. However, with a renewed understanding of how CTR can impact impression share, we are now focused on CTR when testing new ads. As we saw in the example above, if your new ad produces a significantly lower CTR than the existing ad, it will take massive increases in AOV and/or conversion rate to make up for the lost revenue due to fewer impressions and clicks. Therefore, when writing new ads we recommend that you focus on improving CTR (assuming the ads still attract the right audience). This will produce three distinct benefits:
- Greater click volume due to increased CTR
- Higher Quality Score due to increased CTR, which produces lower CPCs and/or higher ad position
- Increased click volume due to higher impression share
We are all familiar with the first two benefits, but the third benefit represents the most value and is the one most often overlooked.
Next time you run an A/B ad copy test be sure to consider the impact CTR and impression share have on your test results. Avoid focusing on revenue/impression, AOV and conversion rate to determine a winner and instead focus on revenue/eligible impression or total revenue. This will ensure that differences in impression share are accounted for, and, ultimately, that the higher revenue producing ad is correctly identified. If efficiency is a key consideration, keep ROI in mind as well.
