Incrementality and cannibalization in user acquisition
Welcome to the fifteenth edition of MAMA Boards, an AppsFlyer video project featuring leading mobile marketing experts on camera.
For today’s mini whiteboard master class, we have Moshi Blum, Head of User Acquisition at Viber, a calling and messaging app with over 1 billion users worldwide.
With competition so fierce in the mobile landscape, if you’re not measuring the incremental growth of your campaigns, you won’t make it too far. Moshi steps in to give us a complete breakdown of everything you need to know about incrementality and organic cannibalization, later offering 6 pro tips to make sure your incrementality measurement comes out on top.
Real experts, real growth. That’s our motto.
Enjoy!
Transcription
What is the true impact of your marketing campaign? What if you discovered that your marketing campaigns are not performing as well as you want them to? Would you do things differently?
Welcome to another addition of MAMA Boards by AppsFlyer. My name is Moshi Blum and I’m the Head of User Acquisition at Viber. Viber is a leading message and calling application and we have hundreds of millions of users around the world. Today, I’m going to talk to you about incrementality and cannibalization in user acquisition. Let’s start.
What is incrementality? Incrementality allows you to understand the connection between your marketing campaigns and the total growth of your product.
Let me give you an example of non-incremental activity, or cannibalization. When you’re searching for ‘Facebook’ on Google.com, you will find the first result to be an ad and the second result to be an organic result. But think about it. The intent of the user, when searching for Facebook, is most likely to go into the Facebook website.
Therefore, when the user instinctively click on the first result, which is an ad, it doesn’t necessarily mean that the ad has particularly influenced him to go to the website. Probably he will do it, with or without the ad, right? What we want to make sure, in our user acquisition campaigns, is that the ads are going to be as impactful as possible in convincing the user to use our products.
Where does A/B testing fit into incrementality?
Before diving deeper into incrementality, let me introduce you to a very powerful tool: A/B testing. At its core, A/B testing is dividing your product into two products, two versions (version A and version B) and dividing your audience into two audiences (audience 1 and audience 2). By comparing the results between audience 1 and audience 2, you can understand which version was better, which is how you back it in growth.
A/B testing is widely used among different teams and different companies; it allows you to optimize and improve your product, app store assets, landing pages, CRM campaigns, email marketing and, in our case, paid marketing campaigns. If you’re running a campaign with more than one ad or with more than one targeting, you’re actually optimizing towards better results and removing the underperforming ads.
Now, to quantify it, let’s use this formula. Result B minus Result A, divided by Result B equals your incrementality index.
What guidelines should we follow in incrementality A/B testing?
We’re going to talk about incrementality index later on, but if you’re using statistics, try to practice it right. For example, use constant factors. While the experimental factor has two versions, the other factors must be constant, since you don’t want them to mess up your results.
In addition to that, make sure that the audience has the same standard deviation or, my tip to this, use as high a sample as possible so it will be normally distributed. And try to make the results statistically significant. Use a calculator to calculate it, but also make sure that the higher results are actually better results.
What is the user acquisition challenge?
Incrementality is a relative concept and, therefore, we should use A/B testing in order to measure incrementality. The unique challenge in user acquisition, however, is that you can’t control your audience; you don’t know the unique identificator of each user before seeing him and therefore can’t segment the users into two audiences.
Luckily for us, we can segment our audiences not only by identifiers, but also by other parameters, such as: geo – if two countries or two cities are behaving the same; time – if two time periods are behaving the same, with users from two different types of cohorts; products – if you have two products, for example: an Android application and an iOS application and they are behaving relatively the same; or demographics – you can segment it by age or by gender, whatever you want.
You need to make sure that every segmentation that you’re doing is mutually exclusive. If you segment your audience according to, for example, NBA fans and football fans, they might overlap. And you don’t want to make an A/B test with overlapping audiences.
I brought with me several examples of incremental growth, segmented by time. Now, as you can see in each of these three examples, the blue line represents the organic results, the red line represents the paid results, and the purple one represents the total results. Note that, although the results are rising in each one of those examples, they are profoundly different.
In this first example, after I opened my paid campaign, I had a positive impact on my organic result, which means that not only do I have an incrementality index higher than one, or 100%, I also have a positive K-factor, which is a topic for another discussion.
In the second result, there is no impact of my paid campaigns on my organic results, right, which is good, it means that the incrementality is actually one, 100%.
And in the last example, when I’m opening my paid campaigns, I see a drop in my organic. By the way, from my experience, most of the campaigns, if they are run at scale, they will behave as we see in example number three.
We will soon check and see whether this is good or not, but first of all, let’s go over a developmental incrementality index formula, an expansion of the formula we had earlier on. Paid plus the organic you have at the time that you started your campaigns, minus the organic in the time before starting your campaign, all divided by paid, will give you the incrementality index that your marketing effort has on your entire growth.
Example
To understand it and to use this formula, let’s take a numerical example. Let’s say that at T-zero before starting your own campaigns, you’re reaching one-thousand results a day. Now, when starting the campaigns, the organic results drop to 500, while the paid results are now 750. Putting those numbers into the formula, let’s do 750 plus 500, minus 1000, divided by 750 and this is equal to 33%, which means that the incrementality index of that specific campaign is 33%.
Now, is 33% a good incrementality index number? That depends; for some companies, that’d be an amazing success; for others, a colossal failure. But, if you want to make sure that you’re not cheating yourself, let’s do some other calculations. Let’s add that the CPA is two dollars, okay? Going back to the basics of user acquisition, you’re going to make sure that the LTV divided by the cost of acquisition is equal or higher than 100%. So, taking the two dollars divided by 33% of the incrementality index, we get six dollars.
If your LTV is higher than six dollars, that’s great; keep on doing those activities, those are ROI positive activities. However, if it’s lower than 6%, think differently. This is how you calculate incrementality and you can do it for all of your activities, or one at a time.
Pro tips for improving your incrementality calculations
To end this session, I prepared several tips that we, at Viber, are practicing, in order to calculate incrementality in a better way.
First one, establish an analytic approach. Most paid marketing teams are focusing on the paid result that they’re getting; however, if you’re actually interested in growth, you want to make sure that you align the team so all members will focus both on the organic and the paid results. You don’t want them to cannibalize your organic results by achieving good paid results.
Second one is clean your data. Some campaigns that you are running are not impacting growth, alright? Some of them are just protecting your brand or brand awareness, which is long-term growth but not directly aimed at achieving short-term growth. Therefore, you need to clean those out, you need to actually separate between the two different activities and calculate them using different formulas.
Get the latest marketing news and expert insights delivered to your inbox
Third, build your benchmarks. In order to understand the incrementality, you must know the organic in T-zero before starting to run the campaigns. If you have that number, that’s great, but if you don’t have that number, my recommendation is to close all of your marketing sources and see what actual organic trend you are getting. If you cannot do that, and a lot of advertisers cannot stop their activity to one week or to one month, do that – close one of the least performing marketing sources.
See what trends there are, what’s the opposite trend on incrementality, whether you achieved less without this resource, and then start by closing all the sources from the least important to the most important and understand whether you have incrementality in your paid campaign. Stop when incrementality is decreasing by a lot.
Fourth, be aware of the fluff. Incrementality has become a buzzword, similar to machine learning or to artificial intelligence. A lot of providers will offer you incremental results, but you want to make sure that their offer is something that’s unique and with value behind it.
Fifth, if you decide to work with a partner and calculate incrementality, make sure you control the raw data of the tests. Don’t be satisfied with just the highlighted numbers; make sure that you can analyze the data for yourself. Get the raw data from the marketing source and compare it to your own data in order to understand for yourself the growth. After all, none of the partners will tell you that their campaigns achieved zero incrementality. You want to make sure that you’re calculating the right numbers.
And sixth, if you’re running a mixture of sources, if you have a portfolio of sources, try thinking about building predictive models. Start using your historical data to understand the contribution of each marketing source on your growth and, with time, you will be able to identify the individual contribution of each source in a better way.
Last word
Some of the advertisers that I present in my theory are asking me, “Why do I need another KPI? Why should I use another metric in order to evaluate my activities?” And to them, I answer that this is not another KPI; this is the most important KPI that you will find in your activity because if you don’t have incrementality, you have nothing.
That’s it for today. If you have any comments, use cases, or remarks, please leave it in the comments section below. If you want to see other amazing MAMA Board episodes, please click on this link over here. Thank you, all.
