Incrementality testing insights from 750 million app installs

Of course, you already know: incrementality testing helps determine whether your ad spend is truly driving new user growth … or merely capturing users who would have installed your app anyway.

But how important is incrementality testing?

And how can you do it relatively painlessly?

In a recent Growth Masterminds I chatted with Jonathan Reich, the CEO of an app publisher with 750 million app installs: how they think of incrementality, how they measure incrementality, and tips for how you can do the same. The publisher is Zedge, which specializes in phone personalization with offerings like wallpapers, ringtones, and AI-powered customization tools so people can create their own one-off phone customizations.

Their flagship Zedge app has been installed over 750 million times, with 25+ million active users and 15 million reviews (!!). It’s primarily ad supported, so smart arbitrage on user acquisition is critical.

Which makes incrementality testing also critical.

Check out our chat here:

Why Zedge started incrementality testing

Initially Zedge grew organically. 

It’s in a cool niche — young smartphone owners are particularly interested in customization — and had such an early start from even before the smartphone era started that millions and millions of users just naturally downloaded the app and used it, thanks to strong ASO as well as SEO.

Eventually, however, Zedge started paid user acquisition in order to accelerate growth even more.

And it clearly worked.

But there was always a nagging question …

“At first, wow, we’re seeing return on ad spend through the ceiling,” Reich told me. “But then the question that came up was, hold on a second … if we were to stop advertising, would we generate those installs and that revenue accordingly?”

And that’s where incrementality testing comes into the picture.

How Zedge does incrementality testing

There are a ton of different approaches to incrementality testing:

1. Holdout group
2. Geo-based
3. Intent to treat
4. Time-based
5. Ghost ads
6. Synthetic controls

Check out much more about all of these, how they work, and how to start in our recently-published definitive guide to incrementality in 2025. 

Zedge primarily manages checking ad partners and campaigns for incrementality in 4 ways:

1. Market-level testing
   In market-level testing, Zedge pauses advertising in select markets and measures the impact on both overall installs and — importantly — organic installs. Depending on the difference, the company restarts campaigns.
2. Budget fluctuations
   If everything in your marketing budget is steady-state, it’s almost impossible to know what’s incremental. So Zedge will dramatically increase or decrease spend to assess the impact on install volume and velocity. How? They’ll often use a “sine wave” approach, Reich says: fixed spend, small spend, big spend, small spend. Comparing the varying investment levels to what you actually know happens in your app — your own first-party data — gives you all the information you need to know about the incrementality of your campaigns.
3. Platform comparisons
   Zedge will sometimes shift budget between platforms to see changes in organic installs and paid ROAS.
4. INCRMTL
   Zedge uses INCRMTL (and Singular data) to assess incrementality on a regular basis.

“There are whole sets of different tests that we have undertaken over time in order to really test the limits,” Reich says. “And we’ve done so without breaking the bank.”

That’s important.

You want to test, but you don’t want to overspend on tools. And you don’t want to underspend on campaigns that actually are incremental to your growth and profitability.

Should you buy your name on the App Store?

There’s 1 really big question every brand has to think about at some point: do you buy your own name on Apple Search Ads and Google Play?

It’s a tough question to answer. And there are potentially massive consequences to getting it wrong.

There is a good side to the problem, but also a bad side.

“If you are first and second, someone’s going to click on you,” Reich says. “However, when you are paying for that and you’re stealing from your organics, that is bad.”

The good news: incrementality testing can give you an answer. And it’s relatively easy to get the answer if you’re in multiple geos.

Simply pause spend in select geos, measure what happens with install volume, and assess whether the price you pay for showing up at the top of your own searches is worthwhile.

Key incrementality testing learnings

Incrementality will teach you things about your marketing that are hard to learn any other way. Here are a few that Zedge learned.

1. Some channels simply won’t work for you
   And some partners simply won’t work for you. No matter how much optimization you do, some partners failed to provide incremental value for Zedge, and you’re likely to find the same thing. That’s OK. Not everything works for everyone. Just find the ones that do.
2. Different channels drive different outcomes
   For Zedge, some were better at driving users who monetized via ads. Others were better at driving subscribers. You won’t know which is which until you test.
3. Don’t overvalue precision
   Seeking ultra-precise attribution data can lead to analysis paralysis. Once you get direction-relevant insights, make changes.
4. Don’t go “cold turkey”
   Completely turning off UA will hurt overall growth. You can run incrementality testing without getting that extreme.
5. Don’t be impatient
   Incrementality takes time to measure effectively. Don’t make changes and poke them with a stick tomorrow. Let them breathe.

Finding success is great. But finding failure is also progress, because you’ve eliminated a possibility.

“We have found that there are certain DSPs that just don’t work for us,” Reich says. “No matter how we test, no matter whether we scale up, scale down.”

That’s why you test before you scale.

How to test a new ad partner

So how do you test a new channel or ad partner? In a word, carefully.

1. Start small
   Give your new ad partner an initial test budget of a few thousand dollars over a couple of weeks.
2. Expand gradually
   If your early results with the new ad partner or channel are promising, increase spend in stages. As you do …
3. Monitor multiple factors:
   1. ROAS trends … up/down/same?
   2. Impact on organic installs … increasing, decreasing?
   3. Quality of new users … engagement, retention, monetization?

You have to test new partners to find the nuggets of gold. But you have to do so carefully.

“We’ll take a small test budget, dip our toe in the water, and begin to take a look at the results,” Reich says. “And then once we validate that something is helping us, the question is, how fast do you press down on the accelerator pedal?”

Much more in the full podcast.

As Zedge learned, incrementality is more art than science.

While data is crucial, success comes from balancing your quantitative analysis with your strategic intuition. The challenge: marketers have to be capable of accepting uncertainty.

Remember, the goal is optimization, not perfection.

“When you start out, you just think, hey, this is all digital,” Reich says. “We can measure anything. We’ll have answers instantaneously. And it ain’t like that.”

Check out the full episode wherever you get your podcasts, or on our YouTube channel. Here’s what you’ll find:

• 00:00 Introduction to Incrementality Testing
• 00:44 Meet Jonathan Reich, CEO of Zedge
• 01:15 The Origin Story of Zedge
• 05:24 Marketing Strategies and Organic Growth
• 07:26 Challenges and Insights in Incrementality
• 09:48 Testing and Optimization Techniques
• 14:44 Key Learnings and Best Practices
• 20:04 Common Pitfalls and Final Thoughts