Strategic Data Filtering for Better Ad Performance — Thomas Petit, Independent Consultant

0:00Hello, I'm your host David Bernard and my guest today is Thomas Bati, an independent consultant focused on subscription app growth. Over the past decade, Thomas has worked with hundreds of clients and helped manage nine digits in ad spend with a positive return. On the podcast, I talk with Thomas about using signal engineering to optimize ad spend, how AI is changing creative testing, and why most people should avoid appto for now. Hey Thomas, thanks so much for joining me on the podcast

0:33again. Hi David, always happy to come back. There's one of one of this time of the year. I look forward chatting to you and browsing some new topics around apps. So yeah, happy to be here. Yeah, I think we're we're going to kind of make this an annual thing. We just the last three years in a row, I think we've done it in the in the late summer, early fall. So, uh we'll just make this a every every

0:54summer kind of thing. All right, that's the date. So, the main thing I wanted to talk to you about today was signal engineering. We've done a webinar on it. You've talked about it quite a bit. People have been blogging about it. But before we get there, there are a few kind of like hot topics that I thought people would really enjoy hearing from you about. And so let's do those first and then we'll dig into signal engineering. And you know, maybe this is going to be a two or three hour podcast. Let's uh see how it

1:19goes. So one of the things I wanted to talk to you about is AI and AI creative generation. And you know, there's been a lot of noise lately about, you know, oh, Facebook's just going to or Meta is just going to, you know, take all your assets and figure everything out and just, you know, 100% automate all your ads at some point in the next like 18 months. Like, it's it's going to generate humanl looking people. It's going to write your taglines for you. It's going to just do everything. It's going to customize it to every individual. and it's just going to be this like utopia of ads. And

2:00that's that's the take I've been hearing from a lot of folks. But I saw a tweet from you recently with a a bit of a contrarian spin on that. And so I'd love to hear how you think about using AI in creative generation in just in the entire ad creation process and kind of what to watch out for and where you think things are headed. There's a lot of focus on using AI specifically for this like for producing creatives and I think that's actually a mistake in the sense of it's hiding the fact that we can use AI for a number of other things that are not creative production. Of

2:36course, it's a natural because because of the cost of production but also like of ideation as well and how we can make multiple variants much easier than before. So it's a little bit of it's so obvious that 95% of the discussion is there. But my first answer would be maybe we haven't done that much elsewhere and I think especially in creative analysis is is where there's a bit of a gap. I'm seeing a couple of tools coming up and some people but it's still a rare conversation like a lot of the focus is on creative production and that's kind of expected in a way. I think if we if we look back since GPT

3:15came out like early on obviously was like mostly text and then we got like midjourney and Delhi but like static pictures at least for apps is is a relatively small amount of of the ads that are actually thrown out there and text are important especially on Google and elsewhere but like for paid social for meta for Tik Tok and for the gaming networks like text is is a very small to significant part so so obviously the acceleration has mostly been around the last 6 12 months which is where video creation became really possible. I mean until a year ago we're in summer 25 until last summer the quality of AI

3:53production for video was not good enough for ad creation and then as everything AI it sort of accelerated so massively that I don't think I know a single team who's not using any AI tool for ad production which doesn't mean that 100% of ads are ad generated are AI generated yet and I don't think we'll ever reach 100% unless one platform decide that there's no other choice but I don't think it's going to be it Like I think it's going to be a bit of a mix and probably it's going to be the majority but it doesn't mean it's going to be the entirety especially around creation like

4:27I mean you can use it not just to generate the picture itself but also to generate like the briefings and I mean multiple varants is another use. So within creative production there's like many layers of how you use AI. It's not like so it could be to create variance, could be to find new ideas and then sometimes you'd use several tools in a row like one to generate ideas and one to classify them and try them and then later one to produce them. There's there's one that's very exciting. I I saw I saw the message last week and I thought, "Oh, crazy." I I remember writing something about it six years

5:04ago, so pre GPT. So it was kind of funny which was like oh it's the first time ever there's a tool that enables you to actually test a creative without spending a single dollar and there's basically this pool of artificial humans that are used they're not humans they're just AI generated to classify if an ad has a chance to become a winner without to spend any money which is which is the dream because I mean on one side we need to produce a lot of stuff to find out what works and the starts out there are like roughly you have to produce 50 creative to have one that is actually a

5:42winner and this winner is going to take 90 plus% of the ad spend. That's kind of a normal winner takes all mechanics that happens in ads. So obviously create because we you need 50 to to find one and sometimes more teams accelerating of creating a hundred a week thousands every month to to find a couple of winners like and so that's the that's an obvious use but uh having like something that tells you without having to do creative testing which one has the most chance to win is something that is still very nent very pioneer and I don't think it's going to fully replace proper creative testing where you put a little

6:19bit of money behind it But it's definitely one way I'm looking forward about acceleration in the next year. Like we're not there yet, but it's something I can see it's coming in analysis as well. I think it's going to accelerate especially around patterns like which hooks have always worked and which not then you can come up and it's it's sort of all entangle into accelerating because of these different layers. One accelerates the next one and amplify let's say the next one. It's all very exciting as of Meta making and Mark Zuckerberg in particular making declaration that his goal is that there's no more assets at all that it's

6:55all generated. I think was voluntarily provocative about the direction it's taking. I don't think we're going to get there in the next one to three years and certainly not 18 months but it's certainly the direction it's taking. I mean Google has done a lot of it early on with UIC and is still doing a lot of it. Meta has done some of it and is clearly now accelerating and I can see why this declaration I mean Zuk said it in a way to prepare people that it's coming not necessarily that this is the ultimate form that they talk about is what's going to be there in 18 months

7:29but it's pretty logical like in terms of personalizing parts of the creative like you could find for example a script of a creative like something that fairly resonates with a lot of your audience but then if you magnify that the text can be customized, that the character in the in the ad can be customized and somebody would add something somebody looking more like them or a little bit older and then you can cater to even more

7:53audience. That makes a lot of sense. Like I'm seeing this happening today in a half manual way, which is like, oh, I've got something that wins. Let's produce 50 of them and one with an older guy and one with a black woman and one with a 20 years old and one with a whatever. And you just try them. No, but I I can see how right now it's more like a marketer's idea of, oh, I've got something that wins. Let's make variance out of it into the platform automating more of this. And so in the in the grand scheme of things, does make sense that we take that

8:24direction? I don't think it's going to be ultimately the the only way the creatives are produced either. Where do you see folks failing with this right now? Or what do you think that things to watch out for? I mean, you kind of alluded to it that these tools that kind of pick the winners ahead of time maybe shouldn't be relied on 100%, but what other things are you seeing that you think teams are doing or, you know, customers that clients that you work with have suggested that you're like, I don't think we're quite like

8:52that's not the right direction yet. It's hard to pinpoint one in particular. Well, one thing that is always a tricky balance, a tricky tra tradeoff is when you find winners, obviously they're they're so much better than the rest that you want to double down on this. So, you create a lot of variance of it. And ultimately, sometimes I look at some accounts and I'm like, okay, but all you've got there is one concept. Like, yeah, you've got a few ads, but it's more of the same. And sooner or later, this is going to go down and and you have to prepare a lot more. So it's kind

9:23of like instead of exploring so many more concept and and the majority will fail like the trick between how much do I have doubled on winners but how much do I prepare what's next doing something that has more likelihood to fail in the short term but more likelihood to be required in the long term is is very hard to balance like and obviously everybody's short-term looking especially in UAE you know we look at the results from yesterday and last quarter already doesn't exist anymore So yeah, long-term preparation of what next and audience expansion especially when you spend at scale like if the account is relatively small. I mean you can tap

10:02on the same concept over and over and with a few variants you can push it around months but everything is accelerating even in ads and I think this is especially visible in in Tik Tok where the trends go so much faster like around music around the specific hooks around the first seconds around even concept of ads like that really accelerating and so you can't just double down on stuff like it just got to create I think like AI obviously enable

10:29us to go faster in like, oh, this works. I'm going to produce 100 version of it. And even if in two weeks this concept is dead, I'm going to have used it in the meantime. So, even more useful for cases like on Tik Tok where fatigue is much faster than for example on YouTube where you probably still want to craft a lot more less ads but go deeper. The other one because the question was where do you sit failing? And here there's something that I don't want to accept, but I still have to, which is we don't understand why winning ads win. And so we have this tendency. Eric Sford is the

11:08extremist about it about don't even try to understand why it works. It's just a waste of time. I'm not that far, but in a way, yeah, I've seen teams fail a lot on I don't know, you'd use an AI tool and say, "Oh, look, those are my 200 last creatives. Those three are the best. everything else fails. Tell me why. And here it's sort of like you will get a result and you will get some patterns but I don't think those patterns push you to the next one where you can actually replicate it so easily like the very specific details of psychology how people react. The kind of

11:41audience that is there is really hard to explain. So I'm still doing it and I still understand why people would do it because then for iteration it brings you to the next one. You want to learn from what you've done to keep progressing and it and it's natural and sometimes maybe it helps but honestly it's a little bit of overstated in the it's very hard to understand why winning ads win and our human criteria is not very good but actually the LLMs they're not very good either like they could spot a couple patterns and all but it's not going to make the next winning ad that much easier either. It's

12:14sad, but in a way it's more like, okay, let's massproduce and see what sticks rather than let's try to really explain why it works. And even ads from competitors, you you'd be like trying to understand why they were and whether you're doing it with a team and brainstorm or you're using doing it through AI or probably the best of is doing both still the outcome is slim and

12:37pretty bad. Well, the next question I want to ask is what tools are you currently seeing success with? But that'll turn this into like a 5-hour podcast. And so I was thinking as you were responding to that last question that we we should just do a webinar like you maybe Nathan Hudson, maybe Marcus Burke, get a few people who are really into this and do a whole webinar specifically on like what tools are working. I think that's going to be changing so quickly that maybe doing it on podcast isn't as good as just doing a webinar and some blog posts that we can update over time. But the next big topic

13:10I wanted to bring up was app to web, the Epic lawsuit, the injunction. If you're listening to this podcast, surely you know, so we're not going to go over it all, but what are you seeing apps do currently? And are you seeing things work in sending people from the app to the web? And then as as a precursor, you probably you've been on vacation, so you probably hadn't had a chance to listen, but the podcast episode just prior to this one, I talked to Zumba and they are seeing success sending people to the web and they've been doing a lot of experimentation. And then as I've shared

13:42on other episodes, I'm also hearing from a ton of folks that they're scared to do it because they don't want Apple to stop featuring them. They don't want to get in a bad position with Apple. Um, and so there seems to be like a lot of variance going on right now with whether you should even try sending people out of the app to the web, who you should send out of the app to the web, what experiments you should do. So, you know, give us a a quick take on, you know, August 2025, where are things at with app to web and where do you see things

14:12going? Sure. If I were to like super simplify like if you got like a big brand or if you've got like some something that people already wants from the get-go, it's much more likely than app to web is going to work. So if you Spotify, it's obvious if even Zumba, but I think the app is pretty recent, but it's the official app of Zumba and like people already know what it is. Like there is a trust factor in there. If you're an indie developer and you're launching a brandless app last month, it's very likely that app to web is a full zeran in the sense that one, you can't do it

14:49all because you've got a small team or it's just you. Two, it's less likely to work because you don't have like this recognition, this brand and people are less likely to like go and three the analysis is actually quite tricky. So you need you need minimum resources to execute this minimum resources not just in designing a great flow which is like a requirement because a poor flow will will make it not work where it could have but actually analyzing properly and comparing the the two is not as easy as it looks like refund works different renewal works different free trial works very different you need to deal with

15:24like the amount of small detail and and one thing that is really interesting from the folks for whom it does work to send people outside of the app is that typically they don't sell the same plans like what work on an app to web flow is much closer to what you see in a web to app flow like where the first touch point is on the web and then you charge and then only they go to the app than what you see through APS like and so if you port the exact same flow and your analysis level is not deep enough to understand all the small details that

15:56are going to change you could actually be misfiring and believe you're doing something great when it's not or vice versa now or actually missing out on a it's not as easy as it looks. So my big advice is that if you're big and you have the resource and you've got the brand, sure, go try it like right now. If you're very small and it's just you,

16:15don't even try. It's not worth it. You're only paying 50% and the complexity, you're probably better off investing your efforts elsewhere. But obviously the majority of people will land somewhere in between. And it's at which time do we make the choice to actually experiment with this. And curiously the the teams I worked the closest with they decided not to even test. Like I was like why? I was like because prioritization is saying no to great ideas. No saying no to stupid ideas. And we can't do it all. And we've got the this road map for the next three six months where we've got like very big

16:49projects going on. we're going to wait for the dust to settle because then there was the other appeal and then we didn't know how Apple would react like I mean now I think the dust starts to settle and we see more or less other examples and Apple might change things later but for now they haven't taken like massive retaliation stuff sometimes there's a first mover advantage like and for example the ones who were advertising on Tik Tok early obviously they like it was a huge advantage I'd rather have people from teams that experiment a not do it and then listen to what they do and shortcut to what they think is working

17:26rather because there's a lot to test there like I mean the different flaws the pricing the how do you charge how do you prioritize the Apple payment versus not and so on like it's like there's a lot of factors so the the the teams I'm working with they're like yeah we're looking at it we're talking to people about it but we're going to push it into the next step of the road map where we think we can execute faster then and we also have a little more guarantee about Apple wants or doesn't want. So I had a couple of folks testing and actually in one case it's not a big app and it does

17:58make a difference for them. So maybe even against my own advice, they went for it and it does work for them. But they just crossed the small business program. So now they're paying 30%. So for them obviously was a little bit of a below 1 million. I mean when you're paying already 15% I don't think like anybody should really try it un unless you're very web based from the beginning like you had a business on the web and you already have a lot of tooling and a lot of experience and you can go but otherwise yeah I mean it's for later phases all in all it's great that there

18:27is the option like I'm very happy that we can do more some people are going to go IP only some people are going to go web payment only and the most sophisticated folks will be able to determine which user which flow flow which moment which price is going to work which obviously requires a lot more work and is not as easy as it looks. I think the only lie here is to tell people that oh it's easy send your people over there and you save 30% of fee which is like a complete lie but uh all the rest I'm happy the option exists and I'm happy to see different for and

19:01and I've seen experiments in both sides so it was really curious and trying to understand like why did it work in this field and not in that field even regardless of the brand power that you have like the brand awareness the vertical also plays a big role like In some verticals I where the the purchase is a lot more instinctive and where yeah the AP is great in this case though there are verticals and verticals there's brand and brands there's your yeah I think it's great to have the options uh but uh yeah not probably for indies it's a bad idea and and for

19:34Spotify it's the best news ever. Yeah. Yeah. No, I think that's a a great summary and a good kind of take for folks right now. If you have the resources, you have the team, and I thought that point was really great that if you're going to do it, you can't really halfass it. You really need to think about it as a as an initiative that like we're going to send people to the web, but then we're going to iterate on that flow. We're going to experiment. We're going to add steps. We're going to remove steps. and and that was the conversation with Zumba that that we had. So, you know, for

20:09folks listening to the podcast who hadn't listened to that episode, if you're curious uh to get a deep dive, you know, they really put a lot of resources in it. And to Thomas's point, you know, they've been selling on the web for decades. You know, they started 20 something 24 years ago selling VHS tapes uh on infomercials. So they already had all the infrastructure, they had the teams, they had everything and and they had the ability as a team to dedicate those resources and and then you know they did a lot of testing and like one of the things that worked for them was actually doing package

20:42selection inside the app and only sending to the web just for the payment piece. So the person had already kind of like selected the package, kind of made a commitment and then the web part is only payment. it's only checkout and it's defaulted to Apple Pay and that's where they saw like the the big win start to happen. So if if you are going to try that's probably one of the first experiments to do is to do package selection in the app and then make the web just the check out and then make that as like quick seamless least confusing no options like you just pay

21:14and then get back to the app. But I think that that was a great summary. So I appreciate your thoughts there. The last thing before we get to signal engineering is hybrid monetization. I mean, this is something we've been talking about for years, but I feel like it's still for for non-game apps and then for, you know, there there's certain categories where hybrid monetization has worked and has, you know, there are some kind of best practices starting to be established, but I feel like it it's going to be one of those trends that's going to continue

21:45to accelerate. And AI is a good example. And to be honest, I'm so frustrated with Claude. I use Claude a ton in my day-to-day work now, mostly just like a brainstorming partner and like get summaries of podcasts I listen to and things like that. And lately, I've been hitting their usage limits and there's no that I don't get just a freaking button like like give me give me 20 more usages for five bucks or whatever. like those kind of things in especially for AI apps or where there where you are resource constrained having some kind of way to get past the those limitations I think is just an obvious thing for a lot

22:23of these companies to start doing but I think there's so many other applications so what are you seeing work and where do you see things continuing to go with figuring out hybrid monetization in apps I agree it's been it's been pretty slow I I've been advocating for not doing only subscription for quite a while and most most the apps I work with they're like it's only subscription and that's it and and and I could see and I could see a few success here and there for very different cases some with ads for the people who don't subscribe but when you retain a lot of subscription apps

22:56they just don't retain the free court at all their their fake premium is you pay or you're going to churn anyway so obviously they're for specific cases AP mostly for upselles and so on and then a lot of ecoms and affiliates head sales and so on, which makes sense for very specific cases as well, but not for everybody. So, I think like you have to pick up the the model, but overall I'm I'm looking back at the last two years and I'm a little bit surprised that it's been so slow and I think one of the reason has been the complexity around it. Like same same as the app to web

23:28stuff. It's it can't really be done easily. like there's no obvious way of oh I'm going to put ads in the app and it's going to increase my conversion or I'm going to put on top of the subscription and no it's really tricky and I think that's the reason it's been so slow and actually all the apps that are have some kind of AI usage are finally accelerating this trend because they've got no other choice and it's true that the big models like cloud and chip and so on they mostly decided to go for tiered subscription so I I don't know, JGBD, you pay $20 a month and uh if you didn't

24:03have any, well, you stop paying $200 a month or whatever. But for a lot of apps that are like usage based, this is a model, but actually the AIP makes a lot of sense where you can buy credits on top. I'm working on a project right now where they have both T subscription and credits on top so that no user face the frustration that you just had that maybe you didn't want to fully upgrade to the super pro package but then for one use case you're working on something today and you would just have added five or 10 bucks of credits and and finish it and

24:34they've got a very smart team of several people that have been thinking through it for months and it's really hard to design well like to avoid cannibalization to not frustrate users to make something that is fair in term of pricing for users but also for the developer like it's very tricky so I think it's mostly down to this complexity that it's been fairly small that it's a natural case for AI just because there's cost so obviously you need to to find a way that you don't lose money but the rest has been slow because of complexity but also because it's sort of the last person of grind

25:11that you get like there's so any lowerhanging fruit around your subscription and I'm glad today there is a normal conversation to talk about pricing and pay wall design and how you package and the free trial and this and that and 3 four years ago this was like still a little bit niche and it surprised me how people didn't realize that they could like increase their revenue massively by iterating a lot because there's been such such an acceleration there and the the overall level has gone up so fast I think eventually it's still going to come that hybrid situation is going to is going to come just taking a very long time

25:46because of complexity and for many people that's okay you should nobody should jump into it I think just at some point I've seen how the the improvement curve of of those experiment on pricing is slowing down that's probably an indicator of maybe maybe it's time to try something different but if you're still making big wins on pricing and packaging just with subscription keep going at it until it slows down and and funny enough I als also use chat GPT quite a bit, but I'm on the pro plan with chat GPT because I use deep research so much in chat GPT and so I probably so chat GPT I actually pay for

26:24out of my personal business because I use it so much more for like personal stuff and for like my side project business even though I actually do use it a ton for like revenue cat work but then claude is actually built through revenue cat as a like employee expense and so I probably should just ask for the higher tier and that's kind of what they're counting done. So you're to your point, if you're not already in a tiered subscription uh model, that's the first step before going to to hybrid. But one of the places I I do think and I've heard this from Tammy Taw at Google. You

26:56know, she spoke last year at App Growth Annual. She did a podcast mini episode for the state of subscription apps report. And one of the things Google has been pushing really heavily because they see the numbers and they see it succeeding so well is working on hybrid monetization outside the US. And so maybe that's another kind of angle where if things are going really well in the US and you're still have lowhanging fruit there, but you have the bandwidth, then experimenting with hybrid monetization outside the US, maybe what it takes to start seeing more wins outside the US of offering people non-recurring purchases, non-recurring subscriptions, offering these one-time

27:36purchases that make it a little easier to to bite off, especially in the countries where subscriptions aren't as widely accepted and and used. So that'd be another kind of vector to think about if whether or not you should be experimenting with hybrid monetization. It made me think differently about it and once you hear it, yeah, it's pretty natural where that where where there's a very high not purchase power but like sort of habit of paying apps obviously the reward is probably further away but then on Android where it's a little bit harder to convert and outside of the US and when you combine the two of it is

28:11much harder like obviously the conversion rate on Android international versus US iOS are extremely different. does make sense to start there. But then it joins my point that it increased complexity in the sense of okay so on US iOS I need to do the tier subscription and then for Latin America I need to add this like payment options because there it's what works because they've got less credit cards especially in Brazil Argentina for example where credit cards international credit cards are a bit limited and then for another region it's going to be something else that's going to work. So obviously if you're a big company and you can dedicate you can

28:46zoom on an area and find out what works for them that's great but I have a lot of apps I work with that operates in 200 countries and even though there is uh uplift to be found the amount of effort that needs to be deployed to find all these individual uplift left and right is a huge work so it really depends the type of organization you're in and the focus you're on. I I know a lot of smaller folks are like, "Yeah, I operate everywhere, but 95% of my effort is going to the US." I understand why for them hybrid monetization is like something that's on the road map for

29:212031 or something. Yeah. No, it makes a ton of sense. Funny enough, I mean, all three topics we've we've had so far are around prioritization is that, you know, they're all good ideas, but you need to make sure you've got the resources in place and the can put in the effort to actually make it work because it's not

29:41just going to be this automatic big win. And I mean, that may be another key too is like if your core product isn't working in the US, like you need to make it work first. like it's not this magic unlock unless for some reason you know the app has taken off in some other country or there's a use case outside the US it makes a ton of sense but other than that it's not some magic unlock that if it's not working in the US if you're not able to charge subscription on iOS and people aren't willing to pay that it's some magic unlock that you're going to do

30:14this outside the US and see all this success so it's like you got to have that core product the value the activation the onboarding flows and everything else pretty well dialed in and it should be working hopefully in the US before you even think about trying this around the world. It's very hard to pioneer anything really. It doesn't mean we shouldn't try but like these models sometimes it's natural like you look at dating for example is is a vertical where there was always a lot more hybrid monetization than elsewhere because there were ads and there were specific IABS and so on and probably if you're in this vertical

30:51you should try earlier than others because there's a lot of success that has been found and users are more used to it and you can follow the patterns that have already been established exactly you can follow some patterns like if nobody's doing it in your vertical there might be a reason that if you're the first one, I mean, you you could create something new and find success, but it's going to cost so much more to find out that Yeah. Sometimes

31:13just not natural. Yeah. All right. Well, let's jump into signal engineering next. So, you emailed me in the spring and you were like, "Hey, David, I've been toying with this concept and I don't even remember if you named it in that first email or if you came up with a name later, but I feel like I'm hearing the term signal engineering all over the place now." But I guess take me back to what is signal engineering and then I mean I know people have been doing versions of this for a long time, but like what what's the origin of of you in the spring thinking that this was kind of the next

31:47hot topic or the next place that that user acquisition team should be putting some effort into. So what we're talking about here is the optimization of what what we call the signal is the data we send back to a non network to an advertising platform to optimize the ads from. So when you tell Facebook, oh I want installs, they're going to send installs, but then it's probably not going to get you very far. And beyond the install, it can be many things. So some people optimize for free trials, optim some people optimize for day one retention or whatever. And this has been very much always existed but in my

32:25opinion underlooked. And so it's it's nothing new but I kind of felt that the discussion over the last couple of years was 90% creative. And it's obvious that the creative is by far one of the biggest lover if not the biggest lover in what's going to make ad spend or paid media work or not. But then doesn't mean is the only one. And I was like, so my intention here was like I want to remind people that there are several factors at play and that if you're sending terrible data back to Facebook and Google, you can be great at creative, you're still going to have a problem. And the other

33:04realization was let's say there's three position of serial engineering. One is the data you're sending back is completely messed up. And that's actually a quite frequent case in small and in bigger organization. And honestly, if that's the case, you're probably better off fixing it before iterating on anything else. Like if if there's something wrong with the signals that you're sending back, it's your top priority to at least fix it, understand what you're passing. You don't have to get super sophisticated, but at least you need to be sure of what you're sending back and having a quality of event that is passing correctly and so on. It's like it's a must have like it's

33:44not like ah nice to experiment all no if it's broken you got to fix it and then there's the majority of cases of people who don't have broken data but haven't thought much through it like okay let's optimize through the free trials and we'll see or or have something set up I I have one particular case where they actually find and it's somewhat not super sophisticated but like a little bit more elaborate scheme where they were sending back different signals when people were taking different types of subscriptions. So like they have a mixed up of of user type and weekly yearly and they just have a multiplier say okay if

34:21it's the weekly subscription from a normal user send one and if it's a yearly subscription from a business user send X5 back to the platform. So, but then they hardcoded it and they iterated for a year and a half on the pricing and so on. And when I came in, I realized that this hard coding that did make a lot of sense back when they did it was actually not representative of the value of users that were coming in. And then it became again a priority to fix in the sense of to to get closer to the real value. So, and and the last phase is

34:56when you get really sophisticated. So I went a little bit around but like if it's broken you really need to fix it find a place where you're comfortable with and dedicate your attention on creative and declassifying and other things but at some point as you grow becoming a little bit more sophisticated about it and having sending something better than just all your free trials combined is probably can unlock value at a different and and most people look to unlock value everywhere. So I it doesn't mean here you need to be super big and have a super big brand to do it. I've spoken to a few people and I have one

35:29case that will be on the revenue cap blog at some point when I finish to wrap up where a fairly young app has found tremendous uplift by changing the events they were sending and they were working in parallel on creative but then it accelerated what they could do uh in a way. So three state broken normal and

35:48sophisticated. Yeah. Well, and and that's what I I I wanted to step back to the to the broken part because I don't know and may maybe this is just my vantage point inside Revenue Cat because we end up talking to so many apps where their data is just completely broken. So my estimate is much higher just because that's who we talk to I guess more. Tell me a little bit more because I'm sure there's going to be people on the podcast listening to the podcast where they're going to hear you say that and think, "Ah, I I don't feel like I can trust my data." So, so

36:21what do you mean by that? And then like where do you see data being broken in ways and you kind of already shared shared one but what are the steps or what are the tools or what are the the kind of basics that you should have in place for for even monitoring to make sure that things don't break even when you think they're working. How do you figure out whether your data is good or

36:43not? So, it's always hard to prevent future breaks, but uh I know when somebody tells me like, "Hey, go go go check my ad account. Tell me what you think or make a note." I don't make so many of them, but like sometimes, maybe it's a friend, maybe it's a portfolio company, so I still do it. And they say, "Oh, here here's the ad manager." And I say,

37:00"I don't care about the ads manager. Show me the events manager." Like, so Facebook call it events manager. In Google, it's called goals. That's the first place I look. I don't even look at the ad account. And the first thing I'm going to check is does the amount of conversion which whatever this conversion is. So let's say it's going to be install and free trial and paid subscriptions to simplify. And I'm going to go check in the event manager. Do the number that the platform reports being sent match more or less what we have internally and very often it's not the case. I really don't care about a 5%

37:35discrepancy here and it's never going to 100% match and that's okay. And there's number of answer of why it's not 100%. It doesn't matter. But in so many cases, I'm seeing 30% or 50%. And this this is going to be a major problem that I need to fix before seeing anything else. So I take Facebook as an example because it's a big example, but I'm going to go to the events manager and I'm going to say, okay, show me last 30 days or last seven days. How many install did you see? How many free trial did you see in total without attribution? Not the ones that

38:04Facebook thinks. not the one that Google attributes to this particular paid activity but like what we're actually sending them because they reported there and I'm going to compare it with something like count or amplitude or mix panel or whatever they have that is internal before we send it. If I'm seeing more than 10% discrepancy here it's going to be a problem. Those tools they're not particularly sophisticated but they're not

38:28particularly easy to understand either. So it's sometimes kind of tricky to match the dates, the geographies, the platform properly and so to debug that actually the data I'm seeing there is the one that is supposed to be there. I remember making mistakes myself sometimes like I don't know one case I was like why am I seeing double the event on Facebook? How is this even possible that there's twice as many users that we actually have? It's just that there are two sources that Meta managed to the duplicate but that doesn't show up like this in the event manager. So the first step I do is what

39:03the platform is receiving is it matching more or less what we think is happening because if that's not the case well after that they're going to try attributing to a campaign and there there's a whole lot of other things that can happen. One that you don't trust attribution which I would understand but one that among those signals that you sent maybe some didn't have the right parameters. Maybe I don't know I remember making a change and and yeah we're like oh only 20% of the conversion are passing like 100% is passing to the event manager but only 20% of what we think is happening is being reported at

39:37the others and actually meta couldn't attribute it on AM and it was only reporting the double opting like the IDFA people until we fixed it and so yeah the the step one is is the total actually matching uh because and I'm saying it because if it's not nothing else can And it's kind of an easy check to do.

39:56Like there can be many other problem. It's just one. But that that it's it's so common and it's so blocking of doing anything that that's always the first thing I'm I'm checking. What drives nuts is that when it doesn't work, the amount of reason that can be for it not to work. But that's something you need to fix yes or yes like uh one way or another. That could be changing the source. That could be sending different parameters to changing the event entirely or whatever. It's also because the default configuration of like for example the the meta SDK or basically you can put the meta SDK and tell meta

40:30okay just map my event on your own and you have like the install for example is the hardest to debug because it's not an event it's something that cames prepackaged so when it doesn't work you've got less levers to move but for example the free tri the paid conversion can be mapped something that happens very often is that a direct purchase without a free trial and a free trial that converts to pay and a renewal would all be mapped to purchase together. How can you actually read anything when you've got this mixed bags of event which I sometimes use a mixed bag of event for optimization but as long as I

41:03understand what's what's actually passing through and I'm going to want all of those individually to make sure like okay I've unbugged that they work properly now I can start getting a little bit smarter around what I'm sending and the default configuration is not as easy as it looks I think is one of the case where making a little bit of effort because it's not something you have to work every other day like it's something like you need to fix, but then you can run on this for 6 12 months. You can never forget about it for the reason I mentioned at the beginning. But uh it's something that you fix once in a

41:34while and also because it gives you options like I was working with an account where they had like massive success scaling on couple different platforms like really impressive trajectory went from zero to eight digits in a couple years like a very beautiful trajectory. And then at some points things started working not as well and we tried everything we could like you know creative and fixing the event and so on. One of the problem is that they put all their eggs in one basket and that basket worked great for a long time but when it broke they had no alternative. And one of the reason to have your your signals properly handed

42:13is also to have optionality in the sense of if what works the best for you is optimizing for free so be it. Or optimizing for value like for revenue optimization so be it. that's great, double down on it and spend 90% on there, but you probably also want to have the other options available sometimes because it's audience expansion. And I would run an event optimization campaign towards free trial in parallel of a revenue optimization campaign and Facebook would bring different users to the app. So for scaling, but also for the many many case where things starts to break, which is much more often than people think. So

42:47I'd rather have different options in my in my toolbox basically. maybe not for today if something works like sure let's let's go all in on it but to have this for for later also to prep to experiment a little bit on the site so yeah uh so one of the reason I started talking about it about this topic was one I had the case I had two cases in in my portfolio where one clearly had their events broken so we needed to fix it and then we thought if we're going to fix it let's fix it and make it smarter at the same time and then I had another client

43:20who who work with a third party. They're already doing some kind of optimization on their own and actually already a fairly sophisticated system. I thought, what if there's something I missed? Let's go and ask some of the best out there about what they're doing and we actually handed it over to them. We're running two tests. One is looking good, the other one not. So, it's not completely uh let's say conclusive, but uh they definitely did stuff that we wouldn't have done on our own and and go one step beyond. So these two things about fixing getting smarter and going on stium they all came at a moment where

43:54at the same time I felt that 95% of the debate on paid media was about creative production. I was like sure should be talked but that's not the only topic. So I like to take my contra hat and say I'm going to talk about something else. Suddenly I also noticed a lot more talks about that. So uh either really good timing or just people got inspired and worked on it. I think it was a little bit of both, but uh I'm I'm happy the topic has picked up a little bit. It doesn't mean that if you've got amazing signal engineering and you have one ad

44:26in your account, it's going to work. That's not that's not going to work. Like Yeah. Um before we move on to to like more specifics around how to do the signal engineering, I did want to ask one more question on the data again, just because I these are kind of questions I think people would be, you know, shouting into their car as they're

44:44driving around listening to the podcast. is that h how do you think about this when you're scaling on multiple networks? So you're you're doing Google and Meta and Tik Tok. Should each of those events managers be seeing 100% of your conversions, 100% of your free trials, 100% of everything? Is that what you're looking for in each individual event manager? or are you looking for Google to be seeing just a percentage of the events and and Meta to see be seeing

45:16a percentage of the events? If you're early in this process, you probably want them all to receive 100%. Like you don't filter anything. You send 100% of these different events. Uh yes, it's almost normal that the same signal is not going to be the best for every platform and it's possible that you end up optimizing for revenue in meta and for a qualified trial in Google and for a filter of direct purchase and trial conversion in Tik Tok to say something like what's going to work is not not going to be universal like every network have different type of user but also different context but also different

45:53ways of optimizing like of of valuing the signal. that you send them and on top of it they don't have the same source. You might be using an MMP and send to all these sources but it's also very possible that you're sending to Google via Firebase and to Facebook via the Facebook SDK and so on. So like the easy answer is yes send 100% of the events to all the different platform in reality one they're going to be different source so they're different by nature and different things are going to work. So, for example, you might have to filter younger audience trials to Tik Tok, but you don't necessarily need to

46:28do it for meta because it works differently. Like what I do is I map these different events. So, let's say I've got my free trials. I send them all, but maybe I'm never using them in any campaign. And then, so I can verify the quality of the data that's being passed. And then I've got my qualified free trials or filtered free trials, let's say, being sent to the different networks. But I map it. When I say map is like I declare it as a different event. So maybe all the free trials are going to be called start trial but then the qualified trial is going to be

46:56called whatever it's going to be maybe a purchase maybe subscribe maybe something actually I I wish that the platform offer us more options in in the number of events that they propose because probably because it's not so common that's why the list is is limited and if we remember two or three years ago Facebook didn't even have start trial subscription it was all purchase from gaming and ecom. So the baseline then is that as you start to think about signal engineering, the first step is that 100% of your data is going to all the different networks and you just verify that you're within that like 5 to 10%

47:29threshold across all the different networks. Then you move on to starting to filter the events and then by doing so then you have a better sense for what numbers should be showing up on each platform because then you can determine I'm not sending free trials of users under 25 to Tik Tok and so you can do the math then of like okay we know that when I was sending 100% it was working and so now I'm only sending you know 75% and so now I have an idea of like what should be arriving at that particular ad network and then that's how you do your data validation once you start doing the

48:06signal engineering right yeah correct I mean all of this well I remember doing this in 2017 and and it didn't come up from a brilliant idea in my mind it just come up from the observation that my free trial rates on Google ads was terrible like we would have like let's say 40% trial to paid conversion on organic on meta like I mean small differences and then we'd have like 15% on Google it was like what what are we doing wrong here and like okay for Google we're going to identify the some of the reason those trials are not converting and we're going to send them a different signal to try to fix

48:41it. It was not like we're not trying to be smart and just do senior engineeric but we started from a problem initially like maybe if you don't have this problem of maybe if your free trial quality coming from from paid is great you don't need to go further for now like maybe you you could do even better but like I started from a from a problem and then seeing that this was moving the needle in ways I said okay now I can get a little bit smarter one of the next step is that historically all these platforms so They they offer event optimization. So they offer instant

49:14optimization which honestly you should never do. There are a couple edge cases where you might but in most cases a very bad idea. There's event optimization which can be different event. They can be filtered and so on. And we can talk a little bit more. And there's revenue optimization. And the problem with revenue optimization is that from the gaming world and e-commerce world this was just revenue. This was just revenue that was generated by the sale. But in subscription, because we've got the the free trial and because we've got the renewals, the revenue that is actually happening that day is actually not what we need to send back to the platform. So

49:49I started getting smarter and say, "Okay, I'm going to engineer the revenue that I'm sending." So it's not a predicted LTV is a predicted value at day x. Maybe it's months two. Maybe it's personally I like to use months 13 because I've got the first yearly renewal and I think it's a better comparison between monthly and yearly. But whatever, it doesn't matter what I like. The real revenue, the one that passed by default on the SDK is not the one that I want to be sending because the free trial conversion is going to come too late because it's going to

50:16overvalue the yearly over the monthly. But maybe or or let's take the weekly. Sometimes the weekly the LTV of a weekly is great because the price is so much higher. If somebody renews for six months on the weekly plan, the revenue where we want to send to the platform is not the real one. uh it's going to typically platform are going to overindex on yearly because all the revenue comes on day one but maybe your weekly plan has a higher LTV when people run you alone. So you're just telling the platform something that is wrong about what you're looking for. And here the idea is not only to trick and I I

50:49remember talking to Andre about senior engineering and and and they summarize it that oh Thomas is manipulating the ad platform. I'm like no I'm manipulating data to send the platform what is the closest value of the users they're sending me. I'm not trying to lie here. At the contrary, I'm trying to fix something that is broken in the sense of the platform is receiving a value that is not representative of my business value. Like the default configuration is not what representative one thing I'm working on right now for example is that users that are not converting within 24 hours but are demonstrating a very high likelihood of retaining for long as a

51:25premium user or converting later. We're going to assign a small value to it. So not the 10 per month or 90 per year or whatever but maybe $1 like because I'm like I like this user and if I send zero there's zero event there's zero value the platform is going to conclude that this is the kind of users I don't want and maybe because they create the network effect they create dau they create late conversions that are very hard for the network to assimilate I'm going to have to send something that shows them hey this user is really zero but that one is actually a little

51:58something the whole scheme there is I want to tell Meta, I want to tell Google, I want to tell Tik Tok about what's valuable for me because it's not natural for them. They just received a very primitive signal. I heard from Meta last week that they were working on optimizing towards different event at the same times like for example different tiers of subscription and stuff and I was very happy because it's something that I always wanted to do and that you can almost only do by engineering the signal and engineering the the revenue behind it. Uh so so we're getting there. It's just I want to

52:29send something that is representative of value so that the platform can do the work they do the best way they can. And they're really really strong at predicting which users are going to complete the action that I'm telling them. But if the action I'm telling them is wrong, you're running on one leg literally. And that's where the whole idea of signal engineering and optimization of the data that you're sending back comes from. It's like, yeah, send the network something better and they're going to do a better job

52:55because they are doing a better job. It's you who are not doing yours. Nice. So, you've you've kind of gone into and given a lot of what I would guess were are kind of more advanced signal engineering practices, but let's step back and like once you get your data right or believe you have your data right, what's the lowhanging fruit in

53:16signal engineering? Where would you say are the first kind of few steps folks should take to start changing the default mappings of how these SDKs operate by default? Like what's the first step? So we mentioned that's what's the second step because the first step is is monitoring that the events are passive properly which is not a given but the second step for me would be either going to what we call qualified trials or filter trials and typically that would be when you see that your free trial rates are either not as good as other networks or not as good as what they used to be and also

53:56because you don't need huge engineering around revenue prediction and so on to do this Like so basically make a regression of okay let's analyze this free trial that are not converting sometimes is a combination of like what I call hard factors which are like the device itself maybe the language like not something the user has told me but something that is like in the hardware itself it's well known that the most recent device converts much better and so maybe I'm going to filter all this very old iPhone actually it's funny when you look at the the specific device the very latest device on on Android the

54:30Google Pixel and the the latest Samsung Galaxy and a couple of others, they convert just as well as an iPhone. Those like they only represent 5 to 10% of the market and the rest have a terrible conversion rate, but I certainly so so you could like try to weigh in those and Facebook has been working on the signal engineering because there's very few other levers that we can do, but Facebook has finally put out of beta one that we used to call it bit modifier for a very very long time. And I could tell Facebook, oh, when you're seeing one of those Google pixel straight assign 20%

55:05plus than any other users because I know they're going to convert easier, they're going to renew longer and so on. But when you're seeing one of those old Android 7 or whatever, don't even show them the ad. But then very often your choice is binary. Like let's take the age because in subscription age is a very common factor for people entering a free trial and not converting. for a very long time the solution was okay let's stop advertising to people below 25 and then I don't have the this problem anymore I was like but I'm actually missing out on something is that there's a lot of

55:35inventory part of what I'm doing I mean my app fits 15 to 25 years old it's just that I have this free trial issue that is going there I just want to factor it I just want to I don't want to exclude everybody below 25 because I'm I'm limiting my options and so this is just another way to do it differently so so Somebody asked me recently, "Oh, but now that I have the bid modifier, I can tell Facebook that the sub 25 are 30% less valuable. I don't need to do all these filtering of trials and like I'd prefer to hammer from both sides personally,

56:08but sure, like it's it's kind of a potential replacement, but it's not available everywhere. The low angle is probably going to be on qualified trials because it's such a common factor, but it doesn't mean it's for everybody." And what is the hard factor? And the other one is soft factor. And the sub factor is something the user does. The most common is a question at the onboarding that somebody answers and that filters maybe they declare this interest and not this one. Like I have a mental health app where when people say they have a high level of anxiety there so much more valuable than when they say

56:40something else. And so we we factor this question. The next level the very sophisticated level would be to actually add questions at the on boarding that creates this variance. And that's what you're looking for. So the the lowhanging fruit is looking at in a regression of the value based on questions like this. And in revenue for example, I was working on with somebody was like super committed. Yeah, I need I need to do something very sophisticated with my revenue. Send to the ad platform. I've got a whole team of expert data analysts and engineers that we can put some predictive LTV back to Facebook and so on. Yeah. Let's look at

57:15your revenue. Actually, you've got zero violence. like you've got a binary case where almost everybody ends up generating this in the same range of value. In this case, they no need to overkill it. But if you notice that you've got extreme variance, then it's probably a case of jumping the qualified trial and go straight at making these revenue buckets because you've got a bunch of users that are going to consume $1A. You've got a bunch of users that are going to take the $20 subscription per month. And then you've got the pro user with the $200 clone subscription. I don't even know what it's worthy subscription. And then if you've got

57:53extreme variance in revenue, I think the lowhanging fruit is to deploy the resources needed because the qualified trial are not going to be enough. Like it's just like step one. So it depends the problem you have like or the problems you don't have. If you don't have variance, don't bother with it. If you don't have a free trial conversion problem, don't bother filtering your trials. Let's dig into exactly how you do this. So, I I liked your example. If you're a mental health app and you have an onboarding question that says, "What are your goals in using this mental health app?" And one of the goals is

58:26calm my anxiety. Another goal is just like, "I just want to be healthier." You know, some like really broad generic goal. And so, you see the people with that broad generic goal, you know, their value is just way lower. And the ones who you know have a specific problem to solve like anxiety their value is way higher. How do you actually instrument this? Is this inside the app that when you map the events you're not actually sending the events and you're and and do you have to prevent the SDKs from just hoovering up all the events so that you can send the specific event that you

59:01want to send? Like how do you actually do this? Sure. I'll answer you but I'll make a a cautious word before which is filtering out like excluding people can work and I've done it a number of times but is a little bit my last resort like if I can find another way to not fully exclude these people because they might generate less value less is not zero so excluding them is actually something that I try to avoid in the first place and when I look let's say an on boarding question and I'm going to have okay high anxiety I want to be healthier or I just want to

59:32sleep better and I'm going to Look, if something is extremely less value, in which case I'm going to try to exclude it. If it's a little bit less value, I'm going to try to find something else. Either I'm going to let it pass and deal with it. I'm going to move to revenue optimization or I'm going to try to find if it correlates with a particular demographic to use the bit modifiers or I'm going to try something else. Like the exclusion is kind of if I can't do any other way, then I'm going to exclude. Let's say I need to filter out some people. Basically some people in

1:00:02the in the onboarding they're just telling you oh I'm just browsing around I don't know really the intent like this basically what I'm tell they're telling you and those I'm probably going to want to exclude in this case either you've got the platform SDK so like or typically you're sending it through an MMP more rarely through like a custom API the custom API you send whatever you want so like I mean you filter the event yourself you send it but it's the least common case through the MP what I'm

1:00:28doing is actually asking my engineers. I'm going to want to add a new event. I'm not going to filter the existing event. The free trial is going to remain a free trial. Okay. If there is a free trial and the answer to this question is this, then send it to adjust apps or singular as being subscribe or whatever and then Facebook is going to start receive it and I'm going to have it. It works the same in the Facebook SDK like you just can have the default configuration but then you can set up your own events. And I'm not talking about custom events. just you can force

1:01:01but that's done on the engineering side that's something the marketers are not doing that's something that requires to open the code and actually do it so I know marketers are getting smarter with code these days but and this is one thing where I called it signal engineering because there's engineering behind it in the sense of I'm actually tweaking the system a little bit but also because very often you do need an engineer it's an interesting dynamic how very few developers are actually interested in all things marketing there's always a little bit of of attention in this ticket because we marketers are terrible to explain our requests and to like properly define the

1:01:37requirements and so it comes into back and forth but also I don't know a lot of engineers who love the tweaking the Facebook SDK SDK that's very rare like something I got a few buddies now that I can rely on but uh it took many years to get to this point and it's actually very easy to execute but the fact that the marketer who's requesting it and the developer who's actually making the change are not speaking the same language at all. It look very complicated and even now I'm surprised sometimes in a couple of this conversation where I'm just answering oh I no idea how you do that but I know

1:02:10it's possible here in the documentation it says it's you can do it here I've described what I want that's your job to deal with it not mine and and it always gets down but like my technical knowledge gets to a limit pretty fast when it gets there but I mean with a with an MPS SDK is not very complicated either I mean once you figure it out once it can get more complicated later like for example for value for the value optimization. One thing that happened is that it doesn't go through an SDK but through an API that sends back the information to the SDK. I'm like okay so I've got

1:02:46this combination of we could hardcode it. We could say okay if there is a this plan that is done on this device having answered this particular question at on boarding then the value is 56 and if it's the other combination it's 23 and we could hardcode this but then every time I need to update and that happens constantly because we change plan currency value change and so on I would need to have the code touch and then another release. So if I know I'm going to do a lot of signal engineering later, what I'm doing is actually there's an API call and say, okay, I've got all

1:03:15these parameter, what's the value right now? Oh, 56 and next week there's going to be 68. And I'm not going to have anyone touch the code anywhere, but I can dynamically change it. And one particular case where you do want it like this is when you operate in countries where the currency tend to fluctuates a lot. Turkey is a very big one. For example, Turkey where yeah, if you hardcode the value there, you're dead. But that's kind of sophisticated and it can break because then you've got the SDK talking to an API like it's not necessarily where you want to start. But eventually uh it does make a big

1:03:48difference. It's a great insight too that uh as marketers it's something to be cognizant of and to work toward is better understanding at the code level what's actually going on so that you can work more collaboratively with the engineering team to successfully deploy these things and then to kind of understand what's going on under the hood to be able to troubleshoot when it's not working well. And to your point, the best implementation is one that that is way more flexible and not hard-coded versus like writing a bunch of hard-coded stuff into the app that every time you want to make any little change, you're having to go talk to the

1:04:29engineering team to make those changes and then when something breaks, you got to push an app update. And so, are there any kind of specifics around building that out that you've seen work really well? like are there with certain clients that you work with do you have kind of tooling in place that's making this a lot easier? When I ask the developers they tell me no but dude this is a very simple API is like there is one call I reply the value it looks it looks once they done it they're like oh what you asked was actually super simple why did we take 3 months to get there

1:05:02because your explanation was wrong so it was also my problem you know it was basically a communication problem I haven't used many tools around this mostly internal tools and then you have to adjust depending on if you're sending to the platform SDK like firebase or meta or via an MMP or via uh even like cat and so on. So you have to adjust but the rest was mostly internal. I'm working now with a company called Voyantis who does this filtering on their own and then send it back themselves. So I discovered a few new tricks. There are very few people who are working on this. I haven't heard

1:05:38particularly of tools that sometimes you might use a tool that's made for something else in in the middle. But technologically it's actually fairly simple. It's just it doesn't always work. There's always a little bit of debate. But uh but in terms of code is actually something that is that very complicated. For me the most interesting part is not so much the execution which is I mean there are issues is regular but there's nothing as a breakthrough there. It's not like it's not that you code it better that the signal is going to be better is really how do you define the signal that is going to make it and

1:06:09all these questions around on boarding and about segmenting users creating buckets of value is actually really interesting because it has a crossing of like I mean there's a lot of data analysis but there's always new stuff we discover like there's so many factors and I like to play around it across different apps and I'll tell you one for example like we noticed with somebody I was working that The completion time of the on boarding was a decisive factor in end value and we were like okay those people who are answering it in less than a minute and there was like 30 screen there was nothing like yeah probably

1:06:44they click next next x next next they start the free trial and they console and I start bringing this to other apps and they're like dude I checked and actually my fastest body completion are actually my best users because they already know what they want and so on. It's not universal necessarily and this criteria is not a great one but there

1:07:00are a couple that are usually very big. So age we mentioned the thing around goals is is very often a very big one and a lot of apps have different types of users. They would have like more pure consumer users like full B2C but then they would have small teams a lot of solopreneurs people with a Shopify shop or influencers who are actually they're one person business like and there's a lot of them if you manage to identify who are like these small business one person business solopreneurs they typically have a completely different bit of course if you're improving their business because the post gets seen more

1:07:38or the video higher quality or whatever Yeah, the the price sensitivity of this because it's their business is obviously very different from somebody who's just trying to make their photo look better. And this is one that is one of the most common things like so yeah, age goals and I don't know how to define this like typology of users between like consumer and proumer and real businesses like and big businesses like uh uh this one is a very big one. Yeah, I know you talk to the MMPs a lot and have some limited

1:08:09insights into some of these ad networks. Is this something you think the MMPs and ad networks could do better? Like should, you know, Appsflower and Adjust and Singular, you know, should they be building these kind of signal engineering tools into their tooling to to make this easier? And then you already mentioned that like the bid multiplier and things like that that Meadow is doing. Do you think these kinds of things are going to get easier and easier over time as the value is recognized and then both the MMPs and the ad networks are going to make this kind of stuff easier? And then just to complicate the question a little bit

1:08:45more, how much of this will ultimately start to get solved with meta specifically, but all the other, you know, major app networks are using more and more AI to figure and machine learning. really it's machine learning they've been doing a long time but uh we'll just call it AI. How much of this is just going to get better and better by default as the platforms are able to figure these things out with better machine learning and more sophisticated

1:09:14tooling on their end. So there's more or less three question I'm going to answer in in a different order that they came in. as of the ad platform offering more options to like put a weight on each value like the bit modifier on Facebook which is now called value based rules even though those two things are not the same but doesn't matter I think we're not going to see a lot of it it's not the direction it's taking the direction it's taking is hands off marketers let us do it we're doing it better than you will ever know like which I tend to disagree with not that I

1:09:44wanted all manual like it was 10 years ago all the machine learning that has been deployed toward world conversion optimization by meta in Google has been insanely successful and we don't want this back. We would be much worse off without it and you can see it that it's where the smaller platform have a problem because the amount of effort to deploy this cost the same to a smaller platform to to a big platform but the reward is not the same. So smaller platform tend to be a lot more primitive and it plays against them and that ends up that we marketers end up concentrating all our money on the same

1:10:18baskets because those baskets are much more efficient and they don't want to give us more control. So Facebook actually does it a little bit but all the others and the direction it's taking is actually less of it. So I'm not expecting to see more of it and that's why I need to do it more on my side because the platform gives me less and less levers to pull. So I'm going to play on a lever that I mean they can't prevent me from deciding what I'm sending them and they do want this signal anyway. So this is where it's still going to get deployed. So that was

1:10:47one question. Two, could the MMP help there? I think they could. They could by making it a little bit easier also because they see patterns like there are only a few MMPs and they have a very large number of clients. So they do see patterns and they could communicate not of oh play is doing this and Tinder is doing that but more like oh in this vertical it's very common to filter or in this vertical be very careful about how you send your ads revenue or whatever and so one on the educational side and I don't think it's been dealt so much and two in actually making it

1:11:23more default the problem here is that everybody's has their own and typically on boarding question is not something that you can have defa default in any SDK it's going to be the the actual developer who needs to declare okay I've got this question is never going to be the same across across two apps so it's kind of hard for them to deploy at scale I think it it should be part of the educational package and like oh you can do this this this I think they haven't done it much because it's so unique to every app but also because in their place it's kind of like yeah as soon as

1:11:56you send us an event we're going to send it to the platform that's my job like and we're already doing it. As soon as you send it to me, I'm I'm going to process it nicely, but maybe they could do a little bit more. And and the last question was, is AI going to replace this on the platform side? And in part, yes, and it's already what they're doing, but they've done as much as they could, and they've gone very, very far. The last piece in my opinion is on our side because it's so unique and because I don't want the platform to deal with my

1:12:27data the exact same way that they deal with the average of my sector. I want it to be the closest possible to my and sometimes even my own internal goal they change over time. I'll tell you one example. I have one particular case where we used to optimize towards a predicted LTV and then suddenly we realized that cash flow was more important and we realized that it was not the same user who bring the maximum LTV C or return after 5 years than the one who bring it after 3 months and we're like okay right now it's more important that we maximize short-term payback okay let's change it you know and this

1:13:03is not something the platform is going to decide for me like I have to decide this for myself but also because every configuration is a little bit unique that's said I believe there will be AI and machine learning but not on the platform side to decide on my side that could come from amplitude of mix panel that could come from revenue who says look we've got access to all your subscribers and we've detected that there's 20% of those that are very low value 20% of those are extremely high value and we're going to package it into here are three events if you want to use them use them if you don't want to use

1:13:34them use them so here. I don't think it's going to be Facebook and Meta who's going to do the job or Apploving or whoever. It's a little bit hard for the MMPs. It could be, but I think it's more going to come from products and subscription data tools like gravity cut. So, actually, it's more of a

1:13:49request. It would be very nice to have. I'm not uh but I think you're in a better place to do it than MMPs, right? The data you have around revenue is much more solid. Yeah. That makes a ton of sense. It came back to you fast here, huh? Yeah. So, I did want to get into the more advanced side of things. So, so we've kind of stepped through this, you know,

1:14:10step one, make sure your data is good. Step two, you know, pick off some of the lowhanging fruit. And then you said, you know, step three is to really dive deep and do some of these more advanced stuff. and you've sprinkled in a lot of the more advanced stuff along the way, but I wanted to give you an opportunity to like really nerd out and like go deep onto like what these more kind of advanced signal engineering processes

1:14:32look like. In my opinion, as soon as you get a little bit sophisticated, you it's going to be done at revenue level. Like you're going to share tweaked revenue, filtered revenue. It's never going to be filtered events. The problem of events is that they happen. It's binary. Either they happen or they don't happen. and the platform they can only optimize toward one event, not two or three. I mean, you can add them up, but it's they're going to have the same value to the platform, which may change in the future. And I hope Meta is going to release this and then eventually Google, but uh we're not

1:15:02there yet. The most sophisticated stuff is done at the revenue level because I can be so much more granular about this users $1, this user is worth 10. That say uh I'll do my own devils advocate against my opinion like some advanced stuff can be done at the event level and I know voyantist for example believes that the timing of which you send the event is actually creating a difference and they proved it against what I betted so they won on that one. I'm still not very good at engineering it myself but they showed me that it's true actually sending it early or later could make a

1:15:36difference. Not too late either, but like sometimes you got a free trial happening, but even the on boarding question and the other criteria we talked to, they're not good enough to decide if this free trial is like highly likelihood to convert or low likelihood to convert. You could preferably prevent the event from being fired, wait from for one or two hours, notice that oh these users, this group of users have done three activities, three meditation, three workout, three whatever within two hours and this group has not and I'm only going to fire those ones and not the others. So you could delay a little bit the event still work on event and do

1:16:13it. Personally, I think all the sophisticated stuff happen at the revenue level because you can be so much more precise about how you send it. My other argument against the the timing is that and I'm going to join the sophisticated question back into the lowhanging fruit is that I'm entirely convinced that whatever comes after 24 hours is useless for the platform to optimize towards which is a very hard ask because a lot of apps especially in gaming need a full week of seeing how users come back to the app, what they do, do they cover later, do they actually start with a small purchase before going up? Yeah, it would be great

1:16:49if or or with free trial is the same case because then you've got 3 days. I had the question recently like somebody was like, "Oh, if I reduce my free trial from 7 days to 3 days, it's going to be much easier for the platform to see this data after the third day." I was like, "No." Anyway, if it's not the first day, in my opinion, it's not entirely useless. It's not 100% useless, but in my opinion, you need to find something that happens on the first day. And that makes that you might prevent the the the event from being shot in the moment it happens, but you want to send it in the

1:17:20first hours. You wait a few hours and then you send it and then that's it. That's what you got. So one of the lowhanging fruit here is don't optimize for your 7 day or 14 day whatever uh is optimize something for for day one. And the advance stuff is move it to revenue so you can tweak it further and further and further and then you're going to edit on the goal and it's going to be and you add a new plan and you can easily add okay I've got a new type of plan or I've got a promo and I'm going to edit the value very close to what I

1:17:48think acts and that can include also like for example you could detect that refund have a particular pattern that some people who are more likely to refund they've done this and this and that and then users who've done this and this and that you would assign a slightly different value. You're never sure. I mean, it's all probabilistic. It's not like at user level, yes, a user converses trial or refunds or renew, but then in campaigns, you're going to have a bag of these users and you're trying to get as close as reality as you can get. And one thing that is actually hard like is I've got this with a team right

1:18:22there where we're monitoring the value we're sending the platforms against the real value that's happening. And we've got like this curve that are following. And as soon as we see that there's a gap in there like we know something is going wrong. As dynamic and sophisticated as the model is, it never follows the exact reality. So we're looking at closing the gap always and always. This developer works on being as close as possible as the real value. But then you can engineer further and I had this other conversation with you can fake it. And typically something that I do is I like to amplify it to make the network's job

1:18:57even easier. So, let's say, let's simplify the situation. I've got users worth $5 and I've got users worth $50. Well, what I'm going to tell the platform is that this $5 are actually three, but that this $50 is actually 100. Like, and I'm going to force it like I'm going to make it a little bit more extreme. The trick here is that it makes that I can't really read the value anymore in the platform. I can't read my rowass on Facebook because it's all fake. I'm going to look at it on my side, but it's okay. I'm going to try to force the platform to gear toward the

1:19:28users that are most valuable to me by forcing it a little bit. And this fun conversation with someone from gaming who said, "Oh, I do the opposite." What? So, when you have a high value user, you take the platform that is a medium value user, say, "Yes, because we've got whales." And then if I really declare a big whale of 500 or a,000, Meta is going to say, "That's it. My job is done for the day. I bring that one user and I'm done." So, they actually cap it. The thing is that it it's a lot more rare in

1:19:56subscription that you want to do that. But let's say you've got this tier of subscription with Chat GPT. Maybe Chad GP is going to fake it that the $200 plan is actually only 100 because otherwise the campaign could derail faster if too many of these user come simultaneously. So you want because at the end of the day the the machine of optimization they're very sophisticated but they're also a little bit dumb and sometimes when you when they see a couple good signals coming they start like completely changing delivery and yeah we're playing with fire a little bit here because one we experiment on the way and I made a few mistake on the

1:20:29way and but two it can derail pretty fast in a way that you haven't predicted and which if you look at it from the network side from app login or from Facebook and you look at it and Uh yeah, it's kind of logical that if they saw all this revenue happen that day, the next day they delivered in a way that basically it went too fast and too extreme. So you need to find a balance because me often I make it more extreme than it is but there is a limit to this like you can't go too far with this like don't don't don't push it. I

1:20:59also had a very interesting conversation with somebody else who was like, is it better to fake it that you're making more revenue so that Facebook will think you're super successful and serve you more than other advertiser or will that actually come to eat your margin and say, "Oh, this guy is making too much money, so I'm going to underdel it." I don't have the answer to this question, but it's kind of an interesting one to think about like how much the platform itself is reacting to

1:21:24the change we're making on that side. Yeah. Uh it's also fascinating. So in this advanced scenario, we're actually manipulating the revenue that you're sending back for the revenue optimization goals. How does that actually play out? So you're looking in like amplitude mix panel or internal dashboards and you're you're determining back to our uh meditation app example that you know somebody who answers anxiety and does these other three actions is our super high value user. So you're kind of backtracking from known data and the signals that you're seeing inside the app and then assigning a value to that. And then to your example earlier as well is like you know if they

1:22:15have this general need but they're still monetizing but they're not not monetizing great. Then you assign the $1 value. How are you doing that and how do you think about that before you go and take risk of of changing the value itself to something that you faked and you you think you're smarter. I think I'm smarter and which can backfire pretty fast. The logic of it is just making a regression of where where is there variance like I know I've got users of different values which are the criteria that creates this variance. Is it an on boarding question? Is it the device? Is it the time code to

1:22:50completion? Is it this? So it's basically crossing this. Okay, I've got this high value users and these low value users and these zero value users. What each do have in common that the others don't? is basically a regression is the reality is that many many times I've explained the data analyst that I want this job to be done and that his goal is for him to find a criteria that I didn't spot by myself like a new criteria the reality is that the criteria they're very often the same they're the one we mentioned they're the big one I guess as you go more sophisticated and I had an interesting

1:23:22talk with Ryantis about this again also about the number of activities that people do and when they do it and the time of day And if they install on a weekend and then it's a different pattern like they go much further than this but I think for most people like basically looking at where does variance comes like regression of which are the criteria that create variance between people who refund a lot between people who renew a lot and usually you've got 95% of the job done like yeah if you look for this where the variance is coming like it should be good enough for

1:23:54most cases. Are any of the tools particularly good at this? I know amplitude recently has like this AI variance detection kind of stuff. Are any of the tools especially good at that? I wish the the problem is that very often the the data is a little bit scattered like amplitude would have very very good but then the connection with revenue cut or similar needs to be extremely good as well like for the revenue to be to be proper like very often it comes from a variety of source I think it's going to come but in the case that I've seen myself it was done manually from back end data like because

1:24:28it's the only place where we had the liberty to dig and query the data in very very specific way ourselves also because we could extract it and the data the data list could run his own model on it and not feel limited by what the those models are offering but also in one particular case because they had very sophisticated internal tooling and they do have amplitude and they do have an MMP and so on but in their own back end they've got the equivalent of radicat and the equivalent of amplitude for every single screen and so obviously this case the data was more complete there if somebody has a mixed panel

1:25:04amplitude conction with render that is really properly done and a very large cohort of users eventually it's going to be it's going to be done automatically maybe there's a joint work to be done here between you and them but most of the ones I've seen they were manual and I think the reason is that one is a little bit of a pirining field still two there's a lot of predict this stuff so it's very unique and three yeah very often you would engage different source of data together and even though you have cut there and even though for example in amplitude I can plug also the

1:25:38MMP data so I can filter by source it's not something that is supernatural it's not something that is like completely well done out of the box because it's made for product analytic people it's not for product managers and not for user acquisition specialist yeah it's not a revenue optimization tool it's a analytics tool exactly it's not what it's made for and it I don't think it's impossible and some people will do it like this but uh I I know I've I've struggled with some

1:26:05limitation in some cases. Yeah, that makes a lot of sense. Well, we've we've gone quite long, but I I I wanted to give you an opportunity for kind of a grabag here at the end of other considerations that we haven't discussed. So, like, you know, one of them that we talked about on the webinar was the trade-offs you make in volume of events versus quality of events. So why don't you cover that first and then just a grabag of anything else that you think people should be considering as they embark on their signal engineering

1:26:36journey. The volume versus quality. I mean we didn't comment it today. So for whoever has missed the previous webinar and not read the article that I haven't posted yet necessarily they've missed it. So I'll go back to it because it's pretty fundamental. But uh yeah, you can have a fantastic event if the platform is seeing one or two every day is not going to work. Like the filtering I mentioned before like for me filtering out people

1:27:01is the last resort. But that's not true. If I I've got a very large amount of data, I would be more inclined to exclude people and is the number of events you said. So here is at campaign level like every campaign basically there's a threshold of optimization that networks there's no exact number but the the number that's floating around is

1:27:19somewhere between 50 and 100 per week. So I simplify in my head and say below 10 event per day per campaign things are going to go a little bit wrong. It's a bit the more the marrier. So if I can have 25 instead of 10 I'd rather not filter and I have 25. And if I have 40 instead of 20 then I have 40. It's better. But once you get there, there's very little upside to sending 300 every day. And there's a lot of upside of sending only 30 every day, but that are super filtered. So in this case, this is where the trade-off between volume and

1:27:50quality is. So if you're too early, basically, you can't do qualified trials almost because you already only have five trials per day. So don't try to filter them like almost you'll want to go higher up before the free trial. say okay these users haven't tried a free trial but they've shown very good intent around around on boarding questions about the first activity I'm going to also send them so that I have more event and so the more event you have the more you can filter around that's pretty clear and this trade-off is is pretty fundamental that day on the webinar we didn't talk too much about timing but I

1:28:22think I covered it decently today also covering the the other part the prosecutive part but uh the timing is important and my rule here is don't wait don't wait too long you can wait a little bit but don't wait too So timing is one and the amount is one. The amount is very critical. It's something because over time maybe the best engineering you can do today is very different from the one you can do in six months. Just because your budget has moved from 100K to 500K enables you to do a very different kind of filtering. Also because your cohort has grown. So before even with the teams that have the most

1:28:56sophisticated model we keep revisiting the the value that we're giving because plans change because we make monetization change because the mix of of channel is different as well and the whole user maybe because there's a crisis maybe because there's this like if you're based on an aggregation of five years of data here is also a big trade-off about how recent or how much data do I have and like sort of I've got this rule for example in in one case which is if we've got enough users in the last two weeks to make a decent prediction we use this but if we don't then we expand to three months and if we

1:29:34don't then we're going to loosen up one criteria a could be a could be a country could be a number question and so on because otherwise and I made this mistake several times like you create useless fluctuation on on very small samples so obviously the more data you have the easier it gets it's not a fixed thing So the advice here the TLDDR is revisit your assumption every now and then every every 6 months or whatever or at least control that gap between what we're sending and what is actually happening is not too big and the thing about this gap is the trap is to look at

1:30:07total like we only have a 2% deviation I'm like yeah but there's 10% of people who are responsible for 80% of this deviation so this is the one so don't look at this gap only in total do monitor that it doesn't go sideways too much and countries is a big one here like country platform. Maybe not every single critter is needed, but at least monitoring the value of the events that you filter are fairly reliable at country level over time is is kind of a

1:30:34good idea. All right. Well, this has been a blast. Uh so much great information. This is why we got to do it annually. You're constantly learning new things and then you're so open to share. And then I get asked all the time, "Hey, can you make an intro to Thomas?" I'm like, "Thomas is busy. He's going to be busy the rest

1:30:51of his life. I'm too busy researching new topics for David for the next one. Exactly. Uh but this is why we'll have to do it annually is that you're you have your stable of great clients that you're making learnings from and so you're you're not hesitant to share like a lot of people. And so I really appreciate like you're always just digging in there and sharing what's working. And so this has been a a master class in uh figuring

1:31:14this kind of stuff out. So thanks again. And then a quick plug, Thomas is actually going to be doing a hands-on workshop at Appgrowth annual. So that's going to be October 14th in New York City 2025. So if you want to go super hands-on with Thomas on single engineering, sign up for Appgrowth annual. The workshops will not be livereamed specifically because we want people to be able to like share and and really open up and have real conversations. if you're wanting to dig deeper, you you got to be in person for the uh signal engineering workshop at Appgrowth annual. So, I'm looking forward to seeing you in a couple months

1:31:54and uh thank you so much again for the conversation today. Yeah, likewise. Looking forward to seeing you in person and thanks thanks again for inviting. Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com subclub.com to join our private

1:32:15community. [Music]