# Creative Testing, Data-Driven Decisions, and Growing Mobile Apps – Alper Taner’s Approach

Channel: Sub Club by RevenueCat
Video: https://www.youtube.com/watch?v=FF2GKgBi6ZI
Duration: 1 hr 3 min
Language: English
Words: 10907
Transcript page: https://viewrankai.com/tools/youtube-transcript/FF2GKgBi6ZI

---

[0:00] A lot of people focus on the execution which is on paper great. But I believe the success of a test is at least 50% dependent on how you plan it, how you strategize it, how you hypothesize it, and how you execute it. And execution is just the what part. Maybe it's even sometimes less than 50%. How you execute

[0:23] matters more than you execute or not. Hello, I'm your host David Bernard. My guest today is Alpert Tanner, head of performance marketing at a stealth mode app studio, working on productivity and utility apps. Alper has been in mobile growth for over a decade, leading UA and Martekch efforts across a wide range of verticals and managing 8 figure annual budgets. On the podcast, I talk with Alper about the competitive advantage of ignoring some best practices, the risk of drawing false conclusions when researching competitor ads, and why poor metrics are just facts until proven problematic. Hey, Alper, thanks so much

[1:04] for joining me on the podcast today. Yeah, thank you for having me. So, it's been a long time coming. Uh you and I first met in person at MAU years ago and we've kind of talked off and on about getting you on the podcast, but recently we were talking about a lot of your contrarian ideas just that you think a little bit differently uh and like to challenge the kind of rules of thumb and I thought that would be an especially fun thing to talk about and so let let's kick it off with that. the podcast shares a ton of like rules of thumb and like hey try this don't try

[1:35] that like oh this didn't work for me so you probably shouldn't do it and you know on the revenue cap blog we share all sorts of of articles like that but you went through as you and I have been talking preparing for this episode and you kind of ripped holes in in some of the like rules of thumb so at a high level why don't you just start off like how do you think about these kind of like best practices and rules of thumb there are definitely best practices and rules of thumb that you should stick I don't know 80% of the time but that doesn't mean they are universal for

[2:10] every single case. So if I were the app developer listening all these ideas of dos and don'ts I would prioritize it based on my current situation data and I would still try them just to see that it also doesn't work. Of course it has to be reasonable but idea is that your data should uh show you that instead of oh we heard someone said XY Z type of an approach. So I think this applies across the way you test the creatives, the way you test incrementality, the way you test your bidding strategy. So I think these are all meaningful tests to a certain extent depending on what stage

[2:58] you're in. If you don't have your own learnings, of course, it's smarter to leverage others learnings and it can just help you to prioritize. If you hear something didn't work for 20 people, just dep prioritize it. But doesn't mean, oh, forget about this because it's not going to work for you because algorithms are constantly changing and improving. And whatever someone tried 6 months ago can work today. And then we see that it's happening across multiple platforms within the platforms as well like the campaign a that or like specific targeting that was working really well stops working. This is like a day-to-day thing or your creative dies

[3:39] right like where the creative that worked 6 months ago that you had to relaunch it works great again. So also it potentially looks much better now because of the late conversions and stuff. So it's not black and white. I think when you hear such advice, you should definitely like take them as suggestions. But when it comes to execution and prioritization, you should do that based on your own data because you don't know the history of that particular account and why particular thing didn't work. And there are several or a lot of reasons why a particular setup would work for one market or platform versus not. And that's why they

[4:22] are all individual cases. There are definitely, as I said, best practices that are universal, but I would always challenge them. I take them as, okay, great. Let me try it. [laughter] Yeah. Well, I I I think you've just created a new rule of thumb. I I I like that that you should spend 80% of your time on the best practices and learn because like you said, if 20 people tried it and it failed, maybe it is going to fail for you. But then take 20% of your energy, your budget, your efforts and do the exact opposite of the best practices [laughter] and figure out for yourself

[5:01] why why they learn. And and to your point, like you know, if you're early and you don't have a lot of resources, you don't want to overindex on being a contrarian. But I mean, that's the whole thing, right? Is that there there's alpha in the best practices and that's why they're the best practices. But then there's also alpha in the things people aren't trying or the things that certain people failed at, but you're going to succeed in that for a very for something that's very specific to your app or your niche or your market or some other thing like that. So yeah, the new rule of

[5:32] thumb. [laughter] Follow best practices 80% of the time and try crazy stuff 20% of the time. I like that. Let's go through some of these like more specific contrarian takes. And one of the ones we were discussing is that sometimes you just need to take something as a fact and not consider it a problem until it becomes a problem. And the example you gave me, and I'd love for you to elaborate on this, is that if your trial conversion is low, and you're like, "Oh, trial

[6:00] conversion being low is a problem." Well, is it a problem? Like, are you getting cheaper installs? Tell me about that. Yeah, like typically people would say like, "Oh, we have, I don't know, install to start trial ratio, I don't know, 5%." And, you know, usual health tier benchmark is more like 15 to 20% and such. And I'm like, "Okay, what what have you done? Like, what have you tried?" and how did you interpret the results? How did you build on those results? They're like, "Oh, we didn't because it's a problem." Uh, you know, so that's not a it's just a fact. I think something becomes a problem after

[6:34] you systematically test it, iterate on the results and try many different ways from let's say radical to iterative and you're still stuck at the same level, then I think I can consider that as a potential problem. [clears throat] Instead, you should always see what you can do or understand the root cause. So, diagnose any factual problem that you consider as a problem and then come up with an action plan and then measure the results to see the impact. It's as easy as that. It might sound very like straightforward but I can tell you like many people bypass uh this kind of stepby-step funnel in figuring out stuff

[7:20] because when you follow such an approach you realize that oh actually it's not a problem like or I don't know it's simple as people say like oh we tried I don't know snap and it didn't work. I'm like okay how did you try it? Oh we spend I don't know 5k in a week and it didn't work. I'm like yeah that's not a problem that snap didn't work. you just didn't test the channel in a proper way that is optimal for the for the channel. Uh you shouldn't test a channel just for a week. And same goes for I don't know

[7:49] creatives to channels or the markets. Oh, it's too expensive for this market or that channel didn't work. So people like to come to conclusions fast and I think because of the priorities like because of this scarcity kind of a mindset like they want to reach conclusions really fast and they want to move on to the next. So when they are in this kind of a mindset I think then they're missing out on testing things properly. Like I think a lot of people focus on the execution which is on paper great but I believe the success of a test is at least 50% dependent on how you plan it, how you strategize it, how

[8:38] you hypothesize it and how you execute it. And execution is just the what part. Maybe it's even sometimes less than 50%. But basically how you execute matters more than you execute or not. Sometimes people also say, "Oh, we tried, I don't know, let's say look alike. It didn't work." Because the way they tried look alike was a 10% run it for a week, that's it. No seed testing, no different ranges of testing and such. You cannot

[9:06] just call the whole thing doesn't work. If you don't slice and dice and try like different things. That's kind of my approach and that's how I deal with day-to-day business to understand and prioritize it based on okay how can I say I tested something and what information I need to call the shots that it doesn't work or it does work. So this kind of thinking I think will force you to think of all the preliminary steps on what you should be considering and so on. Yeah,

[9:39] I like that kind of root cause thinking. I actually did a talk at MAU a couple years ago about the the top three subscription app metrics um that really matter. And that was kind of the thrust of of my talk was that you need to look at the whole funnel, not just individual metrics. And and like what you were saying, it's like if you have a low trial conversion rate or if you have a low install to trial rate or if you overly focus on any one metric and

[10:08] you're like, "Oh, this is a problem." And you like grind and grind and grind on that quote unquote problem when maybe you can get like really cheap installs and the really cheap installs are why you have a low conversion. But hey, that's not a problem. It's not a problem when you get really cheap installs and your conversion rate's not great. Like that's maybe just a natural fact of your business that for whatever reason you're getting incredibly cheap installs and and the example I always use with this is like you know my apps have gotten featured a ton by Apple over the years and getting featured is fantastic

[10:41] because it's totally free zero cost of install but guess what those cohorts convert really poorly. So do I have a conversion problem? I don't know. I know that those cohorts don't convert very well, but like should I overly focus on that or should I be more focused on top of funnel and like finding other ways? And so a great reminder to not get overly focused on any one specific thing to the exclusion of like all the

[11:10] different moving parts in the funnel. And I like the way you phrased it too, like it's just a fact. That's a fact right now. That's a fact. Is it a problem? We don't know. It's a fact. Figure out if it's a problem. Don't just assume it's a problem. Yeah. So, another thing you mentioned you've been thinking a lot about lately is creative interpretation. And you know, we had Eric Seford on the podcast recently and he's intentionally a bit provocative saying like, "Don't even try. Don't even try to interpret the creatives." But you were telling me like you do think there are things that you could learn from trying to interpret why

[11:44] things won, why they didn't. So what is your process for looking at the winning creatives and like finding some value in that process? I think looking at the output of a creative performance definitely matters because that's how you will iterate and find the next winners and or the losers. It is important to look at the relative metrics as well. So maybe what he's referring to is more like don't take it as your source of truth because platform is doing the modeling and such but at the end of the day the output slash the performance of the creative matters and the approach should be more like what

[12:28] can we learn from it regardless of the output in a way like good or bad it's all about the learning okay this creative for whatever reason performed better and other creative performed performed for whatever reason worse. Let's understand why. What is the hook rate? How is the thumbnail? What is the audio? What is the background? And this and that. So once we understand and then we we start changing one variable at a time, then it becomes a lot more clearer on how we can engineer the success, right? I mean that's the whole point of that's how you create winners from the winners because you have a winning

[13:05] concept and thanks to the interpretation of the output you're able to create more winners because you look at the relative metrics that's a guidance for what you should be testing and what you should be not testing anymore because whatever you do in that particular case doesn't work however I have seen from the same concept cept like the third variation

[13:31] worked but the first and second failed. It's the same concept. It's just slightly different messaging and such. So that's why you would always test with few variations just to avoid such cases where where you don't just jump to conclusion of a whole concept and so on. I think that's also important. So there could be of course false positives and so on like you know nothing is perfect but also I'm not also saying like oh you should trust algorithm blindly and and stuff like that. What I'm just saying is like we should be able to interpret the results and then drive conclusions to shape the next creative iterations based

[14:11] on that. So that's when you need to interpret the output and then what matters at the end of the day is your success rate and not the amount of creatives because I think LinkedIn nowadays is full of oh you know what we just tested 500 creatives oh no we tested 700 how much did you test kind of race uh that is not so meaningful because those numbers alone don't matter if your success rate is very low I would more look at your cost per successful creative that is beating the BAU ads performance. And it doesn't matter if you test 20 versus 200. Of course, okay, 20 versus 200 might matter, but it

[14:57] doesn't matter if you're testing a very high amount of creatives with a little success. It matters more. You test less amount of creatives, stronger hypothesis, stronger investment from an analyzing and hypothesizing way with a higher success rate. That what matters more than just the total number of creatives that you're testing. I see potential issues when the creative amount is on the higher end because then the mindset is like hey look I just have to test these like let's just move on and and then having less emphasis on why

[15:36] it worked, why it didn't work and so on. And then sometimes there's a disconnect because typically a company that is testing 500 c creatives a month like tend to have a team and usually there are I don't want to call it silos but you know the guy who is running the UA is not doing the old briefs of the creatives and then analyzing and then giving the briefs and all that typically in the smaller startups yes but in the bigger teams there is that and then so then the UI person's job is just to run them and maybe analyze them if they don't have the analyst team then to give

[16:09] the creative let's say the optimization manager to put them into briefs and get the next production. So when the quantity is so high like I think the level of detail and sometimes also the quality of analyzing and shaping the next input is also being jeopardized a bit because of the high quantity which is not necessarily a win or something to be proud of alone if your success rate is low because also then the mindset shifts like hey look I have 500 creatives I can't spend 5k on all of them. Sometimes this gets into, oh, we only spend $100 on these creatives and that's it. Okay, but how did you decide

[16:50] on that threshold? And I think finding that threshold is such an important thing for every account. For example, there's no universal number that I would give. It's very different from business to business depending on your CPI, depending on your other CPX metrics. But what I would suggest is to look at your cumulative spend and cumulative CPX metrics, whether you're testing on a CPI or cost per trial or just cost per subscription and such, and see when they stabilize. So, because I see a lot of accounts with high spend, they're sometimes just spending 10K, 20K only to realize, oh, this is actually not a good creative. I'm like, oh, really? I could

[17:36] tell it after 500 bucks like look at the data it was already bad like it was already screaming like why did you spend so much? So in order to avoid that inefficiency and scaling it more efficiently, you could easily look at cumulative spend and cumulative performance over time and to see where your data stabilizes because we're not going after the statistical significance

[17:58] like no one has kind of money for that. But we also don't want to spend just 20 30 bucks to see or we don't want to just let the algorithm decide which one to pick. So there the problem is the following like just because algorithm decides to deliver one creative over others doesn't make the other creatives bad creative. I think that's such a misconception or interpreting in a I would say in a wrong way. Like that particular creative that got the 90% of the spend was maybe lucky. We call it like lucky 5,000 impressions because Meta does some sort of a decision making

[18:43] around first 5 to 10,000 uh impressions. And if one of those 5,000 people who saw the ad didn't engage or whatever it signals that meta didn't receive, then it will not deliver anymore, but it will deliver the other one who just got maybe one, two clicks and stuff like that. So, and then I've seen that multiple times where you take out that dominant ad from that adset and all the other ads like suddenly bloom and you're like, "Oh, like they were also great ads, but because of the other better ad or however you want to call it and which is not necessarily better ad as well

[19:21] because sometimes meta call the shots way too early. No conversions, nothing like it doesn't even give a chance. So in those cases, we don't call that creative that got the all the spam successful and all the rest bad labeling. I think this is not universal and this is really I think case by case basis. I've also seen like adsets with like 50 ads live. Oh, let's see what algorithm likes and we will run it for a week and then we will see what happens and then what happens in that week 90% goes to the same ad and because they have this rules of thumb oh we run it

[19:58] for 7 days and at certain I don't know daily budget but then that's useless when the 90% of the budget goes to one creative like that's not a test like you should in those cases for example break the rules and like hey look this reached my threshold this is my decision threshold now I want to uh what other ads going to perform for that minimum threshold that you decide based on your cumulative spend and

[20:21] cumulative performance over time. Yeah. So that's why like there is no one rule that is valid for all. But I personally try to get spend on the other creatives if I see a dominant one. I believe that every creative should have an equal chance. So I democratize a bit within the adset, right? And because I've seen it countless times that the decision

[20:45] that made by Meta was way too early. Yes, we trust algorithms, but we want to guide the algorithms. When we see a 80 90% of a dominant asset, let it be a creative, let it be a country and such, you're not fulfilling the potential of the other variables. And I think this is important. For example, if you put the US with all the other smaller, let's say European countries, there's a 90% chance that at least 70% of the spend will go to US because of its size and everything. Also, you should never do that anyway. Like you should always tear by like CAC or LTV depending on what

[21:24] you're optimizing towards and ideally without those dominating variables. And if there's a dominating one, then you can always take it out, test it separately, give a chance to the rest because it's not meaningful to have, I don't know, let's say like just a couple of thousand dollars spent with a really good CAC, really good ROS. Well, if you cannot scale that and then you try to scale by increasing that existing campaign's budget, but that country that you want to actually scale is not increasing its spend necessarily because if it's getting dominated by the other bigger countries or based on whatever logic that depending on your setup. So

[22:04] in those cases then yes you can experiment with the value based the rules and stuff like that and that we can talk about later in the testing and and the learning side of things. What's the tactical move then there when you do have that you know quote unquote winning creative that's getting 90% of the spend. Is it better to pull the other creatives out because that ad set already knows how to target that specific ad to the right people, to the right geography and everything else. Is it better to pull the quote unperforming ads and give them a shot somewhere else or is it better to pull the performing

[22:40] ad into a a different set? What's what's the best way to to actually implement that? depends on whether this creative is in a creative test adset versus in a BAU adset. If it's in a BAU adset, then I I would create another BAU adset and then I would include those creatives that were under delivering into the new one and I wouldn't touch that something that is working well. But if it is within the creative testing adset and if it already reached my threshold then I would already pause it and then I would put it on the bench to get ready to upload to BAU adsets whenever the team

[23:25] will do the creative refresh. So it is case by case basis but in general rule of thumb is don't touch something is working well but depends on what case we are talking about. Yeah. Yeah, that makes sense. I did want to step back and uh not to put words in Eric's mouth, maybe his point is more to not overly focus on a single hypothesis for why a creative one. So, like going back to the whole like, you know, should you learn from creatives? Because I I do want to dig a little deeper into kind of

[24:01] how you think about what you can learn. Um, and the example he used was you look at a creative that has a dog in it and you're like, "Oh, like people love dogs." And so that's why the creative one and you maybe get overly focused and then to your point, then you go create 200 new assets all with dogs. But maybe it wasn't the dog. Maybe it was just that it was a something cute and a cat one would perform just as well or a panda one would perform even better or

[24:28] ocelot or like who even knows, right? How do you think about forming a hypothesis around that? Because it seems like that that's really the important step is is like not being overly confident in why you think something won, but instead just allowing it to help you form a hypothesis for other things to test based on why a multitude of reasons why that creative could have

[24:54] won. Yeah. I mean, you should never just produce 200 variations of just from a one signal. I think it's always in in phases. So let's let's take the dog example. Okay, let's have the same dog. Let's have a puppy. Let's have another breed. Let's have another animal. So just to test and see if it's the animal or not. So what we want to understand is what makes the difference. Is it the dog? Is it an animal? Is it something else? And once we figured it out, then we would slowly roll out the production and so on. So it's never about oh the dog won now we need 200 variations of

[25:30] the dog. It's more about okay like is it the dog? We want to validate that. So in every step we want to do that and we want to do in both directions. That's why we have both iterative concepts and the radical concepts. Iterative would be oh let's put another breed let's put two dogs maybe it's even more effective and uh but you know the in the radical one you can have completely different hypothesis than what doesn't work based on what works today. but also on based on other sources. So like when it comes to what creative to test, I think you should utilize both the first party data

[26:05] and the third party data. It's not always about the first party data. So first party data meaning like your meta creative output. That's one variable that then that's a valid let's say input. But you also have let's say for particular copy like or let's say if you're testing particular let's say visuals and such that also exists in your app right okay what what is the most popular thing in the app that is more valuable for the users you know you basically look at within your data set like you look at your within your inapp analytics uh what you can learn what resonates with users you look at your

[26:44] high LTV users like why do they stick around what do they use the most what messaging resonates with them and then because our goal in performance marketing is to find more of those high LTV users. So testing like test results from test campaigns is one variable but inapp analytics data is another data input that we can utilize. And then we have the third party data where we have competitor concepts. What works for competitors? We have overall trending

[27:17] memes or concepts or audios and such. And then you could have category level concepts. They're not really direct competitor concepts, but other big players that are doing well within your category. What are they doing? What do how do they advertise their product? So what you can learn from them? So then you bring and merge all these data sets together. Then you decide on okay, you know what? Of course you give a lot more weight to the meta test results. But it is not a 100% hey this is it guys, dogs,

[27:48] let's go. So it is about merging that. So when you hyperlise the next iterations, it's also should come from what worked and what didn't work. Not only for you but also for others. You can check the meta library to see how long competitors are running a particular ad and how many variations of that. If there is one concept with 15 variations and that is a particular the only one that has 15 variations compared other ones having just two and three, you should be able to interpret that likely that creative performing well for them. That's why they're exploring more angles. So it is important to analyze and put enough

[28:27] resources and attention to the planning part of the experiment. Exactly. because of this way and not just act like I don't know growth bro and hey this worked well let's do 200 of them and so on so I think this balanced effort planning and executing matters here how do you think about analyzing competitor ads though because to your earlier point there's a lot of people out there just following the playbook and and creating 15 variations because they formed a bad hypothesis around so how do you actually dig into these competitor ads and try and figure out

[29:07] like if it is actually working for them. And that's something I just see throughout the whole industry broadly is that it's too easy to overindex on something you observe without the data and say, "Oh, this must be working. These 15 creatives must be working because they've created 15 variations." That may have been like a dumb move and you just don't realize how poorly it's performing and they don't even realize how poorly it's performing because they don't have it instrumented well. They're not tracking deeper in the funnel. So like how do you think about looking at competitor data and getting any insights from that when you when you don't have

[29:44] the full picture of why, how, what their process is, how good their stuff is instrumented, how smart their people are, etc., etc. Yeah. So anything you see online let's say should be only as inspiration and not treated as this because I've also seen and heard people saying we tried it it didn't work but I don't know leather is using it like so then they justify and then they defend oh but it worked for them like so as you said like it doesn't mean anything for the brand exactly because of the reasons that you mentioned so it should be just treated as inspiration as another input that's

[30:24] why I talk about a pool of inputs that will shape together the next iterations. It is never about oh let's copy one to one whatever compare is doing. Oh let's copy this trend. It comes down to how I can customize this to my own personas and in my own ways based on what worked for me before and what didn't work for me before. So at the end this needs to all go in that processing machine that considers the personas the past performance the current performance and

[30:58] all your learnings and then you test it. So it's never about oh competitor is doing this great let's do the same. I think that is probably why many people are getting upset I don't know when they just copy the same pay wall or same creative and then finding out oh it doesn't work for us. Well why should it work? uh it's different audience, different funnel, different pricing,

[31:20] different messaging, different promises. So although for example on the web funnels like many of them look very alike especially in the health and fitness space for many years and then everyone copies from each other. So like pretty much a lot of the health and fitness web funnels are similar now. Some are missing of course some I don't know smaller elements here and there but

[31:42] the conceptually like they're the same. At the end of the day, you should just run to see. So they should be treated as inspiration and as a variant in your whole test hypothesis and not as a test hypothesis itself. Yeah. And and then one other kind of elephant in the room in all of this is making sure that you're actually instrumenting things well and actually testing well and can actually interpret the results properly. Bah recently had a tweet that I thought was was amusingly provocative. He said, I won't quote it exactly, but it was something along the lines of like half the people who say the Blinkist payw wall works for them

[32:24] just aren't testing it properly. [laughter] So, how do you think about like in all of this experimentation and forming hypothesis and like running all these tests, how do you think about like even just validating your own process and testing and tooling and and that side of things to make sure you're even drawing the right conclusions from the data? Yeah, I mean that's also I think what we covered earlier like 50% of the issue with something working or not is about how they test it. So I can like totally get like why he mentions that because I hear like similar stories. They're like, "Oh, we tested XY Z pricing because

[33:04] competitor is the same, but our conversion rate dropped and our revenue dropped." Like you shouldn't take these kind of stuff as the hard truth or your strategy. It should be only as a variant and as a a source of inspiration. And in terms of how to interpret the results like everyone has their own inapp analytics tool or their third party tool depending on what test are we talking about. If you're talking about let's say the payw wall tests and stuff like inapp analytics is the way to go. If you're talking about creative tests and so on then we are talking about MMP as a a source of truth relatively speaking

[33:44] here. But if you're talking about incremental impact of a channel, like we're not looking at the last click obviously. So that's also one thing that there is not one single source of truth for every answer you're looking for. So if you want to compare creatives against each other, like MMP could be your best friend. But if you want to compare the impact of UAC versus Meta, MMP is likely not your best friend because one is a push channel, one is a pull channel but and the competitiveness of the vertical and such but typically UAC is a lot cheaper because of the search and which of those higher proportion of those

[34:31] searches also coming from the brand search and so on. So you wouldn't do a budgeting decision or forecast just based on um MMP and such. So you might want to rely on incrementality and mm and conducting experiments around the geo lift and the hold out and the and the blackout test and so on to come to such conclusions. Knowing your baseline, knowing your seasonality problems get different at different stages. like you don't have to worry about incrementality

[35:03] when you're just spending 100k a month. I think it might matter but it matters a lot more once you get bigger because a lot of the decisions at such stage are driven by MMP/ lastclick which is normal because when you are at that stage you don't have all the data infrastructure set up to measure incrementality and and and so on like I'm maybe generalizing a bit but typical startup don't have all those kind of tools inhouse or third party like to make it happen and once to get to certain level I would say 500k plus then these are becoming more a lot more important and prominent in

[35:44] decisions because then you start with the yearly budgets and then the forecasts and everything and the profitability starts mattering a lot more than before and so on depending on let's say your investor vision as well at the end of the day you should just stick to one tool for certain decisions While you should also cross check with other tooling time to time just as a baseline. I'm not saying for example you should use app store connect as your baseline. No. But if your MMP and meta are very different but your app store connect CPP data and meta matches that

[36:24] should give you also some like signals. take it as there are source of truth that you should be sticking to but don't take them blindly as this is it like UAC is the best channel let's double the budget only to see potentially the impact is output is not the double and so on so that's why then it comes down to multiple tools and multiple ways of interpreting it for example incrementality and last click also conflicts or mm um might be conflicting with day-to-day decisions as well depending on at what level you do MM or at what cadence you feed the MM and how you calibrate it. Having worked on so

[37:09] many different apps and audited so many different accounts over the years, do you have any horror stories of like the data teams SQL was wrong and and for a year they were just like making really bad decisions or or the MMP was not hooked up correctly or like do you have any like just crazy horror stories from all these accounts that you've worked with or multiple crazy horror stories of uh just making bad decisions because the

[37:36] process wasn't set up. Yeah, I think story was something like this. They switched the MMP and then they didn't implement the events on the new MMP. I think there are probably plenty, but there are also some must avoid at all cost type of things such as I remember one project switched MMPs and they contacted me after a month and saying like, "Hi, we need some help. We cannot figure out what's happening." And the issue was so obvious. So they switched MMPs. They didn't implement any events on the new MMP and the old MMP was still connected to meta. They still had campaigns optimizing on meta. And

[38:21] this is like a sizable company. But the problem is they have a different team handling everything MMP. They have a different team handling meta only kind of a silos and stuff. And I was like, "How did you not realize this for almost a month?" They're like, "Yeah, because it's not our thing." And so on. So this stuff is

[38:44] like I think on a very extreme level. Other than that, I don't know. I've seen as crazy as someone spend 100k on a creative with 100 plus dollars and did not stop it before. I think it was like 90k something. Didn't stop. Like I was like, really? With a $100 cost per install. Yeah.

[39:07] Wo. Relatively. It was not the crazy high compared to other ones, but it was high enough that you should have killed it after $2,000 max and not wait for 100,000. But so that happens like when I mean it's not an excuse, but it tend to happen in large accounts with no automation guard rails. Yeah, let's keep it this way. like no no guard rails in place and you're just really focused on execution and day-to-day business and you're missing out on hygiene checks, rules of thumb and all that kind of stuff because you're too focused on testing the maximum amount of creatives and so on. So maybe maybe this is another good rule of

[39:47] thumb to come out of this podcast about not following the rules of thumb, but I I don't know how you would divide it up, but maybe 80% of your growth teams efforts should be focused on testing and moving fast and all that stuff, but you want to leave like 10 or 20% to like be continually auditing your processes, continually auditing your data structures, continuing to audit your events structures and and all of those processes that are in place to make sure that that you don't end up in those kind

[40:18] of situations. Yeah. Like another example I just uh remembered is on the positive side of things. We were doing creative testing and and then we did find the winner at the end thanks to interpreting the output and iterating on those output. So it actually uh works when you like do the iteration and and find more winners. And we were able to I think 5x the campaign budget within like two weeks with the same and even decreasing like cost per purchase and we were not applying 20% increase per day. If you see that your daily budget is 2K but Algo spends 3K and you get the same

[41:03] performance. I was like, let me handle the bid and budget for this campaign because I see something I see something very unusual and I was so excited about it that we literally went hockey stick and the cost purchase literally on a downhill and that's like any UA manager's dream and in those cases like but also the budget at some point was spending the double of the daily budget but it is no problem as long as you have the performance and talking about as a problem like there are also companies who put let's say overspend on a Sunday as a problem but then again it's just a

[41:42] fact the my approach is okay what do you do for it have you decreased the bids and the budgets like how did you try to overcome this code problem and then they're like oh like it's just the algo decides on that I'm like yeah but you can guide the algo if you put a lower budget if you have some automation in place then algo be like okay like I don't they don't want me to spend more and so on and there are multiple ways to control it in a way. You cannot control it 100% of course but we can guide the algorithms in that way. Like another

[42:13] example is the the cap inflated budget strategy where we have a million dollars a day budget at the campaign level and which is not a good advice unless you put a campaign spend limits in place. Right? So, as long as you have the guard rails in, you won't be spending 1 million a day. Of course, uh you will only spend up to your total campaign cap. And we tested this to bypass the pacing function, although we were still

[42:46] using the bit caps and the cost caps. And this was like earlier this year when we tested it. And it was also recommended by Meta. And basically it did work quite well because we were able to capture a lot more higher quality traffic. We did not spend a million dollar a day. The spend varied across the days, but we were still able to get a stable performance with such a crazy budget that the algorithm is not pacing the day anymore because it is very large, right? Because the way the daily budget works, it divides the budget to 24 hours in a way that of course it

[43:30] doesn't it never does it equally in a way. You know, you always have this kind of fluctuations. the evening times it peaks up and so on. But by putting a million a day again always with the budget caps and spending limits at every level. So yeah, this this [laughter] is one that could get really dangerous.

[43:48] Yeah. So I think if I'm understanding this correct, is it at the creative or that the daily budget would be at the entire account level? Is that where you're setting the daily budget or at the campaign? Okay, at the campaign level. So you set the campaign level budget to a million and the reason you do that is that Facebook or Meta Meta will try and throttle your performance to not overspend in the morning. So so let's say if you're let's say instead your budget was 100K and 100K is what you

[44:21] actually wanted to spend for the day. And the problem would be that Meta's like oo they only want to spend 100K. we shouldn't serve too many ads this morning because we know the night's going to pick up and so we're going to like kind of throttle the morning ad delivery. So what you're saying is like at the campaign level you set that to a million but then you very carefully go

[44:43] through each bid set and set caps there. But then how do you balance those bid caps so you still don't end up spending or or maybe you want to give it some freedom to maybe spend 200k or 300k or like Yeah. So then how do you dial in those bid caps where it does still allow the algorithm a little more flexibility

[45:02] in in more aggressively spending? Yeah. So of course you start conservatively, right? Like uh if you're cost per purchases 50, you want to start I don't know around 45 and so like or 10% 10 to 20% below and you might not get delivery. Then you play it slowly. And by the way, this is not for let's say the not for beginners like [laughter] so why I'm saying that because this works well at large accounts with large learnings that they are spending 50k a day anyway type of a thing and they want to see if they can spend more than 50k a day if they can spend 100k a day and

[45:45] this is one of the ways to unlock that and again with the campaign limits. So for example, how do you ensure that you you don't spend 200k but you want to spend only 50k? You set a campaign uh spend limit and then you put a 50k as a you say like regardless of how good is my performance I don't want to spend more than 50k. So you put that 50k but that 50k is not stopping your auction dynamics because auction dynamics are based on the bids and the budget and the cap is like as far as I know it is another control mechanism that is not

[46:20] going into the whole auction in terms of pacing and the bidding and and and all that stuff. So it's just a guard rail. So that's the way you control it. So you have the bid as a control mechanism and then you have the safety mechanism by setting up a max limit and then you want to spend another 50k. You're happy with the first 50k then you increase your campaign limit to 100k then you give that room again and then now you want to spend more so you increase it. So basically you are still in control you're just bypassing the auction dynamics in in in this way as an

[46:53] example. So and we tried this for multiple accounts and we had some great results. It didn't work for also some they didn't get delivery or when they get delivery it was too quick too much and then they had to start at like a 10k and 20k and so on per day and so on. I don't know what happened later. I know the cases that it worked and then the cases that didn't work I didn't follow up on the latest status because I'm also busy with my full-time job. But yeah like there are many this kind of things you can test and this is of course one

[47:25] extreme. You should not test stuff only for sake of extreme testing but there are simple stuff as like the valuebased rules. You can test those bit modifiers that exist forever but now it used to be available only through FMPs. Now it's available within the UI. like if you have performance difference above 20 30% among the let's say the placements or demographics and such it is worth testing it and and yeah so there are a

[47:55] lot of tests to do across all channels. So when you want to look for room for growth like you should look at both vertically and horizontally. So horizontally will be like at the channel level okay are we really maxing out at every channel? Do we do I have the response curves of all the channels? Can I really double on this channel or rather spend x% on this channel and y% on another channel channel? This usually will come from the MM output because MM will consider all your various sources of data although mostly it's looking at the impression span and conversions and such but it does know the seasonality

[48:29] because you upload your last 24 months data and such. And then when I talk about the vertical side it is about I don't know on meta I gave already like a few examples on we talked about the audience testing we talked about like clustering based on CAC or LTV depending on how you optimize we talked about the value based like there are like 50 plus I think tests that you can do but the prioritization of this test should come from your current data and then on UAC like it should start as basic as okay what are you bidding on are you bidding

[49:02] on Firebase events or your MMP events. Are you still optimizing towards an install or are you optimizing towards a lowerfunnel? And have you tested excluding your brand keyword? You know, what is incremental value? So at the different stages so starting from so I think we talk a lot about the recent examples were more about like large advertisers with large budgets where they can invest into in-house teams to build all this mm or get a third party tool to basically utilize the outsource the mm but at a small scale you can also just turn on and off like large advertisers cannot afford to do that but if you're a smaller advertiser like if

[49:44] you're just spending Okay, I don't want to uh number now because I said small advertiser. Let's say if you can afford to turn off the channel regardless of your size. Let's put it this way. If you can afford uh just do it. If you cannot afford then you have to use these kind of either the geo lift or the blackout or all that or there are tons of scientific statistical studies you can run for the sake of incrementality and such. But the most basic one is sometimes switch on and off and then and observe the outcome. And again there are no one right way to do things. It really

[50:19] depends on your size. It really depends on your market vertical and such. So I know some of our folks within our industry like they have great advice that are applicable to most accounts but they're not applicable to every single account as well. So that's why I mentioned at the beginning of the talk like take them as inspiration. Take them as like like if you hear something this didn't work at all be like great I will try it but later but don't be like oh we should not even try that unless you have also evidence that it will also not work for you or there are the cues because at

[50:57] the end of the day we have to prioritize how many things we can test and yes there are 100 things to test. Yes it's good to utilize someone else's experience instead of you finding out by yourself. It's good to leverage other people's learnings and that's definitely a a good point. But what I'm just saying is like just don't take it as like don't take out that test from your road map because someone said it didn't work for

[51:20] them was my kind of main message here. Yeah, this is a really fun I I feel like this was also a fun contrast even or not a contrast but a complement to the signal engineering discussion in that Thomas talked about and you know the industry's been talking more about about signal engineering as a way to kind of manipulate the algorithms if you will to to coach the algorithms to influence the algorithms and then so many things that you talked about today are also ways especially that those last few comments around the you manipulating big caps and things like that like the point is there's there's so many ways to tinker

[51:55] and tinkering with bids and caps and all those kind of things and and maybe for some like focus on your creative, let the algorithms do their thing and you you get enough performance and you don't need a big team and like whatever it works. And again, that's kind of your point, right? It's like don't feel like you need to test all of this. Don't feel like you need to try the bid cap, but at some point maybe you do need to scale and you're hitting a wall. There's a hundred different things you can try to break through that wall. And I think you've shared a lot of really fun kind

[52:28] of contrarian or just maybe less talked about tactics to help break through those walls when you are struggling. When you've done the test and realize you do actually have a problem, it's not just a fact of the business and that there really is a problem. There's like so many different ways to tackle it and just try crazy stuff. So, it's been a

[52:46] super fun conversation. Yeah. I mean I also want to add one more thing to what I said in terms of the the testing like sometimes people hear something and they're like oh like we heard I don't know someone said mapping trial to purchase works better like oh should we also do that I'm [clears throat] like what is your reason what is your like diagnosis on like how did you come up with that hypothesis and like why would you change stuff so that's why also like there's a misconception on oh let's just test everything. But that also sometimes contradicts with don't touch if something is working well. And the

[53:25] particular example I'm talking about is event mapping for example because you mentioned also signal engineering and stuff. It's also at the basic level like sometimes you don't need to come up with specific demographics triggered events. Sometimes it's also about optimizing what event you map on meta side. for example sometimes. So there are typically three options I see people do. They either map their trial to trial, they map their trial to purchase or they map their trial to subscribe to give better signal to the algorithm and some people just do it purchase because everyone does it. So it is great as long as it works for you. If you're happy, if

[54:09] you're profitable, if you've been using it for 5 years, if you scaled like don't touch it. Don't try to change it because your whole earnings is based on that. On the other side, if you're not happy with the purchase optimization because your cost per purchase is almost higher than your LTV and then if you're not running the account because of that, then it's time to calibrate slash try something else. And of course, this should not be the main thing to look at. You should first analyze the account to understand why things didn't work and what. And one of the levers also could be on the event

[54:48] mapping/ the signal engineering side of things because when you optimize towards a purchase where the meta has a lot of information on and which is great because it goes after people who actually purchases things. But also you should think about okay these people are purchasing anything. Just because someone bought a skirt doesn't mean that they're gonna try your app. You know, there's also a bit of the relevancy there. So I remember like I think it was sometime last year had an app that had a really high CAC and they because they mapped their trial to purchase and then all we did was mapping their trial event

[55:29] to trial and their cost per purchase/ the actual cost per trial went down by I think around 35%. only with that change. Same creatives, same campaigns, just different event optimization. Also, their CPMs changed as well because if everyone is optimizing towards a purchase, so and then the guy is, let's say, this brand is optimizing towards purchase because his AOV is $200. He's willing to pay up to, let's assume $200 versus a cost per trials worth for your business is $15, let's say, or $10. How are you going to compete with the guy who is willing to pay 200 from the auction dynamics point of view? That guy

[56:13] is spending more and the way to give the auto bid a signal is you increase your budget. You indirectly increase your bid when you have the auto bid when you don't have the bid cap and cost cap. And then that guy keeps increasing and then you're just getting crushed and he's maybe getting the highest intent and you're getting the lowest intent purchasers and stuff like that. So there's a mismatch on like what you want

[56:37] to get and what you tell the algorithm. And then on the other side when you map your trial to trial, you're honest about your kind of intentions and the app because the only way for algorithm to know your business are the events you're sending to your events manager to your meta events manager. So also the events that you map should also logically make sense and should ideally have a funnel logic as well that I've seen the other day someone mapped subscription cancellation to add to cart trial converted to something else and there is a subscribe event purchase event there's everything. So for meta like what is

[57:22] this? Is this a ecom shop? Is this a subscription app? Is this whatever? And all these advertising businesses that are worth billions are built on machine learning because machine learns from all the signals. And if you send this kind of wrong signals that doesn't make a funnel and then the algorithms are calibrating the whole auction dynamics because they're optimizing their own eCPMs with likelihood to convert based on your typical funnel and typical behavior. Like these are like my personal opinion but I'm just interpreting a story here. Like algorithm is not confident to show that advertiser's ad to that user because there is much clearer signal on another

[58:08] advertiser's funnel based on this user's historic activity that is known to meta. He the meta algo will rather show advertisers B's ad to the that user and not A because he doesn't know if this user will convert or not. And he knows that if this user will not convert on the advertiser's A's ad, he will not increase his budget. And Meta optimizes towards the spend/ the advertiser value, right? So, and for meta to work at optimal level, signal engineering is one thing like to improve what you're sending, but fundamentally you should not send mixed signals for the sake of what you're potentially thinking as signal engineering as well. So in

[58:53] general keep it simple, keep it logical. Don't send all possible events because you can. I think that's what also people do. Oh like why don't we track also this because look we can send this event. It's difficult to quantify what I just said, but it is just talking to Facebook product managers like knowing how this platform learns from the data, learns from the signal and so on. Like this is my interpretation of that and I see that the simpler accounts also do better because of that, right? Like it used to be different like you had 50 types of campaigns and so on. Now like you have

[59:29] the broad interest and the lookalike that is all you need. Uh, of course there are more, but I'm just simplifying past versus now because algos know exactly whom to go after based on the signals that it gets and based on the signals that it has on the user. So the signals that your app has is if it's totally different than what users engage with all the other apps and all the other businesses, then there's a mismatch, right? And that's a whole business model of meta. It will find exactly the user. It learns from the users that it showed impression to. It gets the conversion as a feedback and

[1:00:05] then it will try to find more similar people. So when you send this mixed signals mixed funnel that doesn't make sense from the auction calculation point of view then you're also potentially losing out on this kind of opportunities. So having said that like if you're happy with your current event mapping don't try to just challenge it for the sake of challenging. If you are unhappy and thinking there is a room to try something different, I would suggest create another event that gets triggered at the same place. For example, the trial start. Add another trial start event in your MMP. Then map that to let's say to subscribe. Then run a split

[1:00:44] test purchase versus subscribe to see like which event is more resonating. Of course, like the history of each will not be the same. So I would at least wait a week or two so that the events manager will build some history and understand what does this event mean for their algo and so on. Again it's difficult to quantify wait one week versus 2 weeks but it's just to build some data history and then run the test and then see how it is. And that's exactly what we did for this particular project and then we saw a 35% improvement just like that. So sometimes you shouldn't think too complex on okay

[1:01:20] but maybe they also come back on day three then I will fire this super high LTV in predictor event that you can also do that but also just step back zoom out okay what am I telling the algorithm how does algorithm know me like is algorithm XY Z so it should start at this basic level and so on yeah which was to your broader point that there's so many things to play with but don't break things just because people say you should try and break them like form a hypothesis for why this would perform better and then systematically test it versus assuming because everybody said signal

[1:01:57] engineering is going to work that you should be doing that exact form of signal engineering. So yeah, it's been so fun chatting through all of this. Anything you wanted to share as you wrap up? I know you do take on contracting work, but you're also working on your own app studio if people did want to get in touch or are you taking clients or anything like that? Thank you also for your insights and always happy to help happy to chat when it comes to like Mart when it comes to UA challenges when it comes to attribution CRO ASO like that's what I've been dealing daily for the

[1:02:30] past 12 years because of my main project and always happy to chat always happy to learn from you as well and uh learn from the community so that's how we all grow what what's the best place to reach out uh just LinkedIn or what's the best place get in touch with you. Yeah, LinkedIn would be cool. All right. Well, such a fun conversation. I really appreciate you

[1:02:49] joining me. Yeah. Thank you. Thank you so much. Awesome. [music] Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com to join our private community. [music]

---

About this transcript
Read from YouTube's own caption track and laid out by ViewRank AI (https://viewrankai.com).
ViewRank AI finds the videos already beating a creator's own average on Instagram, TikTok and YouTube Shorts, transcribes them from the audio itself in more than 60 languages, and turns what worked into new ideas and scripts.
Free transcript tools, no account needed: https://viewrankai.com/tools
How to read any video this way: https://viewrankai.com/llms.txt