# The Post-Attribution Playbook for Growth — Eric Seufert, Mobile Dev Memo

Channel: Sub Club by RevenueCat
Video: https://www.youtube.com/watch?v=0dZ-wwsNLWY
Duration: 54 min
Language: English
Words: 11391
Transcript page: https://viewrankai.com/tools/youtube-transcript/0dZ-wwsNLWY

---

[0:00] Hello, I'm your host David Bernard. My guest today is Eric Suffort, media strategist, quantitative marketer, author, and investor. Eric currently shares his musings in the mobile dev memo newsletter, blog, and podcast, and invests via Heracles Capital, his early stage venture fund. On the podcast, I talk with Eric about how measurement dysfunction paralyzes growth, why diversifying channels for the sake of diversification actually hurts performance, and the futility of trying to interpret why ads win. Hey, Eric, thanks so much

[0:37] for coming back on the podcast. Great to be here, David. Thanks for inviting me back. All right. So, we put a Google form out there for folks to ask questions and we got some really good questions in and so I'm going to get to those in a sec, but I was kind of surprised that nobody asked about how to use AI, what's going on with AI. I mean, it's just like that's all everybody's talking about on Twitter. Uh, so I was kind of expecting more questions around AI. So, I'm going to selfishly lead with my own question because I I do feel like we're we're at a bit of an inflection

[1:06] point where, you know, things are still early, but it feels like you're going to be left further and further behind if you're not at least starting to experiment. So, what I wanted to ask you is like where do you think the lowhanging fruit is right now? Like teams that you're seeing be successful, what do you see them doing and using that's like effective today? not like oh this you know 6 months from now this will actually get good but like what's good today there are a lot of dead ends that I see companies pursuing with with AI right and I think my advice here is is if you

[1:36] are going to embrace AI and like I think it's important to just kind of maybe take a step back and and sort of define what what we're talking about when we say AI right it's sort of like a catch all term at this point but it shouldn't be though it shouldn't be so like and so I think when you're talking about AI you're fundamentally talking about replacing sort of human decision-m with some other mechanism, right? Like that's that's ultimately a big concept what you're talking about, right? And when people talk about AI, they tend to focus on the outputs. I write a prompt in a chatbot

[2:08] and there's a bunch of text that that gets generated, right? Or I I write a prompt in a text to image tool and and there's some images that get created, right? And I think that's probably not the substance of how you transform your business with quote unquote AI, right? Here's my advice like if you want to actually embrace this in a transformative way for your company and not like in a superficial way. First of all, start with first principles. What

[2:30] does AI mean to your business, right? So, you know that I'm among other things that I do. I'm the chief strategy officer at the fabulous. We make health and wellness apps. They're all subscription, right? So, I've got a a genuine reason to be here on the sub club today. And so, I'm leading this effort at at Fabulous. And what I did was like I just worked with the founders and I was like, okay, let's let's start with what are what are our principles around AI, the use of AI within the company. like let's just define what we want to actually achieve with with AI, right? So this company Zapier, they

[2:57] released this document that they were using internally which broke each organizational function down into like a line item on this matrix and then going across the columns were like the the levels of implementation of AI from like unacceptable or like no implementation to like the full embrace of AI, right? And they just sort of described what each team should do to sort of cascade across these columns. And so we did the same thing, right? This isn't like a top- down mandate, right? We presented this to the teams and we said, "Right, fill it in. You tell me what is the sort of complete sort of transformation of

[3:35] your organizational function through AI and you tell me what is like an unacceptable sort of non-implementation of AI and then we'll just sort of plan out a road map to get from no implementation to the complete transformative implementation and then we'll decide what resources are needed right and so the teams themselves get to decide how they define that roadmap and pursue that roadmap. But

[3:59] there is no option to not do that. That's how I would start essentially. How do we absorb this into the culture of the company and make sure that every single functional team feels enabled, right, to do to do this and and they also feel sort of like that they have the agency to define what that implementation looks like because I think that's really important, right? So that that's where I would start on the just kind of on the marketing side. I think again not focusing on outputs but actually focusing on automation and replacing the sort of human effort with this sort of like machine-handled mechanism thinking about anything that

[4:36] would move the needle right so if if you want campaigns to be optimized in real time like that's not something you're going to build you're going to rely on Facebook doing that or Google doing that or Pinterest doing that or Amazon doing that like they have those tools or Tik Tok doing that right like those are those are sort of like platform imperatives that's not for you to build and to your earlier point. Those are actually machine learning models, not generative AI models. And I think Zuck tried to make that distinction on the uh latest earnings call is that the generative AI stuff is not what's

[5:08] powering the increase in the efficiency of ad spend on meta. It's actually just getting better and better at the machine learning models. And that is a little bit of a catch 22 right now and kind of a a confusion in the industry is that generative AI, chat bots, image generation, all these kind of things, it seems so magical just like plastered over everything, but there's a lot of things it's not good at like data analysis and other things where machine learning models which now kind of just get lumped into this umbrella term of AI like you were saying earlier. And so part of it is also like picking the

[5:41] right tool for the right thing. And so like you know to your point the platforms are going to be so much better at that optimization and they're not using generative AI to do that. They're using these machine learning models that they've built up over decade to do that and all the data that you don't have and

[5:56] all that kind of stuff. Yeah. And that's that's really important, right? The data. And so that's a great distinction. I think a lot of people don't make it. Right. So what what Meta calls those those buckets, right, is core AI and Gen AI. and Gai. I mean to to be fair I mean they have they said they have 2 million advertisers using their Genai products for creative production but a lot of that is still pretty superficial. I mean they have animations which is kind of a big deal but a lot of it is still just like ecom you know sort of swapping out the backgrounds but that I mean his his

[6:25] point there was that hasn't yielded that much extra efficiency yet because they're still in the early stages of rolling that out right but what has and what I called out in my earnings analysis but I also wrote another piece uh called like AI is not the metaverse saying like the AI is really generating truly substantive efficiency for advertisers right this second it's not this far-flung far-fetched sort of like destination. It's it's working on behalf of advertisers right now in generating sort of regular, you know, sort of improvements or optimizations to their ad campaigns. But those are those are things like uh Gem, which they've talked about, Lattis, Andromeda, and I had Meta

[6:58] VP of of AI on the show to just talk about all those tools. But it's really interesting, and it's like yes, every quarter they're pointing to those tools and saying they drove 5% more efficiency. And I think that's something that people misinterpret because they say 5%, who cares? Five, but no, but that compounds. That's 5% a quarter or that's 5% every half year and then also when you unlock 5% efficiency in ad spend what do you get more ad spend the next cycle right and then so that grows faster so I think people like sort of they don't know how to interpret that so I would leave that kind of heavy lift

[7:26] into the platforms you can't do that so what can you do right and again if you're not focusing on the output you're not focusing on just creating a bunch of creative right that's probably not going to move the needle that much what should you be focusing on just automating tasks right and a big one of those is the way I see people using AI is for creative prospecting. So just looking at what

[7:45] your competitors are doing. Yeah. Pulling that information into an S3 bucket, whatever that information is. Maybe it's ads, maybe it's other other stuff, other things that are visible to you. And then using some sort of agent to sort of uh interpret it and that that would have been a full-time job 3 years ago. That that's a full-time job. And and every you know, every big scaled app advertiser was doing that. They were

[8:05] looking at the Facebook ads library. They were putting that into a dock into a Google doc, and they were sending it around. components like hey here's what our competitors are doing this week what what lessons can we take from that but now you can do that in an automated way using tools like LLMs to sort of interpret what you're seeing interpret the sort of concepts from these ads and tell me why you think they're working right and so that's a big thing that I think a lot of people don't appreciate it's like creative synthesis but like scaled way beyond what people were doing with like one person working on that

[8:34] maybe full-time or halftime interesting vector to be thinking about this on and to your exact point is that one of the the challenges we still have especially with generative AI crunching numbers is hallucination like I tried to just get chatpt I used uh pro I use thinking I still haven't done deep research but I just tried to get it to translate a list on a web page into like a formatted list that I wanted and exclude some things with a specific criteria right it did great for the first 40 things on the list and then it just started hallucinating and making up, putting stuff in the wrong place, like

[9:12] it just like really fell down. And so that's a really great example of like understanding the the limitations of generative AI. You probably don't want to try and like have an AI in between your ad buying and making decisions on pricing and stuff today. There's just so much risk there. But to research creative, especially with competitors, and come up with a hypothesis and things

[9:35] like that, hallucinate all you want. We're going to go test that anyway. Okay. And the ultimate deciding factor is Meta's algorithm that's going to pick the winning creative anyway. So, you're just like generating ideas. Hallucination isn't going to break things, cost us tens of thousands of dollars if we're trying to buy ads based

[9:52] on it or whatever. You know who Jason Lumpkin is? He's disaster. Oh, yeah. Yeah. He's an investor in Revenue Cat. Yeah. Very. Yeah. Is he? Oh, wow. Um, so he he was chronicling his experience with Replet, like building an app, vibe coding an app from scratch on and he was just like every day he was kind of almost like

[10:07] journaling like here's what I did today. And then one day he's like project's over. Replet deleted my production database and like it's I'd have to start over like so I'm I'm I'm giving up. That kind of stuff happens. I mean I think these tools do really well at very specific discrete tasks. They don't do well when those tasks are chained together. And like there's this idea of like temperature in machine learning like you want to add a little bit of noise whenever the sort of output is being selected like with an LLM essentially what you're doing is you're doing this this very like conditional probability to predict the next word

[10:37] right and if you think about like attention and and like the transformer mechanism like what the the big sort of innovation there was it could look at like a very long context chain and figure out how each word sort of affected what what words coming next. But when you do that you still add in like a little bit of noise to determine like what the next word could be. Now you you imagine like you're doing that and then you know you're stacking that and so like when you give it like especially a very complex task particularly the next word it's not that difficult but like a very complex task

[11:01] you stack that noise that stochasticity it compounds and that's where you get like hey here's a 40step process every step in a way there's a little bit of stoasticity a little bit of randomness that's being thrown in there and like by the time you get to the 40th it's like a game of telephone the way I like to approach these tasks is like very discreet and I'm intermediating everything right here's the output okay I'm taking that output and then I'm sort of checking and I'm giving that back to you and I'm saying now do the next thing. And so I think we're still in that phase where there needs to be guard

[11:30] rails either human oversight or just throw it at something where the downside risk is pretty limited and contained. Yeah. No, that's fantastic. Any other specific examples before we move on to the other questions of things you see working today within those limitations? When you talk to companies, especially in the app world about the use of AI, they immediately jump to creative. And I think that's probably the least valuable place to apply this. I mean, yes, you can go from like 10 creatives to 200, but a lot of times it's just people are taking their 10 creatives and those themselves are variants and they're getting 200 variants of one concept,

[12:05] right? That's not actually doing anything for you. There's just diminishing returns on taking 20 variants of one concept and going to 200 and and they're they're like vanishingly small gains, right? What you really care about is the concept. And actually, it's coming up with concepts that you yourself couldn't come up with because if you could just come up with them, the AI is not doing that much more to sort of influence the performance. Sort of like aha moment with this was the last company I worked at. I built this tool called Draper. And that's just what it did. It just it just created variants of of ads. And this is like 2018, so people

[12:36] weren't really talking about like AI at that point. This wasn't even like machine learning. It was just created a bunch of variants like all permutations of these different ads. What I would do is we would just deploy these on Facebook all the time. Like there's just this constant cycle of this deployment and then we had like a standup every every week with the whole company and I would say here's the ad that worked the best this week. I have no idea what it's going to be, right? But like no person touched that. That was just this process running in the background. If you could mass experiment like that then you could

[13:02] find and and then Facebook was was at that point there was still VO was like a year old and so that was the way that they did the audience pairing was is VO based on the on the the sort of value estimate. I couldn't foresee what audience this was going to be targeted to anyway. like let me just feed the beast with as much creative as they need and they find those audiences those high value audiences and they test all this different stuff and like that was a big awake like aha moment for me I shouldn't have any sort of preconceived notions about what's going to work like and I

[13:27] actually posted about this maybe a month ago and I got a lot of push back but my my point was given the use of pmax it's it's pretty text based but but advantage plus like there is no point for you to try to interpret why an ad won or or didn't win right there's no point what you should be interpreting is if when you get a win or the win rate increases is the process worked. Now maybe the process took a new input and that's the learning, right? But it's not the output because that was random. That was utterly random. Why that worked was utterly random. And if you try to sort

[13:57] of deconstruct it and take a learning from that, you're just wasting your time. What you should do is okay, how am I sort of changing the inputs such that I'm getting a higher win rate and let the machine do its thing. That output is irrelevant. That output is is cannot be interpreted by you. You can't understand it. You can't understand why that worked. Don't even try. If it worked though, what did you change about the inputs? And that's what you learned,

[14:20] right? The process worked, not the ad. So, I guess part part of what you're saying is use this generative AI because it is noisy because it is going to just come up with random stuff. Let it go crazy on concepting to generate ideas that you're just would never generate and then feed the beast. to your point earlier like all the major platforms now do all sorts of crazy testing where if you can just feed the beast and I like that like what's the process to feed the beast and iterate on the process to feed the beast versus thinking you can like figure something out and then make 10

[14:52] more creatives that are going to win because you've found some insight like just create an insightful process to feed the beast and let the beast do the work because because at the end of the day you have no idea you have no idea who that ad was even shown Yeah. And and and but but even if you did, even if you did, and here's where we're we're heading to, this is really the reason why people push back against this. I mean, people are saying, "No, that's of course you could learn from that because then that that creates a feedback loop." And I was like, it's it's a kind of a I mean, I don't want to

[15:20] say it's a difficult or complex point, but I think it's just counterintuitive that like, no, no, what you care about was the how did you change the process to improve the outcome, not what can I learn from that outcome to then sort of like optimize the inputs, right? That's not something you can interpret, right? But I think and and so that kind of scares people because ultimately it's saying artists aren't important. There's no creativity. It's a it's just this is just this optimization problem. And I think people I understand that why that's like uncomfortable. But and I I can understand the push back too. I mean

[15:50] like look anytime you're talking about sort of removing humans from this process, it's scary. Look, I mean, you read the news, especially in gaming. I mean, King, they they just laid off whole studio, the the company that makes Candy Crush, and they said or like some of the people that were laid off said, "We spent the last 6 months training an

[16:08] AI tool to do our jobs." And so, it's this is having an impact on employment right now. And so, that's scary, right? Like I I had a professor in undergrad, an econ professor who said, you know, you you can't talk in this sort of like dispassionate tone about dislocation from innovation because people are losing their jobs. And so if someone loses their job, they don't they don't care about the rational argument that like well actually the economy is going to be more efficient

[16:32] because we have this innovative thing. They feel they feel like it's a conspiracy against them personally. You do have to approach this with like total empathy and sensitivity, right? Um but that but I mean that's that's just the reality. I understand why it's a sensitive topic, but I think where we're heading and this I think again some people view this as like kind of dystopian, but what if it didn't even like what if the the ad was just incomprehensible to you, but it triggered something in your brain and and that's I think ultimately what people don't want to recognize that that's already really the case. Now, it

[17:02] may be a picture of a shoe and it's a dog skateboarding with wearing this pair of shoes and you people like that cuz the cute dog. That might not be the reason at all. I mean, we're just assuming that we're making a lot of assumptions when we sort of try to deconstruct these ads to understand and a lot of times they're just post hawk rationalizations. Oh, I like dogs, so everyone must have responded to this ad because of dog. We have no idea what caused that impulse to click and we shouldn't try. I I I think we we can't we can't we must acknowledge that we

[17:28] just can't do it. The last thing on AI before we move on to the questions is just where where do you see all this going? I mean, you've kind of dropped hints along the way. Zuckerberg, I think, said on a interview or maybe one of the earnings calls that he sees in in the not too distant future and and as you just alluded to that that they're

[17:46] just going to generate all the creative. They're going to generate the ideas. They're going to know the person so well that each ad is going to be personalized to the individual and iterating on creative is just going to kind of go away because they're going to be so much smarter at that. So that that's one vector. Happy for you to dig deeper into that one specifically, but what other ways do you see this going over the next

[18:06] 12 to 18 months? My theory is that Facebook could deploy a change tomorrow that achieves what you just described. Like they are slow rolling this because they know that they have to get people on boarded and comfortable with it. And if they did that push tomorrow where like, "Hey, by the way, just give us the money and we're taking care of everything else." Like people would be reluctant um because they wouldn't trust it yet. And so they have to kind of be very deliberate and measured with with

[18:33] the pace that they roll these tools out. And so on that timeline, just given that restriction, given that that sort of like comfort restriction, my sense is in like 18 months we're at kind of brand specific full fidelity video ads being generated and without prompt, just based on past performance. and and they're not auto deployed, but you can go through them and say yes, no, yes, no. And and I think that's probably like on the 18-month timeline. They could do that right now. People would be very reluctant to use it. Where you go from there is just auto deploy, right? Like just do it and and I don't need to be a

[19:12] bottleneck. But I think within 18 months, you get you get to that point. Another like sort of hypothesis I have, it would be regulated by comfort. Could you imagine if you just said, "Hey, Meta, optimize my landing page. I'll put some code in here." you've got the the pixel. I'll import another JavaScript library and you just render that in real time. You decide for this user like what and and here's, you know, I'll give you some guidelines. I'll give you some guardrails. Here's usually what the onboarding looks like, but you optimize it for that person, right? I was on a podcast called like the marketing operators and I was talking about this

[19:44] idea of signal engineering and one of the hosts I could see like it it really clicked for them and they wrote later on on Twitter and said the idea of signal engineering is not limited to the existing like events that you have. You can do whatever you want. You could create hurdles for the user to clear that potentially are good proxies for LTV. And if and if that is, then that's what you do. You created some problem for the user to solve or or some hurdle, some some sort of obstacle to accessing the product. But the most high intent users, they will do it. And that could

[20:14] actually be a very strong signal of ultimate value. And so one of the host posts on Twitter after they created a capture for no reason other than to test the user's intent. and it drove like a 40% increase in rorowaz or something. So like but that's that's the essence of signal engineering. It's to create that test for intent and if the user clears it, they're a high value user. They send that back to the ad platform. Let them optimize in real time against that, right? And so I think that kind of stuff is really fascinating. Now what if Facebook could do that for you? What if

[20:45] Google could do that for you? And would you think that they would do better at that than you could? Yes, I probably would. Now, where that gets a little, you know, just political is like, are you going to get the product manager handing that over? Because I think the marketing people have been comfortable with this for a long time, especially at SMB type, you know, direct response advertisers. Like, everyone that we're talking to right now is like, "Give me, give me, give me as fast as you possibly can." And someone at like PNG or like Clorox is saying like, "Over my dead

[21:15] body." There's a mentality divide here. Like now imagine you start bringing product managers into the mix which tend to be a little bit more murial. They they look at themselves as like artisans I think to a greater degree than like a user acquisition manager would. And they're saying no way. Like I control this product experience. I control the first touch point that the user has. I'm going to give that to Facebook. Like no way. I I could see there being a little

[21:37] bit of tension there. Yeah. Uh Thomas and I talked quite a bit about signal engineering on the on the last episode of the podcast. And one of his points though was that and this this gets back to like Pmax things that Google's done, things that you Facebook is kind of doing with their value optimization stuff is that do you really want to hand over control of the outcome to Facebook in that you know if they know your exact rorowaz and they're optimizing to get you one penny of profit on that rorowaz whereas what signal engineering does provide you an opportunity to do is fake that in a way

[22:14] that gets you what you need out of the algorithm while still maximizing for profit on your end versus giving it all over to Facebook, which is I think an accusation a lot of people have made against Google's ad uh products is that they'll hit your target rorowaz and then keep spending but sending it to all their crappy, you know, ad placements that they know aren't going to perform, but hey, they already they already got you what you asked for, so they're optimizing for their own, you know, revenue and placements and filling inventory than they are for optimizing to to your ultimate outcome of being the most profitable that you can be. So that

[22:52] that's another level of comfort that UA teams will have to get over and the kind of cynical view of these bigger companies and the way the algorithms are going to perform in order to hand off things like signal engineering to the bigger algorithms. I've clashed with Thomas on this point since 2016 when UAC was rolled out, right? I I I wrote about UAC when it was introduced and I said, "Look, the cynical interpretation of this is that their goal here is to maximize spend, right, and to hit exactly your rorowaz target, right?" And so like that became a lot more relevant. UAC was the precursor to PMAX and AAA uh automated

[23:28] app ads was the precursor to uh advantage plus. So we've been dealing with this for years and years, right? These tools are not new, right? I wrote this or I did this podcast a while back called uh commerce at the limit because people are scared of this stuff right like for all the reasons I just spoke about like well it's you know it's disintermediates humans from the pro like what what am I here for then and but my point was like look I mean this has been happening this has been the modus operandi since 2016 with UAC I mean that was a precursor to to PMAX

[23:56] that the reason they started with the app environment is more controlled the apps go through review they're downloaded in the app store you had prior to AT you had a totally transparent view line from the ad to the to the usage. And so they could just much more easily in this sort of sandbox environment, understand all those signals and automated. It got hairier with with ecom. But look, there's still people working on apps, right? That didn't displace everybody. You know, things a lot more efficient on the marketing side. I wrote this piece

[24:23] called Satisficer's Remorse, right? Where this idea that like do I wish they were optimizing for my spend adjusted rorowaz? Yes, I do. But I also understand that if I was running this, I might not have even been able to spend as much as I did at that level of rorowaz. They're probably more efficient. So that that satisfies like I'm accepting the outcome that meets my rorowaz target. And then the remorse part is like I know they could have done better, but I couldn't have. I know if they were a nonprofit, I'd be making more money, right, than I am now. If they were a nonprofit and they were just

[24:56] looking out for me exclusively, I would be making more money. But they're not a nonprofit. If I was in charge of doing it, if they didn't give me the benefit of these tools, I probably would have performed worse. And I think when people say, "No, I could do it better than they can." For the most part, I think most

[25:10] people are kidding themselves. Yeah. Yeah. Yeah. You know, it didn't occur to me until just listening to you talk through this, but the ultimate kind of dystopian, especially for the app industry endgame of all this is that at some point Meta and Google know better than even we do the the goal and the outcome that the individual wants out of a piece of software and then they just

[25:36] generate that in real time for the user. I I I mean that that's very dystopian, very far off because you do have a lot of problems around like you said with um Jason Lumpkin where the AI just deletes the production database. Like I think we're a long way from you know fully AI generative experiences that's going to have pers persistent storage and like track your food and macros and calories over time. It's like as you were saying that I was thinking of your earlier statement about the team at at King that was training the AI that eventually replaced him. In some ways, we're going to be teaching Meta's algorithms better

[26:11] and better exactly what the user outcome is and what they care about and what they value. Uh, and that's very valuable data, very valuable information, but I think that's uh a long way off if not impossible. So, we'll see. You could take this to any sort of extreme you want. Like I I prefer to focus on because I'm like an AI maxi, right? I I believe this is like transformative. I think this is going to benefit society sort of inflection point for the better. what I like to focus on and and you like taking it to that like sort of extreme where like Facebook is just producing an app right for me to

[26:41] download in real time and all the benefits uh acrew to Facebook was like okay well then no one's clicking on anything no one has any money then the economy collapses what I think is really exciting though is this idea that sort of anybody can be an advertiser right so like if anybody can be an entrepreneur because of these tools then it actually becomes a lot more difficult to get attention because you're competing with a lot more products And so that's like an issue on the app side, but like I think there's a whole class of entrepreneurs that exist already that are doing things like lawn care or they

[27:15] run a barber shop or they run, you know, a bike repair service or something and they're not advertising because it's just out of scope. I It's just it's totally unrealistic. But but what if it wasn't? Like what if it actually was just I'm typing in a prompt for what my business is and I'm actually not competing with that many people because it's like more of like a locally oriented business. And so this is really just unlocking a new group of existing entrepreneurs that can be on boarded into the advertising economy, right? And could benefit from that and drive more business as a result, right? Like and so

[27:46] that to me is really exciting. It's like you just empower a whole tranch of people that run small businesses, like truly small businesses, one person companies or just all like sort of like locally oriented companies. You empower them to reach as wide of an audience as as is relevant, right? And that's really exciting. Yeah, and we we've already seen that. I mean, DTOC is a fantastic example, like the proliferation of very niche products. I I don't use Instagram a ton, but when I do, they're so damn good at finding the exact product that is like missing from my life. I mean, it's ultimate like consumerism and

[28:20] stuff. Do I actually need it? like, you know, all the that aspect of like dystopian, whatever. But there's some really cool products that only exist because Facebook allows those creators to reach an audience like me that's niche enough that they would have never been able to like, you know, sell it in a grocery store or sell it at Walmart, but people like me actually want that product. And then there's infinite niches like that that exist in the world. And I know, you know, people go back and forth and you're you're also kind of an ad maximalist and a lot of people just wish advertising didn't exist, but this this is the getting

[28:58] attention for this innovation. Should that just be free? Should you should you be able to build something and then how then do you get attention for that? And advertising is a very effective market driven way to build something innovative and then get attention for that thing you're building. So I I go back and forth a little bit. You know, I'm a little creeped out sometimes on all the data collection, everything like that, but man, I bought my Meta glasses and freaking love those things and create it. It's like at some point there is a benefit to society more broadly. The benefit to individuals there are negative externalities, there's problems

[29:36] like it's not all like, you know, sunshine and unicorns and stuff. But yeah, it's like I think understanding fundamentally how advertising creates new markets and empowers entrepreneurs is really powerful and it's important and too many people just crap on the ad industry without really understanding it and what it does generate in consumer surplus. And Facebook's not taking all the profit like people wouldn't come advertise with them if they were taking all the profit. Now are they trying to maximize their profit but can you also maximize your profit on top of that? And the answer is yes. I mean that's why there's been a proliferation of entrepreneurship and these DDC products

[30:12] and everything else like that. So yeah, I'm not quite as ad maximalist as you, but I I appreciate its role in the broader market and in empowering entrepreneurs. And I think exactly to your point, it's like we're going to see that accelerate, not decelerate in the coming years. One argument that I just find like disingenuous in the extreme when people understand this this market is like well advertising has always existed. It didn't need you didn't need data like you could always just advertise on TV. You could always just advertise on radio or magazines. You didn't need all this data. All this data

[30:46] collection is just a privacy violation. It doesn't doesn't change the fact like it's not it's not enabling advertising. It's like yeah that form of advertising always exists. It continues to exist. But you know what didn't exist? DOC. You couldn't have D TOC without personalized advertising. Like you, it just would not be possible for the reasons that you pointed out. It's too niche. It's too niche. You can't do a TV ad campaign, a national TV ad campaign for this niche product. The economics won't work. But if I could reach the individuals that would find this relevant. And by the way, when you reach those people, the

[31:13] clickthrough rates are still sub 5%. Right? And and people look at that as an indictment. No, that is not an indictment. That's showing you the sort of natural sort of reluctance or sort of friction there is to advertising in the first place. Right? If there was actual manipulation happening, those clickthrough rates would be 100%. Right? If this person is deemed relevant and I was manipulating them into doing something they wouldn't otherwise do or they don't want to do, then the click rate would be 100%. The conversion rate would be 100%. It's sub 5%. On a good day, it's it's four. Like for most products, it's it could be sub two, sub

[31:45] one, and those could be profitable campaigns, right? But the thing is like this enabled new sectors of the economy, and that is growth. And there are are there bad aspects? of course, but you don't throw the baby out with a bath water. You identify the things that you want to remedy and you in a surgical way, you sort of remove those from the workflow. You don't just throw

[32:06] everything out and trash everything out. And like people push back when I kind of make these sort of sweeping statements and they say, "Ah, that's a straw man. No one's actually looking to kill personalized." Yes, they are. Yes, they are. There is there is a a bill that was just resuscitated called the Banning Personalized Advertising Act. They did ban it in the EU. They did. Effectively, it's banned for the biggest platforms, the gatekeepers. And so, yes, there are people that want to ban personalized advertising. I'm the one who steered the conversation to DTOC, but to be honest, it it's the app industry, the proliferation of the app industry and

[32:41] subscription apps specifically. Meta and Google deserve almost as much if not more credit for the proliferation of apps and the variety of apps that we see today. and and so many profitable app businesses for the same reason that they empower DTOC's like they allowed app developers to reach those audiences more effectively, more efficiently. What we see today as the app industry I think owes a a debt of gratitude to Meta and to Google um because they they empowered it just like they empowered DDC. Well, you and I could talk about these sorts of things for hours, but I did want to get to the questions since we since we

[33:23] did allow people to submit questions. Question one, an hour in. Um, given the dominance of a few paid UA channels, how risky is it for a fintech subscription app, and I would say any subscription app to have over 80% of its spend on Meta and Google? What practical diversification levers actually work in this vertical? So, first, is it risky?

[33:44] And then if so, how would you diversify? So I wrote a piece about this a while back, a couple years ago. And like the point I make was I think people feel compelled to diversify because they have this sort of like abstract notion that being totally concentrated in like one or two channels is a bad thing. And yeah, there's risk there, but the reality is that diversifying adds a lot of overhead. It adds overhead in terms of having doing data integrations of and having to create new creative formats, right? and having to, you know, a lot of times these companies, each platform has like sort of a different way that the

[34:18] ads are exposed and so you have to sort of accommodate your measurement to that. And so diversifying for the sake of diversifying is often times a bad idea. And the question is like and then how much could I spend, right? So like I might onboard like a lot of times you hear this kind of like common refrain like uh well the performance is great but at really low spend well okay but that's actually not great. Like what I really care about is my my spend adjusted rorowaz, right? and not just that the rorowaz is high on this particular channel at low levels of spend because could I have taken all of

[34:49] that overhead eliminated it for supporting that new channel and then allocated that budget to meta or Google and seen the same level of rorowaz because if I did I'm actually worse off if I could have right so it's actually when you have a new channel it's the rorowaz doesn't have to meet the rorowaz of the other channels that could have absorbed that budget has to exceed that because you're supporting a new channel and so I think like diversifying for the

[35:11] sake of diversifying is often a mistake. You diversify when you've reached saturation on the existing channels I think and then you look for other channels or when you feel like there's some sort of interaction effect that that channel could produce that boosts the performance on other channels and that often times is the case right often times that's the benefit of running on some other channel particularly if it's more like brandoriented channel so I think that's just that's the way I like to think about it is I wrote a piece a couple weeks ago called optimization models in digital advertising I talk about optimizing towards rorowaz and

[35:42] optimizing towards row adjusted spend and they're very different things, right? And so if I'm optimizing towards rorowes, then I really want to kind of keep spend as low as possible spread across many different channels because I'll get the max rorowaz per channel, right? Because the rorowass and the spend tend to move in opposite directions, right? But that's often times that's not really what I'm doing. I'm optimizing towards maximizing spend with a rorowaz constraint. And in that case, what you want to do is this what I call the waterfall method. Max out the biggest channel, right, until it hits my rowass threshold. Then move on to channel two, which would be smaller

[36:12] like potential spend there. Max out channel two until it hits my row threshold. Move on to channel 3. That's always the approach that I recommend companies take because it minimizes overhead and complexity. And so what do you see as the risk or do you think it's just actually not risky? The risk is just missed opportunity. like how would you classify risk and how people should think about the risk of being so

[36:38] dependent on one or two channels? So yeah, it's missed opportunity. You you could call it opportunity cost risk, but what most people kind of mean when they talk about the risk there is just that performance degrades. And if I've got five channels and one of them bombs, then okay, like there's 20% of my spend at risk. If I've got one channel and it absolutely tanks, which can happen, then that's all my spend. Now, the the issue here is these these things tend to be pretty correlated. Like, why would performance bomb on a channel? Probably because a competitor came in and is out bidding you everywhere. And if they're

[37:11] out bidding you on Facebook, they're probably out bidding you on Google and Snap and Tik Tok and wherever else. So, the issue is I think the risk is not necessarily per channel. That's what I think a lot of times people think about it that way because everyone's had experiences where like, "Hey man, what happened to Facebook yesterday?" Like if clickthrough rates got cut in half or they got cut down by two/3s or whatever for whatever reason and there was just like a blip. But if you're thinking like a structural change a structural change to performance that probably would be correlated across every channel because there's either uh just a change in

[37:38] consumer sentiment or more commonly a competitor came in and just crowded you out. All right. What's the single biggest pitfall you see across mobile growth teams right now? Something that even sophisticated companies consistently overlook. The thing that I see most commonly is just a very sort of chaotic approach to measurement. Just lacking any set sort of like coherency that could materialize in a couple different ways. Like either I see people using competing tools essentially not really knowing which one to trust or how to interpret the output of one relative to the other. I see this as just misalignment across the various stakeholders, right? like often times

[38:16] finance and UA or UA and product and not really being totally aligned on like what you know good looks like, what these metrics should look like for success. I see it as having a bunch of tools that aren't working in sort of concert, right? Like just a couple different data points, right? Like I don't know how to interpret them as a as an ensemble. I just looking at the individual ones and I don't really know what these these things mean as a as a whole. I call it like measurement disorganization is the most common thing I see. And I think the way to overcome that, it's like this is not a satisfying

[38:46] answer and it's it sucks as a process. I don't do that much consulting anymore, but like this was probably like the most common project I was asked to come in on and and it just it's really really challenging and it's stressful getting all the stakeholders together and then just kind of coming up with some sort of plan that satisfies everybody, some sort of model for measurement. And when I say model, I don't mean like a machine learning model or whatever, a regression model. I mean like an operational model for how we're all aligned around what success looks like and like all of our needs are being met by this measurement

[39:18] apparatus because I think one thing that companies just tend to sort of under appreciate is the fact that your measurement model your measurement system is essentially like the heartbeat of the company. I mean, everything flows from that and you really need to be doing it correctly and in a way that's credible, but also in a way that sort of like everyone understands and appreciates, right? And and that serves their different use cases. It's like getting a a group of people together in a room and just finding alignment on

[39:50] that. That that's that's the solution. And that sucks. That's difficult people oriented work. It's not like, oh well, we'll come up with a new machine learning model or, you know, we'll build this new dashboard. It's like, let's get a bunch of people together, understand their needs. The CFO has got totally different needs than the UA team, and the UA team's got totally different needs than the product team, and like, let's just get everyone in a room and understand what they need to receive as output and also get everyone to agree on

[40:16] what good looks like. That's a great answer. Right. I I wouldn't have guessed you'd go into the people side of things, but I mean, we're we're just bags of meat making decisions and uh that's that's a huge point of friction, a huge point of challenge and disagreement. So, you know, one of one of the things we actually see at RevenueCat is we sometimes have a marketing team come to us specifically because the engineering team just won't

[40:44] align with them on their priorities. like the engineering and product teams have their own priorities and and UA and the marketing teams are just like, hey, go figure it out. They're left out on an island. They can't get engineering resources. They can't get time with the product teams to push to get the engineering resources. And so, yeah, it's like it's a problem we see in prospects coming into Revenue Cat trying to solve the engineering problems because their engineering teams just won't give them the time of day. The canonical example in my mind and like I've experienced this and I have like Vietnam style flashbacks of this like

[41:16] very difficult challenging humanoriented problem is product team on boards a different LTV model or a different LTV product because they don't trust the one the UA team's using. That's the kiss of death for productivity. You're going to spend the next two months arguing and that relationship's over. Yeah, man. And it's a great place to go. Like solve your people problems. The technology is not going to fix everything. You need to solve the people problems in in concert with solving the tech problems and data problems and all the other problems. All right, next question. In subscription apps, and then again this maybe was the same person especially in fintech, but this probably

[41:52] applied to all apps. You often have to make budget allocation or bid changes before key metrics like LTV retention or incrementality are fully baked, sometimes within days of launch. How do you balance speed versus accuracy in those early optimization decisions and what frameworks do you recommend for making confident calls with incomplete data? That's a really good question. Yeah, that is a good question. So, a couple things, right? So, one is I wrote this piece a while back called like it's time to retire the LTV metric. And it's, you know, it's a little bit of just a clickbait headline. Um, I don't really mean that. I what I mean is what you

[42:30] often see teams doing is like they'll try to calculate like a terminal LTV on the basis of like 10 days of data or something you only launch one time right but like a product and so you can just kind of wait and you extend the soft launch but like when you're talking about like a campaign level LTV it's like well okay I'm not going to get that quickly and so what I like to do is just try to because again ad spend and rorowaz tend to be inversely correlated so as as ad spend increases this rorow goes down, right? And so what I would like to do is just sort of prove out

[43:00] these frontiers. And so I'm going to spend 5k a day or whatever and you know if I'm hitting 150 rorowaz, great. And then what I'll do is let those cohorts age, understand how they progress, understand what their day 20 rorowaz is, understand what the day 30 rorowes is, and like just build more and more cohorts where I can track them over time. And then I'll push that rorowaz frontier out. So like now I'm not trying to hit 150 on day three or 200 on day three. I'll like let me just see actually these cohorts seem to be progressing to where they're at like you

[43:34] know 150 by day 30 or okay that's great. Now what I'm going to do is I'm going to push and then I change my bid accordingly right and and grow the budget and then what I'll do is track the cohorts more and see where they land at day 60. It's like okay well that's 120 so I'm going to increase the budget more and get and then that'll decrease the row. And so then what I really care about is like where does that land at at like 110 or something? But really what I'm trying to do is is just just iteratively progress that frontier of my rorowaz target starting from a place

[43:57] where like yeah okay if I can't hit 150day rorowaz at day seven with very low spend probably not going to be able to grow this and so it's back to the drawing board with whatever I'm adjusting. That's the way I think is the right way to approach this especially like in a launch phase. If you're just talking about like launch like flighting new campaigns really what you're talking about is the performance of the creative and it's a question of like well when to kill a creative as soon as it's obviously not a winner and that's often times very quickly like same day it could be next day this is getting no

[44:26] delivery not a winner maybe it could perform at the average or whatever but it's not going to grow my ad spend so it's a loser and I kill it. The way I approach creative testing is really just I'm trying to identify losers as quickly as possible. The winners take time to prove out, but the losers are pretty

[44:40] quick to prove out. Yeah, great answer. Next up, what's one opportunity for growth you think most companies in the app industry are missing right now? Either because it's too early, too messy, or doesn't fit into standard user acquisition playbooks. The real answer is the opportunity for growth is just that your measurement doesn't support true growth. It's it's broken or it's flimsy. And so, you're just really trying to replicate what you've done in the past. Like if you don't believe that your measurement can adapt to new channels, new sources, new ad types, you're just going to do what you've been doing and by definition you won't grow. And so if you need to break

[45:16] out of that, you need to build truly like robust incrementality focused measurement. And that's a challenge, right? Challenge for the human reason we talked about, but there's a challenge is a technical challenge too. So that's that's the real answer. And then and then you know assuming that's in place, then you have the world as your billboard, right? like I can advertise in influencer channels, I can do out of home, I could do digital out of home, I could do CTV, could do podcasts, I could do all these things that are supported by the measurement. But often times that lack of growth, that stasis is just a

[45:49] function of like I don't trust the measurement to capably run attribution on these new channels or interpret the changes across the portfolio in in a reliable way. And for that reason, I'm just going to keep everything the same, right? And so that's that's often times what I see. It's like people are just paralyzed by as a function of their

[46:08] measurement not being robust. The fundamental problem in marketing uh half my marketing is performing. I just don't know which half. And that will that will be the perpetual problem that needs to be solved in marketing. You you can you can say that because this is your podcast, but I've actually banned that phrase from my podcast. Uh I I just feel like it's it's too reductive and people misinterpret it. I love that phrase. I hate the way it gets used. The way that people use it is to say this measurement's all smok and mirrors. No one really knows. We need to be telling a story. We need to be connecting

[46:42] emotionally. The only rorowass curve I care about is is the curvature of a satisfied customer smile. That's marketing. That's adver. And you this measurement stuff is just a bunch of hocus pocus. That's not what he was saying. I mean, John Wanaker was like a pioneer of advertising. He's one of the first people to take out a full page

[46:58] newspaper ad. He understood measurement. what that statement is saying is not is that's not a bad thing. It's like I understand the outcome given the inputs. I don't know the mechanics of it. I don't need to know necessarily that No, no, you're you're 100% right and I love that you brought that up because it it brings up such a good point is that there will always be some amount of uncertainty in marketing. The good marketers embrace that uncertainty. Not in a way like you said of just like ah throw their hands up and like oh we're just going to do brand stories or whatever. They embrace that uncertainty

[47:29] to say how much can I be certain on and what are the certainties I can build a process around and that's what good marketing is. It's not throwing your hands up. It's figuring out how to be more and more certain but within the the constraints that you understand you can't get to 100% certainty. The people who think they can be at 100% certain are are just as diluted as the people

[47:53] who think they shouldn't be measuring. So there's this concept of Vickenstein's ruler, right? And I've talked about this a bunch, but like I wrote a piece about it years ago called Vickenstein's ruler and adds measurement. And to your point, like so the idea of Vickenstein's ruler is like I take a ruler to measure a table. Uh well, if I don't trust the accuracy of the ruler, the table's also kind of just measuring a ruler. That measurement apparatus isn't telling me anything about the table. And in fact, the thing that I'm measuring might tell me more about the measurement tool. And I say that because that WMaker quote,

[48:22] the one reaction is like, throw your hands up. it's not possible. So, let's go do brand campaigns and go to can and sip rosé, right? And the other reaction is, oh, good point. I should only do marketing on channels that are deterministically attributable. And then what you do is you fool yourself into thinking that anything is deterministically attri. And so, a lot of times on teams, it's like, hey, we we cracked the code. We found a loophole and and we're running all these campaigns and we're getting this deterministic attribution using all these hacks and workarounds. I'm like, you're kidding yourself. you're telling me just more about your inability to

[48:54] understand what's actually happening than you are telling me about how precise your measurement is. And that's a very bad signal. When you get teams that are like building a consumer product and they're looking for investment and it's like, well, we've got a secret sauce for advertising because we figured out how to hack these signals together to get deterministic attribution. I'm like, no, you're you just convinced yourself of that, but actually you're just getting noise and you're going to be you're going to be wasting a lot of money and that's probably no better than just doing a

[49:17] more like a holistic probabistic model. That question went places I didn't think it would go, but I'm glad it did because that was really fun. Um, last question and we'll wrap up. What's one area you see growth teams pouring too much time and budget into that you believe will matter less in the next two years? Genai, creative tools. I think you could use off the shelf stuff and get most of the almost all the value that you're going to get. Again, I don't think there's that much value there period unless you're doing more of the fundamental stuff that we were talking about at the outset, which is like

[49:46] concepting and that there are not really any like off-the-shelf tools for you. You have to kind of build a system yourself. Maybe there will be at some point. It's probably a good startup idea, but just just cranking out. We went from 50 variants a week to 200 or 2,000, but they're all the same concept. There's no value ad there with the last 1,800 or something. You know what I mean? Like that's that's one thing. It's the testing 50 shades of blue. Like the incremental lift. There's so many other problems you could be solving for than

[50:15] testing that 50th shade of blue. Yeah. Exact. And then the other thing is why are you investing any more time in ad attribution kit that you think that's a genuine source of value or you know sort of a competitive advantage? Don't burn resources on that. It is so baffling to me and I I don't want to dig up this can of worms because we could talk about it for another two hours, but it is so baffling to me that Apple went through all of the things they went through, even the negative press, the tumult in the industry, the loss of app store revenue that AT caused and then

[50:48] didn't actually build something useful for attribution and then just let everybody fingerprint anyway, which is almost more insidious. I mean, I I get it. I think the one thing that came out of AT that was good is that it did at least break a lot of the data broker workflows that you could deterministically find one person and just track them everywhere even when they're on their cell phone and the IP

[51:14] address is different or whatever. Yeah. Anyway, I don't want to dig up this whole can of worms, but like it's just such a mess and it just baffles me to no end that Apple didn't take that opportunity to build an attribution tool that actually mattered and then just let everybody fingerprint. It's so baffling. I have a couple theories here. I think one of them though is they are kind of their hands are kind of tied from a privacy perspective. if you want to honor the sort of like religious zeal that they have towards privacy and I do think they're genuine about that. My interpretation is it was it was a

[51:47] competitive maneuver. That was the motivation there. I don't think it had anything to do with truly had anything to do with privacy. But once you invoke privacy as the stalking horse for that competitive maneuver, then you've got to adhere to the privacy principles of the company which are genuine. And then you when you're trying to build attribution

[52:04] that way, you just can't not functional. like you take their commitment to differential privacy that just breaks the data set and like even if you you know don't implement it in a way that that truly sort of like aderes to it like the principles there even if you do that like okay it's is sort of superficial it still just kind of breaks everything and then and then you you introduced this sort of like crowd and that they they did which is kind of like a reverse form of differential privacy and then you had the scheduling stuff they couldn't implement this to achieve what they were trying to achieve which was competitive

[52:33] disruption without applying ing their sort of culture of this privacy zeal. They just couldn't do it because it was a they they portrayed it as this privacy maneuver and and therefore they couldn't do SK ad network now add attribution kit in a way that is actually functional because when you have to apply all these

[52:50] privacy protections to it, it breaks it. Yeah, I'm I'm uh resisting the urge to dive back in and and talk for another 30 minutes on this topic part two. Um yeah, but man, Eric, this was so much fun. uh really enjoyed the conversation. I think there's, you know, so many things for people to take away if not practical, which I feel like we got to a lot of practical stuff. I think it's really important to think at a higher level, at a almost philosophical level about a lot of these things, which I think then plays out in your decision- making being better because you have a better

[53:25] intuitive sense of like how all of this works together as a market, as a economy. And so, yeah, on so many levels, this was such a fun chat. So, thank you for for joining me today. Cheers, man. Always a pleasure. Hope to see you in Austin soon. Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com

[53:51] to join our private community. [Music]

---

About this transcript
Read from YouTube's own caption track and laid out by ViewRank AI (https://viewrankai.com).
ViewRank AI finds the videos already beating a creator's own average on Instagram, TikTok and YouTube Shorts, transcribes them from the audio itself in more than 60 languages, and turns what worked into new ideas and scripts.
Free transcript tools, no account needed: https://viewrankai.com/tools
How to read any video this way: https://viewrankai.com/llms.txt