Sub Club Podcast: Lessons from 121 A/B Tests - Kenneth Schlenker, Opal

0:00hello I'm your host David Barnard and with me today Revenue cat CEO Jacob iding Our Guest today is Kenneth schlanker founder and CEO at opal a screen time app with a mission to empower humans to focus better every day on the podcast we talk with Kenneth about scaling to 5 million in ARR on paid ads positive and negative results from 121 AB tests and why they still have build an Android app hey Kenne thanks so much for joining us on the podcast today thanks so much for having me David very excited to be here and Jacob nice to have you joining us today as well I'm

0:41excited to talk about screen time and limiting it with opal today so let's go Kenneth I wanted to kick things off talking about opal I'd love for you to share just what the app is and why you created it sure yeah opal is a screen time app available on the App Store we today are the leading application for screen time management and and yeah it's an app that basically saves you an average of an hour and 20 minutes per day in less screen time and helps you be more balanced in how you use your phone awesome so I know you've been thinking about building opal in this

1:153phase approach and I think that this approach is something a lot of apps should consider is that when you first launch there's certain things you should be thinking about and then as you kind of move along you change focus and grow so let's talk through those different phases of how you think about opal and how to grow it as a subscription app Business what's the first phase and kind of how you kicked off launching and finding product Market fit with opal we've been at this for three years now and started off with just thinking that there's an opportunity to build better screen time software than what's

1:52available by default on every single iPhone and actually Android today but we did this in phases that you said the first phase was really about okay let's prove the opportunity not only that people are willing to use a better screen time software but also that they're willing to pay for it and we've been in that phase for over two and a half years so it's a long journey you know in this first phase the main metric of success for us has been d8 return at spend so essentially what we try to figure out is not only are people willing to use an app but are they

2:25willing to pay for it and then are we able to acquire customers in a way that makes sense economically with just immediate cash payback and you shared with me before that y'all hit 5 million in ARR which is an incredible milestone for a subscription app and I would have thought three years ago in your founding like oh we're going to build a screen time app ah it's built into iOS do people really need it are people really going to pay for it I'm sure you never heard that Kenneth in your it's first time in that first phase like how did you think about proving out whether or

2:59not people actually care enough to use and pay for and can you build an app that like is a better version of what comes for free in the system I think something amazing about subscriptions and obviously this is music to your ears at Revenue cat but something amazing in this early stage of the business is that essentially if you get people to pay for your products as a consumer that's a pretty strong signal that you have something that's pretty valuable and so there used to be a sort of trend of consumer apps that well okay let's build a use case let's get people to use this

3:29and then we'll figure out how to monetize later but actually we took it the other way around which is let's build something that's good enough for people to pay for it first let's figure out who these people are let's figure out how we can actually acquire people like these people specifically and that's sort of the original product Market fit you mentioned we recently passed yeah 5 million in AR we basically did this in a year starting from almost zero a year ago to 5 million just recently and this has been done with basically focusing relentlessly on how can we build something that people are willing to pay for and then figuring out

4:01who those people are and figureing out how to acquire more of these people and you know specifically for us what's interesting Is We I actually initially launched a business thinking well this app will be used mostly by I would say you like genz kind of younger maybe students but it turned out that the people willing to pay for the app are more sort of knowledge workers 25 to 40 who care about this issue so much they're willing to pay for it and they kind of also see direct Roi of the time they're saving how they can actually be more productive at work spend more time outside of work with their family for

4:32example and so it's been a really interesting way to essentially use subscription as discovery engine as a product Market fit engine just focusing on this one metric which is d8 return out span I'm curious in those early days I don't stare at everybody's stuff but once in a while I'll see somebody pop to the top of a leaderboard and last year I've seen opal be like oh you guys figured something out right in those early days as you were like iterating on Ras were you dumping money in from the very beginning and like how much like how did that practically work because you kind of need some amount of critical

5:04mass to make those measurements and did you do that literally on day one was there no like period of no monetization just like experimentation or how did that play out there was an MVP phase which was just getting the product to work which mostly test flight only a few hundred people but immediately once we were on the App Store we immediately started testing paid marketing and so basically with a pay wall and I think we hooked up Revenue cats from the very first week that's when we met yeah yeah on the App Store and yeah it gave us a lot of signal really fast gave us volume

5:36you know I think in the early days what you want is you want 100 downloads consistently per day more than like you know drip of like a thousand downloads on a weit list so having a drip of 100 people per day paid marketing can actually deliver that quite well and then you're able to push a new release every single week one thing we've done also from the very first week that we were on the App Store until today is we've released one app version every single week on Monday that's really sort of a discipline in the team that creates a rhythm where you can actually experiment things and by Thursday you're

6:04like let's get something in because I want to see the number change you know the next week right yeah right and then sometimes it's a very small thing sometimes it's much bigger but that allows us to organize our work around this really cycle and it aligns really well with we look at day eight Ras because we have a one week free trial and the big advantage of that is you get like very short duration cycle so if you make something new on Monday eight days later you get a sense of like did you get an increase or decrease in Ras and you can keep iterating with very short

6:32Cycles so that's yeah I think it's a really good way to discover the True Value what you're building I also think you know the landscape changed so much in like the last few years I mean what was true of three or four years ago is not the case today and today it's much harder to break through organically as an app maker and many people do it I don't know necessarily how to do it so for me it was okay you let's figure out something that people willing to pay for that we can actually acquire customers for and now we're thinking we're kind of going going the opposite way of thinking

6:57okay like how do we actually from here now build a lot more organic traction did you have specific milestones in those early days like we're going to launch and we're okay for now as a learning at 50% day eight row as or 25% and was that the case I mean when you first started spending on ads was it like 20% row outs at day eight so you knew you were going to spend money to learn and then it just kind of inched toward that 100% or how did that go early on we spent a very small amount of money in the early days as a test

7:30but very soon we reached 100% Ras I mean part of the reason for that is we priced quite high from the beginning it's $100 subscription on annual basis when we start off actually we're at 69 and we ended up like Rising the price but throughout this past year we went from zero to 5 million in AR we've been operating at over 100% day eight or as so basically we don't have a marketing budget like we're only reinvesting with 8 day cycle I think the point you make about using 8 day how much money you realize Revenue in 8 days over ad spend is really smart the subscription model

8:03has enough like inputs and variables that it can tend to get confusing even I think for the smartest people to totally reason about and so I think it's really helpful to cut and slice something that you believe is going to be indicative and optimize you know what you were saying better for that iteration cycle than total correctness right you're going to be wrong sometimes like you're probably going to overspend probably some of those cohorts weren't fully positive based on measurements and whatever but as long as you have some number that you believe in the grand Arc is going to relate to your goal reducing that measurement time is the most

8:35important and then you couple that with a rapid iteration cycle that's rocket fuel right just keep hitting it and hitting it and hitting it and hitting it until you're where you want to be but it's interesting you know definitely get super confusing I mean the main value for me is like having one metric that you look at is like it's hard enough for the CEO and you got to communicate it to a bunch of people right that aren't as familiar with the model and all that stuff but honestly it's enough even just for you right the product's complicated enough don't make the business model super complicated too you know yeah that

9:06was a good learning of like just Pi one metric that you can iterate against it's just simplifies so much I mean I wouldn't say we didn't get there right away like by the way what I'm talking about right now is like this clear one phase where we only did date R it didn't actually happen like that from the beginning it evolved into that and decided to drop everything else and just focus on this single metric but once we did that's really when we were able to grow yeah I think folks like think they have to build a whole company first right you don't you just got to get one

9:36little engine of something going right and forget about all the other things forget about tackling too much and the reason that that's okay is you will need to do all those other things but it's much easier at 5 million in ARR than it is at zero you just have a little more resources you have a little more attention you have a little more experience so did you just stick with like the tiny little drips of AD spend and then one day you're like okay let's go and then just turn the money on how did you decide and then what was that like to trigger the infliction we tested

10:05until we got close to we didn't get to 100 right away when we start scaling but we felt like we had a path to get to 100% and that's when we started to scale at spend and then basically we pilot at spend based on that like every week we look at what was the you know Ras the previous week are we able to expand budgets or have to lower them if we're below 100% we just basically cut back to get back to 100% And yeah week over week basically driving that Evolution as long as we're above 100% we scale so that's how we started spending a lot more and

10:34then we're able to grow there's no like Grove goal that comes from like just I want to get to 5 million it just like happens as a consequence of setting the bar in terms of overas tactically how were you measuring that so were you just doing Blended return on ad spend from a weekly cohort so you would just say everybody who opened the app during this week counts towards our spend that week what's the ACT ual calculation of day eight row as yeah so now we look at cohorted based on install so someone who installs the app within that week and then we look at the revenue that

11:10generate 8 days from install we also look at day 30 when we started out it was much Slimer than that we just looked at how much are we spending this week and how much are we making a week later with a eight day delay that's sort of works you know at the beginning and then you end up at a point where you have a lot of like late conversion so people download the app in January and end up paying in May and then so the numbers don't line up and so you need to have a cohorted view but we started off in the most basic way possible the other thing

11:36we do is we do attribution in a very simple way which is we ask people on the on boarding where they came from and we found that to be actually the most reliable way of doing it so I mean we also have like all the fancy like attribution software and at the end of the day just asking people you know where they heard about us and if they say Facebook it's one thing and they say Word of Mouth something else and then basically you segments the cohort based on that answer and then we look at unblended roas based on those cohorts another good example of keeping it

12:06simple you could tie yourself in a knot being like well not everybody that installed that week probably saw an ad that week they probably saw it the week before or whatever and then like suddenly you're tied up all in your cord and you're falling over or you just go like yeah I don't know we'll just say everybody who installed this week saw an ad this week right and it's like is it correct no but is it useful probably and it lets you make decisions faster you know it's funny I mean I should probably talk to you guys more about this is you probably have a ton of experience

12:30working with other apps but attribution is one of the things I talk to other app Founders most about like how do you guys do this what are the best practices everybody has a different view there's not like one single fully approved method for doing this so there's everybody's doing their but yeah I thought it's just asking people where they come from is just the dumbest but also the probably most reliable way of doing it you know even if you had perfect instrumentation on every single campaign and every single creative and all that stuff you probably as a team your size don't have the time to action

13:01on all of that data anyway right so you have to kind of think in Broad terms like how much are we spending on this channel on this broad creative strategy you know so I mean I guess falling off of that I sent you an email the other week because I got an ad on Instagram for opal that was so good which like maybe talk about that so you talk about spend and campaigns and stuff but there's like the whole building the creative and like all of that stuff how are you splitting your time between app development the performance actual marketing and putting those things out there and then like the creative

13:30development part thank you for that feedback I loved receiving your email yeah it's basically for people who haven't seen it maybe you have but we did release an ad actually a few weeks ago that took over our entire campaigns and basically this ad it's much more of a organic type video that talks about our mission in general more than the it doesn't even show the product yeah was so good though I was like I was like amped up it was like 1984 like the famous Apple ad it was great we'll link it in the show notes there's a fun story also about this what happen happened is

13:59someone made this video before we ever reached out to them and it wasn't about opal it was just about the space but we like to talk to creators who are sensitive to the mission that we have and it was very align ra mission right so we reached out to them and they wrote back and said actually I use opal and I love it we're like well can we use this video as an ad and they said yeah sure you should just take it and we made a deal with them we tweaked it slightly but really it was made as organic content and worked really really well

14:30and actually now we have more it was the second time we're doing this wasn't the first time but now we're doing this a lot more where we're finding that basically whenever we talk about the mission it's working quite well so it's probably 80% of the work is creative 20% is sort of campaign management performance so definitely creative is play a big part we've tried everything we have tried several agencies where including like fully outsourced creative agencies that basically like make ads will run it for you and you pay a percent of spend we've tried working just with creative agencies that just send us Concepts and we're more and more

15:03doing things fully inous with I mean Lally right now it's seem of one and then a little bit of Contracting what we're finding works right now is finding creators in our space that like care about what we're doing ideally they using the app and they love it and then producing stuff with them but I mean in terms of like spending my time I actually have someone great in the team right now like that's really specifically creative Partnerships manager who's working just on the creative side and we're looking to bring someone else with more the sort of typical user acquisition I would say more sort of technical side of campaign

15:32management and data we're a team of seven we have one person that just does creatives basically speaking from CEO position the folks that come in and they're most excited about Revenue cat in our mission they bring a certain Something Magic that is hard to find and bring that in in your creative work and and that's one of the reasons with the exception of a few Fringe activities like I don't love working with contractors especially for core things right and for an app Business the market marketing is very core it maybe is marketing it's maybe not part of the product technically but as a founder and a builder it is right this is probably

16:06half of what you're focused on all the time is like the acquisition aspect definitely I think it's probably half products half marketing would be a good spit of my time and it's interesting because yeah marketing informs product product informs marketing it really kind of works together we have to understand why people are willing to pay inside the app like what features so for example one of the things that people love is the ability to essentially like lock yourself out of distracting app without any way of canceling or bypassing the normal I know I dismiss I dismiss screen time every day I'm like goodbye computer I have power over you ignore all day so

16:39we for you Jacob have built this special feature it's called deep focus and basically there's no way to cancel our bypass that's just simple you know value proposition this is one of the reasons why people like this app and like to pay for it and so of course you want to build creative around that and that tends to work well just like you know understanding why are people using this for work during the day or they using it in the evening to kind of Disconnect and unplug with family or with friends like what's the use case that you can highlight in ads and then also when you

17:08see an ad or creative that works that talks about I don't know sleep for example you're like well actually maybe we should make a bigger deal of sleep inside the product and prove the product so it kind of works both ways and it's I would say probably 50/50 right now of like my time and attention between product and marketing well we could talk about this for another hour the zero to one stuff is so fascinating and all the ads up but I did want to move on to the second phase that you're now moving into after getting this dialed in and that's the fremium strategy so tell me about

17:37why you decided that this was a needed second phase to move into as a company so we believe that there needs to be great screen time software available for everyone we think focus is very important in the 21st century and what that means is like we don't really have I think the luxury of being like a smaller Niche product by n i mean it could be a very profitable business with like a few hundred thousand people they are subscribed or but we really think that this product we're building needs to be used by millions of people one of the things we see is students for example right like we get a huge amount

18:12of organic traction with students but the business model we have right now is focused on converting people to a pretty expensive yearly plan that doesn't fit at all this audience like very few students at scale I think will pay $99 a year but there's a huge amount of fraction they really really need it and I'm going to say students I mean college but also it goes down to junor high school high school the mission for us is very important and so we'd like to get in the hands of people who need it and there's probably millions of them so we don't think we have a choice to kind of

18:41stay just smaller and the second phase that we're entering is about building the best screen time app for all knowledge workers all students and in a way that allows them to use a daily for free and that means we need to build a better premium funnel smarter upsells most of what we done the first phase where we focused on day eight Ras was about converting people right away onto a payall or they pay an annual plan mostly I would say like based on intention I would call this the gym membership effect where you're going in you want to change and so you're paying for an expensive subscription not only

19:14because you want access the gym but also because you want to change and make a commitment to yourself so that's worked really well in the first phase but in the second phase we're moving away from this and building a more premium funnel where you can start to use the product for free do a lot of things and then like there's different ways in which you pay for the app subscriptions and also you know in app payments I would say like smarter upsells so we're kind of doing maybe the opposite of what for example dualingo did where you know they started off completely free focusing on retention and then figured out okay like

19:41how do we actually get people to pay we're doing the opposite which is we figured out how to get people to pay a smaller number but now we're looking at how do we expand this to many many more people including people using it for free right now the paid penetration so if you look at the monthly actives of opal 177% are paid subscribers and I think if you compare that to dualingo for example they're 78% paid penetration so you know what that means is we can probably have a lot more people using the product for free and convert a smaller percentage of people while still having a great business why do I think

20:15it's necessary also to have a premium funnel to grow much faster I think it comes down to it's just a lot easier to talk to your friends about an app if it's a free app than if it's a paid app I think lingo onboarding says learn a language for free forever that's like on the onboarding for a screen I think that the perception even though actually their app isn't really free I mean there's a lot of paid features but like the perception as a free app is so important in organic referrals and so yeah that's what we're working on in the second phase and the other aspect is

20:46when I look around at other subscription businesses and maybe have more examples of this but I think there's many subscription apps that kind of hit a ceiling at maybe 10 maybe 20 sometimes 30 million doar in AR and they end up with a model where they can be dependent on paid advertising mostly converting people in the onboarding and it's really hard to break the seiling we want to avoid the seating by working on it now essentially if you scaled the RAS eventually you will have attracted all of the available people you can attract you can localize and things like this but eventually your churn and your

21:19acquisition will reach an equilibrium probably around in that range it is fascinating to see the DU lingo move reversed right and also has to do capitalization because dualingo raised a ton of money to basically run unprofitably for a long time before they finally and then they did turn on a big money machine which is the bet but it's interesting to see a more Capital efficient probably way of doing this which is starting from this position and widening out one thing I'm curi say was it going to be to be able to maintain you come in it's a premium product it's a gym for Millennial knowledge workers

21:52but then also like that same app has to communicate to the 19-year-old freshman in College who's just trying to like Focus for their midterm I think it's hard the way I look at it is like this phase one we pass level one right now like passing level two it's just as hard to pass this level two I mean the conviction we have is that if you look at dingo right it's like they were able to democratize learning language make it fun and make it like a game we think like the ability to focus it's not just screen time management but it's about like learning to focus is something that

22:24is just as big and maybe even bigger in terms of like how many people need to use this learning a language is a real luxury good you know what I mean like you only do that if you have a lot of time and like you maybe have aspirations to travel and things like this but I feel like screen time is something that plagues everybody right yeah that's the bet and so I think even if it's hard we have this conviction that it's really Mass Market a lot of people that could use this and so the tricky part is what we did so the first phase was simple

22:52like day eight oras is the main goal right the main metric phase two is a little bit more complex we look at retention of course as a main metric whether people pay or not like just how many people are still using yeah usage and then the second thing is arpu we want to make sure that we can increase retention while having an arpu which is lower than it was before but doesn't go below a certain level so basically we have those numbers and it's if we can increase retention where we want it to be and then reduce our arpu you know the first year to above $2 we can change the

23:26trajectory of the business massively and actually we've done some of it already so you know one the things we did for example is make it easier to dismiss the original onboarding pay wall so that more people start using the app and more people start using the free version of the app and we were able to do this in a way that increased activation and decreased arpu in a way that's manageable so basically like the net is positive in the long run that's the way we calculate so like if you can make those bets remove a little bit of friction for payments early and then that increases retention and then make

23:58sure that it doesn't destroy rpu too much and if you can have those two work together I think that's the key it's a fun example and just evolving your model a little bit which I found I've had to do it several times at Revenue cat like every year and sometimes you forget and then like you wake up one day and you're like the numbers I'm supposed to be caring about don't really represent what I care about anymore and you have to go back and You' be like okay actually these are the numbers we care about you have to sort of recast the business in a new set of metrics and it's good I mean

24:24that's the right way to do it it's like think about that first think about what you're trying to do state your hypothesis right then state how you're going to measure it then go do things that's the real scientific method so yeah it's so interesting that you also come across I think it's fascinating to do this and speaking of like how you spend your time as a CEO and like what you focus on definitely the more you can also have time to take a step back and say okay like are we optimizing for the right things what have we learned in the last few weeks that either confirm or

24:53deny the strategy were on is very helpful all right and then the Third phase and this is coming I don't know how soon so maybe you can tip us off when you think you'll be able to start moving into this third phase but the third phase being expansion so after you start getting this premium product dialed in what's the expansion plan from there premium phase two as mentioned and then phase three is expansion into basically other platforms so we're only iOS right now Android Mac desktop any device that you may own or come across maybe others Vision Pro and other in the future because of course screen time

25:29matters everywhere and also expansion in terms of distribution where we currently do e2c direct to Consumer we know that we also have a B2B angle that we can sell to colleges universities schools but also employers and many others that would be really phase three phase two the goal is to work on this until end of next year and then beyond that enter the next phase of business you're working backwards from what a lot of apps tend to do like start start with flutter we want to be on every platform from day one we're going to launch iOS Android web Mac you know whatever why did you

26:06start iOS only I mean at 5 million AR and not having an Android app is kind of surprising compared to what we hear in the industry oh you should be on Android and so why did you do that and how's that playing out it's working pretty well there's just so much to do to figure out the right kind of product and the right kind of way to onboard people that we felt it was better to do it on oneform than multiple platforms but also there's a timing thing where Apple essentially introduced a screen time API back in November last year I mean with iOS 16 and we were the first app to

26:38benefit from this API and to build on this API so the focus was very clear like there's an opportunity right now to build on iOS and to build well I do think that there is a lot of value in being a crossplatform like just even Mac plus iOS like if you work like me you know with your iPhone and then also your Mac there's natural extension there so we're going to get there but you know also quite resource constraints I mean we just want to do what we can with the resources we have but this will come very soon yeah I think it can be a trap

27:05for a lot of absolutely it's a trap it's very resource intensive it's not linear in platforms like two platforms is not twox difficulty it's probably three or fourx the difficulty and it's not 2x the revenue yeah the ri is lower than the first one and if you're still learning in your core platform and like you're still iterating those lessons not all of them but a lot of them will translate to your new thing and also like you're a moving Target right now right this phase of transitioning into premium may change how the product looks and feels pretty fundamentally potentially and like you don't want to have to do that twice yeah

27:40that's right do it once I mean you can use tools and things like this but like I do think that especially as you think about word of mouth too like that's one of the big reasons to have an Android app is to make sure that every time somebody says opal there's a listener on the other end that can install it and benefit and from what you've told us from the mission obviously that's going to have to happen but yeah for folks like it can really be a trap don't feel the pressure and this is why metrics are important too cuz like if you don't have a metric that's telling you like you're

28:04not good enough yet you're G to be like H maybe let's build an Android app that's a good idea and you're G to waste a bunch of time it's like no focus on the RAS focus on whatever when you get past a certain scale you'll have enough threads and this is an interesting transition point where you guys are at seven people or 10 people is 20 30 40 once you get to 30 40 50 people then it's like okay we can put four people on an Android app and not derail the company's progress right but at this stage you have like one thing you can think about as anization basically

28:34that's so right I mean the other thing I think depends I mean for us it's a relatively new news case like people don't have a screen time software on their phones I think 25% of Americans check their screen time stats others don't so it's still pretty nent so there's a lot of Discovery to figure out like how do you build a product that helps most people focus and so it's easier in that Discovery phase to do on one platform you're saying like when you have two you kind of stuck the very different user bases too right like I mean eventually and this is I think talking into that expansion and

29:02distribution phase it's like you'll build a brand that then you'll become the ones educating the world on screen time and that actually has an advantage because when you go into those platforms you already have this like body of brand and experience and makes the conversion much simpler I think absolutely that's the plan I've also to back up what you're saying Jacob I've seen like a lot of horror stories from other startups and actually end up like launching too soon and when you ask them what's your greatest gr they'll say well actually we went on Android too soon or we went on desktop or Chrome too soon and it's just

29:32a version of the spreading yourself too thin too fast we did it for years over and over again until we really started to understand like how much can we chew at any given time but once you do and that's the power loss right of compounding as your company doubles or doubles and doubles like one day you wake up and you're like okay we actually have a lot of capacity to like tackle more than one thing and that's what you should wait for right it's like wait so it's all about being frenetic and fast but also patient very difficult thing to balance so we'll do a follow-up podcast

30:01in the year will'll be be all over every platform every screen speaking of fenetic and fast you've been on my list of people to invite to the podcast for a while now but you tweeted recently something that reminded me I gotta get Kenneth on and that was you tweeted a screenshot from notion with this massive list of ab tests that You' run and then somewhere in the Tweet you said we've run 12 21 AB test so I wanted to spend the rest of time digging into this testing I mean that's a high velocity of tests the first thing I want to ask is what is your setup for testing like how

30:40do you actually manage 121 test as a small seven person team going from zero to one it's a lot that's a lot I don't advise everyone do as many I think we probably could have avoided only do the ones that work out yeah exactly save half of them big better no but we did do that I mean I think one of the advantages of having like a single metric that you look at in this first phase which is the r it makes it that much easier to AB test on a single metric and as I was mentioning earlier it's also like much shorter iteration Cycles so if you run an AB

31:14test you can get in our case because we have a 7-Day free trial you probably get most answers within eight days if you have enough volume especially in early days where you're looking for big uplifts but our setup is I mean we use Firebase for AB testing but we run off tests through amplitude the data cohorts data analysis through amplitude basically with a user parameter on amplitude for Baseline versus variance and then we look at uplifts and figure out statistical significance because we AB tested a lot on day8 Ras it has a lot to do with testing stuff very early in the funnel the earliest in the funnel

31:51the easiest it is to get statistical significance quite fast let's talk through we're going to have to kind of speedrun these but we created a list of your top three positive results and your top three negative results so the first one that you did that was a positive result was showing the pay wall early so tell me about that test like how you executed it and then what the results look like yeah it's almost like common wisdom but it is true that we found we tested many many iterations like where to plays the onboarding where to plays the pay wall what kind of pay wall how

32:21easy is it to dismiss or not and when you say how easy to dismiss you mentioned that ear you mean like fading in the X versus not or like full hard payall or not it dance around when you try to click on it that's right like those old ads in 2002 we we didn't try that one we probably should it worked for Bonsai buddy I know that no but we yeah definitely hiding or making it as discreet as possible the dismiss button for example was a winner but I would say generally like what we learned is you need to get the pay wall to be as early as possible in the

32:57onboarding but not too early and I think what you're looking for is the peak motivation moment where if you need a few screens to build motivation and build intent and then display the pay wall that's better than like putting the pay wall right away in the first screen that's what we found at least so for example for us what's interesting is you download the app you might find out if you do this you know we're GNA ask you what's your screen time we're going to ask you a few questions and then we're going to build this mini report that will one of the things it'll do is it'll

33:23give you like a lifetime projection of how much you're going to spend on your phone and the average for a US person is actually 22 years so you know no terrible so the average American gets a smartphone age 12 77 year old is the sort of average life expectancy and then you spend five hours and eight minutes on average which is if you look at only wake hours because 16 hours awake during the day basically adds up to a pretty large portion of your day and then you project that onto lifetime so anyway we make that projection and then after we make that projection we tell you hey but

33:58the good news is we can save up to three four five years of your life together that's the moment where there's High motivation and then people willing to sign up to a free trial and a little bit like a gym membership effects for sure I'm going to commit to this and commit to this change and the benefit of this is you get people who of course pay but also are quite motivated to try the product and improve it and so yeah we I think through various iterations we went from about 7% of people who start a trial to 177% of people who start a trial from download be the number one

34:28all right so the second one was on boarding to a scheduled session this is an interesting one yeah so what we found is if you onboard people to committing to something where they don't need to remember later to do it like an automatic action basically so for example for us when you download the app and you tell us which apps distract you most so you might say like Instagram for example or Tik Tok and then automatically we're going to create this Monday to Friday 9 to 5 work hour schedule where when your work hours start these apps will be blocked and so what happens is you on board you might

35:04like exit out of the app do something else and then the next day you don't need to remember it'll just start to block the app and then you'll remember oh I committed to this now I'm in that's clearly something that was a huge win for us yeah that's a really cool one we talk a lot about retention and onboarding and all these things on podcast and yeah it's just such a powerful thing to onboard to something that's going to be right back in their face soon it works for this and I think also I add like one thing is so you basically make it an automatic action

35:34but you allow editing editing should be very easy so like you make the commitment but you can always come back and like change it or cancel out of it there's no problem but the default is that you on board and then it just works you don't need do anything else don't don't dump somebody into an empty you know like you should have an exact idea of how you want the for every person's experience to be and send everybody right I used to describe onboarding it's just like you want want people to slide down into their first experience right like no you don't want them to get

36:00caught no bumpy elbows no nothing they just click app launch and like they don't do anything basically and then then boom they land and they're figured out right simultaneous to that you're pitching them you're doing some other things describing the value and the thing is it's like you measured it it actually helps and that's because that first three minutes three to five screens is so highly leveraged right it's like every single dollar you're spending on ads hits that every single user has that experience they're going to form an opinion about your ad probably semi- permanently in that first three to five minutes I sometimes see people design tests super poorly so it's

36:33not surprising to me that these are the tests that you saw have really high output because you want to test something big and you want to test something that's frequent right that people see a lot what would you test these against like were your variants just like status quo then new idea is that kind of how you evolve that you know ideas to test yeah mostly just like a binary test cuz we didn't have a ton of volume sometimes we did variants where like a different message or maybe like a different placement they wall position three versus bayall position five that kind of thing but mostly

37:00Baseline versus variant and then adopting the winner and then like doing it again the less extreme your test is the longer it's going to take to get a result or potentially not get a result that's I think something unintuitive about testing is that it's not just good or bad most of the time it's inconclusive and also the time it takes to get conclusion is related to the difference in the two right so if they're identical the time it takes to reach a conclusion about that is actually pretty long so the bigger swings you take the better yeah I will say one thing so we didn't do well 121

37:31is a lot but we could have done better at prioritizing the bigger swings like having a clear view what do you think the uplift is likely to be and that's a little bit like guess workor but I mean you can come up with an estimation and then priortize especially early like the bigger swings if you get like a few percentage points uplifts it's not going to be enough you need massive uplifts at the beginning sometimes it's a very simple things like just putting the payall to screens early two screens before can be like plus 10% and trial started and that's a massive win or sometimes you can spend a lot of time to

37:57build this new feature that will end up moving things by a tiny piece I've done that I've buil like long buildout feature that has very little impact even though it's beautiful right and it's literally just changing the opacity to the x button on the pay wall to 80% increases Revenue 20% you're like okay why do I even try and it's sounding also to stay motivated now when you've done these tests for the team like you work five weeks on something or 3 weeks on something and then changes nothing or it changes like 1% but yeah I think definitely testing is you have to be in the spirit of a lot of things will fail

38:28I think those failures are good though because I think it teaches you reality I think too many people have some sort of like magical thinking about how everybody's Steve Jobs and if they just apply their big brain product brain and think of the perfect thing and then they make it look beautiful and they ship it it's going to matter and sometimes that is true but I think most people have a unrealistic understanding of how the world works and like AB testing is a really good way of kind of putting your nose in it and being like nope nobody cares or maybe they do but I think the sooner you

39:01get over that as a product Builder and that's all levels right you were saying it's like the engineers being demotivated and even the folks coming up with ideas it's like teaching them where are the levers of change on our product that's a thing I think a team because once you've got it baked into the culture of your team two years from now or a year from now and some bright-eyed new product manager coming in they're like what if we they're like no like we did this test number 63 total failure which you know may or may not be useful but that understanding of the physics of

39:30your business and what moves the needle and what doesn't I think that's I mean Facebook was really good at this that's what drove them you know they just learned like connections like that's everything and goes back to this one simplified metrics and then also the iteration Cycles you guys there's no way you did 121 test if you weren't shipping every week right that would be impossible definitely and so you have to also pick a metric that allows you to do that but that's helpful I mean another interesting example is tiered referral rewards so basically we have in the app you can invite people and if you invite

39:56five people you get the subscription for free like for a year that's kind of like an organic mechanism but one very small change that we did is we basically mostly like make it look better this referral program and then like have a tiered reward system where like you invite five people get the free year you invite like 20 you get a lifetime sub you get a special prize at 50 and kind of like putting that into a beautiful UI that just like gives people a goal of how to invite people and that led to 80% Improvement in how many people send the referral to at least one person it's

40:28really interesting to think about how because that's a very different user that you're trying to optimize around than the person who's buying the $100 annual subscription during onboarding that's the kind of thing you're going to have to figure out as you it sounds like you already have started to figure out like okay make sure that you're coming in are you 25 to 35 professional okay you're going to go down this way are you 17 to 22 you're GNA go down the referral program actually we do that and I would actually advise everyone to do this as well like you just ask people their age and different CH on

40:58boarding yeah yeah yeah yeah that makes sense all right so tiered referrals was the third positive result but we got to speedrun the top three negative results and to Jacob's point this platonic ideal of like we're going to run this test and it's always going to be meaningful but it's meaningful whether it's a positive or A negative result and usually want more negative results so tell me about the top three negative or we'll go one by one but what's the first negative result that was meaningful because you learned so much sure I mean we failed a lot for sure um so these are the top three most meaningful negative results

41:36but the first one it's very hard to beat the blinkist pay wall so I don't know if you know the blinkist kind of pay wall I mean it was published and I think in the mobile app sub Community it's probably a very well-known fact but basically this transparent payall that shows you hey you're going to start the tree trial now and then like in six days you get a reminder and in seven days you'll pay your for subscription it kind of like decreases the anxiety of people to sign up to a free trial so we have that and that's the main pay wall right now in

42:04the on boarding and we tried dozens of others and actually keep trying but we've never been that pay wall so a lot of failure around that placement though you seem to have tested well but the actual content it's not a huge lever yeah that makes sense the content hasn't been that huge of a lever I mean we did experiment to get to that blink Pall but once we've had it so far we haven't made gains too much choice in the onboarding is another one so we tried in a few different ways offering people a choice on how to disconnect so you I mentioned earlier that the default way right now

42:36is work hours 9 to 5 Monday Friday we're going to block all distracting apps and then you can go back and edit and choose no I don't want work hours want something else whenever we tried to give people more choice in the onboarding it failed and that's a little bit to your point Jacob of like you want the smoothest got to be slick yeah there's limited fatigue budget users only going going to make so many choices and you need to make sure you're spending that mental energy on exactly the things you have to people live very busy lives and their onboarding they might be doing something else they need something very

43:06simple to commit to that was meaningful in the sense that we know now that the automatic action is really the best way to do this and we're still experimenting around this but this is a clear one and then yeah the other thing is we've done a lot of tests that didn't work on re-engaging people after the initial on boarding in the first day and a lot of it failed with one notable exception I'll say like when people cancel their trial We re-engage Them of an offer and that works quite well it's really interesting that they're non-committed enough to be like no I'm not paying for this but then you present them with a

43:39deal and everybody's like well I did get a deal like I almost wonder how much of it is that it's like well I better take advantage of well who cares right you don't need to know why so how would you do that but you would detect the trial cancellation via revenue cat.com and then you would just fire an email because you have their email or a push notification or what was we actually do so we do use Revenue cats by the way this is not an ad um it is an ad don't let them confuse you so we do use Revenue cat for subscription status we actually use a

44:10software called one signal great software currently we send notifications with a 50% off offer actually right now but it's being AB tested you might not get this offer you might get a different one this is this is yeah don't call kenth do this somebody's going to send you a nasty email at opal I heard on the podcast that I could get % off this is not legally binding and then also in app kind of modal when you come back say Hey you know okay so yeah making sure you get the visibility on it but it's interesting that you're doing it very much targeted in the app experience

44:44right which probably makes the most sense it's people have the context and all that well I think emails can work too we used to collect them we don't anymore as part of the onboarding process we have a phone number sign in and imagine that was an AB test that collecting the email caus more drop off still to be determined but basically we have a social feature inside the app where you can see your friend screen time which I find is very cool speaking about the ideas that might not turn out into yeah I don't care about other people's screen time but I think if I showed people my screen time I think it

45:11would make me better about it right like that's got to be the idea yeah there's all this thing about positive peer pressure in the sense like there's sure it's positive well you know in like addiction science you know AA and others the support group that you get of people that are like supportive but also kind of checking in I'm like what are you doing can help and so that's the idea but so far it has been AB tested it was a slight uplift we do get people that invite their friends but it's not in the three most meaningful wins so far but we're still working on various ideas to

45:40essentially focus better with your friends in different ways and have a support group and so the phone number bur authentication kind of allows for that I think that's a great place to wrap up21 AB tests a lot of failure how many are you doing in 2024 200 you know it's exponential it's like the AR good thousand you're doing something right all right it was so much fun talking to you Kenneth thanks for being on the podcast yeah thanks so much for the invite David and Jacob and congrats on Revan cats amazing software so excited to be using it every day oh geez thank you but I do want my t-shirts

46:12please okay I'll get your T-shirt don't worry thanks thanks so much for listening if you have a minute please leave a review in your favorite podcast player you can also stop by chat. sub club.com to join our private [Music] community