Running Effective In-App Experiments — Giancarlo Musetti, Ad Hoc Labs

0:00[Music] [Applause] hey you're listening to the sub club podcast a show dedicated to the best practices for building and growing subscription app businesses we'll share insider secrets from the top subscription apps on the app stores let's get into the show hello i'm your host david bernard and with me today revenue cat ceo jacob id our guest today is giancarlo lucetti growth product manager at ad hoc labs ad hoc labs makes several apps including firewall and dialed but is most well known for burner an app that allows you to create multiple phone numbers to protect your privacy and better manage communication on the podcast we talked with john

0:43about soft versus hard pay walls how to think about product experiments and why removing friction from onboarding didn't actually help hey john carlo thanks for taking the time to be with us on the podcast today hey david thanks so much for having me so i wanted to kick things off talking about uh the company you work for ad hoc labs uh it's one of those companies you know kind of like fishbrain and some of the others we've had on the podcast for it's a huge business but you don't really know about it unless you know about it uh so tell us a little bit about ad hoc labs and uh burner your

1:14flagship product yeah so ad hoc labs are the makers of firewall burner and dialed so firewall is a robocalling solution burners are flagship product and that's a second phone number app so you can get a second phone number make calls and text from the app and that's very popular for people working in sales that want a specific area code a specific city popular for people who just generally want to keep their business number private um and just generally you know have an anonymity um when they meet people yeah so when i get uh when i get sales calls that come from my tiny three thousand person town i know who to blame uh you

2:05can blame bird my wife has actually used burner professionally uh one of the many apps we've had on the podcast that i'm a fan of um she's a licensed professional counselor and so when she puts her number out there she doesn't want it you know to go to her cell phone and doesn't want callbacks to come from her cell phone um so yeah it's a it's a super cool product and you mentioned uh leading up to the the podcast when we were chatting dating i hadn't thought about that as a use case it's like you know if you're if you're putting your number out there on

2:35tinder and and other dating apps like you don't you don't want your real number that that people are gonna have forever uh so yeah it seems like there's there's like all these all sorts of use cases professionally personally um you know putting your number out on craigslist or facebook marketplace it's like there's just so many use cases to have that second number that you can just burn which is cool like the the whole like name and branding around burner but yeah so you know one of the things you put in in your notes for for our conversation today it's it's a top 100 150 uh revenue

3:07grossing app in in the app store it's a huge business so a lot of people have this need how long have you been at the company and um you know what have you seen growth wise or like what is really resonating with people yeah um so i've been with a company for a year now it's funny that you know burner had steady growth for you know several years i mean it's been a top five revenue grossing app in the utility category for i think over five years now and yet in the last couple years growth has like has started to grow even faster and i think a lot of that is thanks to

3:48the experimentation um that we've been doing which has been awesome to see i want to know more about experimentation tell me what i got to do to make my app go unselected it's clearly that whatever you discovered is universal and will apply everywhere that's a well-known fact in apps so oh yeah of course yeah so there's several uh experiments i wanted to dig into um one of the ones you talked about was soft paywall can you tell us a bit about that experiment and and what kind of results uh you saw definitely so the soft paywall was uh there was a time when you could you

4:22know download the burner app and just create a sample burner um without starting a free trial and so generally we've been shifting towards this subscription model where people need to subscribe to use burner rather than just paying one off for a number and so the soft paywall was basically a seven day free trial uh paywall where you you subscribe to a seven day free trial sku and it's a soft paywall because there's a maybe later or either maybe later button or just some button at the bottom that lets you like pass the paywall and get into the app um and so the soft pay will we notice

5:05uh really helped increase subscription revenue and then a little bit after that we tested the hard day wall and that's where we removed um that like bypassing button and people just had to start a seven day free trial and that's where we we started to see tons of growth that's kind of roughly when i joined the team and then since then we have been breadcrumbing our way from our previous revenue model which is credits to subscriptions and so my time here has been focused on increasing monthly recurring revenue and we've had several different experiments that have helped us get there yeah before we move on to those experiments i wanted to ask

5:50a few more follow-ups um yeah it sounds like uh the hard paywall was done around the time that you got there do you recall and have you talked to any of your colleagues did apple hassle yell at all i know like in the past that the hard paywall is something they've been sensitive to like a free app has to have some kind of free component like when you say a hard paywall is there any functionality in the app at all if you don't uh start a free trial or subscribe or is it just completely blocked there's nothing you can do in the app without subscribing actually if you start the free trial

6:27and then you decide you don't want the subscription you use prepaid numbers instead of subscribing so subscriptions aren't the absolute only way okay did you have to start like some iap trial to get past that screen though yes okay yeah i mean we've done that i mean i this is not me personally but elevate we've done that i know apple's okay with it but it's interesting because you can always like start the trial and cancel and then and then and then you have to have it's it's it's odd because at that point you have to have a free experience like you can't like kick somebody back out like once they've

7:01canceled a free trial um maybe you can't after it expires i suppose but um they've been okay with that um but it doesn't so so the way you described it it sounds like the soft pay wall performed less well than the hard pay paywall yeah which this is one of these things that like and why testing and data are always so powerful over any sort of intuition because you think like ah that's a bad user experience that's gonna turn a lot of people off and you're like yes but also you're showing it to a lot more people right like just like the way that usage decays and interest decays so

7:36quickly in an app usage like the more you push to the beginning the more like urgency kind of you drive for the user experience it's not necessarily in the book of like user-friendly ux it's not oh i wouldn't say always it's the most user-friendly right like user you most user-friendly will be here it's a free app that you know it's going to run out of money and crashing you could really can only figure this out through experimentation um there are some like high-level things that work but um you know every app is different interesting to me though is i i have a there's this app called open phone which

8:09is a phone number for small businesses it's like if you're a plumber you run a small business you can have this like shared phone number and stuff like this this was a big problem they actually had was people trying to do the burner use case with their product because they had free trials up front um and they were like not they were not tooled they were not tooled to um accept this kind of usage right where it was like get a phone number because because of the backing costs like the underlying cost of getting a number and things like that going back to the discussion and maybe

8:39we can expand a little bit here on how you've navigated this transition of usage based or like consumption based pricing to subscription isn't there some difficulty with um you know somebody wants to use a hundred numbers does the subscription package like give them infinite numbers or is it like some number per month or how does that how did you package it yeah so we you can't get infinite numbers you right now can get either one one three or ten phone number subscription package okay and you can still purchase prepaid numbers in addition to that so if you want seven numbers you can get a three phone number subscription

9:17and then a few other prepaid numbers right the way we packaged was like there was you know some discussion around should we be able to give them the ability to purchase any number of subscription numbers right four five six seven eight and we basically opted not to do that because the ux would be kind of weird especially like upgrades and downgrades and and um just skew management on your side as well it's such a pain right right and and one thing we've noticed is that sku management does take some time but doesn't take that much time and so we've generally when we are considering creating a new

9:58sku we're not too worried about the skew management part of it um but we do kind of think about what's what's going to actually help the user in the best way and also increase revenue right and so we ended up launching the 10 line experiment um and that was basically a 10 phone number subscription package which is doing really well um a lot of users have liked that i guess if you have habitual use like you need as many as you can get right you know if you're somebody who uses it every day like 10 is probably not even enough right yeah and it also depends on the use

10:35cases right because there's people who have one phone number they use that specific phone number every day but then there's people who are like dating for example and they want a separate number for each person right because if one person becomes spammy they want to delete that number but they don't want to delete all of the contacts they have right and so the same i've heard this in like the business use case as well where someone has a ton of different contractors and has a different number for each contractor and has like 50 different numbers you know so one other takeaway there is talking to users is really interesting because you

11:10learn a lot about your app and how it's used especially for an app like that where the it's a utility that has a number of different applications right did you line the subscription products up with like tiers of consumables you were already selling like did you on the consumer could you buy credits at any granularity like 1 2 10 20 or like did you have packages as well there are packages for credits okay did you align them like the subscription ones and the no so you have like they're they're all offset from each other yeah so the way it works is to buy a number you can either buy a prepaid burner or a

11:44subscription number the prepaid burners there's a list there was like five of them um there was like a mini burner which would give you x number of text messages per month and why minutes of calling and so there were different types of prepaid numbers that offered different amounts of calling and texting in order to buy a prepaid number you had to buy credits and there was like five tiers for buying credits you could buy like 25 credits at once okay and the credits had a different like they had the the different tiers of number had like a credit cost yes exactly interesting that's much more like uh zynga land like when i used to

12:23do free to play stuff right you had like you had some intermediary currency right um because like what i was gonna i was curious about is like how did you discount because because you know people are gonna buy bundles of numbers um because they get some discount right like why would i buy three you know assuming i'm not maybe i don't need three maybe i only need two and then i lost money because i didn't use all of them for the month i was curious like how did you think about pricing for those like how did you think or like actually not even how do you think like

12:51right now like what how much do i save if i buy three on a subscription versus three ad hoc yeah for sure so generally subscriptions are gonna be way cheaper and we intentionally did that because subscription revenue is more reliable it makes our business more valuable and it's also giving a discount to the user so it's like a win-win situation right um and that's kind of been the reason we've been kind of aiming towards increasing mrr yes is it like 10 20 do you know offhand like roughly like what i can save by signing up versus uh buying ad hoc so just comparing the like one phone number plan

13:34to the ten phone number plan is a significant discount it's like it can be 15 20 um and then like going to the annual plan from the monthly plan for any like the three line or the 10 line is another it's like 20 percent sure so the idea is you know the more you pay upfront in like the subscription you're going to save a lot more yeah i mean you're buying a you're you're selling a futures contract right you're saying like we'll give you a discount if you commit you know if you put up this much capital now and then you'll have access and then it's up to the consumer

14:08to decide like what their future consumption is going to be um but then you can afford to discount right and then like you said it gives your business a lot more price predictability and like um and stuff like that i'll just pause and say you all have a very complicated setup compared to most apps right that's what i was going to jump in on and say that like my like eyes were glossing over trying to like picture all these i know what i'm i'm grilling you giancarlo and like exactly what your tears are and so i'm trying to put this together but this is great because this

14:39this is something i think a lot of apps and myself included are scared of the complexity like don't force users to make choices you know too many skus is going to just confuse them and so i i'm curious your perspective because it sounds like you know hitting that inflection point recently was in part just trying to push everybody to subscription instead of having all that complexity is am i understanding right that part of that inflection it was that simplification or is there still like a lot of skus and a lot that the consumer has to to understand to use the product and subscribe yeah i mean i imagine the inflection

15:18isn't just that specific experiment right right growth growth is happening for many reasons we have some going on on the marketing side as well we have a lot of stuff um in the experimentation world that the growth team's working on and then we also have things outside of that too so i think it's a mixture of different factors that's causing the growth um and not just one thing but today is it is there less cognitive load like how many skus do you present in that initial pay wall right right so in that initial paywall we just offer right now a three phone number package that has a monthly and annual option

15:58and part of the breadcrumbing kind of plan of moving from the credit a la carte model to the subscription revenue model um the hypothesis was that there are too many products right and that's a little too confusing and so one of the most successful experiments was removing a bunch of prepaid numbers rather than offering five different prepaid numbers we just offered one the one that was um it's called the picture burner and it kind of has like texting calling and you can send pictures and when you say when you say remove numbers you mean these different tiers like these different flavors of numbers with limits and

16:36capabilities yeah just like removing four products that you could purchase to buy a prepaid number so rather than having five which we did before the treatment in this experiment only had one and it just drastically outperformed control interesting yeah so that makes sense because all of that complexity you and jacob were going through about the prepaid and i was just well when i heard there were three different levels i was like oh boy we just added another dimension yeah this one from uh this one from a a spreadsheet to a tensor real fast so so that's great to hear so so it was really that simplification process that

17:18that was the successful experiment and so so you initially present something very simple and then if they decline that then they can buy those prepaids and then they kind of get so are how many different paywalls do you end up showing like so once they get rid of that initial paywall and they get dropped into the app where they can buy these prepaids is that a completely separate paywall or do you still have the the uh free trial option with that single product monthly or annual once they go past the paywall they can go to subscription management to manage their subscription upgrade to submissions with higher numbers or they

17:57can go to another place in settings to buy the prepaid numbers yeah okay yeah yeah that's also fascinating uh i i this is where i wish we had slides you know we should be doing this like a webinar yeah i'd love to like have this if it's all about webinars now we can but i did want to talk through some of the uh unsuccessful experiments i think you know everybody loves to talk about oh this did a 10 lift that did a 25 lift and this is what you should do but understanding the experiments that failed i think helps inform uh the kinds of experiments you want to try and and

18:34just not enough people talk about the failures so uh there were a few here that that you listed as failed experiments so i'd love for you to talk through the the annual subscription price testing and what failed about that experiment yeah so that experiment was testing the current annual subscription price on the paywall 59.99 against 49.99 and 69.99 um and you know the kpi for success here was roughly i think like there had to be 15 more often for the 49.99 package or 15 no less than 15 a 15 drop an opt-in for the 69.99 to break even yeah on the price difference and that unfortunately didn't happen we were

19:24just looking back at this experiment a couple days ago and it's tough because you can't actually determine whether the treatment is successful you can estimate whether it's successful by predicting the retention for the annual subscribers right but like for you know there was a significant increase in opt-in for the 49.99 plan right you dropped the price of the annual sku more people are opting in but you can't actually measure immediately what the long-term benefit to the business is because you don't know how that cohort of users is going to retain are they going to re-subscribe after one year presumably it'll be higher retention compared to like 59.99 or

20:1069.99 right yeah and then what was it the control price you said was what 59 99 60 bucks so you saw some elasticity standard right like you got higher conversion at a lower price and lower conversion at a higher price but then um was it that you didn't feel confident not knowing those renewal rate changes to make a decision or was uh was the where the change is just not big enough to justify we estimated the retention and you know we landed on oh it looks like it's roughly break even yeah but i i'm very excited to like revisit this a year from now and see if

20:47we're wrong right because yeah that's the beautiful because you have cohorts right you actually started behind those plans so you can look at it you know but a year after yeah really agile yeah it's not and so we obviously we try to be agile part of the part of like the experimentation flow is we need roughly one to two weeks for an experiment during onboarding to get statistical significance and i like to generally run onboarding experiments in parallel with experiments down in the funnel as long as they're not like price oriented or going to like confound yeah you change the wording around the thing you just

21:28showed somebody or something like this right yeah um but that's great i mean a price test i when you first said like oh 49.99 or 79.99 whatever it was i was like oh those prices are too close like you the closer you get like the more data you need to show a significance right mike who does huge volume too though so right but but there's there's benefit there if you can't limit the uncertainty on the renewal rates that you're only going to be able no matter how much conversion rate data you get you're only going to be able to get so accurate until you have like trustworthy renewal rate data and

22:03like it's hard like the app store might look hell like the app store might change the way renewals work in the next 12 months right like there's all kinds of reasons to not like you know so you could you know you could have said like we'll assume that they're the same but it's like well for the 49.99 case you're adding a marginal whatever it is 10 more users right and you it might be a lot to assume that those users who were already on the price margin are going to behave in a similar way like i think it would be pretty foolhardy to be like they will

22:32be at or above the average in terms of renewal behavior as the folks who are all buying it you know 59.99 right so you almost have to assume it's going to be worse right because they're already marginal customers um so yes um price experimentation is hard we know this like we've been working on this price experimentation product for years um the easy part is running the test the hard part is the predicted ltv because there is just like a lot of parameters and and often that the model is very sensitive to right like small changes in your renewal rate create can create huge swings in your

23:07lifetime ltv um and so it's pretty easy like let's say you were testing ten dollars a year and sixty dollars a year you can get significance there but you're pretty fast but anything more fun it gets closer so how did you like how you were talking about like different amounts of experiments you run like how does how did that experience inform the kind of tests and things that you you focus on now because that's what i always found with running a test is like when you have a test that's inconclusive it's not really a failure i feel like that's actually it's a null result is actually interesting because

23:41it tells you like hey there's unless you just screwed it up which happens but like a lot of times it tells you like this was a path that's probably not super rich right so like how does it focus how did you change your philosophy after going through that yeah i think philosophy is generally pretty pretty much the same after that right because pricing in my opinion is still very very important right like it's i think one of the biggest levers and i think even though this experiment in the long run probably has roughly the same ltv for both of those cohorts there's a lot of opt-in in one of the cohorts and not the

24:20others and so that speaks to the power of price right and that's kind of backing up um the other experiments we've run the soft pay wall the hard pay wall removing specific prepaid burner plans and so on and another experiment you mentioned was removing the screen that lets users choose their full number i'm curious about that one because i mean again i have used the product and i think you remember it was like years ago when i set up for my wife and so i'm assuming it was like a whole screen where you get to like go through a bunch of different numbers and i remember like kind of like

24:54when i got the option to pick a number i um you know you spend time like you get invested in like oh i want to find the best possible number with the right you know maybe i can spell something or whatever um so yeah i'm curious about this specific experiment and then why you considered it a failed experiment yeah so to give context um before we jump into the experiment um i break down in amplitude which is a analytics platform for those that don't know it's a it helps you understand the user journey and i break down each step of the funnel for onboarding just to see what is the drop off at each

25:33screen right every screen has a drop off there's always some amount of users that stop using the app at every screen and so the idea is if the screen doesn't add lots of value to the user let's test removing it so that we can get less drop off on that part of the onboarding and so there's a screen so the way the the funnel works is you have a welcome screen when you open the burner app you then enter your phone number then you select an area code and then after you select an area code you get a list of a few different options with a full burner number

26:13and the idea here was choosing the area code is really important but users probably don't want their full number they don't want to select the full number it's a lot absolutely load it's a lot of numbers i have to pick it's like nine numbers i have to pick lots of lots of information and things you have to enter and yeah users are trying to like get into the app really quickly and so like let's try to remove that friction um and then we tested that and the way we set up the experiment is you choose your error code and then it jumps you to the the screen

26:45that usually shows after the full number screen which is like a congrats you have this number and then we also added this little button at the bottom of the next button that basically takes them back so that if they want to go and select the full number they can actually go do that and so it kind of creates this loophole like okay if someone really cares about the phone number they can go back and choose that if not um they can click next and get to the paywall um but in the end this didn't actually outperform um and so the the primary kpi here was free trial start

27:20it's very difficult to get like lots of data on what percent of each of users in each cohort are actually paying for a subscription seven days after the free trial starts so for onboarding i think we were mostly looking at like the free trial opt-in as like a higher level metric to give us uh an indication of whether this was gonna like make us lots of money and in the end we decided um to keep the control yeah so i kind of spoiled the results because i mean it was exactly like my use case is that after i spent a minute like finding that perfect number and

27:59fussing around like finding a number i was like more invested and more likely to start a free trial um so it was like so maybe the cognitive load was actually creating a little bit more buy-in a little more sunk cost along the path to starting that free trial did were you able to like measure um or did you look at as part of this experiment like how much time was spent on that or like how many different numbers people cycled through before they moved to the next step how much time i i don't think we have that data conversion rate that's all you care about you know like you're talking about your

28:35funnel it's like yeah it's just like well yeah we're on this page how many people made it to the next page and then ultimately how many people started a trial right like that's what you care about um and then you know the the the details are probably all not all all not that informative unless you you know maybe one day you'll hire a product manager who's like the product manager of phone phone number selection right which then they'll have the time to dig into it um but it's interesting funnels are especially in consumer just so important right you have only have a limited amount of time to make a pitch

29:06to somebody and and it really can just be how you present it is like do i convert this person or not right like some amount of people probably were never going to convert some amount of people probably always were but there's i think i don't know what number is but there's people in there that if you just presented in the right i know i'm like this like i judge a product like this if i'm three pages in and i don't feel like okay we're heading towards me solving my problem i'm bounced like give me i'll try another app i'm out of here um and especially if i'm not impressed with how

29:31the app's put together i i it doesn't bode well for the rest of the usage experience right um but so it's really important to study these pages but nothing's in a vacuum i always found this with experimenting with onboarding funnels like you'll take something out but it's all you know you might just be you're right and that there's drop off in every page but it doesn't necessarily mean that page is causing that dropout right like you pull that page out and then that dropout is going to spread between the page before and the page after potentially um so not always the case right like you could put up a

30:02screen that is just like very confusing and that would probably cause more dropout um but you know generally yes the fewer pages is better but also sometimes not if they're really good pages right uh and that like like david was saying it creates anecdotally um we did this at elevate where we had a very very light onboarding when we launched it was like three questions sign up bang you're done and then launched you into some brain training games and then we added this like literally added 25 steps to the onboarding like as an experiment it was like a personalization test kind of thing and i was like

30:37this is not going to work this is going to ruin the funnels this is this is going to make our app everybody's going to churn whatever so we a bead it and yes it drove down a lot of like number of users getting to the end but it turned out it didn't affect trial start rate it didn't affect day one retention it didn't affect a bunch of numbers that i was convinced it would ruin but honestly it was just they were good pages and it was shaking off people that were gonna turn anyway right it was just moving some of that turn further up um so

31:05anyway this all goes back to the point of like testing is important because there's literally no way you can think about this and figure out what's gonna happen i'm curious did you end up keeping that 25-step onboarding funnel i haven't used the app in a while but yeah it became a big it was called on boardio it was a whole game uh but but it was this like questionnaire we had you play like a few mind games like and then we actually that became the seating data for what customized training you received after that um and it wasn't like super crazy scientific like do you like do you want

31:35to work on words okay yeah you'll get some more words don't i'm sure it's very scientific whatever but like we were you know trying to ship uh and get it out and but it created it gave us it was two things one it gave the opportunity for us to show people what the product was gonna do more directly right um and i assume like numbers is kind of a good way to show that like hey like we have control of the phone system and you can type in a phone number that's kind of cool so it showed people more what the product would do before they had to

32:02commit anything right like they were just playing um and then uh two yeah it did kind of turning people out that are not going to convert to meaningful users earlier or later doesn't really matter right in fact it's probably in the benefit of your company if a user is not going to be a high value or like good for your brand or they're not going to be satisfied it's probably better you turn them out as early as possible right um and so by saying like hey this is kind of what the experience is going to be like people were like bounced and so yeah it turned out it was a it was a good move

32:31it made the product it didn't affect any metrics negatively it might have even i don't remember everything but i remember being one of these things where mr smart mapped me with like down you know like funnels and all this stuff was like this is no way there's no way this is gonna work right i like to think of um onboarding funnels like slides like you just want users to be able like if they just like take one step onto your onboarding like whoops suddenly i'm a user right like i think it's just like impossible not to um but i've since learned that that's not always the case

32:58especially i'll just last complicating anecdote is in b2b which is what i work in now all that physics changes because people have like a very different experience with a product and you can actually have quite a bit of friction and it doesn't affect things too it may affect like no this is everybody's going to tweet at me about how painful entitlements are in revenue cat now like i know i know we're going to fix it but people will work over it right and so like but yeah i still think it's it's good to have that mentality that like make it as smooth as possible but you know it doesn't always it's not

33:32always the case right like yeah sometimes a little bit of onboarding friction creates a better product for sure yeah and to david's point it sounds like that creates a little bit more commitment right when you have to choose your full not only the area code but like the full number and so if anything it it makes me wonder what else should we be adding yeah like asking them like hey what are you going to use well maybe that's not good for a thing called burner but like yeah i've actually considered that though yeah like what's your use case yeah i mean it's helpful for down down

34:01the funnel customization sure like what type of content are we feeding them what type of modals are we sending what what's the email yeah and now we're starting to really optimize for that for crm um and and that's going to be very exciting i think for in a especially a product in a utility category like if you can like smartly integrate some of that stuff that's gonna help on church you know like it's agreed it's not commoditized you guys have a brand but like you know i can set up a twilio account and make a burner thing right but so like the more you invest in yeah getting people to

34:35lock in there's this it's a billion years old now but this like back in those evernotes when they were raising their series b their deck leaked and it was really interesting how they thought about inputting data creating lock-in right and they credited that a lot with their really great consumer subscription growth and i think not enough apps especially in prior utility categories like think about smart ways to do that right and so yeah if you can figure out ways to get users to invest in your app that's going to just help you right down the line so we've been talking a lot about um experiments and

35:10getting into specifics and tactics and everything but i do want to take a step back and talk about the ideation for road map and experiments um yeah i'd love to just get your thinking about like what experiments to run and you know sometimes it's so often counterintuitive like jacob was saying like you you don't want to test the 25 page onboarding but you probably should so how do you think about like what crazy experiments to run what you know outside the box thinking is maybe going to actually um produce better results and how so how do you pick what to work on yeah so it usually starts with a brainstorm and

35:50it's important to me that the brainstorm has like a bunch of different people uh ideally an analytics person a designer hopefully an engineer customer support people like i i want everyone who touches the product to be included in that brainstorm a lot of the time and that's just really helpful because there's so many different things you can possibly test and you never know where the next great idea is coming from um and so that's like step one is having that brainstorm and then from there uh discussing technical complexity with engineering to figure out how long will each of these take um right because there might be

36:30something that's like medium to high opportunity but insanely high complexity and it's going to take months to build right and so at that point is that worth it or is the high like the medium to high opportunity with like a very low lift worth it right like that second option is going to be where you're going to want to start and as an engineer you're going to want to find that out early in the process right right and so this is all happening before the roadmap is set um and so you know just mixing the brainstorming with the complexity estimates another thing is also looking at like the market right

37:08there's a bunch of second phone number apps there's a bunch of apps in general i i've gone through dozens and dozens of apps screenshotted every single screen in their app put it all in figma and then use that as a starting point for the brainstorm to like get everyone's creative juices flowing and i think that's really helpful as well to see how other people are doing it and then you know try to take the good things that they're doing and improve on those from there just synthesizing into a roadmap how do you balance the technical complexity component because some of the some of the experiments you know like i

37:44mean again to get to back to jacob's story like putting 25 pages and asking for information that you then have to store and then personalize the rest of the app like that was probably a massive technical lift and you know and jacob intuitively was like this is not gonna work luckily i had a visionary boss that pushed me through it so when do you when do you push past those technical complexities is it there's a hypothesis that it will be a huge benefit that there's going to be this massive change and how do you think about that yeah i think sizing the opportunity is really helpful right so a good example

38:23is that we're working on a premium burner like a plus burner plus and we're thinking through what are the different features um and kind of going through this entire process that i just mentioned there's unlimited numbers on there i've been meaning to ask you about that so actually all of our subscription plans have unlimited calls and texting uh not unlimited numbers that would right that would be crazy i don't think that's the cost basis i think with like the backing networks right it's like they charge you per number so it might be undoable but i was like somebody's gonna pay a thousand dollars a month for

38:54that yeah we have to we have to deal with spam yeah right yeah you don't want to create we don't want that to be honest yeah sorry so you're saying opportunities right like the sizing opportunities yeah and so part of that involves um sometimes it's just estimating right like oh if x percent purchase at this rate um what is like the revenue gonna be and then for something as big as like a plus product you wanna make sure you're including the right features right so we're doing tons of surveying to make sure like what are people not only interested in but what are they willing to pay for

39:36um and so i used two pricing models to do that the gover granger model and the van westender model and so these are two different survey models that basically ask questions about pricing i think uh the van west endorp model is something like are you interested in buying burner plus with xyz features for this price if yes survey done if no what about for this price ten dollars cheaper and then you start to see the percentage of people who are extremely likely likely not that likely definitely not likely to purchase and then you calibrate that with like assumptions and there's like step-by-step instructions online that you can find on youtube uh youtube is a

40:23great source i highly recommend it and so we use that to come up with a pricing a price point that will you know balance maximizing revenue but also getting as many users as possible on it and we also are doing smoke testing so basically showing users burner plus and seeing if they're interested in purchasing before they can actually get it and so then we measure yeah like a cta that's like actually purchased or one that's like tell me later or no like actually purchase and then you give them the womp paywall like or that's modal that's like sorry not ready yet but thanks for your interest

41:02coming soon coming very soon i love consumer because you have so many users you can like just tiny little thing you can do a little weird experiment like that and like can be so informative same with the pricing experimentation right like you have so many users with a consumer app like you can get surveys out and get significance on something that that that finicky right and i i don't think i'll say from personal experience like in my past net was not nearly as bullish on surveys because like i think i was just i didn't like talking to people right um surveys are great you don't actually have to talk to them like

41:34you just you send the link you send an email and people tell you and it's and it's in some way scientific and stuff like that so that's interesting i want to put a link in the notes to those models because i think that's something i've never heard of before that it'd be interesting i'll send that over well there's like a million other things i wanted to talk about but we we do need to wrap up any last thoughts as we wrap i know uh burner is hiring um are you hiring on your team specifically or or any uh specific job postings you want to want to shout out and anything else you want

42:03to share with the listeners as we wrap up yeah we burner is hiring um we'd love if you're interested in the product um if you're interested in working at a high growth successful business this is a great opportunity the team is really great um we didn't i don't think i touched on like why i joined berner uh but the reason was uh to be completely honest i didn't know too much about burner when i first got my like thumbs up for the first round interview i just applied to a bunch of places and yeah um murder came back and i was like okay so let me do some

42:36research did lots of research it was really cool and then talked to my manager in the first round i was like oh this person is really cool i would love to work with all of these people and that's something that i value a lot in the workplace it's just you know a great environment and we are hiring on lots of different teams i think we're hiring a director of customer support and finance lots of engineering um you can check out the the roles online but if you don't see the role that you think you would like where we accept all applications we just want great talent um and if we think

43:14it's a good fit like we're going to make an opening for it how many folks are there now 30 30. oh that's a great size like we're slightly a little bit bigger than that but like we're like 50 but like it's small enough you can know everybody and yeah and but big enough that you have like help um and like you can do things totally um it's a great time especially for like if you're like focused on an app that's super fun so that's awesome i would have thought you were much bigger i mean having these apps david huge leverage there's huge leverage in apps yeah

43:44it's funny i mean as an indie developer i often get people emailing as if like i've got like 20 people on staff and it's like that's just me part-time in my developer part-time and a designer we contract with a few hours every once in a while uh and then but then i make the same assumptions right yeah four years ago burner was a huge company so that leverage is incredible it's it's amazing what y'all are doing with just a 30 person team thank you yeah so we'll we'll include links to uh the hiring page and and to giancarlo's uh twitter and linkedin and all that kind of stuff

44:16uh in the show notes uh but again thank you so much for being on and um a lot of great uh uh stuff that we were able to talk about today so thank you thanks so much for having me this was a lot of fun to make sure you never miss an episode subscribe to the show and your favorite podcast player thanks so much for listening until next time [Music]