Stop Celebrating Conversion Wins Before Checking Renewals – Sara Grana, Yousician

0:00So you really need to understand your whole set of metrics and how do they all work together because most of the cases something goes up and something goes down and you need to make sure you know what is going on. Hello, I'm your host David Bernard. Today's conversation is shorter than usual and will be featured in Revenue

0:20Cat's state of subscription apps report. Each episode in this series will explore one crucial topic and share actionable insights from top subscription app operators. With me today, Sarah Grana, who works on revenue strategy at Musician. On the podcast, I talk with Sarah about the cost of not tracking your experiments and decisions, how refunds and chargebacks quietly erase your paywall wins, and why stacking AB test wins should compound your growth, but almost never does. Hey Sarah, thanks so much for joining me on the podcast

0:53today. Thanks for having me, David. So you spent almost seven years at Babel and recently transitioned to Musician. And one of the the things you told me when we were preparing for this was that the first thing you did at Musician was you asked like where's your log of experiments? Where's your log of business decisions? I don't think a lot

1:13of companies keep that kind of record. [laughter] So why was that the first thing you asked? and and how do you recommend doing that? So when um I start a company and also when I started at bubble in my role in revenue strategy I really look at okay what is the map of our revenue over the years. So from your revenue will come from subscription business. You can either get money from people that never had a subscription and start a new subscription. People that upgraded from a subscription to another, people that renew a subscription or people that used to have subscription then turn and then came back. So like these four buckets,

1:53you have like these four buckets. Having the history of how these four buckets evolve can also tell you a lot. So when sometimes you cannot find experiments or what not you can see, oh I see that from February 23 all of a sudden the new subscriber revenue went really you know like really big like what happened and they're like oh yeah this is when we started the lifetime subscription or I you know like this type of things. So sometimes it's about finding yeah it's great if they have a log and then you can go through experiments but you need to differentiate okay what is what is

2:27important versus not because sometimes you run like some companies run a lot of experiments so it's not really useful to go through all of it so I would also recommend let's map the revenue understand what are the big difference that you see and then try to map okay something happened here what happened and then that would tell you a lot about the history and how things work together also in in that particular company or

2:53sector. How would you recommend actually tracking that or like in your ideal state? Is that just in a notion doc and a Google doc or is it in a spreadsheet and like the revenue changes that happen like what's the ideal state of something like that look like? basically the revenue bucket so to say to me it's like Excel like I would track in Excel like how how are moving and then from that maybe have a little presentation for me or like then to share and then you map okay here you know this line this happened because of this thing and then you can like link to whatever data taker

3:26or whatever documentation that is the company or you can put like within the same Excel file if you stay in Excel the links of what happened when but yeah I'm I mean I'm a bit of an Excel person. But yeah, like everyone needs to find their own way when you're trying to optimize conversion, retention, when you're working on, as you do, revenue strategy at a company. All these little things add up to become the product in a way that is hard to untangle from each other. And so having this record means that, you know, when you ran this experiment, you can then go back and look at that cohort. Did it, you know,

4:03turn at a higher rate? Did they engage in the app in a different rate? How did they engage in the app differently? And so all of that leads to kind of making better decisions over the long haul and being able to look back historically and understand those decisions and maybe retest some of the assumptions from the past. But one of the big things is conversion over optimization. I think a lot of apps are falling into this trap these days of getting too focused on earlyFunnel metrics at the expense of downfunnel metrics. So what are some of

4:35the pitfalls you've seen in that? I think there's two major things that I keep seeing over and over again. One of them is this not letting enough time for the cohort to evolve and see what happens later. Right. What you mentioned like what happens with the renewal rates like okay we have a price increase. Oh great, you know we did amazing. We rolled this and then six months later you look at the cohort and say oh actually the control group actually is performing better than the test group because they are you know renew more. So one of them is like this not looking you know like giving the time sometimes it's

5:10fine to roll out right it's fine you have a win you roll but then look back right and then the second one is within the same moment in time like the same snippet in time but not looking at the right metrics like there's sometimes that we forget to look at some metrics I give like a really clear example sometimes with a winback win back campaign okay or like people that cancel their auto renewal okay when They do that, we're going to send an offer, right? Okay, they cancel to renew. We send an offer because then maybe we get them back. Okay, great. We do this great

5:42amazing numbers. Okay, you look a bit more deep and you realize, oh, a lot of people are cancel, you know, like putting the outre at the beginning and what they're doing is they're asking for a refund and then they're getting the offer. So, you are actually like net negative. So I think that's also something that tends to happen especially with refunds and chargebacks and because people tend to forget about those like they never happen and um yeah you might be even without waiting in time you might be messing up with your system. So you really need to understand your whole set of metrics and how do

6:17they all work together because most of the cases something goes up and something goes down and you need to make sure you know what is going on. I hear this all the time with price testing where you double the price and you exactly match the revenue. So conversion cuts in half. Sometimes you'll get that win where you double the price and you do get a 25% lift in average revenue per user or something like that. But you always got to be looking for those downstream things. Like you said, one thing goes up, another goes down. You know, when you look at the entire life cycle of a subscriber, any one movement

6:55here can have downstream effects if you're not really carefully looking for it. And I think we're just in this mode as an industry of chasing payback as quickly as possible. And sometimes it's like it's just a business decision you have to make. you have to make that decision of like we're going to sacrifice long-term revenue for being able to hit that rorowaz at day whatever 7 30 90 whatever you're targeting but I think what's really important and what you're hinting at is like you got to know what you're sacrificing you got to know as you push this number up what number is going down and are you willing

7:30to make that trade because a lot of times people aren't tracking those downfunnel numbers and don't know the trade they're making. What are some of the specific examples you've seen of that kind of overoptimization and and how how it does impact the long term? It happens also a lot when introducing new plans like when you didn't have the yearly subscription or when you didn't have the lif I mean lifetime is a huge example because the value that you get from the get-go like it's really high and then every time like every time you do a test like lifetime versus something else like always the something else is

8:02going to lose because people are not looking at what is the LTV of that something else right like if they would have bought a tour subscription they are not bringing you the the €100 they are bringing you like €300 so don't compare it with the you know 250 compared to 300 um and things like that. So I think like every time there's like a new plan that always like people please look at the lifetime value that this plan would have had. And then another thing that um happened I said before like a lot like refunds and chargeback is something that people tend to forget about. So if you

8:36are introducing a new payment method and things like this also like especially like how look at those metrics look what is happening there. any other specific places you think people should be paying especially close attention to? I know, you know, payw wall optimization is a huge thing and I've been frankly surprised at how big of moves folks have been able to see on the payw wall. Like over this past year, one of the like really big paywall optimization things that that people have been doing is having a toggle to enable the free trial. So by default, you don't have a free trial. you, you know, tap a button

9:12or slide a slider and it enables a free trial. And now Apple has started to reject some of that. So, I don't know if that's going to be something that is allowed long term, but I've been shocked at like, you know, what a big lift that can have. But again, I haven't talked to folks who looked at that cohort 6 months, 12 months down the line. I mean, we're just now I mean, I think that that started about 12 months ago, so you're going to start seeing those cohorts maturing. any things like other things like that that you've seen that that that you saw very specifically lead to

9:42downfunnel problems? Sometimes it's not so much what we said like downfalling problem and more like you are just putting the revenue at the beginning right is just that more revenue than at the end or not but one thing that happens that is really clear like with churn people usually tend to just think churn is a product thing like churn is a product problem but most of the time I not most of the time let's say most of the time but a lot of the time like the commercial strategy that you have what people buy when they buy it and which price they buy it are going to have such

10:13an enormous impact on your renewals and extensions and what happens is that people like in the u marketing side they don't really you know see in the index and then the product side they also don't have a view on what is happening at the beginning of the funnel in a way for example something that I think is going to happen right now is we have now the web check out right in iOS you can send people to buy in web what I've seen in different companies is the web renewal rates are significant ificantly higher than the up rate. So what is going to happen now is a lot of teams a

10:47lot of product teams are going to be like wow like our product is amazing we have increased our you know renewals by whatever 20%. If they were to slice from what is a cohort of users that bought you know natively in app what is the cohort of users that bought through web maybe the renewal rates are flat and what just happen is you are acquiring users through other means. The same can be said of course for the subscription plans like if you have like your share of users that by one month subscription is bigger or like growth for whatever reason your renewals are also going to

11:23get better. if you look at the to overall number. So I think when you're looking at data like cohort of the different users what they bought when they bought it if discounted or not discounted is also another clear example like sometimes you do a discount you bring a lot of revenue but then those users they are not going to renew or sometimes you do a price increase you bring more revenue because your conversion goes down but not as much as your as the price increase goes up but then those users you have like less pool

11:52to upgrade users and things like this. So there's a lot of a lot of um different colors to this problem. So what are the specific metrics and ways you dive into the data to better understand that? Are you looking at both subscription retention and usage retention of very specific features like how are you watching that entire funnel?

12:15What's your preferred way to look at it? What are the metrics you most look at? So what is more most important to me is which plan did they buy discounted or not? and where like you know web versus app. Then sometimes in some and this is something that you need to like look for your company right sometimes the marketing channel might make up for different users and cohorts right so understand what are the different one and once you have that then you just look at that you just look at that cohort and then if you see that the cohort is going up or down you know okay

12:46probably it's a product thing what usually happens is that this h the share for those different cohorts is changing over time and this is what bringing you the fluctuation on renewal rates. It might be different when you're um yeah when you're doing a specific experiment, you know, that might change that. But I think it's really always good to look at what are those things that make a difference that are upper funnel as upper funnel as it gets so to say and then uh look for that. And then when it comes to when I'm looking at a test that I do in the pay wall, what not then I'm

13:19looking at revenue per user of course, but then I'm also having a look always on preback. So I always wait like a couple of weeks what is happening there because that's like an immediate effect that you can see and then always after some months recheck the cohort you know like what has happened with these people. Sometimes it's a bit frustrating because you have the majority of your users buying a 12 month subscription. So you won't see it until later on but maybe there's something that you can do if you see that the one month behaving in a particular way. Maybe you can assume that the same would happen for

13:50the 12 months. You can also look when you have the result make some scenarios of renewal rates of those different you know if the one month was was to renew at this as this you know like 10% less or 5% more whatever you want based on the result of your test what would happen and then you can know once you know like you have some like really easy way to check in a way like in a sense like you can you keep an eye okay if it goes below 10 then I know we are in trouble I know that probably the the winner is actually the loser And this

14:22gets back to the what we started with was keeping a detailed history. Is that when you do run an experiment, do you set like an alarm like you know three months from now? Yeah. Yeah. My Google calendar up there has this experiment recheck with my like the data analyst. Yeah. Gotcha. So you are looking back at at specific intervals and do you do that for I guess you wouldn't do that for every test. What's the criteria for the things that you're most career to look back on? I think like anything that is price changes I really care about because for example if it's about oh the

15:00layout is different or like the copy is different then sure maybe there's a different in renewal by why would it right like so there's certain things that like probably not so I don't know like you shouldn't be spending a lot of time the thing is also the more you do it the easier it gets in a way it's just like a another part of the of the analysis so to say it's already set up that way. So you have already a cohort it's like okay you pull back okay yeah here and sometimes it's even with some tools is even there that you can just recheck this test how it perform so it

15:35doesn't need to be like really time consuming but usually everything everything that is with price or discounting or thing like this I would always uh recheck after yeah depending on what we are selling you know like 3 months 6 months and then if you have to wait a year is yeah it seems like too maybe the the priority would to make sure you're checking back on the things that made the biggest move. So maybe there was a copy change on the payw wall button that that did a 25% lift, which is sounds crazy, but it happens sometimes where you have these like crazy lifts. And so maybe those

16:13kind of things would also kind of fall into that bucket of like anything that made a surprisingly big lift. Mark that one to go back on because especially if then you are not seeing because something that happens a lot is oh we have all these experiments all positive but then when you're looking at the numbers you know you're still flat

16:32you're still improve but not that much. So Especially if you see that then it's time it's more time to recheck like actually I would say if you see that the growth of your company follows all your AB test increases then it's less of a you know like red flag maybe you're like oh maybe you don't need to retest you seem to be doing fine I think that because I think a lot of the times we're like yeah all this like you know I look all the list of you know experiments with like 5% 10% I'm like but wait a second like what why we are not growing

17:03you know like 30% year on year if we have all these wins and I think this happens also like with product like with with um you know like a new feature what not yeah this lift this lift but then when you look over time it's not really holding together so I think there's also like I don't know why it happens right but um there's this like mismatch between what your AB tests are telling you and what the long-term effects is telling you That is that is such a great point. It's like you can stack all these wins. Oh, we got 10% higher completion

17:39in the onboarding step. Oh, we got 15%. And it should compound. It should even compound. So, it should be even like way more than than that. And it's not. So, yeah, tracking and and u making sense of all this is such a challenge, but that's why you need to do it. You're not actually getting that 10% increase if you're not actually getting that [laughter] 10% increase. and you're never going to know if you do these experiments in isolation and aren't tracking everything. So, it was so fun to chat with you about all of this. Thank you so much for coming on the podcast. Anything else you wanted to

18:12share as we're wrapping up? I know you've just started this new role at Musician. Any uh job listings you want to shout out or anything else like that? Yeah, we are we are looking for people. It keeps changing. So, I will just go to musician.com careers. But yeah, it's a really fun company to work at. So like helping people learning to play an

18:31instrument, that's a great thing to do. Yeah. Awesome. Well, thank you so much for joining me. This was great. Thanks a lot. [music] Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast [music] player. You can also stop by chat.subclub.com to join our private community. [music]