Lean Analytics: How to Focus on What Matters – Ben Yoskovitz – MicroConf 2013

0:00[Music]

0:07[Music] so I'm going to apologize in advance for people who are getting two copies of the book because it's a gigantic book by business book standards so the careful with your wrists give them out quickly because they're really big book but it turns out when you write a book about analytics you have to be specific and and that's what we've done and we're gonna get into some specifics today about data and what's important to measure for your business before I do that because most people won't know who I am I just want to go through a little bit of a background so I hate talking about myself but it's sort of a

0:43necessity of this sort of thing so I'm these things I'm really a product guy I've been an entrepreneur for a long time since about 1996 when I started my first business was actually a service company I'm a blogger as Rob mentioned I'm also an angel investor which is an interesting experience to see companies from the other side and try to help them out from an early stage I've learned a whole lot of things in my career this was my first business it was a service business it actually pivoted a couple of years in we switch from being a service business to a product business and I

1:14know that that's a huge challenge it was a big challenge for us we were lucky about the timing with that this is project management software we were doing web design we were doing fourth $400,000 ecommerce sites back in 96 to 98 when Shopify didn't exist and you could do it really easily converted that business to probably business the dot-com crash hits we have this fledgling little product all the service business drives up have this little product sort of saves the day but as we grow that business and we're sort of that I don't know it's about a million dollars in revenue we get comfortable it's this dangerous place for

1:48entrepreneurs to get to and I stayed at this business much too long and I got zombified and you see this with a lot of companies and a lot of entrepreneurs they're making just enough money they're comfortable they get married they have kids which are is wonderful things and and they get cozy and and that's a dangerous place to be so if you're if you're loving your business and you're excited and you've been at it for 10 years or 20 years wonderful but if you feel like there's more to life you got to think about moving on and that's what I did and I moved to another business called

2:15standout jobs which I was the CEO of and I learned that recruitment sucks this was a venture backed business the first business was bootstrapped we raised 1.8 million dollars we then went on to spend 1.8 million dollars essentially crash-and-burn we did exit this business but was four cents on the dollar and and this was again b2b most of my career has been in b2b space and I learned recruitment sucks but I really learned here about the importance of talking to customers and and that sort of flows into the next thing in terms of lean startup and other things that I did so then I said well you know enough

2:48about being the CEO which is a super painful job banging my head against the wall for three years incidentally stand-up jobs was 2008 to 2010 building a recruitment business during a recession is a terrible idea don't do that so I started a startup accelerator so instead of being in charge of stuff I just watch other people do it and these were we invested in five companies so I'm kind of sitting on the other side of the table but not really this is sort of a TechStars like model but we actually spend up to 12 months with the companies we did five companies they all happen to

3:16be consumer applications three of them went on to raise additional capital which is kind of the unfortunate vanity benchmark for accelerators one of them exited which is the real benchmark that you're attempting to accomplish was a company called local mind was acquired by Airbnb so that was cool decided not to do that again because accelerators are really really crappy way to make money so I said I need to make more money this doesn't make sense you can't scale these things really well although there's some examples of that so then I joined go instant and I moved from Montreal to Halifax this was 2011 a year later we were

3:50acquired by Salesforce go in since a tech company b2b I'm now I don't know employee nine thousand nine hundred and eighty six I make that up but we're part of Salesforce big company learning a lot of interesting things about being inside a huge company and trying to build this fledgling startup we required very early with Goins and we were about two years into that business so very important for me to note that I'm not in fact an expert in analytics okay this is not what I've spent my life doing I've spent my life sort of moving from different business to different business as an entrepreneur somewhat investor kind of

4:24person focus on product I'm also not a data scientist right so this is not what I do for a living but I decided to write a book on analytics anyway and I realized after writing the book you know the first question people ask is well why did you write this book and so that the actual story for this is that I was asked to write it so I wrote it with this fellow by the name of Alistair cruel Alistair works very closely with O'Reilly which is our publisher he's previously written books complete web monitoring it's it's even bigger than I think lean analytics which is incredible and they were

4:56talking to him about writing a book in this lean series running lean from ash Maurya and there's a few other books in the series that are pretty good and he called me up said Ben you want to write this book and Alistair and I had worked on year 1 labs together I said that sounds like fun let me give that a shot I'm so it's kind of a bucket list thing for me so that that's why I wrote the book but but there's other reasons why I wrote it and this is sort of the number one thing I kind of realized after being an entrepreneur for a long time is that

5:20we're all liars all right I'm a liar you're a liar we're all liars but in a good way to an extent to be an entrepreneur to get up every morning and fight the good fight to do what you guys do all the time takes a little bit of line it takes this reality distortion field that we put around ourselves to say we can do this I can accomplish this thing that nobody else has necessarily accomplice right I'm gonna get up and do this we also have to lie to a certain degree particularly early on with our businesses because they're they're not realized yet right we're selling a

5:53vision and a story and ideas before they've actually been proved before customers have validated this completely so we have this reality distortion field but what we realize is that when that gets too strong and relying to ourselves too intensely what happens is we end up running into a wall crashing burning and dying and it's a horrible painful death and if you haven't failed in a business I'm you're probably not trying hard enough at this point but nevertheless lean analytics and and the use of data is about poking these holes a little bit into reality into our reality distortion fields so it is a lien book it's about

6:27lean startup I'm not gonna talk a lot about lean and I think everybody's familiar with it at this point but this is kind of my reasoning or thinking or realization about what's important so this is the build measure learn cycle from Eric Ries right this is the core print of all you want to get through this cycle as quickly as you can iterate and learn until you can find a scalable successful business and so the truth is that everyone's pretty good at the idea of stuff right everybody here has ideas everybody thinks their ideas are great my ideas are better than everybody else's here right that's kind of what

6:54we're like and most of us are pretty good at the building part right people love this part developers love doing this and it's really important of course we have to build stuff and get it to customers but there's a little bit of danger here of course and the dangers that we build too much we build the wrong thing right and we don't get the appropriate amount of validation quickly enough to figure out what's gonna make a successful business so ideas build we're pretty good at it's when we get to the measure part with a lot of startups and having worked with many startups you realize this is where things start to

7:22fall apart they're not sure what to measure they're not sure how to measure it they're not sure why they're measuring certain things and without that with no data there's no learning right so if you can't figure out what to track and why to track it and how to track it you can't learn quickly enough and you won't scale instantly I will put all of these slides on SlideShare just so everybody knows so this is why I wrote the book and so I don't know if you'll read the book you probably won't read all of the book cause it's too long but there's pieces of it that I hope

7:48you'll find valuable but what I hope you get from the book but also from this talk are the following things first is the importance of intellectual honesty right so following the Lean model it does genuinely become harder to lie especially to yourself and that's a good thing and I believe very strongly in intellectual honesty and when I talk to a lot of startups this is where they often fall down right there just sort of believing their own their own story the other thing is using your gut properly so I would never tell somebody all you need is data that's nonsense right everybody has to go with their gut and

8:21so instincts are experiments data is proof you have to combine your gut and what you believe to be true and you have to measure it to see if it's in fact true thirdly better decision-making abilities I think the advantage that data gives us the right kind of data is that it genuinely helps us make better decisions and that's critical because in a business of any size that's what we're spending most of our time doing is making these decisions that could be small but they could be incredibly pivotal for the success of our company long term and finally focus so startups fail because they run out of

8:54money right this is why startups fail you just run out of dough but we derail often times because of a lack of focus and this is very hard for any size of business and there's so many things to do all the time from operational hiring to marketing to sales to product development to cash flow accounting there's just a ton of things to do if you can't focus on the right things at the right time you're really gonna have a hard time succeeding so here are the six things that I'm going to get into when we get into sort of the meat of things we'll talk about what makes a

9:26good metric we'll talk about the dimensions or the types of metrics that are out there talk a little bit about analytical superpowers the lean analytics framework the one metric that matters and the lean analytics cycle and I've got a bunch of case studies here most of them are not small business case studies there are some consumer ones I'm warning you in advance that there are some consumer case studies in here but hopefully you can take interesting lessons from it nonetheless so first of all what makes a good metric so the definition is important for me in terms of what I believe analytics is and it's the measurement of movement towards

9:58business goals so if you don't have a business goal in terms of what you're trying to accomplish what are you actually measuring what's the point you need to measure things that are helping you understand whether you're getting to your goals or not a good number a good metric has to be comparative right so we think of a number like 2% conversion right that's a pretty good number but an increase in conversion from one week to the next or one month to another month is better so a comparative number typically is over time typically when we're doing something like cohort analysis so a good number think about

10:29the numbers you guys are tracking and maybe when you go back you'll look at these analytics and you'll come up with some other ideas for things that you want to track so a good metric has to be comparative it also has to be understandable so ideally the metrics that you're tracking today the stuff you're focusing on really defines your business right it defines what you care about and it should be something that you you know if you're working with a team you're putting on a TV on a wall and everybody's looking at and everybody's focused on it everybody understands it or if you're talking to somebody else they get what that number

10:56is all about a good metric is a ratio or a rate right an absolute number like number of users is a pointless metric right percent of users that are active starts to be better percent of active users month-over-month is even better so ratios and rates are inherently comparative and to help you make better decisions and finally most importantly good metrics change how you behave right if you're looking at numbers that you're tracking right now and you're not sure you know the number goes up it stays sideways it goes down I'm not sure what I would do with this number then it's a bad number and so

11:27this is the the lesson for me if a metric won't change how you behave it's a bad metric this doesn't necessarily mean you don't track these numbers I would never say you only track one little thing at any given point in time even though I'll talk a lot about focus but if you don't know what you're supposed to do with it track it put it in the back worry about it for later on so let's talk about some dimensions of analytics or metrics so the first thing is qualitative versus quantitative so qualitative is the fuzzy stuff right it's the warm and fuzzies this is often the stuff you're collecting early on

11:56although you should always be doing this by talking to customers it's the in Lean Startup it's the problem interviews the solution interviews it's getting that feedback that you can score it necessarily very well you can't aggregate it but all the insights into what you're doing and whether you're doing something worthwhile is in the qualitative the quantitative is just the hard numbers right and those hard numbers don't necessarily give you answers they just help you ask better questions so discover qualitatively this is the lesson for me you discover things qualitatively you prove things quantitatively so I'm going to give you an example of that so this is Airbnb

12:28this is a case study from the book I think everybody is is hopefully familiar with Airbnb it's a pretty large business at this point and they had this gut instinct they had this qualitative notion that professional photography was going to help the people who got those photos done get more business rent out their rooms or their apartments or their homes more frequently so they did an experiment with this and this is what they did they had this sort of hypothesis that professional photography is going to get more bookings but instead of building a lot of software to do this instead of building this automated system to figure out how to

12:59handle requests and route photographers and scheduling them and all this complicated stuff they built what's called a concierge MVP has anybody heard of that before that concept right so a concierge MVP just is you give out it you know you have this promise that you're delivering to customers we're gonna you know get you you know give you professional photography they don't really care how it's done behind the Sene so Airbnb did it by hiring twenty feet on refers and just randomly throwing them out there and getting them out there there was not any software to automate this or built around this right they just wanted to test it and it turns

13:32out it worked professionally photographed places got two to three times more bookings so they took that qualitative information they built this concierge MVP they got the data that they needed and then they kept iterating and iterating on top of it to try to scale the professional photography and then they built a whole system behind that to automate that process to make it easier for people to book these photoshoots and in in February of 2012 so that's that that date is pretty old they were doing 5,000 shoots a month and you can do this for free with Airbnb it's so valuable to the bookings that they get so fundamental to the success

14:06of their business that they've got a whole entire system and people at that company dedicated to nothing but professional photography all started with this you know fake system that didn't actually exist so let me talk a little bit about vanity and actionable metrics so vanity metrics are the things that make you feel good right these are when you look at your graphs and they look at your dashboards these are the numbers that are going up into the right the problem is they don't really give you any actual insight into what's going on the actionable stuff is really about what's gonna change your behavior and so there's no more message here except for

14:36vanity metrics are bad right I've listed just some examples of them I think that the worst culprit for me is the followers friends likes right I've got a hundred that I don't but you know imagine somebody coming to me and saying Ben I've got a hundred thousand Facebook fans I'm like okay that's great for you like good job the reality with something like that is if you can't get those people to do what you want them to do when you want them to do it it's basically a waste it's a vanity thing right it just makes me look popular and cool so let me talk about another

15:05dimension of analytics exploratory versus reporting so exploratory is the kind of stuff when you're looking into your data and you've probably because collecting data is pretty easy you probably have a bunch of data about your business looking into it to look for insights and new learnings about what's going on with your system and the reporting metrics again very simple this is the day-to-day managerial stuff but the exploratory stuff particularly for early-stage businesses is the interesting stuff so let me give you an example of that there's a company called circle of friends circle of friends started in about 2007 or so and when Facebook had about 50 million users this

15:41is a consumer business and so they leverage Facebook early back in the day you could spam the hell out of all your friends like crazy and they grew to 10 million users on Facebook and circle of friends was basically Google circles but inside a Facebook so 10 million users is a lot of users even by today's standards is a lot of users back then 2008-2009 was an incredible amount of users and the problem which is pretty obvious here was engagement sucked right so Mike Greenfield the CEO founder of this business goes and looks at his data and he's like holy crap nobody's using my product right there's a fundamental

16:14problem here so what does he do he goes and he starts looking at his data and he starts trying to figure out what's going on aside from the fact that it was insanely viral and what he found out was that moms are completely crazy but they're crazy in a good way because he found this segment in his data of people in this case moms who are using his product like crazy by every metric imaginable they were getting a ton of value so Mike makes the decision which you know to the outside looking in may have looked crazy because look at all these users he abandons most of his users and circle of

16:47friends become circle of moms now he grows that business back up to about 4 million users and eventually exits it but now it's based on these fundamentals that make sense because people are getting value and using the product and engaged so that's the the benefit of this explore excuse me the exploratory data to figure out stuff about what's going on that you may not have realized so finely lagging versus leading metrics or often these are called lagging versus leading indicators so a lagging metric or indicator is something that tells you reports the news right it's a historical value whereas a leading indicator helps you predict or tells you what's going to

17:23happen it makes the news and so the example I like to give is in lagging is churn churn is the percent of users or customers that abandon your service over a period of time right by the time they've turned they've left they're gone right very difficult to get them back now you can still act on turn because you can go and say you know what I know my turns too high I have to do some new stuff bring a new cohort of users in and see if I can change but you've lost a whole bunch of users the leading indicator for churn could be something like customer complaints

17:52customer complaints are going up there's a very good chance churn is also going to go up the difference here is that you can respond almost in real time to something like customer complaints because you can track it constantly right and and and then try to address it maybe a product update is causing problems maybe your customer service group is overwhelmed you can try to address it and figure out what's going on and solve customer complaints and in turn solve churn most businesses when they start it's it's the lagging metrics that they're tracking because they don't have a ton of data so they're focused on historical information you want to get

18:27to figuring out what the leading indicators are in your business so let me talk about analytical superpowers or what Alastor and i think of what the heck is growth hacking which is a term i'm sure you guys have heard before and thought a lot about I'm going to show you a chart and maybe you've seen this example before the numbers don't really matter but there's two lines on this chart one is ice cream consumption and one is drownings right and it turns out that look at these pretty lines right they actually work together as ice cream consumption goes up so do drownings now if we were idiots which were not thank

18:59goodness we might look at this and say well jeez if ice cream consumptions going up people are gonna drown we'll stop selling ice cream and that's gonna solve our problem clearly that's not the case right something else is at play here and that obvious thing to do is the seasons right as it's the summer and it's getting warmer people go people go out and eat ice cream it's warmer it's wonderful and if they happen to be living by the ocean they go in the ocean they also die right they take the ice cream they walk into the ocean they die that's what happens right so these two

19:28numbers are correlated right and if we looked at data and we said okay these numbers are correlate what can I do with them we might not get the right answer we want so here's the difference between something that's correlated and causal right correlated numbers are related to one another but probably impacted by something else whereas causal is one number that's actually affecting another number and in our case it's summer that's causal to ice cream consumption and drowning whereas the two are correlated so this is the superpower correlation helps you predict the future because it gives you an indication of what's going to happen I screams going

19:59up we know drownings are going up but imagine if you could change the weather if you could change the weather then you actually know both of those numbers are gonna go down and if that was your business stopping people from eating ice cream or drowning that would be fantastic so you look for a correlation in your data and then you start testing for causality and trying to figure out how these numbers work together and when you find causality if you can find it it's very difficult to do then you can optimize the causal factor and now you're starting to hack the future for your business so let me

20:30get into excuse me the lean analytics framework so we look at two major variables in businesses we look at the business that you're in and we look at the stage that you're at now you'll see six business models here um business models are very very complex and we had to sort of pick six sample business models that we felt that a lot of people would find some kind of relationship to but the truth about business models is that most businesses combine these things right I'm a SAS but i'm on mobile i'm a two sided marketplace with some kind of user-generated content so it's very difficult to really hone in on

21:06specific business models but we identified six and started to look at the kinds of numbers that mattered and then we we came up with these lean analytics stages and we believe that most startups if not all startups you know go through these stages towards succeeding empathy stickiness virality revenue and scale and I'll talk a little bit more detail about these so let me first go into business models now I'm gonna show you a slide that breaks all the rules of slides right pretty pictures with big words and it looks like this right you don't have to read this because this will go online but it's in the book as well this is our way

21:41of trying to describe in this case the SAS customer lifecycle doesn't apply for every business but it's our way of trying to understand how a visitor in this case flows through your business maybe free trial offer maybe not moving from an engaged user to paid ideally maybe raising the capacity limit so that there's tears now and your pricing and and what you see when you look at a model like this and we did it for all of the businesses a it's complex it's tricky and depending on the stage of your business if you're very early on you're just hoping to get visitors in and maybe convert and start using

22:14product validated once you've done that you want them paying or you want them paying more money once you've done that you realize maybe customer complaints are an issue and people are churning out so as you move through this lifecycle you realize the metrics that you care about at different points in time for your business change and so this is one example of the SAS customer lifecycle so I'm gonna do this is a little bit contrary and I think to the recurring revenue theme but and I'm a huge fan of recurring revenue of course but this is an example of a company focused on business model so this is a company

22:44called clear fit they're actually in the recruitment space one of the few businesses in the recruitment space that I actually like and they're at the revenue slash scale stage it's a SAS business even though it's our recurring revenue business they started by charging $99 a month targeting HR people and they were getting some amount of traction but not enough to really get them excited that they had sort of crack this nut and so then they started talking to their customers and they realized that HR people don't like buying things on a recurring model particularly things that they feel are tied to job ads even though this

23:17business is actually about assessments and so what they did was they switched from $99 per month to $350 per job posting right giving up that that beautiful idea that we're just gonna you know get these you know build up this recurring revenue stream they went to this other model which is a bit more traditional in the recruitment space but it turns out that HR people like buying things like that and what happened when they made that change was they 10x their revenue off of 3x volume in sales and Ben Baldwin I don't know if everybody can see that the quote here but he says people don't do subscriptions for

23:50haircuts hamburgers and hiring and and the lesson here for me is that you have to really understand your customer how they buy why they buy where they are in the in the ecosystem of their business how do they currently budget things HR people in many instances you know they get to get a job request like we need to hire this person immediately it's like great give me the budget spend all the money on a crappy job boards boom get a flow of resumes trying to figure it out they go over and they do it again and again and again they don't as much as we'd like them to always be hiring

24:18they're not always hiring most businesses don't do it that way they switch they were very successful so the lesson here at least in my mind is don't just follow the leader right you have to really test these things out you have to test your business model out you can do this don't just assume that a recurring revenue model makes sense if your customers don't like buying like that you may not want to get into the business of innovating on business model and trying to slam that value prop down their throats so let me talk a little bit about the lean analytics stages so we have these stages empathy stickiness

24:52virality revenue scale and we have these gates that we defined where we think these are the things that you sort of have to accomplish in order to get through from one stage to the next stage so we started empathy this is right at the very beginning you have an idea you you get out of the building as Steve Blank says and you start talking to customers and ultimately what you're trying to do is find a problem so painful that people are willing to pay you for it which doesn't always apply in consumer businesses but certainly in b2b and that's really what you're looking for and if you find that you move to

25:21stickiness and stickiness is all about building that Minimum Viable Product right that MVP you're building something for presumably some early adopter type crowd and you're looking for stickiness you're looking for engagement use of the product and retention that they stick around for long enough that you've clearly created some amount of value for them right and if you can get to that point in your business you move to the next stage which is virality now as we you know as we know in the consumer startup virality and the viral coefficient you know one active user invites another one in a perfect world I can sort of add people at infinitum

25:53that's an aspect of virality and virality can help in lowering your acquisition cost but this is really about acquiring customers in a cost-effective way so you've got this small base of customers now you need to move to the next stage and see if you can grow that in a cost-effective way if you can do that and start to scale that then you move to revenue now you may be and and I'm a big believer in this charging upfront right and that's fantastic but at that point in time you're charging just to collect some dollars right just to see what's going on you're not necessarily building this economic

26:26engine this small little economic engine that makes sense now that you've gone through these stages and you've got this acquisition and this funnel is starting to make sense now you're gonna focus on whether the math works and the economics work so this is what I'm really talking about revenue I'm not necessarily saying wait charge money I'm saying you don't focus on optimizing on revenue and lifetime value versus cost of acquisition until you sort of get through these other stages and then you move to scale and scale can mean a whole host of things it really depends on your business it might mean taking your product and targeting

26:56and targeting it at new markets it might mean using AP is to build a partner ecosystem it might mean business development or hiring salespeople but this is about now you've built this engine that could and you want to scale the heck out of it um so I'll give this example a case study of a company called buffer I think they were mentioned earlier today and and and how they sort of went through these stages and buffer is a social sharing app it allows you to to share things on your social networks and today they're focused primarily on customer acquisition what is interesting is that they did in fact charge from day

27:27one which is pretty rare for a business that's kind of targeting consumers or maybe prosumers right social media experts and that kind of thing or freelancers and that's that's pretty rare we don't see a lot of that but they did it they did do it now they didn't do it to build a big scalable business from day one they did it to prove that anybody cared they put up a dollar amount and said we're gonna charge people from day one to see if we're solving a problem that's painful enough that's one aspect of what they were trying to do and they start getting some traffic to their site and seeing what

27:56happens and so these are the numbers that they shared with us so 20 percent of visitors creating an account it's a pretty decent amount these people are paying a small amount of money starting to prove that they're solving an interesting enough problem this is the empathy stage for them then 60% of signups returned after one month that's also pretty good for them right so they're pretty pleased with this and now they're sort of proving a certain amount of engagement usage of the product and stickiness 20% of signups are still using the product regularly after six months so from empathy fully through stickiness then buffer makes the choice

28:28and they move to a freemium model right and they do this because they've decided they want to build a or attempt to build a bigger type of company and they're gonna use freemium as a marketing ploy or and a marketing attempt for virality but of course now they also need to focus on revenue and building this engine that makes sense and so they get about two percent conversion which is pretty good pretty typical for a freemium type business now even at this point we're not even we're not a hundred percent sure yet if they're ready to scale and focus on putting tons of people through the top

28:59of the bucket but their turns also pretty low their turns also about 2% and so the buckets not that leaky and so they know that if they pour people through the top of this funnel there's a very good chance if it follows this they've got a successful scalable in this case freemium business and so the lesson here for me is is that you should skip steps at your own you know skip steps but do it at your own risk right circle of moms 10 million users looks fantastic from the outside in inside the guy's miserable because nobody's using this product that's jumping to virality before you've proven

29:30anything if you can't build that fund the fund the fundamental economics of your system don't work or the fundamental math of your system doesn't work it's going to fall down at some point so how does this all come together well we take the business that you're in you take the stage that you're at and you look for the one metric that matters so this is a big sort of thesis or concept within the book is that at any given point in time in your business depending on these variables there's one number that you can really focus on it doesn't mean you ignore everything else but there's one number that really

30:02matters to you that you want to focus on and try to improve and there's two pieces to this you have to pick that number and you have to draw a line in the sand so drawing a line in the sand is is is essentially setting yourself a target right so let's say you're gonna pick conversion from on your website for people signing up right and let's say that's you know 0.5% you know it should be higher but you're not sure what it should be you have to set yourself a target and work towards that otherwise you're not going to know if you're successful now finding lines in the sand

30:34is is hard because you know there's not a ton of data out there about it so we did some research we found some numbers that we thought were interesting these are not absolutes by any stretch of the imagination you can look at analogous businesses to yours you can talk to other people who are not competitive but in similar businesses look for lines in the sand you can do lots of research about this as well but ultimately you need to find that target so let me give you an example of a company that that is it has in fact done that it's a company called office drop so they do scanning

31:04of your paper they're originally actually started by scanning paper and then allowing you to do collaboration on them so it's an interesting business it's a freemium business they have about 180,000 users and for them the one metric that matters at least for now because this number is going to change over time and does change over time is paid churn so it's the number of paid people that abandon versus the number of new paid people that are coming in and that's their one metric that matters but they also have a line in the sand right in this line in the sand is is sort of a goal or a

31:35target that they're trying to set at any given point in time and for them it's under 4% paid churn is what they're looking for a month over month and for them they're actually around 2% and of course it varies month by month and then they have to dig into it but what they told us was and and you know we see this of other sort of SAS businesses is that over 5% typically means you cannot possibly build a business that's going to work the economics are just not going to scale you're going to be putting too many people through the top of the funnel and it's going to cost too much

32:03and they're gonna they're gonna flow at the other end so his numbers are pretty right so he has this paid churn as his focus and line in the sand is just under 4 percent paid sure and so if it shifts a little bit he's ok with it if it gets up over that he's got a real problem and so here's how a company would look at something like pay churn is what I call sort of a a business health indicator right looking at this number it's on a TV screen everybody knows what that number should be everybody's focused on that number so let's say paid churn goes

32:30down right that's a good thing then you can dig into well why did that happen right it doesn't give me the answers it's just an indication that something is going well less people are abandoning my service so maybe it has to do with a marketing campaign that was successful maybe we were able to lower customer complaints we made an investment in our support and our efforts in support maybe it was response time for example maybe we did a product upgrade that a lot of our older customers were really looking for and we validated that it made sense to do so they didn't abandon and if I

33:00can figure out why patron dropped then I can do more of that and hopefully continue to draw up a turn and keep these happy customers if it's sort of net neutral I want to look at that and say well can I acquire more valuable customers what can I do here that's going to help my business and of course if patron goes up that's really bad so now I'm looking at things like well all the new customers that I acquired were they just bad cust maybe not the right segment for me did I have a marketing campaign that I invested in that just failed was there a

33:27product upgrade that went wrong was customer support sort of falling to pieces what are the reasons for this number going down so again it doesn't give you all the answers it just gives you the place to start digging um so here's some interesting benchmarks now the thing about benchmarks also is that they're very dependent on your type of business right so an example of that is something like time on site right so this is for social sites this is for the Pinterest of the world the reddit's of the world this is not for a Productivity software right if you have productivity software and somebody spending 17

34:00minutes in your application there's probably a problem with your application right and in some cases it might even be a vanity metric right they're spending 20 minutes on my side or in my application but it's in the help files right so it's really dependent on the type of business that you're in but some of them that I think are interesting the and I'm going to sort of pick them at random but CLV to CAC is customer lifetime value to customer acquisition cost and time and time again when we talk to companies they're looking for a three to one ratio so the customer lifetime value for every $3 of customer

34:32lifetime value I can spend $1 to acquire that customer so that's kind of an interesting benchmark don't take these as absolutes but you can look at your numbers and maybe compare them and see where you are I think page load time is interesting you know performances you know gets talked about but often doesn't get focused on but there's a significant sort of correlation with slower you know page times going up and people abandoning which is pretty obvious but not where a lot of people make an investment and when we talked to some folks I believe this was chart beat who ran a lot of this data for page load

35:03times it was under five seconds if you're under five seconds that's a good place to be when you get up to about 10 seconds you're gonna start losing people so let me go through just to end this I'm going to go through what we call the lean analytics cycle so it's trying to take all of these concepts and put it into sort of a pragmatic practical approach to to doing things whether it's you know validating an idea or building a product feature or running a marketing campaign or whatever the case may be so ultimately this is all about business problems right it's about solving business problems so you

35:35looking for a key business problem that matters to you you're picking the one metric that matters that's related to that you're drawing this line in the sand and you're getting started and so this is how it works we pick the KPI we draw the line we find a potential improvement right so now we're saying okay I've got this KPI I'm gonna use churn as my example right churn is 5% I need churn to be down to 2% that's my line in the sand I look for potential improvements you might run multiple experiments at the same time maybe it's product features that you know people are abandoning or not using again it

36:04might be customer support or something else so you find a potential improvement and you decide on what you're gonna do now if you don't have a lot of data about how to make an improvement you basically make a guess this is your gut what does your gut tell you about your business you should be the person that knows it the best so if you don't have data just make a guess if you do have data you're looking for commonalities in that data right you're looking for something in the data that thread that tells you what you should try to do as an experiment then you're gonna go and

36:33write a hypothesis right and the hypothesis is basically if I do this I believe this will happen which will give me this outcome well and I really believe in in in this structure and I also believe in writing this kind of thing down right put it on a piece of paper put it on a whiteboard so that when you're looking at it every day you're like that's what I'm trying to accomplish that's what I believe to be true let's go find out find out if that's the case then you go and you can do two things right you can sort of make changes in production just launch your

36:58changes add new features change features whatever it is or design the test right in some cases if you're building you know if you're talking about features here maybe you deployed only to a certain number of customers maybe you're a/b testing different things but ultimately you have a hypothesis now you've got to go run this experiment you're gonna measure the results and see what happens with that and then you're gonna ask yourself okay well did we move this needle right did churn go from 5% to something else what is it now and if you got let's say in my case churn from five percent to two percent

37:25success you won right so take a bow have a drink pat yourself on the back and then go to the next KPI right after turn what's the next thing that matters well maybe it's you know what now I can put people through the top of this funnel so let me go figure out which of my acquisition channels is the most cost-effective so it's always about going through this and moving from KPI to KPI now if you weren't six this is kind of the doom and gloom part you sort of pivot or give up I think there's a little bit drastic when I say it out loud now it's kind of like I

37:55didn't get turned from five to two percent so I just quit so don't do that but but maybe there's something in the data and some of the experiments that you've done that gives you an indication of where to move right that's the point of going through these kinds of cycles and doing it quickly is that you've learned something maybe you can pivot your business maybe you change the business model maybe you change the target market that you're going after off more often than not there's sort of two things that you might do is you might draw a new line or you might try again right so drawing a new line what

38:24this means is let's say I got turned from five percent to three and a half percent and it's not quite what I wanted but I'm pretty okay with it you know I look at all the other numbers and I look how that's affecting my bottom line and I look how that's affecting profitability and revenue and everything else and customer happiness and so forth I'm like you know what I can keep optimizing or trying I'm not entirely sure what to do and I may be over optimizing for my business right I picked 2% because you know that's what Ben told me to do but maybe it's not

38:50maybe it's not the right thing for me and the economics of my engine are ok at 3 and 1/2 percent so you draw a new line so we sort of drop that down right that gives you don't drop the line because you hit three and a half percent and you think well I'm done and I don't know what else to do right don't cheat but if you can draw this line or change the line which typically means lowering it but you know you're creating the value for customers you can do that or ultimately you try again you said I still I went from five to four I'm going

39:15to continue to experiment with this and keep going um so this is the sum total sort of the strategy of how I think for any aspect of your business you can iterate your way through the use of analytics hopefully towards success well thank you very much you on your chart did the needle move what if the needle moved and it was because it was summer sorry well what if it move because it was summer you get when I'm gaming how do you know that the needle moved then well is it summer well in yeah well yeah these things are not these this is these aren't all perfect right this is not

39:48perfect science right but ideally it's because you have a hypothesis and you're running a particular experiment there of course could be other variables that are affecting the business you know whatever those might be so you don't necessarily know perfectly but ideal you're running reasonably enough controlled experiments to say I'm reasonably certain that this is why it happened anything you could put in your analytic so that's just that's just well I think it's I think it's a question I mean what will happen is so let's say you run some kind of experiment and we'll you you know churn goes down and you're like aha turns down I can go focus on you know

40:22acquiring more customers that's great and then it's summer and all of a sudden churn spikes right so then you're because in general you're gonna be collecting a lot of data you're gonna go back and be like holy crap we fixed it why is it now broken and then you're gonna go into the data and explore and try to understand what's going on so that's what you do before you do your hypothesis yeah yeah I'm curious about the the without data make a good guess part could you maybe give some tips and advice on how to make good guesses about things you don't have good data for yeah

40:56so and and I would say in most businesses none of your data is statistically relevant right from and so we won't get into a stats kind of discussion but you you have to try something right most of the time you're gonna know what your problem is whatever and there's you know most businesses have multiple problems but you're picking one problem and you have to make a guess in terms of what do I think could solve this problem and what I would just suggest is try to keep those experiments really short right as short as you possibly can because you don't want to run something like well I think

41:28the products not engaging enough so I'm gonna go into a back room and build more features for six months and then launch that and see if that fixes it right that's not that's not good right cuz the the the the cycle time is too long so I think you have to keep those experiments as short as you can and you know maybe you can run multiple experiments at the same time and sort of try to understand which one might be the thing that's working but ultimately you have to do something because you know you sitting around and waiting is not gonna solve the problem we have time for one more

41:57question

42:02our company which is mobile games and something cash my attention which you say about freemium model a lot however when you put some numbers my company gets by try and buy which is no Freeman used try twenty percent of conversion rate but you talk about too much of a premium is carries if everyone goes to premium all even they got a high rate on such conversion try and buy should have got with the freemium model as well or I should stay the same it's it's I mean it's impossible to know without really knowing your business we do cover games mostly free games or free mobile apps in

42:45the in the book and and it's a fascinating space you know the trend very clearly in mobile applications is freemium the economics around it are its it's really like razor thin right the volume of it and you know I'm thinking about games and I've actually with year one labs we did it an investment in a game company but you know it's razor razor thin so it's difficult to say so that's really just sort of fall unfortunately that's sort of following the trend of well you know everybody's going freemium and trying for in-app purchases you know obviously it works in some cases doesn't in others but it's

43:20really a math game right it really really is and I actually had a case study and I took it out because mostly because of Jason actually because he was like you know I hate free and mobile and I was like better take this case study oh but it was it was for a it was for a company called sincerely so it's in the book so you might want to look at that and some of the things that they did where they built these there's a couple of tactics they took they built these off-brand apps so not branded for sincerely they built these off-brand apps just to test basic economics to see

43:50if they work to see if they could get people to do the kinds of behaviors that they wanted to do and and they built about nine to twelve apps within twelve months so it's a really interesting example of a company trying to understand the razor thin margins and the economics in free mobile