Victories & Tragedies: The 3 Year Journey Building Balsamiq Cloud – Peldi Guilizzoni

0:00[Music]

0:11hi everyone welcome all right so I really tried to get to that 80/20 practical advice and 20% inspiration I enjoy my career speaking of microcon for a lot because you can really get into the nitty-gritty and like I said like they said I come back for the community all the time I've made lots of great friendships through this conference alright so today we're gonna be talking about our web app called Masonic cloud and so it's gonna be a journey off from going from the beginning until now so let's start with a quick show of hands how many here are doing a SAS startup ok how many are

1:01doing software but not SAS alright so and then how many here on the spectrum between being client-side people and server-side people so for client-side front-end UX a few yeah quite a few but about ops IT machines Linux yeah quite a few and then the rest I feel like maybe both we'll see anyways the first question that I have for you is are you SAS literally sure that you want to do SAS because like I say it's a bit of a painting the SAS you know there are other options you don't have to have run servers you know there are other ideas so I'm hoping today to make you think

1:55basically scare you to death about running a SAS we'll see how that goes I did not want to start a I have a SAS version because I wanted to be a micro preneur I was a big fan of the micro preneur Academy and start small stay small that was my goal I wanted to have a one-person company and I wanted to be able to sleep at night and I wanted to be able to spend time with my family and I wanted to be able to take vacation so I said no SAS this is a blog post from 2008 where I said no I'm not gonna host it myself

2:29I don't want to do it it's just I have to keep your password safe it's just too much stress I want I want to do maybe a desktop version or you know you keep your own data I don't want to have any of that the other thing is DevOps right I was by myself a lot of you want to be solopreneurs or a small company and so you have to be good both at the client side and at the server side and it's really hard to be really good at all of it it's also part of the challenges why I want I started a company I wanted to

3:04get good at all of it you know I wanted to learn what what it was like to do it all but I mean these are very different careers right there's front-end developers back-end developers there's IT people there's it's hard to do it all the other thing is with the sauce these days unfortunately you sort of have to have a global team from the get-go because people expect you to answer their emails you know if if it goes down and you're sleeping right you're sleeping half of the 24 hours all right you know at least what happens right so you have to hire someone somewhere else to at least be able to answer the emails

3:45not even restart the servers but at least deal with the fire as it happens right so it's expensive it's not easy so I said no I'm not gonna do it can't do it then of course I passed and our customers started screaming for it they really really wanted it a competitor came out that was identical to us but they did a web app and so we're like alright we got to do it so I had a I hired an employee who was much more of a now ops guy and he said I'm fine staying up late I'm fine being on-call to answer if it goes down and I said alright let's

4:23do and so in 2011 we launched what's called my balsamic I don't know if any of you have used it but it was a web version of our wireframing editor so in 2011 we launched and it was it went pretty well we learned a lot this is an interesting slide because you can see how the revenue grew and grew and grew and then in November 2015 we did a release that added yearly subscriptions so that's that's why it went up and then we have a release since we did not other than security updates we have not released my balsamic and so of course the revenue is going down but

5:09the lesson here is that the revenue is really sticky people just won't leave I expected this to drop off much quicker I was hoping they would drop off my Stricker so that they would move to a new one but it's still making $50,000 a month and it hasn't been updated in four years right so when you're stressing out we need you we need more features we need to continuously pump our features no actually you don't really have to it will continue and this is also the reason why the web is full of these zombie SAS apps they have not been updated forever but people keep them

5:43around because they bring in a couple thousand dollars a month right so so that was interesting anyways the reason why we stopped updating my balsamic is because in 2015 we decided to rewrite our whole code base which is the thing that Joel Spolsky is to say it will kill your company it should never do it well now now actually people do do it and so this DHH gave this talk about how they've decided to rewrite Basecamp and basically he was saying after a few years you f2 you've learned so much from running yourself your company that you want to compete with your old ideas and so they what they did was they did

6:24version 3 which was a completely new app and they left the old world running in parallel and I saw this is brilliant we should do the same thing and so we did we built more Summa cloud and when we shipped it we said my balsamic will keep going until for a long time if you're happy with it don't you don't need to change anything and as we saw a lot of people are still very happy with it so this is a good trick because it allows you to cut features because it's a new product right it allows you to to not have to go to feature parity before you

7:01ship the new one right so and in a way it is risky of course it takes a long time we're going to talk about that today but it allows you light like tha says to be your own best competitor instead of having some other startup compete with you because the market changes that while you want people want changes alright so let's start the journey this is the end of 2015 we did the my balsamic release with the yearly [Music] subscriptions and then we decided all right let's let's rewrite everything so we have two parts to our to our app one is the editor so the wire framing editor

7:46where you do the actual work and then there's the wrapper we have a desktop sort of experience we have a Google Drive experience that embeds the editor and we have our web or web version cloud which has you know people management etc but then once you add it it's the same editor so we have different teams we have two teams internally that work on these things so in q4 2015 these are you'll see a few of these lives these are taken straight from our wiki we have quarterly goals for each teams and these are the quarterly goals so in 2015 again for the editor we had new editor one web

8:28right let's try to try to see if we proof of concept you know if it's gonna work if the technology is going to work same for cloud we're like alright the old stack was building built-in Grail Grails which is a sort of a dying technology bad bet that we had made it was on AWS but it was a Mis it will take two hours to build there was you know we were not happy with this so we thought okay what's the what's the new hotness in 2015 for for server-side stacks and so we looked at we looked at server list a SS lambda had just released and I was

9:13really excited about the promise of just write the code will take care of all the scaling will take care of all the security updates will take care of everything and so we try that it was a disaster it was not built for web apps it was it's built for maybe little pieces of it micro services transaction stuff but at the time when they launched their at a time out of like half a second for a query if anything was longer than half a second it was going to timeout and we might also get some things that took like a minute to come back anyway so we sadly give up on it

9:49now if I like to start now I will give it another look but it's still not really designed for it for that we're still not there so I really love this is one of my favorite quotes if you're excited to play around with something it probably doesn't belong in production this is from my shade shows mostly from sorry about that from pinboard and I love it you know they're more boring the technology is the safer you are in production because it means that they're it's been tested it's been documented there's a giant part of Stack Overflow about it right but it's kind of hard to find developers

10:27that want to work on boring technologies but it's still fun so we went with what for us was a mixture of boring and exciting so on the client side use react we as graph QL which was still pretty young at the time to talk to the server the server is built on node so which is JavaScript so we we can move people from the client to the server more easily and then my sequel for database and Redis for cash so these are sort of tried-and-true technologies and we we run it on a table yes just like we've been running for years the one thing that we added at this point was

11:09convex convex is kind of like kubernetes kubernetes I don't know how to say that it's it's an open source library that allows you to do updates scale up scale down you know helps you orchestrate all the different services that make up your web app all right so in 2016 the editor version the timeline was all about okay make it more into a real editor go beta and then really go beta and then really really really good beta because it always takes a lot longer than you think so same for the cloud version first quarter was okay let's set up production environment staging environment that go beta and that we were not ready so do it

11:59again and do it again so that 2016 is really the time where you have to sort of build the building blocks for the app and so what are some of those this is signing up and logging in you don't have to you're understand it just has to look complicated which it is you think in your email and pick a password and you're in right well not really there's email verification and there's I I never got the email verification and then you have to have them accept the Terms of Service and then if you want to do social login like Google like log in with Google that's a

12:43whole other flow and then logging in oh I forgot my password so these are all screens that you have to design you have to implement and it gets even more complicated so this is the flow for when someone who's already enough' who already has an account wants to invite a colleague to their workspace right well there's a million things you have to think about because oh you got an email it's both signing up and getting out into something at the same time and you might be logged in with another account so it's it's really complicated authentication has a ton of hidden features that you don't see like

13:24two-factor authentication you can you know you want to offer that or people want to change their username how do you deal with that or even just picking the duration of a session if you want to do a short so that's secure but you don't want people to log in every day because they complain about that so how do you do that then there's all these features like show me where else I'm logged in what other devices I'm logged in on or lock me out of everything or on the back end you want to do rate limiting on the login API so that people don't brute-force and Dorne then I will

13:56service so all these things nobody ever sees right or when you log in from here Google says hey this is an unusual location for you right this is all stuff that has nothing to do we've got what you're actually trying to build right you're your value-add is not here this is table stakes and it thanks for freaking ever the other thing to build at this point is this sort of complicated chart about the lifecycle of an account right start with a trial and then if they might expire or if they pay it goes to paid then if there's a problem with their credit card we give

14:35it a we move it to grace which means you have a few days to fix that other one other after data goes to expire really delete it and then really really deleted you know what I mean right and then there's your admin team your support team is supposed to be able to also like extend the trial or move something you know so this is a lot of stuff you have to do before you even work on what you're actually trying to build okay so a couple more tips if you're doing b2b how many are doing b2b yeah that's easier for sure so one thing that we've

15:14learned from my balsamic that we put in cloud is to always have if you have the concept of a shared workspace for your customers always give them an option to have two owners not just one admin always two because one might get fired and then the company writes you hey we want our data and legally it's actually their data right so or one goes on vacation the other one wants to I I'm in the account so always have two owners the other thing that we added is the billing only administrator so we do it all the time whenever I signed up for a new service I always say don't send me

15:49the card send it to joy because she's the one with a credit card and she wants to pay but don't let joy mess up my stuff so only give her access to the building area right so this is something that we we learned over time and we did and this is the revenue for 2016 we hadn't launched anything so it's all zero what's happening okay oh no these are not that's not correct I got a second we're missing a slide yeah there's a all right well let me see if I can remember 2017 2017 is beta right so we've done it we took us a year

16:36instead of three months like we were hoping but they're 2017 is let's get to beta let's do the beta and then let's do gamma the gamma is what I'll tell you about it in a minute but it's a paid beta that we do every time whenever we ship a new product and then the editor was let's get it to shipping quality right so 2017 was all about shipping so what we did was this gamma which is basically early access program is just like the beta were you inviting people whenever you want but now they have to pay yay but it's great so it's like the product

17:12is basically launched but it's not linked from your marketing website you have to give people you send people the login the page to sign in the URL to sign in this is perfect for working out all the building bugs right all days of the lifecycle of you know of an account here's where you can test it you can test people credit card expiring and all that stuff in the meantime you can train the support team on supporting this new product because it is a new product and it's ready so you can you can write the docs take screenshots is basically live but you're not really telling anyone yet

17:51and then you work on the support or administration UI which is all it's a whole other app for extending trials looking up people etc and you work on your ops because even during the gamma you you try to get enough people that will show you the first bottlenecks in your server-side architecture and so you fix those before really launching right and then you work on the legal stuff there's a service privacy policy right and you're doing this what money's coming in so really you can do this for a year and basically removes all the stress out of the launch because if you do a gamma at launch day all you do is

18:34write a blog post saying hey look we've been alive for six months we have two hundred paying customers that's it there's no there's no release it's already been released right so here's a few screenshots of our what we call super admin UI where you can do all sorts of things obviously search you search for people you see everything about them what their sessions you can log them out and four spaces you can make them free on credit restart trial so this is all stuff that you're gonna need again it's invisible and again has nothing to do with whatever you're building but it's something that you're gonna need to

19:14build at this point it's also a good time to start working on your getting into a good groove of release cycles so weekly releases so you practice all that dance merge from master to release kick the bill blah blah blah blah blah right this is a screenshot of our load balancer so you want to do zero downtime releases right it's a dance where you spin up three new servers with the new version you attach them to the load balancer when they're all healthy you take the three old ones away and and there's no downtime and everybody's migrated you keep the three ones warm for a while I guess you have to rollback

19:52right so these are this is a time where you have to you work on these things and because it's still very it's live but not public so if it goes down it's not the end of the world right and also you add all the auto scaling basically what you're trying to build is a system that is self healing and self scaling right so based on usage it goes up or down and if it goes down every starts quickly so that you if you're asleep you don't have to you know nobody ever notices ideally it spins up a new instance in seconds and and you can deal with that in the

20:31morning right so that's all different sets of rules that we set up for all the different parts of the app to to do this kind of orchestration I mentioned legal stuff privacy policy now's a good time to work on that especially now with gdpr you gotta be have a really good one you got a list all the people all your all your vendors all the people that you send your customers data to and so here you're trying to get as few as possible right and you are big names and I know this is bad news for those of you who are building software for other startups but as a customer when I sign

21:18up for something I don't want to see my data being sent to 400 little players around the world that I've never heard of what are they going to do with my data right so so try to pick your vendors carefully we'll talk about this a little more later same with the service look at all the stuff on the left you have to cover a lot of a lot of legalese but if you want at the bottom like Terms of Service it says hey if you're a startup and want to copy/paste this and change our name to your name go ahead so feel free but have it checked

21:51by a lawyer we spent a lot of time on this this is 10 year old 10 years of of tweaks so we're pretty happy with it anyway so this is the revenue for 2080 17 so as you can see up until October so let's see in say March we went gamma so we started getting people to pay and then in October we had already about $2,000 a month in revenue and we said ok we think we're ready to open it up and so we did the blog post and it showed up and so we were very happy and it's all good so so you know that's a good six

22:34months of gamma that made us be able to handle the load once we launched all right 2018 does anything now the app is live right everything's great now we get into our regular rhythm of releasing what people are asking for because whenever it's a version one it's gotta have some rough edges it's inevitable so as soon as you launch you're gonna hear about it you're gonna hear what are the the features that are clearly missing that you have to work on first and for us it was these things called revisions and notifications search permalinks these are all the four big things that people were asking for and then we

23:16noticed that two things one is we had decided to save data in a database that was making the data bass too heavy so we thought about that kind of data is really a document it should go to s3 right and that's something we didn't notice during the gamma and then off zero which is a library it's a service that helps you with all those authentication features went from being ten dollars a month to eight hundred and fifty dollars a month and we're like okay and let's break that in-house we were not it was not giving us enough for that value so until one we decided to do that via the

23:55editor team moved to rewriting the desktop apps instead because the web was pretty good but you know but don't lose anyone meaning let's leave some people in the web version so if there's bugs in cloud we can fix them but but basically yeah this year was all about let's give people what they're asking for let's do just regular normal stuff but then in q2 something happened which we'll talk about in a minute and then in q3 it's more about okay now let's slow the F down we're gonna just a little bugs we're just gonna take it very easy and as you noticed notifications which was supposed to be done in q1 and r0 they're

24:38still not done in q4 right we basically slowed slowed a lot down and the reason is got this message of May 11 from my lead developer in cloud and he said we have a fire I deleted some cloud projects and I was driving and I said alright twenty minutes and I'll be home we'll deal with it so it was bad it was really bad we took a few a couple of days to run queries and understand what was their real magnitude of the damage and in the end we had deleted twelve hundred and fifty eight projects from our young web app these are you know you've used our product you know what

25:23you know what that means right it's you spent months of work working on this design in your home that's their one source of truth that your developers look at gone gone I told my wife well we had a good run I thought this was gonna be the end of balsamic but it wasn't surprisingly enough it wasn't and a large part of why it wasn't is this email that we put together the support team drafted it and then I worked on it with them and I put it in its entirety in the slides so that when you download the slides you can use that and copy paste should hope you never have to use

26:10it for your sake but it's all here three pages I mean at the end of like this is horrible it hurts if you want me to meet if you want me to meet with your boss I'm happy to do that like right because people were pissed right a lot of a few people were pissed but most people were incredibly accommodating they were like you know we do software to happens and we were we were bracing for a disaster and only three people trying to threaten to sue us so what mostly because of this email so absolutely take a look after when you get the slides because apparently we did

27:02something right there mostly the lessons though is if you're moving data from one kind of storage to another kind of storage third the backups are the kind of stories to do that because because we're stupid even after ten years of being in business right you should not make these mistakes but they happen so always there's backups on and set up daily backups with at least seven days retention and then a few tips on when your things to think about when you're writing code that deletes data okay so first of all dogs do it on your own get someone to code review do a presentation and this is why

27:46what I'm planning on doing get a few get some feedback before you even start writing the code this is another nice tip if you have automated tests which I hope you do instrument them so that time can go faster right because you're deleting stuff you usually you keep stuff in for a few days and then you really delete it but to test that you have to wait a few days so instead make it so that there's a fake concept of time that can go in in one minute and and then show you this is a little bit better work but it's worth it use a trash never really delete anything

28:21until you really want to for it's a liability to keep people's data forever right and gdpr right to be forgotten right you have to delete at some point but don't just go you know keep it in a trash for a week or so right and there's another one our routine we caught it after 1200 it could have deleted a hundred percent of the data but we caught it after an hour it only had to leave at 1200 but making it make it delete one thing a minute right it takes two weeks to delete what you wanted to leave but who cares right at least after a day if you notice you

29:00have a fire you've only deleted a hundred things and not everything right and then take a freaking snapshot right before shipping the deletion code right so there if you do delete you have something we were able to recover thousands of things from because we had the last three days and then this is what happened to us those ship the least of stories right after coming back from vacation but that was part of it because the developer had it already one of the gator came back this has been ready let's go right and we didn't think about it you know it wasn't the right mindset all right so about picking your

29:40third-party vendors we use PubNub to do a real-time collaboration you can do Co editing and when we shipped they said oh you're paying 3000 now we are changing our thing from now on you have to pay 30,000 a month for for this service because we added all these features which we don't care about and so that was stressful I have to get on a call with them to say please give us a year you know 2000 flat rate or something so that we can get off of your platform because we're not your customer anymore so we rebuilt the whole server with socket IO etc another thing

30:17that happened was we had log Matic as a service nice European startup to gather logs from all the different machines they got acquired by data dog which is a big massive thing it's expensive and again we saw the logs actually have a lot of personal identifiable information for my customers so we thought let's bring that in-house so that it's one less vendor that has all this stuff and so we built our else ELQ stack elasticsearch Cabana and I forgot the third thing anyways so we built that in-house so when you pick yourself party vendors again try to pick someone who's not gonna get acquired and it was not gonna raise the prices too

31:00much on you it's not easy because these these take months to replace right it's work that your customers never even see anyways 2018 revenue went up you can see may there's that big dip people left they were pissed they left but then miraculously they came back and it kept growing so now we come to this year and this year we're back we're sort of pushing the accelerator again we've recovered from the from the fire and but we needed more people to work on the desktop app so these teams are like five people and three people we're not they're not enormous right but we need it we really wanted to get the

31:45desktop app rewrite finish so we hire more people for the idea team we moved some people from clouds to editor so cloud this year new app one person one developer that does everything because we said let's no down it's going well our customers are happy support says it's a great to support not to be not another request so so we slowed down a lot and we worked on little feet on these features that you've seen in the previous slide too so the things that we keep trying to work on and never get to but we're doing that now and really the name of the game after after a year you know after your

32:28web app is a couple of years old is all about ops it's all about scaling our one developer has really worked on setting up all the alarms and all the metrics you know CPU usage data memory data to make sure that as soon as there's something that's a little bit dangerous we know first before things go south right disk space going out there are hundreds of these I have no idea what they are thankfully the stage of my career where I don't have to but you know it is a lot of work again completely visible to your customers but there you have to do all this stuff especially right now that is

33:09scaling gradually and and then this is another slide that remember this this was from a few years ago Apple WDC keynote they did a Mac OS 10 update and famously that this very controversial slide which was we're not adding any features we're just fixing two decades of technical death right and and actually this is kind of grainy because there's no pictures of this event anymore they kind of cleaned it all up but that's kind of the mindset you have to be in like try to the features you build before you ship then it's all about making a scale right if you have a large team shirt do what everyone but

33:57even in that case I would say focus on the back end a year for a year or two and then once that saw that then you can add more things bringing more customers so this is now this is the whole revenue for cloud since 2016 without the gamma is this little thing right here you you know to put it in context and this is what we're expecting for for the yearly revenue so overall we're very happy with it but I hope that in this talk I showed you that there's a lot more to assess than what you read online so thank you [Applause] [Music]