We Accidentally DELETED All Our Customers Credit Cards 😱

0:00I want to take you back almost 10 years to a hot summer Sunday evening in Fresno California it's 11 p.m I'm thinking about wrapping up what I've been working on when suddenly my phone starts vibrating I checked the screen and it's a contract developer and Friend of Mine Derek Reimer who was doing contract work for me on hittail and on this new project I was working on that was later to be named drip the problem was Derek and I didn't call each other we always texted so I answered the phone with uh oh is everything okay and his response his first question was do we have a backup of the database it turns

0:36out through a slip of the finger he had forgotten a where clause on an update statement and had effectively deleted every customer credit card we had on file this was one of those moments where you think to yourself I wonder if we're gonna make it through this and in this video I'm going to tell you the full story and also talk about four key takeaways that I learned about dealing with this situation I'm Rob Walling I've started six companies five of them bootstrapped I've written four books on entrepreneurship and I've invested in more than 150 startups if you stick around till the end of the video I'm

1:07going to tell you how we ultimately arranged things so we would have had very little downtime even if we lost our entire database so Derek and I are on the phone we're trying to troubleshoot how to get this data back in the database we know that we have a database backup but we have a contract DBA who was a freelancer but he was a guide worked with for quite some time he was in Poland so he was on a European Time Zone I think it was 11 o'clock Pacific time where I was which put it around 7am his time so I go into Skype this shows

1:40how long ago it was we didn't use slack at the time and I emailed him and we tried to get a hold of our DBA to figure out can we restore this table it should be possible right if we have backups this should be relatively easy I will say as a developer if you've ever forgotten a where clause on an update statement or a delete a row statement I've done this myself at least twice Derek had done it in production in essence in drip in the early drip database we add 100 maybe 200 paying customers he felt terrible and this is where those of us who are our hardest

2:14critics the person who is most angry about this is usually ourselves and I told Derek look don't feel bad this is a mistake we'll figure out in the future how not to make it but for now let's sit down and figure out what our options are so before I tell you how the story wound up let's look at what we can learn from this experience I have four items to walk through the first is that if you're building startups catastrophes are inevitable I've seen Founders or their companies get hacked I've seen folks lose customer data I've seen cease and desists lawsuits a critical platform Banning or blocking API access every one

2:51of those and more has happened to companies that I'm invested in companies that I've owned it is inevitable if you do this long enough and you do enough interesting and risky things the catastrophes will happen so the key is to stay calm and look at all the possible solutions rally your team or yourself and get to fixing it you can Panic for a minute 5 minutes 10 minutes then take a deep breath and think what are all my options the second step when this happens is to do just that it's to look at all the solutions that you have put them in order of which we want to

3:23chase down sometimes if you have a team you're going to be chasing two simultaneously and then act then get to fixing things as Derek and I discussed our options we knew that the best option was to get a hold of our DBA if we could and restore the table from the database backup but we realized maybe the DBA was out for the day maybe the DBA wouldn't respond or maybe it's always possible that your backups don't work and that we wouldn't be able to restore the database table so then what was Plan B Plan B for us was realizing that the actual credit card numbers were stored in stripe and

3:56what we had lost in the table was a customer token that linked us to stripe we hadn't lost the actual credit card numbers so we could have would have been a pain we could have manually mapped each customer from our database to stripe using the name field or an email address or some other ident identifier it would have been manual but it probably would have gotten us at least 95 percent of the cards maybe all the way to a hundred percent not the ideal solution but at least it's something that would work if plan a wasn't coming through and then we came up with a third

4:26option which was if somehow mapping them manually wouldn't work we can email all of these customers and ask them to input their credit card number again this is far from an ideal option I bet we would have lost 10 20 30 percent of our customer base it would have been a large number and it would have been disappointing but it's not business ending that's a thing to keep in mind is even things that sound like a catastrophe like losing all my customers credit card information these are just speed bumps they're not roadblocks it wouldn't have ended our business and in fact grew to become a multi-million

4:59dollar company that I sold for enough money that I never had to work again and we had many catastrophes like this along the way and each one we sat down we planned and then we figured out how to fix it the third lesson to learn from this is to plan for bad things to happen and to hopefully mitigate them in advance so there's a balance here you don't want to build a ten thousand dollar fence around a thousand dollars in cash but there are things that are pretty likely to happen as you grow including things I've already mentioned like data loss getting hacked being sued having a cease and assistant to you

5:32having application downtime how can you spend a small amount of time and or money to protect against each of these having backups of your database paying a third party like Tiny Seed Company glitch secure to do pen testing or security audits having Insurance in place having a protective corporate structure in case you do get sued that you don't get personally sued there are so many ways to think about mitigating some of the more common things that can happen to you and if you prepare for bad things to happen without expecting them to you can do your future self a favor and the fourth thing to learn from this

6:06is specifically with database backups because data loss is something that is actually quite likely to happen in the life of your startup whether accidental whether it's on purpose whether your hard drives fail or whether you miss a where Clause it's to not only have backups but test your backups test your backups over and over I've heard the same story 10 20 times about companies that back up their database but they never test the restoration process I have seen entire startups doing six figures in ARR I think there was one that was in seven and they had to shut down they had to shut down because they

6:41had backups that just didn't work that didn't restore so what I've always done is paid a DBA a database administrator a few hundred dollars a month to not only make sure the backups run but to test those backups once a month and our DBA would restore the backup and then run several queries against it to ensure the backups were working with things like mitigating data loss a little bit of effort and planning goes a long way I'm sure by now you're wondering how did this all end up did we lose the credit cards did we need plan a plan b or plan C the end result of our journey to get

7:12the credit cards back ended very quickly within a half an hour our DBA was awake he said no problem he grabbed the most recent database backups did a restore and in no time we had all of our credit cards just like we'd had an hour before it was a fairy tale ending for something that could have obviously gone pretty poorly but following the steps that I outlined in this video we had a couple of backups even if that initial plan hadn't worked in a minute I'm going to tell you how we eventually got to the point where we would have had almost zero downtime even if we lost our entire

7:44database but before that I want to recommend if you like this YouTube channel and you're into podcasts I have two podcasts to recommend that are related to this channel startups for the rest of us is the first one I'm approaching 700 weekly episodes I've been recording about 30 minutes every week for 13 years it's all about building launching and growing bootstrap and mostly bootstrapped SAS companies just like we talk about on this channel the other one is the microcomp podcast just search for microcomp in any place you get Podcast that's where we pull some of the audio from these YouTube videos like you're watching as well as a lot of our

8:18microcomp talks we put them in an easy to consume podcast feed so how did we get to the point where we would have had almost zero downtime well this is not something you want to do from day one because it's expensive but once you get further along you have a lot of paying customers you have Revenue we paid for a complete hot swappable replica of our database it was a live backup it was expensive but it meant we would have had close to zero downtime if something happened to our database which was one of the only single points of failure in our application because everything else

8:46was spread across multiple servers the front end interface the apis are queuing all of that was a bunch of servers running but with our database since it was a single server that was a single point of failure if you want to hear more about drip which is the business that I was building when all this happened check out this next video to get the backstory of building a SAS app to millions of dollars in annual revenue