We Got HACKED... Here's What I Learned

0:00even though it was almost a decade ago I remember it like it was yesterday I got a text from my co-founder on a Monday morning who said overnight Sunday basically Sunday night into Monday we had dozens of accounts created from Russian IP addresses these hacker spammers were sending phishing emails to unsuspecting PayPal users trying to get their login credentials and they were sending it through our software it's that moment in a business where you know you've arrived because you're notable enough that someone did try to send spam through your application but it was devastating and our IPS were blacklisted and for those who don't know if you're

0:37not familiar with email sending it basically means that our emails sent from our platform even from our reputable customers were going into Spam folders and as an email marketing tool your one job is to get emails into people's inboxes it was a very difficult experience I remember panicking and thinking this is it like I don't think our business is gonna survive this and in today's video I'm going to talk about how it all played out and what I learned I'm Rob Walling a Serial entrepreneur with multiple exits I've written four books on entrepreneurship and I've invested in almost 150 companies so I arrived at our tiny little office which

1:14was basically a 20 by 20 room in a co-working space and there were four of us that worked out of this office the it was the second drip office and I was just panic-stricken and the interesting part is it was so early in drip's history that we literally didn't have a way to just shut down an account so someone could sign up for a drip account and if the trial expired we disabled it but if they were a paying user and their credit card was good there was no way for us to disable accounts we just hadn't built it yet you know early on in a product oftentimes you don't even have

1:46billing set up until you need to Bill someone so you can have users and be like well 30 days from now I'm gonna need to Bill folks or you don't have delete buttons in the app we didn't have a way to delete campaigns and delete draft emails that you hadn't used because we just didn't need them yet and so there we were with several accounts that we identified as sending these PayPal phishing emails and we could look at the email and it looked just like a PayPal email it honestly made me really sick it made me feel like I had done something wrong you know that we hadn't

2:13been responsible and I feel like you could argue it either way I mean we were a bootstrap startup that was still losing a bit of money each month I was paying out of pocket to keep us all fed and paid and we were building exactly what our customers needed to get the job done and that was the priority and so the first thing we did was my developer who later became my co-founder Derek Reimer he jumped into the code and he pulled their account IDs and manually put them in the code if accounts ID equals this don't let them log in don't send any emails it's the hackiest thing

2:45you could ever do if you're a developer listening to this you know how hacky this was and we pushed it to production because we needed to stem the bleeding the spammers who had set up these accounts they knew that we were in the US and that at midnight one in the morning on a Sunday night or I guess a Monday morning if you think about it none of us were going to be awake and that's what they did they sent it from 2 A.M until until we woke up and stopped the sending and so they knew what they were doing and it's obvious they had done this before and they were trying to

3:10steal people's credentials so once we got the accounts blocked the emails stopped sending then we had to think about what do we do about our blacklisted IP addresses because if they stayed blacklisted then our customer emails were not going to be able to be delivered and again you have one job as an email marketing tool and it's to get your emails into folks inboxes and so that's what we said about doing we started contacting The Blacklist and let them know what happened we said Russian hackers we're sending spam it's been remedied and how long do we need in order to you know in essence be taken

3:43off your Blacklist and it turns out some of them had automated means of doing this where if there was not a Spam or a phishing report within x amount of hours you automatically fell off this was really our first foray into this I mean this became a full-time job we had a single individual devoted to just monitoring Blacklist at a certain point in drip's history right as we were sending 100 million emails a month you can imagine this is a huge job dealing with all the providers but in these early days when we had hundreds of customers the first time we had spammers we had to think about this from square

4:11one so as I was working on getting rips off The Blacklist Derek set about to write some code to figure out how could we have anticipated this right and he and I got in front of a whiteboard and we looked at the signs what signs could have triggered an approval process right where a new account can't just start sending emails willy-nilly but if they have a credit card that's prepaid if they have a credit card from certain geographies if they exhibit high-risk behaviors and we defined those on this whiteboard how can we have a process that either allows them only to send a few emails or maybe not at all until one

4:45of us manually approved them in the end even though I thought it was business ending it was not and in fact we dealt with spammers for the rest of the time that I worked a drip and it became an ongoing battle it's just a cost of doing business unfortunately it's just something you have to deal with when you send email from your app and one thing I started to realize at that point is there were going to be a lot of these speed bumps along the way because as an entrepreneur you're making hard decisions with incomplete information and you kind of have a new crisis that

5:13crops up every day or every week and I started to think I can't let these make me think the business is going to end every time because if you get in the headspace as an entrepreneur it can really drive you crazy and it can make your time as an entrepreneur not enjoyable so over the next year or two we grew the business to multiple seven figures in Revenue we sold it for many millions of dollars and it all turned out good and it all could have been killed that one day if we hadn't reacted so now I want to talk about Five Lessons that I learned on that day back almost a

5:42decade ago the first lesson is to not panic bad things are going to happen you're going to feel like you want to panic and you can Panic for a minute or two and then you need to take a deep breath and you need to get to work because panicking is where your heart races your hair stands up and you're not very capable of troubleshooting an action actually fixing things when you're in that state of panic so I guess if I don't panic I mean Panic a little bit and then get over it and get to work the second lesson I learned was if you're the founder or the executive you

6:12need to lead and that was part of where I came in and we all kind of knew what to do but I had to take charge and say hey so this is what's happening here's what's muddled in my mind I think we have three or four priorities right now we have the Blacklist we need to stop the emails how do we keep people from doing this in the future how do we even triage this do we need to talk to our customers about this you know have we been hacked is it just spammers like it's all this stuff and so we divided and conquered and I said can you

6:35research this can you write the code such that we're not sending these emails anymore and just one at a time we hammered through it but it took someone like myself to kind of coordinate it and say this is the plan of action let's attack it the third thing I learned is something I touched on earlier it's that very very few emergencies are business ending things are rarely as bad as they first appear and if you can stay calm and meet with a few you smart people on your team and figure out what the steps are to remedy the situation you're going to get so much more done and have a

7:06better end result the fourth question that I would ask myself after something like this happens is how can I keep this from ever happening again some Founders are process people and they think about how to turn everything into a process and that can be good you can also take it too far and have too many processes too early and you see this with folks who work at Fortune 500 companies who then come and try to work at a startup and they want to process for everything but it slows you down too much on the flip side some Founders fly by the seat of their pants all the time every time

7:36with no process and you can move really quickly make a lot of decisions but then you start doing things over and over that really should be in a process and hand it off to someone else so there's a balance with a crisis like this where everything really hits the fan you always have to have that post-mortem of okay this happened and we fixed it or it's being remedied right now what are our next steps over the next days or weeks to make sure this doesn't happen again and whether that's writing code having a policy in place whatever it is this event for us was the start of a

8:06policy of having some trust for new signups but really kind of being wary of letting people send too many emails when they hadn't built up their trust score in drip and the fifth thing that I took away is that people who run blacklists and vendors see we were sending emails through email apis you can think of like send Grid or mandrel or mail gun and it was their IPs that were being blacklisted so we had to contact vendors and even some customers were reporting oh I'm monitoring your IPS and I see they're on Blacklist we communicated with all of these folks and they were willing to work with us because we said

8:40look this happen we are working on it we're going to have it remedied as soon as we can and what we found is that some people were unreasonable but most people were willing to work with us and they trusted that we were doing the best job that we could to get this fixed if you like this video and other videos on the channel and you want to hear more content from me talking about bootstrapping and mostly bootstrapping software startups head to startups for the rest of us.com or just search startups for the rest of us in any podcast player I've been doing that show for more than 650 episodes it's come out

9:11every week for 13 years and I dig into building and launching and growing and all the things that you need to be successful as a SAS entrepreneur as I mentioned at the top all of this happened while I was building my last SAS app drip which was an email marketing tool if you want to hear more about how I built that business over several years check out this video to see the steps I took to grow it and eventually sell it for Millions thanks for watching I'll see you next time