The Day Podcasts Stopped

Go behind the scenes of the first DDoS cyber attack in Buzzsprout's history and learn what we're doing to prevent outages like these moving forward.

Read the full DDoS Technical Post-Mortem Report on our Blog

Charities we supported:

• Feeding Texas
• Front Steps
• Project Vesta

Special thanks to:

• Jack Rhysider, host of "Darknet Diaries"
• Jordan Blair, host of "Dreamful"

Review Buzzcast in Podchaser or Apple Podcasts to let us know what you think of the show.

Contact Buzzcast

• Send us Fan Mail
• Tweet us at @buzzcastpodcast, @albanbrooke, @kfinn, and @JordanPods

Thanks for listening and Keep Podcasting!

Transcript

Copy

Priscilla: 0:00

And then I saw something about a hacker and I was like, well, that's crazy an attack? What?

John: 0:06

I get a text from Kevin that says, hey, alert the FBI. This guy must be some kind of joke, something going on. And then Tom FaceTime me,

Kevin: 0:21

I got that text and started digging into everything that was going on. And then I realized, sorry, kids, it's time to hop in the car, we need to head back to Jacksonville.

Travis: 0:44

On Sunday, February 21, Buzzsprout was the victim of the cyber attack. Now, none of our customers podcasts or personal information was ever at risk. But it was still a big deal. Normally, on the show, we talked about what's going on in the podcasting industry and dive into new features that Buzzsprout is rolling out to our podcasters. But this week, we wanted to do something different. So in this episode, we're going to take you behind the scenes and share the real stories of what it was like to be inside Buzzsprout during a tenuous 27 hour period, and explain exactly what happened. Don't worry, though, the story has a happy ending. It all started late Sunday morning, when Tom one of the cofounders of Buzzsprout got a notification on his phone.

Tom: 1:32

I think we started receiving a notification about 1140. And it was definitely something I hadn't seen before I was at church with my family, and just got up and left. And they had no idea I didn't say a word because I just I had to get out and go look and see what was going on, and was immediately in touch with Brian traywick, who does all of our operations. And he was already on the case and investigating what the issue was,

Bryan: 1:58

I was actually pressure washing my house at the time. And I got a notification on my phone that the website was down. So I immediately put everything down, ran inside to my computer started looking into what was happening, and saw that traffic was significantly higher than normal, about 10 to 15 times higher than normal levels of traffic.

Priscilla: 2:23

I had done some in the morning cleared, set out some of the problems that I could get to and then was going to get back on in the afternoon. And I looked at my phone before starting a project in the house and saw someone had posted in Basecamp. And Tom had said looks like something's going on. like there might be we might be getting hacked. I called Megan. And I was like, all right, Megan, what's going on, as she and I were kind of going through the information we knew to kind of craft a response to support people and to everyone writing in who didn't know what was going on. And then we were also trying to communicate with Tom and with Brian to figure out eta is how serious this is what's compromised. If anything's compromised, all of that kind of stuff.

Alban: 3:05

I went to a museum with some friends. And I'm driving back and Priscilla starts texting me from support, like, are you seeing this, and I get back home, sat down at my chair and did not move from about 230. Until 1am. It was like time did not really work.

Travis: 3:22

From then on. As word traveled quickly, everyone started jumping online to see what they could do. But just to give you some context, Buzzsprout rarely goes down.

Alban: 3:34

If ever, you can be like, okay, we're down for a minute. That's a big deal. We're down for five minutes. This is a bad outage, you're down for an hour, and you're like, Oh my gosh, that probably only happens once a year. And then by the time I you know had been there and we got back up after three hours, you know, is the biggest outage I think in Buzzsprout, probably 11 year history at that point.

Travis: 3:56

Now at this point, the entire ops team, support team and marketing team are all hands on deck. And it was a tough go there for a few hours on Sunday, when we were trading blow for blow with the attacker to see how we could get ahead.

Bryan: 4:12

The first thing was put our site in a mode and I'm under attack mode to try to keep it online despite an ongoing attack.

Travis: 4:20

That's Brian, our senior Site Reliability engineer,

Bryan: 4:23

you may be unable to access the site at times, it most likely was very slow, because we were trying to cut down on the amount of automated traffic coming from the attack that was hitting the site at a time and as a result, some legitimate traffic, quite a lot of legitimate traffic when that mode is enabled was also being affected. And then the next phase was to analyze the traffic of the attack and to find patterns that we could actively block to prevent just the attack and therefore allow as much legitimate traffic as possible through to the site.

Travis: 4:55

I asked Megan what was going on with the support team during this time.

Megan: 4:59

So You're kind of starting to see it in patterns. Because as the attack was active, we would be shutting down certain things and then re activating certain things just to kind of counteract all the attacks. So you would start to see a lot of things that are like, I can't even log into my Buzzsprout account. And so then you'd have to be like, Okay, look, we're getting hacked, like, keep trying, you know, we're doing our best. And then once we were able to get in, they couldn't click on anything, because our servers were being so overloaded. They couldn't click on anything in the account. So they would be able to get in and then it was like you're frozen. And just a lot of people being concerned about their content. Was it safe, a lot of people concerned about their personal information? Was it safe, which Luckily, we were able to reassure everyone that this was not a data breach, and they were okay with all their information was fine. all the episodes were okay. But I think Sunday especially, it was a day that a lot of podcasters put up their episodes or scheduled their episodes for the week. And so there was just a lot of anxiety around when were they going to be able to upload these episodes to keep their audiences happy and aware, and what should they be telling their audiences? So trying to communicate all of that?

Travis: 5:56

Yeah, to say that our customers noticed that the website was down and that none of the episodes were playing is a bit of an understatement. Here's how many emails we normally answer in customer support on a Sunday,

Megan: 6:08

we usually do between like 100 to 150. On Sundays, it was at least over 800 that day, a lot more than what we typically see on a Sunday. And between that over that Sunday, and that Monday, both Priscilla and I answered over 2000 emails, which like a busy day in support is 500 emails would be like a really busy day. And so for it to be like doubled. That was a lot.

Travis: 6:31

So at this point, early Sunday afternoon, it's all hands on deck fighting against this cyber attack, which we pretty quickly figured out was a distributed denial of service attack, or a DDoS. But what exactly is a distributed denial of service attack? And why are networks like bus routes, so susceptible to them?

Jack: 6:51

What a denial of service is, it's when somebody attacks you in a way that your services cannot work anymore.

Travis: 6:58

That's Jack Rhysider. He's the host of the Darknet Diaries podcast, which focuses on the world of cybercrime.

Jack: 7:05

Now, this could be flooding your internet connectivity with so much traffic, that legit person trying to access the website or podcast can't get to it. That's a volumetric based DDoS attack. But you can, you can have a DDoS attack, that's just one packet that can just come in and take down your network. And you know, if one packet takes down your network, that's a denial of service as well. But in your particular case, what happened was that you had a lot of traffic coming to your systems and servers that made it just it's almost like a crowd of people were filling up your business with making it so that nobody else can come into your business. So yeah, it's kind of like that. There's just so much stuff in the way that you can't get through legitimately.

Travis: 7:48

Now, if this is the first time that you're hearing about a distributed denial of service attack, and what it's like, you're certainly not alone. And you might wonder, well, is this something that happens frequently? Or does this happen rarely at all?

Jack: 8:00

It's common, because it's simple and effective. So you can get the tools necessary to do a denial of service attack, in a matter of hours, you can be all ready to go, you can use your own systems to do it, or you can rent systems, people have set up botnets and different things. And then they rent out those botnets for whatever people want to do with them. And one of the most common things is to launch a denial of service attack. So that means there's 1000s of computers all trying to connect to your system at once Sending big payloads and packets. Yeah, it's common just because it's easy to do and effective because you want your servers to be open so that the whole world can download your podcasts or your or your content. So you don't want to restrict it. And here, you've got 1000s of servers all trying to download stuff all at the same time. And it's what is what your systems have been built to do is to accept this kind of thing. So it's, it's really difficult to defend against because you you want that door to be open. And what they're doing is just kind of blocking that doorway. So yeah, it's it's common, it's it happens quite frequently. It's part of doing business online.

John: 9:04

This is not a 15 year old programmer, or kid who's just hacking away.

Travis: 9:09

That's john Buzzsprout, VP of programming.

John: 9:12

This is someone who actually, I believe, understood the podcast industry, and understood user agents in traffic behavior. So we believe that sometimes he was masking himself as Apple podcasts and going after feeds, which are also requested barely often, and they're getting cycled. So the IP addresses and the URLs that they're attacking, constantly change. You're looking for patterns of behavior, and he's constantly seeing that you are stopping him. So he's changing his pattern.

Travis: 9:40

So in a nutshell, a distributed denial of service attack is when someone sends your servers, your network, more traffic, more page requests than you can reasonably handle. And you may be wondering, Well, isn't that something that Buzzsprout should be prepared for? Like isn't that something that we gameplan for getting These spikes. Sure, but it's not quite that simple.

Jack: 10:04

People might be mad at you for being so easily knocked over. But it's a weird thing where you could drive your car through a restaurant. And it's like, well, you know, how much do you blame the restaurant for easily being shut down because a car got driven through it. And it's one of those situations where you can't really expect like the entire massive amount of load, because every day you're taking inventory on how much load or servers your servers have, and how much do we need for the future. And let's build out for that. And you can even expect some spikes because there's some very exciting episodes that you know, your top shows or have put out or something, right, so you can kind of expect, alright, let's let's prepare for twice as much traffic or even three times as much or four times as much, but you don't expect for 30 times as much, or 50 times as much. And that's something that is really difficult to defend against, because it's such a massive influx of traffic that it's just very difficult to prepare for, and plan for and reduce in the moment as well.

Travis: 11:01

What's the motivation here? Like, why does this guy want to shut down Buzzsprout? Well, we checked Twitter and found out

Tom: 11:10

and somebody saw that there was a message, somebody had sent a request by Twitter asking for money to stop the attack. We already knew at that point that we were being attacked. We just didn't know what the reason there's, there's various reasons why somebody might do an attack like that. We didn't know it was going to be extortion.

Travis: 11:27

That raises an interesting question, do you pay him to go away? Because at this point, Buzzsprout had been down for a couple of hours, and are podcasters who are trying to upload episodes, look at their statistics, share their podcasts with their listeners, were unable to do so. So what do you do at that point? What do you do when you know that at least hypothetically, you can pay this person and make it go away?

Tom: 11:57

We had posted immediately, we posted on Twitter, immediately posted on Facebook, we wanted everybody to know that we're aware there's an issue, but then we find out there's a ransom demand, you can make this whole thing go away. So we know that our customers are being hurt as a result of this attack. And we know, or at least, this person is saying that if we send him money, it can stop.

Kevin: 12:15

You know, we're aware that these things happen. And we're aware that people try to extort money from businesses, especially if they can exert some sort of control over them. And so from the very beginning, it was one of the first things Tom said one of the first messages he sent right after he let us know that we were being attacked was I think he sent a ping that said, we don't negotiate with terrorists. And so I realized, okay, this is what we're facing, we cannot give in to criminals who are trying to extort money out of businesses, because it's just going to further proliferate criminal activity.

Travis: 12:43

So there's a couple of problems that arise when you decide to pay the ransom to make the attack stop. The first problem is, you have no idea if it's actually going to stop that it is going to take that ransom money you gave to him and use it to fund further attacks to then ask for more money, there's no guarantee that it will actually ever end. The second reason is because even if he stops attacking us, he will then take the money we gave to him and continue to negatively impact other businesses. So we knew that the right answer was not to give into this person's demands, not to give them any money, but instead to just focus on shoring up our infrastructure to fight back. But that doesn't mean it was an easy decision, even if it was really clear what the right answer was. Because we knew at the end of the day, it wasn't just us that were suffering. It was our podcasters. But there was one thing that really made a difference that really gave us the confidence to keep pushing forward, knowing that we made the right decision, even if it meant that this attack was going to go on for a little bit longer.

Tom: 13:49

I'm glad we landed where we did, and I'm man, I'm so encouraged by our customers, they were all behind it. As soon as we told him, we're like, Look, guys, here's a situation. There's a criminal and they're demanding payment. We don't think it's right to pay them. We think if we pay them, it's just going to result in more attacks either on us or other people in the podcasting industry or anybody else that if we pay them, they're just going to have a target on their back. And so we made the decision that we were going to not pay and we dragged it out as much as we could. And we use that time to go ahead and reinforce our defenses and get ready for another attack.

Travis: 14:23

That's right, you guys, our podcasters were with us 100% behind the decision to not given to this person's demands. Now, at this point in the story, the first attack had stopped, he was waiting to see if we were going to pay him and in the background. We were making decisions about how to shore up our infrastructure to get prepared for what we knew would be a second attack that could happen really, at any moment. Now that we had a good look at the exact type of attack we were facing, we could gameplan and implement a surgical defense.

Tom: 15:00

The attack relented, we didn't know when it was going to start back up. But we also know that Monday is the busiest time for Buzzsprout. So Monday we have a ton of activity on our servers, not only are people uploading episodes, but Monday's are also when we send out a weekly email that tells people their stats from the previous week. And those stats, emails are really server intensive when they run and generate all those reports. And so we know, in the back of our minds, we know that the worst possible case is for the attack to resume on Monday, and we have to just prepare for it. And so that's why we worked through the evening, implementing the best things that we could find in terms of preparing for that attack

Kevin: 15:45

through some communication that they were pushing our way, we had good reason to believe that they'd be back, we didn't know exactly when they gave some clues unintentionally, and some of their communications about what part of the world that they might be in. And so we thought that they might be turning in for the night. And so we figured we would have an opportunity to fortify a little bit before they woke up, or we're waiting for us to wake up and resume the attack. And so we had an idea that we might have a certain amount of time. And so Tom and his team, were putting a plan together of what measures can we put in place in the time that we have to be able to defend ourselves against the next attack,

Bryan: 16:17

it was only a matter of time before he resumed the attack, it could have been at any moment. And we didn't know how long we had. And so we immediately started working to protect our infrastructure as best we could, to put us in a better position where we wouldn't be affected by some of the worst aspects of the attack and allow us to actively fight the attack. The next time it occurred more effectively.

Travis: 16:39

Everything is up and running. Our customers are happy, but they don't know that the quarter barbarians is out there just waiting to attack. So they're like, oh, everything's fixed. But we know it's only fixed until the next attack. Now while the operations team and the support team were working diligently to put things in place to help us be prepared for the next attack, Kevin and Alban were deciding how to best communicate with our customers. Because even though we had a couple of 1000 reach out to us in support, not everyone was aware of what was going on. The question on everyone's mind is, well, how long is this going to last? And when is the site going to be back up?

Kevin: 17:17

And that's not a question that we could offer an answer to. But what we could do is the next best thing is give them continuous updates. So we didn't want to push an update, like once an hour. Now we want to be updating them like every 10 or 15 minutes. And anytime somebody asked a question on a social network, we want to try to be as real time in our response as possible.

Alban: 17:34

Our company culture has always been very open, telling everything that we're explaining, we actually assume everyone will understand us the way that we mean things we understood. And there's positive intent, we think we've built up a lot of trust over the years. So there wasn't ever really a question not to send it, I think this is really true. People want to know that you actually are going to tell them what's happening, and you're honest with them and treat them like adults. If you treat everybody like you can't handle the truth, you don't really need to know you're not going to take this the right way you're going to use this against me. If you act like that with everyone, they know that you're not telling them the truth, you're keeping a lot back. And when you do that they move from this posture of I'm giving you the benefit of the doubt, to this posture of I know you're trying to pull the wool over my eyes. And now I have to become a personal investigative journalist to figure out why my podcast is broken. And so I think by going a bit more on the openness side, what we end up doing is we end up getting a lot of good grace, maybe I mean, we had so many 1000s of people reaching out on Twitter, on Facebook, personal emails that I still haven't gotten to now 1000s of emails that went into support, all of them saying like, I can't believe this guy is attacking and extorting you, we're behind you 100%. Even if my show is down for two days, we're still behind you.

Travis: 18:58

So we had informed everyone that had a Buzzsprout podcast, what was going on, we were doing everything that we could to shore up our infrastructure to get ready for a second attack. And sure enough, Monday morning, he was back.

Alban: 19:10

There were a lot of rough patches night if we were never down, but there were a lot of like people running into bugs. And by 9am, the guy was back with bigger threats than ever. And he was able to deliver on them, at least in part, because Buzzsprout went back down about 9am.

Tom: 19:27

My thoughts Monday when he attack resumed was how long ow long can this person afford to attack us? Is it going to be an hour, two hours, four hours, ix hou

Travis: 19:38

so the attacker was back in full force. But even though he took down our site temporarily, we were much better prepared. The second time around

Tom: 19:46

everything that we had done Sunday night came to bear on the attack that came on Monday, we were able to use everything that we did Sunday night to prepare to help mitigate the attack on Monday, and then around 2pm we had effectively knocked down enough of the traffic that the extortionist didn't see as worthwhile anymore. So you can see the attack finally subsided. And we found out later that they moved on to some other podcasting companies after they stopped the attack on us

Bryan: 20:14

around one to 2pm on Monday, and we were working with a cloud provider to actively fight the attack. And they were able to identify an aspect of the attack that allowed us to surgically block the DDoS attack while allowing as much legitimate traffic through as possible. And pretty much as soon as they put that block in place, the attacker gave up

Travis: 20:37

after the attacker finally gave up, we were able to quickly get everything back online and get Buzzsprout fully functional again. And while we wouldn't necessarily have called it a fun experience, there were certainly some silver linings that are invaluable. Now, looking back, the thing that really stuck out to Tom, the co founder of Buzzsprout, was how the team pulled together,

Tom: 21:03

I was so encouraged, going into that battle that I wasn't alone, it wasn't just me and the ops team and the support team, everybody, we were all there for the fight. Everybody on our support team, all the developers, everyone was answering questions posting on Twitter, interacting on Facebook, on the code and the ops side, we were doing everything we could on the back end, to squash as many fake requests as we could to be able to keep things running, helping people, people who had critical episodes that they needed to upload, we were working with them to get those episodes uploaded and posted. So it was it was a lot different, it was much more encouraging. On Monday, I felt like there was an army of us fighting back. It's still a lot of barbarians that were fighting. But I felt like I had a whole army with us and our whole our whole team and all of our customers all of our podcasters. man it was it was encouraging was painful, but it was encouraging.

Travis: 21:59

Another really encouragi g thing that happened during thi period was that severa of our competitors, t least competitors on paper, reached out and offered their su port to help us fig

Kevin: 22:10

So Monday morning, while we were in the throes of the attack of another podcast host Spreaker had been attacked by the same we assume the same criminal network within hours of them attacking us again. And so they were able to connect with us they reached out proactively and offered any support they could and and so some members of our technical team hopped on a call with their technical team and started exchanging emails and documents. And when you're fighting these attacks, there's, again, I can't get into the specifics of the tactics that we use to mitigate the attack. But the strategies that seemed most effective for them, they were able to share with us. And some of the code that they use, they were able to share with us. And so that was amazing to see, like a competitor in the space, come to our aid in a time of need. Like proactively, like that was absolutely amazing. And so I can't thank the spreaker team enough. And the attacks didn't stop with us like once we got on the other side of the attack, then pod bean was victimized and so was captivate and so we tried to repay that, like continued to pay that forward to those teams as

John: 23:10

Podbean, Spreaker and I ere talking today, as they w re dealing with the attack things that they could do to hel mitigate it, Mark and Kieran from Captivate are sharin their information. So we're aking all information, and we want to start a little GitHub repository. Because one thing as been great about this experi nce is while we're talkin together over something that's very stressful, very agoniz ng, and frustrating. The podcas community has been there so inc usive, they're so great. And ev n competing platforms are gettin together and helping each o her.

Travis: 23:45

But at the end of the day, the real MVP was you guys,

Alban: 23:49

the amount of like, understanding and kindness and good wishes that people sent our way was unbelievable. 1000s of people were reaching out saying you're doing the right thing. We're behind you. I had somebody write us and say I'm on the free plan. But I'm telling you, as soon as I can log in, I'm upgrading, and you have a customer for life. And I'm like, you can't even log into the website right now. Like, I don't know what we've done to deserve these kind of customers.

Priscilla: 24:17

I just was blown away by our community of podcasters. I feel like, you know, right in the beginning, obviously, there is a lot of worry. And there's a lot of unknown and especially if your podcast is possibly at risk, especially in the beginning, when you just don't know what's going on, that frustration is totally warranted. And I feel like people would write in and be like, what's going on, and we would tell them, and they would write back the kindness things and be like, Oh, I'm seeing this on Twitter. Oh my goodness. Don't even worry about it. Thank you so much. You guys are doing a great job. Keep up the good work. And I you know, as you're sitting there in the support inbox, going email, email, email, email without looking up and just typing as fast as your fingers can type and then you get an email from someone who's like, Hey, I just wanted to shoot you an email and let you know that we're rooting for you. And you got this and thank you for everything. Or even when you get an email that says, hey, I have a question. But I know you're in the middle of stuff. So don't prioritize this. That kind of stuff from our people. It felt so good in the moment. And even now looking back on it, I'm like, oh, my goodness, it just feels so good. And then you just go, how did how did we luck out with all of these people in Buzzsprout? I just don't I don't get it. What did we do to deserve all these people? I don't, I don't know.

Travis: 25:32

So what happens next? What is Buzzsprout doing to make sure that if something like this happens in the future, that we are more prepared than we were the first time,

Kevin: 25:42

this is the first time in the 11 plus years that we've been operating Buzzsprout that we were attacked with something this sophisticated this severe for this amount of time. And so we've had, the team has done an amazing job of putting operating procedures in place for things like this. This is a trial by fire, right? Like, you can only do so much to simulate the scenario until a scenario hits. And so what I was thrilled about, although I didn't realize it in the moment, of course, because you're scrambling, like nobody panicked. everybody understood what was happening, and what each of our individual jobs were in that moment. Again, it started on a Sunday. And there wasn't any like nobody, I don't think it had the thought of well, it's Sunday, like, How bad is this? No. I mean, it's like our customers need us. And our team needs us. And this is I wouldn't use the word fun. But it's an opportunity to rally with your team and support customers who have supported us for years. And I hope that we've done a good job over those years of building that trust bank with them that when they realized that they went to log in on Sunday, and things weren't working the way that they would, that they trusted, that we were on it. And then we were going to fight hard. Regardless of what was coming at us, we're going to try to do the right thing. And we're going to be upfront and transparent with them and do everything we could and drop everything to support our customers.

Tom: 26:59

So moving forward, we have a action plan, we've experienced it, we've gone through it, it's it's something, it could happen to anyone at any time. So we've always known that we could be the target of an attack like this. So in some ways, it's a little bit of a relief that we've been through it because we've always known it could happen. Well, we've been through it, we've learned a ton, we've got a great infrastructure in place to be able to mitigate it in the future. I can't guarantee that we won't have issues that we couldn't be taken down for a period of time as a result of it. But I can say that we have the best team with the most experience in how to mitigate this when it comes to Buzzsprout podcasting,

Travis: 27:41

being the victim of a cyber attack, like what we experienced this past week is never something that you hope happens to you. But at Buzzsprout we're big believers in turning negatives into positives, which is why we're going to do something special with the money that the hacker had hoped that we would be sending to him.

Kevin: 28:00

So now that we've had a couple days to process and recover, like what we want to do on this side of that experience. And so we've done two things. One is we want to put our money where our mouth is, and like we have said that there's the decision to not pay the ransom had nothing to do with the money. And so like, let's make that real. And so there are people in Texas right now, there's people all over the world. But there's people right in our backyard in Texas, who are just you know, they're suffering from this weather crisis that they were hit with last week, there are people who are still without power, the food chain has been disrupted, they don't have clean water. And so we've taken the money that the criminals demanded as ransom holding our business captive. And we've donated that to two charities in Texas. One is called feeding Texas and the other one is called front steps. And the other thing that we've done is that we realized that DDoS attacks require a lot of energy to run. I mean, you're spinning up 1000s of computers for hours doing this one useless task, and there is a negative effect to that. And so we've also made a donation to project Vesta that will more than offset the carbon footprint of the attack. So the end result of the attack is that podcast episodes were delayed through the Buzzsprout platform for a few hours. But people in need are now going to get food, water and shelter and the global environment is going to be healthier overall.

Travis: 29:14

Thanks Kevin for helping me end this episode. on a high note. If you'd like to read the full DDoS technical post mortem, you can click on the link in the show notes to read it on our blog. And we'll also leave links to the three charities that we donated to as a result of this series of events in the show notes as well. If you would like to make a donation there to special thanks to everyone on the Buzzsprout team that gave their time to share their stories of what happened behind the scenes. Jordan host of th Dreamful podcast for our epic pisode artwork. And thank you or sticking with us through hick and thin. We're so excited o be able to continue to serve ou and help you with your odcast. Well that's it for this eek. Thanks for listening. And s always keep podcasting