A 'Worst Nightmare' Cyberattack: The Untold Story Of The SolarWinds Hack
MARY LOUISE KELLY, HOST:
Late last year, a long-simmering cyber conflict between the United States and Russia broke out into the open.
(SOUNDBITE OF MONTAGE)
UNIDENTIFIED REPORTER #1: Authorities say the Russians targeted some of America's most sensitive and important computer systems.
UNIDENTIFIED REPORTER #2: Hackers breached SolarWinds to infect at least seven U.S. government agencies.
UNIDENTIFIED REPORTER #3: Government agencies were caught off guard by an unprecedented attack.
KELLY: That's right. Hackers targeted SolarWinds, a Texas software company you'd probably never heard of. But they make a piece of software that is indispensable to thousands of IT departments, including many of the nation's biggest companies and U.S. government institutions.
ARI SHAPIRO, HOST:
Russian hackers were able to sneak malicious code into this widely used software, which gave them a vast list of potential targets. NPR investigative correspondent Dina Temple-Raston got an exclusive look inside the sophisticated attack. Here's how it unfolded.
DINA TEMPLE-RASTON, BYLINE: The routine software update may be one of the most familiar and least understood parts of our digital lives, but this last spring it became the vehicle to launch an epic cyberattack. Hackers used a software update to slip into some of this country's most sensitive computer networks, which allowed them to take aim not just at the economy but at our national security, too.
ALEX STAMOS: I mean, it's one of the most impressive and effective cyberespionage campaigns of all time.
TEMPLE-RASTON: Alex Stamos is the director of the Internet Observatory at Stanford University and the former head of security at Facebook.
STAMOS: They were able to get access to some very sensitive companies and government organizations without getting caught for quite a while.
TEMPLE-RASTON: And when he says quite a while, he means almost a year. And for all that time, the hackers roamed around the networks of companies like Microsoft, Intel and Cisco and government agencies like the Treasury, the Department of Energy and the Pentagon. And the hackers might have kept skulking around those networks were it not for the vigilance of one company - FireEye, which happened to be one of SolarWinds' customers.
STAMOS: That's right. I mean, this is - we only know about any of this because they made the mistake of attacking FireEye, which is, like, a professional incident response company.
KEVIN MANDIA: You know, us doing investigations is kind of like The Beatles entering a battle of the bands. They're going to do really well in a battle of the bands. We're going to do well investigating.
TEMPLE-RASTON: That's Kevin Mandia, FireEye CEO.
You're former Air Force intelligence. Is that right?
MANDIA: No, I was in what's called the Air Force Office of Special Investigations. I spent from 1996 1998 responding to what I would equate to as the Russian Foreign Intelligence Service.
TEMPLE-RASTON: So late last year, when he saw some suspicious activity in his company's network, it felt familiar. The first clue that something was wrong came when the FireEye security team noticed something unusual - someone trying to register a second phone onto the company's network. So they called.
MANDIA: And the gentleman said, no, I did not register that phone. So who did?
TEMPLE-RASTON: Whoever it was, they were roaming around the network, looking like an employee.
MANDIA: It just felt like the breach that I was always worried about.
TEMPLE-RASTON: The FireEye security team started tearing apart their servers, looking for the intruder, and it took them weeks to trace the problem back to that SolarWinds software update. And once they were certain that's what it was, they wrote a report and sent it to the head of cybersecurity at SolarWinds, Tim Brown.
TIM BROWN: And the report was detailed. The report said, we decompiled your code. We see this malicious code here. We see proof that, yes, we had shipped things that had malicious content inside of it.
TEMPLE-RASTON: And what was going through your head?
BROWN: You know, it's kind of a nightmare idea for any security person. You know, we deal with little, tiny incidents often. But this had the potential to affect thousands of customers, right? This had the potential to do a great deal of damage.
TEMPLE-RASTON: Brown went home, packed a bag and was prepared to stay at the office for the rest of the week.
BROWN: I would say Sunday, Monday, we knew that the attack itself and the code that was inserted itself was pretty purposeful. So we quickly understood that the attacker was on a mission.
TEMPLE-RASTON: This wasn't just a hacker in a hoodie. This looked like a nation-state. So they brought in someone who knew how to deal with these kinds of attacks.
ADAM MEYERS: Hi - Adam Meyers, and I run intel at CrowdStrike.
TEMPLE-RASTON: CrowdStrike is a cyber investigation company, and Meyers has helped them unwind some famous hacks - Sony in 2015, the Democratic National Committee a year later. So he knew a nation-state attack when he saw one, and this looked like one of those.
MEYERS: I started rolling up my sleeves and started actually looking at the code. And the backdoor itself was 3,500 lines.
TEMPLE-RASTON: A backdoor is a little portal into the software.
MEYERS: And, you know, there was quite a bit of things that it did. And the tradecraft of this threat actor was phenomenal.
TEMPLE-RASTON: That little blob of code was the tiny, beating part of the attack buried deep inside the SolarWinds software.
MEYERS: We're hoping it's going to have, you know, variable names or maybe some comments in Cyrillic or Mandarin, give us some clue that - who wrote this thing.
TEMPLE-RASTON: But as the CrowdStrike program kept chewing its way through the code, Meyers' heart began to sink. The crime scene was a bust.
MEYERS: They washed the code. They cleaned it of any human artifact or tool mark. And that was kind of mind-blowing that this threat actor had the wherewithal to just hide anything that a human might have inadvertently left behind as a clue.
TEMPLE-RASTON: Experts like Meyers can often find gossamer connections inside the code. Some hackers have little tics. Others copy and paste from previous hacks. It's like a nerdy calling card. And nation-states typically have teams whose whole job is to try to break into other countries' systems. This happens so much, there's actually a convention to name them.
MEYERS: So if I say it's bear, it's Russia. If I say it's panda, it's China. North Korea is Chollima. You know, we always kind of use the official state animal. And I think when we looked, that was the official state animal of North Korea, which was just what we were hoping for - an imaginary flying horse.
TEMPLE-RASTON: To Meyers, SolarWinds felt like a bear operation, but he wasn't sure. He started looking for hints in the hack itself, which it turns out started earlier than anyone thought - all the way back to September 2019. That's when the hackers tried to insert a little snippet of code into the SolarWinds update to see if it would end up in finished software. It worked.
MEYERS: They modified the product. And so at this point, they know that they can pull off a supply chain attack. They know that they have that capability.
TEMPLE-RASTON: After that initial success, the hackers did something they never do. They disappeared for five months. They returned in February 2020, armed with code that allowed them to build their own SolarWinds update. But their version had a little addition - code that gave them that backdoor, that secret portal into SolarWinds' customer networks. Then came the trick. At the last second, they swapped their version in.
MEYERS: Right. Like, I - when I was growing up, you used to have to check your Halloween candy because somebody might have put a razor blade in your Reese's peanut butter cup, right? But imagine those Reese's peanut butter cups going into the package, and just before the machine comes down and seals the package, some other thing comes in and slides a razor blade into your Reese's peanut butter cup.
TEMPLE-RASTON: The package gets sealed. It's put in a box and goes out to the store and into plastic pumpkins everywhere. It wasn't complicated so much as crafty. Here's what really worried Meyers, though. This bait and switch could have worked on anyone.
MEYERS: It could have been reconfigured for any number of software products. We realize that this could be elsewhere.
TEMPLE-RASTON: To this day, no one knows where the hackers have been or exactly what they have done except, of course, for the hackers themselves. SolarWinds is still investigating. Typically, no one talks about a hack. But the CEO of SolarWinds, a man named Sudhakar Ramakrishna, thought he needed to.
Why have you been so open about all of this? It's very unusual for a company to be this open.
SUDHAKAR RAMAKRISHNA: You forget about competition and competitors. And in that context, the right thing to do is to report. The right thing to do is to give them the ability to fix those issues and protect their customers, right? And we can compete on value. We can compete on price. We can compete on other factors. But you don't compete on that.
TEMPLE-RASTON: Ramakrishna wasn't running SolarWinds when the hack happened. He was hired just before the breach was discovered and stepped into the top job just as the full extent of the attack became clear. So when he published a blog post laying out an 11-point security plan, it was seen in two ways.
IAN THORNTON-TRUMP: One interpretation of that could be, we learned a valuable lesson from what the hack was. The other interpretation could be is that there were at least 11 material deficiencies in the actual security we had.
TEMPLE-RASTON: Right.
THORNTON-TRUMP: I see the 11-point plan as actually an admission that things were not good in the security house.
TEMPLE-RASTON: Ian Thornton-Trump used to work at SolarWinds. He was on the company's security team until 2017. He says he left because SolarWinds refused to spend enough money on its own security. Now he's chief of cybersecurity at a threat intelligence company, Cyjax, and he says he wished he'd done more to convince people at SolarWinds that a big hack was coming.
THORNTON-TRUMP: There's an emotional component of me that is just super-sad about this. Something bad was going to happen. And, you know, we always say in cybersecurity, it's when, not if, right? It's when you're going to get data breached, not if you're going to get data breached. And this was a whopper.
TEMPLE-RASTON: But you have to wonder, of all the software companies to target with this huge, complicated attack, why did the hackers choose SolarWinds?
RAMAKRISHNA: I've thought about this quite a bit as to why us. Why not some somebody else?
TEMPLE-RASTON: And Ramakrishna has come to the conclusion that the hackers chose SolarWinds because they thought they would be able to cast a wide net and possibly hack 18,000 customers with just one sophisticated attack. This wasn't just a hack, though. This was really about espionage. The White House thinks Russia was behind this and specifically that it was a group linked to Russian intelligence - APT29, known as Cozy Bear. Alex Stamos of Stanford says this was a high-end job. The hackers did their homework. They spent a lot of time studying the adversary.
STAMOS: They demonstrated not just technical acumen, but the way they did this demonstrated that they understand how tech companies operate, how software companies operate.
TEMPLE-RASTON: And that's the other thing that makes this hack different. The attack on SolarWinds was a bit of a bank shot. A nation-state wanted intelligence about the U.S. and hacked a private company to get it. FireEye's Kevin Mandia says that's what's new.
MANDIA: We would have landed at this day sooner or later. But to see it happen, that's where, you know, you have a little bit of shock and surprise. OK, it's here now.
TEMPLE-RASTON: And since it is here, new ideas may be required. For example, some people are suggesting there be a more formal way to investigate big cyberattacks. Stanford's Stamos likes the idea of starting something like the National Transportation Safety Board but for cyber instead. He thinks we should be looking at cyberattacks as carefully as we look at plane crashes.
STAMOS: When the Boeing 737 Maxes started crashing, there was a government agency whose entire job it was to gather up the facts of all of those different crashes and then to come up with a theory of what needed to be fixed and then oversaw the fixes that went into that.
TEMPLE-RASTON: And Adam Meyers, the man who found that little blob of code inside the SolarWinds software - he's busy as ever fending off other attacks.
MEYERS: This was an intelligence collection operation meant to steal information. And it's not the last time that's going to happen, right? This is going to happen every day. And, you know, I can't tell you how many investigations I've worked on since. It gives you a sense that this is continuing to happen. And I think there's a lot that we all need to do to work together to stop this from happening.
TEMPLE-RASTON: Dina Temple-Raston, NPR News.
(SOUNDBITE OF DECEPTIKON'S "THE WAY OF THE SAMURAI")
Copyright © 2021 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.
NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.