solarwinds data breach case study

Essential Guide

• Risk & Repeat: Breaking down SEC charges against SolarWinds
• SolarWinds fires back at SEC over fraud charges

SEC charges SolarWinds for security failures, fraud

SolarWinds hackers still active, using new techniques

• SolarWinds attacks come into focus
• SolarWinds warns of zero-day vulnerability under attack
• Autodesk targeted in SolarWinds hack
• Malwarebytes breached by SolarWinds hackers
• Mimecast certificate compromised by SolarWinds hackers
• SolarWinds Office 365 environment compromised
• SolarWinds chases multiple leads in breach investigation
• SolarWinds backdoor infected tech giants, impact unclear

SolarWinds hackers Nobelium spotted using a new backdoor

• CISOs on alert following SEC charges against SolarWinds
• SolarWinds CEO on life after Sunburst
• SolarWinds response team recounts early days of attack
• Senate hearing: SolarWinds evidence points to Russia
• SolarWinds hackers stole Mimecast source code
• SolarWinds backdoor used in nation-state cyber attacks
• FireEye red team tools stolen in cyber attack
• SolarWinds backdoor shakes infosec industry
• SolarWinds breach highlights dangers of supply chain attacks
• SolarWinds attack almost certainly work of Russian spooks
• SolarWinds confirms supply chain attack began in 2019
• How SolarWinds attack will change CISOs' priorities
• SolarWinds hackers attacking more IT supply chain targets
• White House: 100 companies compromised in SolarWinds hack
• SolarWinds puts national cybersecurity strategy on display
• Senate hearing raises questions about SolarWinds backdoors
• Microsoft, SolarWinds in dispute over nation-state attacks
• FireEye releases new tool to fight SolarWinds hackers
• Microsoft, FireEye deliver kill switch for SolarWinds backdoor
• SolarWinds struggles with response to supply chain attack
• Biden picks cyber veteran to reinvigorate security response
• SolarWinds attack stumps SecOps pros
• SolarWinds CEO sets out rescue plan

SolarWinds hack explained: Everything you need to know

Hackers targeted solarwinds by deploying malicious code into its orion it monitoring and management software used by thousands of enterprises and government agencies worldwide..

• Saheed Oladimeji, Sean Michael Kerner

2020 was a roller coaster of major, world-shaking events. We all couldn't wait for the year to end. But just as 2020 was about to close, it pulled another fast one on us: the SolarWinds hack, one of the biggest cybersecurity breaches of the 21st century.

The SolarWinds hack was a major event not because a single company was breached, but because it triggered a much larger supply chain incident that affected thousands of organizations, including the U.S. government.

What is SolarWinds?

SolarWinds is a major software company based in Tulsa, Okla., which provides system management tools for network and infrastructure monitoring, and other technical services to hundreds of thousands of organizations around the world. Among the company's products is an IT performance monitoring system called Orion.

As an IT monitoring system, SolarWinds Orion has privileged access to IT systems to obtain log and system performance data. It is that privileged position and its wide deployment that made SolarWinds a lucrative and attractive target.

What is the SolarWinds hack?

The SolarWinds hack is the commonly used term to refer to the supply chain breach that involved the SolarWinds Orion system.

In this hack, suspected nation-state hackers that have been identified as a group known as Nobelium by Microsoft -- and often simply referred to as the SolarWinds Hackers by other researchers -- gained access to the networks, systems and data of thousands of SolarWinds customers. The breadth of the hack is unprecedented and one of the largest, if not the largest, of its kind ever recorded.

More than 30,000 public and private organizations -- including local, state and federal agencies -- use the Orion network management system to manage their IT resources. As a result, the hack compromised the data, networks and systems of thousands when SolarWinds inadvertently delivered the backdoor malware as an update to the Orion software.

SolarWinds customers weren't the only ones affected. Because the hack exposed the inner workings of Orion users, the hackers could potentially gain access to the data and networks of their customers and partners as well -- enabling affected victims to grow exponentially from there.

How did the SolarWinds hack happen?

The hackers used a method known as a supply chain attack to insert malicious code into the Orion system. A supply chain attack works by targeting a third party with access to an organization's systems rather than trying to hack the networks directly.

The third-party software, in this case the SolarWinds Orion Platform, creates a backdoor through which hackers can access and impersonate users and accounts of victim organizations. The malware could also access system files and blend in with legitimate SolarWinds activity without detection, even by antivirus software.

SolarWinds was a perfect target for this kind of supply chain attack . Because their Orion software is used by many multinational companies and government agencies, all the hackers had to do was install the malicious code into a new batch of software distributed by SolarWinds as an update or patch.

The SolarWinds hack timeline

Here is a timeline of the SolarWinds hack:

• September 2019. Threat actors gain unauthorized access to SolarWinds network
• October 2019. Threat actors test initial code injection into Orion
• Feb. 20, 2020. Malicious code known as Sunburst injected into Orion
• March 26, 2020. SolarWinds unknowingly starts sending out Orion software updates with hacked code

According to a U.S. Department of Homeland Security advisory , the affected versions of SolarWinds Orion are versions are 2019.4 through 2020.2.1 HF1.

More than 18,000 SolarWinds customers installed the malicious updates, with the malware spreading undetected. Through this code, hackers accessed SolarWinds's customer information technology systems, which they could then use to install even more malware to spy on other companies and organizations.

Who was affected?

According to reports, the malware affected many companies and organizations . Even government departments such as Homeland Security, State, Commerce and Treasury were affected, as there was evidence that emails were missing from their systems. Private companies such as FireEye, Microsoft, Intel, Cisco and Deloitte also suffered from this attack.

The breach was first detected by cybersecurity company FireEye . The company confirmed they had been infected with the malware when they saw the infection in customer systems. FireEye labeled the SolarWinds hack "UNC2452" and identified the backdoor used to gain access to its systems through SolarWinds as "Sunburst."

Microsoft also confirmed that it found signs of the malware in its systems, as the breach was affecting its customers as well. Reports indicated Microsoft's own systems were being used to further the hacking attack, but Microsoft denied this claim to news agencies. Later, the company worked with FireEye and GoDaddy to block and isolate versions of Orion known to contain the malware to cut off hackers from customers' systems.

They did so by turning the domain used by the backdoor malware used in Orion as part of the SolarWinds hack into a kill switch . The kill switch here served as a mechanism to prevent Sunburst from operating further.

Nonetheless, even with the kill switch in place, the hack is still ongoing. Investigators have a lot of data to look through, as many companies using the Orion software aren't yet sure if they are free from the backdoor malware. It will take a long time before the full impact of the hack is known.

Why did it take so long to detect the SolarWinds attack?

With attackers having first gained access to the SolarWinds systems in September 2019 and the attack not being publicly discovered or reported until December 2020, attackers may well have had 14 or more months of unfettered access.

The time it takes between when an attacker is able to gain access and the time an attack is actually discovered is often referred to as dwell time. According to a report released in January 2020 by security firm CrowdStrike, the average dwell time in 2019 was 95 days. Given that it took well over a year from the time the attackers first entered the SolarWinds network until the breach was discovered, the dwell time in the attack exceeded the average.

The question of why it took so long to detect the SolarWinds attack has a lot to do with the sophistication of the Sunburst code and the hackers that executed the attack.

"Analysis suggests that by managing the intrusion through multiple servers based in the United States and mimicking legitimate network traffic, the attackers were able to circumvent threat detection techniques employed by both SolarWinds, other private companies, and the federal government," SolarWinds said in its analysis of the attack .

FireEye, which was the first firm to publicly report the attack, conducted its own analysis of the SolarWinds attack. In its report, FireEye described in detail the complex series of action that the attackers took to mask their tracks. Even before Sunburst attempts to connect out to its command-and-control server , the malware executes a number of checks to make sure no antimalware or forensic analysis tools are running.

What was the purpose of the hack?

The purpose of the hack remains largely unknown. Still, there are many reasons hackers would want to get into an organization's system, including having access to future product plans or employee and customer information held for ransom. It is also not yet clear what information, if any, hackers stole from government agencies. But the level of access appears to be deep and broad.

There are speculations that many enterprises might be collateral damage, as the main focus of the attack was government agencies that make use of the SolarWinds IT management systems.

Who was responsible for the hack?

Federal investigators and cybersecurity agents believe a Russian espionage operation -- mostly likely Russia's Foreign Intelligence Service -- is behind the SolarWinds attack .

The Russian government has denied any involvement in the attack, releasing a statement that said, "Malicious activities in the information space contradicts the principles of the Russian foreign policy, national interests and understanding of interstate relations." They also added that "Russia does not conduct offensive operations in the cyber domain."

Not the first time

The SolarWinds hack is the latest in a series of recent attacks blamed on Russian operatives. It is believed a Russian group known as Cozy Bear was behind attacks targeting email systems at the White House and the State Department in 2014. The group has also been mentioned as responsible for the infiltration of the Democratic National Committee's email systems and members of Hillary Clinton's presidential campaign in 2015 in the lead-up to the 2016 election, as well as further breaches around the 2018 midterm elections.

Contrary to experts in his administration, then-President Donald Trump hinted at around the time of the discovery of the SolarWinds hack that Chinese hackers might be behind the cybersecurity attack. However, he did not present any evidence to back up his claim.

Shortly after his inauguration, President Joe Biden vowed that his administration intended to hold Russia accountable, through the launch of a full-scale intelligence assessment and review of the SolarWinds attack and those behind it. The president also created the position of deputy national security adviser for cybersecurity as part of the National Security Council. The role, held by veteran intelligence operative Anne Neuberger, is part of an overall bid by the Biden administration to refresh the federal government's approach to cybersecurity and better respond to nation-state actors.

Naming the attack: What is Solorigate, Sunburst and Nobelium?

The SolarWinds attack has a number of different names associated with it. While the attack is often referred to simply as the SolarWinds attack, that isn't the only name to know.

• Sunburst . This is the name of the actual malicious code injection that was planted by hackers into the SolarWinds Orion IT monitoring system code. Both SolarWinds and CrowdStrike generally refer to the attack as Sunburst.
• Solorigate . Microsoft initially dubbed the actual threat actor group behind the SolarWinds attack as Solorigate. It's a name that stuck and was adopted by other researchers as well as media.
• Nobelium . In March 2021, Microsoft decided that the primary designation for the threat actor behind the SolarWinds attack should actually be Nobelium -- the idea being that the group is active against multiple victims -- not just SolarWinds -- and uses more malware than just Sunburst.

The China connection to the SolarWinds attack

While it is suspected that the initial Sunburst code and the attack against SolarWinds and its users came from a threat actor based in Russia, other nation-state threat actors have also used SolarWinds in attacks.

According to a Reuters report , suspected nation-state hackers based in China exploited SolarWinds during the same period of time the Sunburst attack occurred. The suspected China-based threat actors targeted the National Finance Center, which is a payroll agency within the U.S. Department of Agriculture.

It is suspected that the China-based attackers did not use Sunburst, but rather a different malware that SolarWinds identifies as Supernova .

Why is the SolarWinds hack important?

The SolarWinds supply chain attack is a global hack, as threat actors turned the Orion software into a weapon gaining access to several government systems and thousands of private systems around the world. Due to the nature of the software -- and by extension the Sunburst malware -- having access to entire networks, many government and enterprise networks and systems face the risk of significant breaches.

The hack could also be the catalyst for rapid, broad change in the cybersecurity industry . Many companies and government agencies are now in the process of devising new methods to react to these types of attacks before they happen. Governments and organizations are learning that it is not enough to build a firewall and hope it protects them. They have to actively seek out vulnerabilities in their systems, and either shore them up or turn them into traps against these types of attacks.

Since the hack was discovered, SolarWinds has recommended customers update their existing Orion platform. The company has released patches for the malware and other potential vulnerabilities discovered since the initial Orion attack. SolarWinds also recommended customers not able to update Orion isolate SolarWinds servers and/or change passwords for accounts that have access to those servers.

The greater White House cybersecurity focus will be crucial, some industry experts have said. But organizations should consider adopting modern software-as-a-service tools for monitoring and collaboration. While the cybersecurity industry has significantly advanced in the last decade, these kinds of attacks show that there is still a long way to go to get really secure systems.

The Nobelium group continues to attack targets

The suspected threat actor group behind the SolarWinds attack has remained active in 2021 and hasn't stopped at just targeting SolarWinds. On May 27, 2021, Microsoft reported that Nobelium, the group allegedly behind the SolarWinds attack, infiltrated software from email marketing service Constant Contact. According to Microsoft, Nobelium targeted approximately 3,000 email accounts at more than 150 different organizations.

The initial attack vector appears to be an account used by USAID. From that initial foothold, Nobelium was able to send out phishing emails in an attempt to get victims to click on a link that would deploy a backdoor Trojan designed to steal user information.

Podcast: SolarWinds attacks come into focus

Download this podcast

Need for software bill of materials highlighted in aftermath of attack

In the aftermath of the attack, the U.S. Cybersecurity and Infrastructure Security Agency issued guidance on software supply chain compromise mitigations. The guidance provides specific tactical recommendations on what organizations should look for to identify and remove potentially exploited components.

As it turned out, the SolarWinds incident was one of multiple attacks in 2020 and 2021 that highlighted risks with supply chain security. Incidents such as the Colonial Pipeline attack in May 2021 and the  Kaseya ransomware attack in July 2021 demonstrated how attackers were able to exploit vulnerabilities in components of the software supply chain to affect a wider group of vendors.

Modern software applications no longer rely on a monolithic stack of discrete software components. Developers now build applications out of many components that can come from many sources. Any one of the components that makes up an application could potentially represent a risk if there is an unpatched vulnerability. As such, it is critical for developers, organizations they work for and end users that consume applications be aware of all the different components that make up an application. It's an approach that is known as a software bill of materials (SBOM). An SBOM is like a "nutritional label” that is present on packaged food products, clearly showing consumers what's inside a product.

The need for SBOMs was mandated by an executive order issued in May 2021 by the Biden Administration. The executive order led to the National Telecommunications and Information Administration report released in July 2021 that provides guidance on SBOM best practices and minimum requirements. The executive orders also mandated that U.S. government agencies only work with software vendors that provide SBOMs.

"Those who operate software can use SBOMs to quickly and easily determine whether they are at potential risk of a newly discovered vulnerability," the Executive Order stated.

SEC legal action

In June 2023, the U.S. Securities and Exchange Commission (SEC) sent SolarWinds a Wells notice at the conclusion of their investigation. It informed former and current executives that the SEC intends to recommend civil enforcement action, alleging that SolarWinds broke federal security laws in public statements and internal controls related to the hack. For example, the company continued to distribute updates infected with the APT29 malware after the initial breach.

The Wells notice states that the SEC intends to bring legislation against SolarWinds but is not a formal charge. The purpose of a Wells notice is to give the recipient time to argue that the charges should not be laid.

SolarWinds CEO Sudhakar Ramakrishna will explore resolution with the SEC and maintains that SolarWinds responded appropriately to the attack. The company has maintained that the attack was unforeseeable, highly sophisticated and backed by a world power.

SolarWinds also settled a class action lawsuit October 2022, paying out $26 million to shareholders who maintained that SolarWinds neglected internal security preceding the breach and misled the public about its digital security.

In October 2023, the SEC sued SolarWinds and CISO Timothy Brown, stating the company concealed its cybersecurity vulnerabilities before it was attacked. This is the first time the SEC has sued the victim of a cyberattack. SolarWinds plans to fight the charges in court.

Lessons learned from high-profile data breaches

Related Resources

• Panda Adaptive Defense 360 Technologies –WatchGuard Technologies, Inc.
• Our Security, Delivered Your Way –WatchGuard Technologies, Inc.
• AWS Certified Security Specialty Exam Study Guide –TechTarget
• SASE: Improving Cloud and Work-From-Home Security –Cloud Gateway

Dig Deeper on Security

Judge tosses most of SEC's lawsuit against SolarWinds

Security Think Tank: Attackers leveraging the supply chain

Wavelength is the distance between identical points, or adjacent crests, in the adjacent cycles of a waveform signal propagated ...

A subnet, or subnetwork, is a segmented piece of a larger network. More specifically, subnets are a logical partition of an IP ...

Transmission Control Protocol (TCP) is a standard protocol on the internet that ensures the reliable transmission of data between...

Exposure management is a cybersecurity approach to protecting exploitable IT assets.

An intrusion detection system monitors (IDS) network traffic for suspicious activity and sends alerts when such activity is ...

A cyber attack is any malicious attempt to gain unauthorized access to a computer, computing system or computer network with the ...

Labor arbitrage is the practice of searching for and then using the lowest-cost workforce to produce products or goods.

Data privacy, also called information privacy, is an aspect of data protection that addresses the proper storage, access, ...

Product development -- also called new product management -- is a series of steps that includes the conceptualization, design, ...

Organizational network analysis (ONA) is a quantitative method for modeling and analyzing how communications, information, ...

HireVue is an enterprise video interviewing technology provider of a platform that lets recruiters and hiring managers screen ...

Human Resource Certification Institute (HRCI) is a U.S.-based credentialing organization offering certifications to HR ...

An abandoned call is a call or other type of contact initiated to a call center or contact center that is ended before any ...

An outbound call is one initiated by a contact center agent to prospective customers and focuses on sales, lead generation, ...

Lead-to-revenue management (L2RM) is a set of sales and marketing methods focusing on generating revenue throughout the customer ...

• Skip to main content
• Keyboard shortcuts for audio player

Untangling Disinformation

A 'worst nightmare' cyberattack: the untold story of the solarwinds hack.

Dina Temple-Raston

An NPR investigation into the SolarWinds attack reveals a hack unlike any other, launched by a sophisticated adversary intent on exploiting the soft underbelly of our digital lives. Zoë van Dijk for NPR hide caption

An NPR investigation into the SolarWinds attack reveals a hack unlike any other, launched by a sophisticated adversary intent on exploiting the soft underbelly of our digital lives.

"This release includes bug fixes, increased stability and performance improvements ."

The routine software update may be one of the most familiar and least understood parts of our digital lives. A pop-up window announces its arrival and all that is required of us is to plug everything in before bed. The next morning, rather like the shoemaker and the elves, our software is magically transformed.

Last spring, a Texas-based company called SolarWinds made one such software update available to its customers. It was supposed to provide the regular fare — bug fixes, performance enhancements — to the company's popular network management system, a software program called Orion that keeps a watchful eye on all the various components in a company's network. Customers simply had to log into the company's software development website, type a password and then wait for the update to land seamlessly onto their servers.

The routine update, it turns out, is no longer so routine.

Hackers believed to be directed by the Russian intelligence service, the SVR, used that routine software update to slip malicious code into Orion's software and then used it as a vehicle for a massive cyberattack against America.

"Eighteen thousand [customers] was our best estimate of who may have downloaded the code between March and June of 2020," Sudhakar Ramakrishna, SolarWinds president and CEO, told NPR. "If you then take 18,000 and start sifting through it, the actual number of impacted customers is far less. We don't know the exact numbers. We are still conducting the investigation."

On Thursday, the Biden administration announced a roster of tough sanctions against Russia as part of what it characterized as the "seen and unseen" response to the SolarWinds breach.

NPR's months-long examination of that landmark attack — based on interviews with dozens of players from company officials to victims to cyber forensics experts who investigated, and intelligence officials who are in the process of calibrating the Biden administration's response — reveals a hack unlike any other, launched by a sophisticated adversary who took aim at a soft underbelly of digital life: the routine software update.

By design, the hack appeared to work only under very specific circumstances. Its victims had to download the tainted update and then actually deploy it. That was the first condition. The second was that their compromised networks needed to be connected to the Internet, so the hackers could communicate with their servers.

For that reason, Ramakrishna figures the Russians successfully compromised about 100 companies and about a dozen government agencies. The companies included Microsoft, Intel and Cisco; the list of federal agencies so far includes the Treasury, Justice and Energy departments and the Pentagon.

SolarWinds CEO and President Sudhakar Ramakrishna inherited the attack. He was hired shortly before the breach was discovered and stepped into the job just as the full extent of the hack became clear. Demetrius Freeman/Pool/AFP via Getty Images hide caption

SolarWinds CEO and President Sudhakar Ramakrishna inherited the attack. He was hired shortly before the breach was discovered and stepped into the job just as the full extent of the hack became clear.

The hackers also found their way, rather embarrassingly, into the Cybersecurity and Infrastructure Security Agency, or CISA — the office at the Department of Homeland Security whose job it is to protect federal computer networks from cyberattacks.

The concern is that the same access that gives the Russians the ability to steal data could also allow them to alter or destroy it. "The speed with which an actor can move from espionage to degrading or disrupting a network is at the blink of an eye," one senior administration said during a background briefing from the White House on Thursday. "And a defender cannot move at that speed. And given the history of Russia's malicious activity in cyberspace and their reckless behavior in cyberspace, that was a key concern."

"The tradecraft was phenomenal"

Network monitoring software is a key part of the backroom operations we never see. Programs like Orion allow information technology departments to look on one screen and check their whole network: servers or firewalls, or that printer on the fifth floor that keeps going offline. By its very nature, it touches everything — which is why hacking it was genius.

"It's really your worst nightmare," Tim Brown, vice president of security at SolarWinds, said recently. "You feel a kind of horror. This had the potential to affect thousands of customers; this had the potential to do a great deal of harm."

When cybersecurity experts talk about harm, they're thinking about something like what happened in 2017, when the Russian military launched a ransomware attack known as NotPetya. It, too, began with tainted software, but in that case the hackers were bent on destruction. They planted ransomware that paralyzed multinational companies and permanently locked people around the world out of tens of thousands of computers. Even this much later, it is considered the most destructive and costly cyberattack in history.

Intelligence officials worry that SolarWinds might presage something on that scale. Certainly, the hackers had time to do damage. They roamed around American computer networks for nine months, and it is unclear whether they were just reading emails and doing the things spies typically do, or whether they were planting something more destructive for use in the future.

"When there's cyber-espionage conducted by nations, FireEye is on the target list," Kevin Mandia, CEO of the cybersecurity firm FireEye, told NPR, but he believes there are other less obvious targets that now might need more protecting. "I think utilities might be on that list. I think health care might be on that list. And you don't necessarily want to be on the list of fair game for the most capable offense to target you."

Kevin Mandia, CEO of the cybersecurity firm FireEye, said the Russians didn't just attack SolarWinds, they took aim at trust. Demetrius Freeman/Pool/Getty Images hide caption

Kevin Mandia, CEO of the cybersecurity firm FireEye, said the Russians didn't just attack SolarWinds, they took aim at trust.

The SolarWinds attackers ran a master class in novel hacking techniques. They modified sealed software code, created a system that used domain names to select targets and mimicked the Orion software communication protocols so they could hide in plain sight. And then, they did what any good operative would do: They cleaned the crime scene so thoroughly investigators can't prove definitively who was behind it. The White House has said unequivocally that Russian intelligence was behind the hack. Russia, for its part, has denied any involvement.

"The tradecraft was phenomenal," said Adam Meyers, who led the cyber forensics team that pawed through that tainted update on behalf of SolarWinds, providing details for the first time about what they found. The code was elegant and innovative, he said, and then added, "This was the craziest f***ing thing I'd ever seen."

Like razor blades in peanut butter cups

Meyers is the vice president for threat intelligence at the cybersecurity firm CrowdStrike, and he's seen epic attacks up close. He worked on the 2014 Sony hack, when North Korea cracked into the company's servers and released emails and first-run movies. A year later, he was on the front lines when a suspected Kremlin-backed hacking team known as "Cozy Bear" stole, among other things, a trove of emails from the Democratic National Committee. WikiLeaks then released them in the runup to the 2016 election.

"We're involved in all kinds of incidents around the globe every day," Meyers said. Typically he directs teams, he doesn't run them. But SolarWinds was different: "When I started getting briefed up, I realized [this] was actually quite a big deal."

The attack began with a tiny strip of code. Meyers traced it back to Sept. 12, 2019. "This little snippet of code doesn't do anything," Meyers said. "It's literally just checking to see which processor is running on the computer, if it is a 32- or 64-bit processor and if it is one or the other, it returns either a zero or a one."

National Security

U.s. slaps new sanctions on russia over cyberattack, election meddling.

Investigations

Why russia may have stepped up its hacking game.

The code fragment, it turns out, was a proof of concept — a little trial balloon to see if it was possible to modify SolarWinds' signed-and-sealed software code, get it published and then later see it in a downloaded version. And they realized they could. "So at this point, they know that they can pull off a supply chain attack," Meyers said. "They know that they have that capability."

After that initial success, the hackers disappeared for five months. When they returned in February 2020, Meyers said, they came armed with an amazing new implant that delivered a backdoor that went into the software itself before it was published.

To understand why that was remarkable, you need to know that finished software code has a kind of digital factory seal. If you break that seal, someone can see it and know that the code might have been tampered with. Meyers said the hackers essentially found a way to get under that factory seal.

They began by implanting code that told them any time someone on the SolarWinds development team was getting ready to build new software. They understood that the process of creating software or an update typically begins with something routine such as checking a code out of a digital repository, sort of like checking a book out of the library.

Under normal circumstances, developers take the code out of the repository, make changes and then check it back in. Once they finish tinkering, they initiate something called the build process, which essentially translates the code a human can read to the code a computer does. At that point, the code is clean and tested. What the hackers did after that was the trick.

They would create a temporary update file with the malicious code inside while the SolarWinds code was compiling. The hackers' malicious code told the machine to swap in their temporary file instead of the SolarWinds version. "I think a lot of people probably assume that it is the source code that's been modified," Meyers said, but instead the hackers used a kind of bait-and-switch.

Adam Meyers, vice president for threat intelligence at CrowdStrike, said when he became familiar with the SolarWinds attack, he knew it was a big deal. Oscar Zagal Studio hide caption

Adam Meyers, vice president for threat intelligence at CrowdStrike, said when he became familiar with the SolarWinds attack, he knew it was a big deal.

But this, Meyers said, was interesting, too. The hackers understood that companies such as SolarWinds typically audit code before they start building an update, just to make sure everything is as it should be. So they made sure that the switch to the temporary file happened at the last possible second, when the updates went from source code (readable by people) to executable code (which the computer reads) to the software that goes out to customers.

The technique reminded Meyers of old fears around trick-or-treating. For decades, there had been an urban myth that kids couldn't eat any Halloween candy before checking the wrapper seal because bad people might have put razor blades inside. What the hackers did with the code, Meyers said, was a little like that.

"Imagine those Reese's Peanut Butter Cups going into the package and just before the machine comes down and seals the package, some other thing comes in and slides a razor blade into your Reese's Peanut Butter Cup," he said. Instead of a razor blade, the hackers swapped the files so "the package gets sealed and it goes out the door to the store."

The update that went out to SolarWinds' customers was the dangerous peanut butter cup — the malicious version of the software included code that would give the hackers unfettered, undetected access to any Orion user who downloaded and deployed the update and was connected to the Internet.

But there was something else about that code that bothered Meyers: It wasn't just for SolarWinds. "When we looked at [it], it could have been reconfigured for any number of software products," Meyers said. In other words, any number of other software developers using the same compiler may also be on the receiving end of a cyberattack, he said, and they just don't know it yet.

Picking and choosing targets

Meyers said it's hard not to admire just how much thought the hackers put into this operation. Consider the way they identified targets. The downside of breaking into so many customer networks all at once is that it is hard to decide what to exploit first. So the hackers created a passive domain name server system that sent little messages with not just an IP address, which is just a series of numbers, but also with a thumbnail profile of a potential target.

"So they could then say, 'OK, we're going to go after this dot gov target or whatever,' " Meyers said. "I think later it became clear that there were a lot of government technology companies being targeted."

The hackers also reverse-engineered the way Orion communicated with servers and built their own coding instructions mimicking Orion's syntax and formats. What that did is allow the hackers to look like they were "speaking" Orion, so their message traffic looked like a natural extension of the software.

"So once they determined that a target was of interest, they could say, 'OK, let's go active, let's manipulate files, let's change something,' " Meyers said, and then they would slip in unnoticed through the backdoor they had created. "And there is one other thing I should mention: This backdoor would wait up to two weeks before it actually went active on the host. This was a very patient adversary."

None of the tripwires put in place by private companies or the government seems to have seen the attack coming. Christopher Krebs, who had been in charge of the office that protected government networks at DHS during the Trump administration, told NPR that DHS' current system, something known (without irony) as Einstein, only catches known threats. The SolarWinds breach, he said, was just "too novel."

Christopher Krebs, who was in charge of protecting government networks during the Trump administration, said the SolarWinds breach used techniques that were "too novel" for the current system to catch. Drew Angerer/Getty Images hide caption

Christopher Krebs, who was in charge of protecting government networks during the Trump administration, said the SolarWinds breach used techniques that were "too novel" for the current system to catch.

"Upwards of 90[%] to 95% of threats are based on known techniques, known cyberactivity," Krebs explained. "And that's not just criminal actors, that's state actors, too, including the Russian intelligence agencies and the Russian military. This was a previously unidentified technique."

And there is something else that Einstein doesn't do: It doesn't scan software updates. So even if the hackers had used code that Einstein would have recognized as bad, the system might not have seen it because it was delivered in one of those routine software updates.

The National Security Agency and the military's U.S. Cyber Command were also caught flat-footed. Broadly speaking, their cyber operators sit in foreign networks looking for signs of cyberattacks before they happen. They can see suspicious activity in much the same way a satellite might see troops amassing on the border. Critics said they should have seen the hackers from the Russian intelligence service, the SVR, preparing this attack.

"The SVR has a pretty good understanding that the NSA is looking out," Krebs said. "What the SVR was able to do was make the transition from wherever they were operating from into the U.S. networks. They move like ghosts. They are very hard to track."

The hackers didn't do anything fancy to give them the domestic footprint, officials confirmed. In fact, they just rented servers from Amazon and GoDaddy.

Early warnings

There were some indications, elsewhere, though, that something was wrong.

In early July, Steven Adair, the founder of a Washington, D.C.-based cybersecurity company called Volexity, saw some suspicious activity on a client's computers. "We traced it back, and we thought it might be related to a bad update with SolarWinds," Adair told NPR. "We addressed the problem, made sure no one was in our customers' systems, and we left it at that."

Adair said he didn't feel he had enough detail to report the problem to SolarWinds or the U.S. government. "We thought we didn't have enough evidence to reach out," he said.

That was the first missed sign.

The second came three months later when a California-based cybersecurity company called Palo Alto Networks discovered a malicious backdoor that seemed to emanate from the Orion software.

In that case, according to SolarWinds' Ramakrishna, the security teams at SolarWinds and Palo Alto worked together for three months to try to pick up the thread of the problem and walk it back. "None of us could pinpoint a supply chain attack at that point," Ramakrishna told NPR. "The ticket got closed as a result of that. If we had the benefit of hindsight, we could have traced it back" to the hack.

Palo Alto Networks had agreed to speak to NPR about the incident last month and then canceled the interview just an hour before it was supposed to take place. A spokesperson declined to say why and sent a few blog posts and wrote: "I'm afraid this is all we have to help at this time."

"Just 3,500 lines long"

It was the cybersecurity firm FireEye that finally discovered the intrusion. Mandia, the company's CEO, used to be in the U.S. Air Force Office of Special Investigations, so his specialty was criminal cases and counterintelligence. In the intervening years, the kinds of patterns he learned to recognize in special investigations kept appearing in his cyber security work.

The first indication that hackers had found their way into FireEye's networks came in an innocuous way. Someone on the FireEye security team had noticed that an employee appeared to have two phones registered on his network, so she called him. "And that phone call is when we realized, hey, this isn't our employee registering that second phone, it was somebody else," Mandia said.

Mandia had a security briefing a short time later and everything he heard reminded him of his previous work in the military. "There was a lot of pattern recognition from me," he told NPR. "I spent from 1996 to 1998 responding to what I would equate to the Russian Foreign Intelligence Service, and there were some indicators in the first briefing that were consistent with my experience in the Air Force."

He called a board meeting the same day. "It just felt like the breach that I was always worried about."

What his team discovered over the course of several weeks was that not only was there an intruder in its network, but someone had stolen the arsenal of hacking tools FireEye uses to test the security of its own clients' networks. FireEye called the FBI, put together a detailed report, and once it had determined the Orion software was the source of the problem, it called SolarWinds.

Brown, vice president of security at SolarWinds, took the Saturday morning phone call. "He said, 'Essentially, we've decompiled your code. We found malicious code,' " Brown said. FireEye was sure SolarWinds "had shipped tainted code."

The tainted code had allowed hackers into FireEye's network, and there were bound to be others who were compromised, too. "We were hearing that different reporters had the scoop already," Mandia said. "My phone actually rang from a reporter and that person knew and I went, OK, we're in a race."

Mandia thought they had about a day before the story would break.

After that, events seemed to speed up. SolarWinds' chief security officer, Brown, called Ron Plesco, a lawyer at the firm DLA Piper, and told him what had happened. One of the first things companies tend to do after cyberattacks is hire lawyers, and they put them in charge of the investigation. They do this for a specific reason — it means everything they find is protected by attorney-client privilege and typically is not discoverable in court.

Ron Plesco, a lawyer with the firm DLA Piper, has made cybercrimes a specialty of his practice. "I've been in situations where, while you're in there doing the investigation, [hackers are] watching your email, they're compromising your phone calls or your Zooms," he said. Kriston Jae Bethel for NPR hide caption

Ron Plesco, a lawyer with the firm DLA Piper, has made cybercrimes a specialty of his practice. "I've been in situations where, while you're in there doing the investigation, [hackers are] watching your email, they're compromising your phone calls or your Zooms," he said.

Plesco, who has made cybercrimes a specialty of his practice, knew that once the story broke it would be saying "to the world that, ready, set, go, come after it," Plesco said. "So that puts you on an accelerated timeline on two fronts: Figure out what happened if you can and get a fix out as soon as possible."

The company worked with DHS to craft a statement that went out on Dec. 13.

To investigate a hack, you have to secure a digital crime scene. Just as detectives in the physical world have to bag the evidence and dust for prints for the investigation later, SolarWinds had to pull together computer logs, make copies of files, ensure there was a recorded chain of custody, all while trying to ensure the hackers weren't inside its system watching everything they did.

"I've been in situations where, while you're in there doing the investigation, they're watching your email, they're compromising your phone calls or your Zooms," Plesco said. "So they're literally listening in on how you're going to try to get rid of them."

By mid-January, Meyers and the CrowdStrike team had isolated what they thought was the attack's tiny beating heart. It was an elegant, encrypted little blob of code "just 3,500 lines long," he said. The best code is short and to the point, like a well-written sentence. This little encrypted strip, Meyers thought, might help them figure out who was behind the attack.

Little blobs of clues

Think of forensic cyber teams as digital detectives looking for patterns. Coding tics can sometimes help identify perpetrators or sometimes forensic teams find small cultural artifacts — such as Persian script, or Korean hangul. When an elite Russian hacking team took over the electrical grid in Ukraine in 2015, it had more literary aspirations: It sprinkled its malicious code with references to Frank Herbert's Dune novels. That's why CrowdStrike found that little blob of malicious code so intriguing.

Plesco shows a timeline of the SolarWinds hack on his computer. Kriston Jae Bethel for NPR hide caption

Plesco shows a timeline of the SolarWinds hack on his computer.

After weeks of working with the code, Meyers convened a Zoom call with leaders at SolarWinds and members of his team from around the world. He shared his screen so everyone could all watch the encryption fall away in real time. He began walking the spectators through the code as it was revealed, like a play-by-play analysis of a game. Meyers kept watching for the big reveal. "We're hoping it's going to have, you know, variable names or maybe some comments in Cyrillic or Mandarin to give us some clue who wrote this thing," he said.

But as CrowdStrike's decryption program chewed its way through the zeroes and ones, Meyers' heart sank. The crime scene was a bust. It had been wiped down. "They'd washed the code," Meyers said. "They'd cleaned it of any human artifact or tool mark. And that was kind of mind-blowing that [they] had the wherewithal to hide anything that a human might have inadvertently left behind as a clue."

Holy s***, he thought to himself, who does that?

Just type "solarwinds123"

Against such a sophisticated hack, it is easy to suggest this could have happened to just about any software company. But there were some troubling signs at SolarWinds that may have made it a target.

Consider its online marketing website. It contained a list of clients, including specific companies and government agencies, that ran its Orion software. While a lot of companies do that, the SolarWinds site was very specific. It was, two cybersecurity analysts told NPR, like a shopping list for adversaries.

Ramakrishna pushed back on the criticism. "Lots of companies do it. That is their badge of honor, saying all these customers rely on my technology," he said. "I wouldn't say that was the reason for why we were targeted." Ramakrishna said the hackers were "a lot more sophisticated" than that. Shortly after the attack, though, that particular page on the marketing website was taken down.

There was another unsettling report about passwords. A security researcher in Bangalore, India, named Vinoth Kumar told NPR that he had found the password to a server with SolarWinds apps and tools on a public message board and the password was: "solarwinds123." Kumar said he sent a message to SolarWinds in November and got an automated response back thanking him for his help and saying the problem had been fixed.

When NPR asked SolarWinds' vice president of security, Brown, about this, he said that the password "had nothing to do with this event at all, it was a password to a FTP site." An FTP site is what you use to transfer files over the Internet. He said the password was shared by an intern and it was "not an account that was linked to our active directory."

Ramakrishna said it was a password for a third-party site where some of SolarWinds' tools and apps were available for download. Ramakrishna admitted, though, that while the matter was unconnected to the breach, it was a problem to have that kind of password on a site that contained something someone might download thinking it was a SolarWinds product.

The SolarWinds attackers were masters in novel hacking techniques. The White House has said Russian intelligence was behind the hack. Russia has denied any involvement. Bronte Wittpenn/Bloomberg via Getty Images hide caption

The SolarWinds attackers were masters in novel hacking techniques. The White House has said Russian intelligence was behind the hack. Russia has denied any involvement.

"We used that as another opportunity to reeducate everybody on password policies," he said. "I do not want to minimize it or be casual about it, but I want to highlight that it had nothing to do" with the attack on Orion.

Ramakrishna inherited this attack. He was hired as the SolarWinds CEO shortly before the breach was discovered and stepped into the top job just as the full extent of the hack became clear. In a way, that has given him an incredible freedom. He can't be blamed for something that happened before he got there, and the changes he's making could be seen in the context of a new man in charge instead of a response to the attack.

Shortly after he arrived, he published a long blog post providing what was essentially an 11-point plan to improve company security. "Armed with what we have learned of this attack, we are also reflecting on our own security practices," he wrote in the blog post , adding that his goal was to put in place an "immediate improvement of critical business and product development systems."

Ramakrishna said he planned to transform SolarWinds into a truly "secure by design" organization with more robust threat protection and detection tools across its network, with a particular focus on where it developed and built software — the places that the SVR hackers used to break in.

He said he would establish privileged accounts and all accounts used by anybody who had anything to do with Orion and the company would enforce multifactor authentication, or MFA, across the board.

"If I come up with an 11-point plan to improve my company's security, one interpretation of that could be that we have learned a valuable lesson from what the hack was," said Ian Thornton-Trump, chief information security officer at Cyjax, a threat intelligence company. "The other interpretation could be, is that there were at least 11 material deficiencies in the actual security we had. I see that the 11-point plan is actually an admission that things were not good in this security house."

Thornton-Trump used to work at SolarWinds and was on the security team. Thornton-Trump left the company in 2017 because, by his own account, SolarWinds' management (Kevin Thompson was CEO at the time. Ramakrishna wouldn't arrive for another three years.) didn't want to spend enough on security.

I'll Be Seeing You

How the u.s. hacked isis.

Can A Computer Catch A Spy?

Thornton-Trump concedes that the hackers who broke into the company were so sophisticated it would have been hard for anyone to defend against them. "But if you're driving drunk, rolling down the road, and it was raining and you smash up your car," he said, "why are we focused so much on the damage to the car, instead of what actually led up to the series of events that led to the great undoing?"

In other words, does the overhaul of SolarWinds' security practices add up to an admission that something was wrong, or is it simply a responsible upgrade?

Ramakrishna said it was both. "Oftentimes what happens is people conduct investigations, identify learnings and then implement something like this," he said. "Can we do things better? Absolutely. And honestly, even after implementing these 11 things, I'll be looking for the next 11 things to work on because the adversaries are becoming smarter and smarter every single day."

Ramakrishna said he wonders why, of all the software companies it had to choose from, the Russian intelligence service ended up targeting SolarWinds.

"I've thought about this quite a bit as to why us, why not somebody else," he said. "And that goes on through any investigation. As you think about this, we are deployed in more than 300,000 customers today. And so we are fairly broadly deployed software and where we enjoy administrative privileges in customer environments. So in a supply chain attack like this, the goal will be to try to get a broad swath of deployment and then you pick and choose what you want to do from there."

Whatever the reason SolarWinds ended up in the crosshairs, the attack revealed the U.S. cyber community's spectacular inability to connect the dots. Not just the early warnings from Volexity or the investigation with Palo Alto Networks, but a simple discovery from a lone cyber researcher in Bangalore suggests that something is not right in our digital world.

Bigger attacks

"It's one of the most effective cyber-espionage campaigns of all time," said Alex Stamos, director of the Internet Observatory at Stanford University and the former head of security at Facebook. "In doing so, they demonstrated not just technical acumen, but the way they did this demonstrated that they understand how tech companies operate, how software companies operate. ... This certainly is going to change the way that large enterprises think about the software they install and think about how they handle updates."

Intelligence analysts, already years ahead of the rest of us, are paid to imagine the darkest of scenarios. What if the hackers planted the seeds of future attacks during that nine months they explored SolarWinds' customer networks — did they hide code for backdoors that will allow them to come and go as they please at a time of their choosing? When hackers shut down the Ukraine's power grid in 2015 and disabled a Saudi refinery with computer code a year later, they showed it was possible to jump from a corporate network to system controls. Will we find out later that the SolarWinds hack set the stage for something more sinister?

Even if this was just an espionage operation, FireEye's Mandia said, the attack on SolarWinds is an inflection point. "We ... kind of mapped out the evolution of threats and cyber," he said. "And we would have landed at this day sooner or later, that at some point in time, software that many companies depend on is going to get targeted and it's going to lead to exactly what it led to," Mandia said. "But to see it happen, that's where you have a little bit of shock and surprise. OK, it's here now, nations are targeting [the] private sector, there's no magic wand you can shake. ... It's a real complex issue to solve."

The Biden administration is working on a second executive order — beyond the sanctions — that is supposed to address some of the issues SolarWinds has put in stark relief.

Anne Neuberger, the deputy national security adviser for cyber and emerging technology in charge of the SolarWinds attack response, is preparing an order that would, among other things, require companies that work with the U.S. government to meet certain software standards, and federal agencies would be required to adopt basic security practices such as encrypting data in their systems.

In addition, software companies such as SolarWinds could be required to have their so-called build systems — the place where they assemble their software — air-gapped, which means they would not be connected to the Internet. Those elements are all still under discussion as part of the executive order, NPR has learned.

Anne Neuberger, deputy national security adviser for cyber and emerging technology, is in charge of the SolarWinds attack response. She is preparing an order that would require companies that work with the U.S. to meet certain software standards, and federal agencies would be required to adopt certain basic security practices. Drew Angerer/Getty Images hide caption

Anne Neuberger, deputy national security adviser for cyber and emerging technology, is in charge of the SolarWinds attack response. She is preparing an order that would require companies that work with the U.S. to meet certain software standards, and federal agencies would be required to adopt certain basic security practices.

Another idea starting to gain traction is to create a kind of National Transportation Safety Board, or NTSB, to investigate cyberattacks in a more formal way.

"When the Boeing 737 Maxes started crashing, there was a government agency whose entire job it was to gather up the facts of all those different crashes and then come up with a theory of what needed to be fixed and then oversaw the fixes that went into that," Stamos said. "We need the same kind of function in the U.S. government."

The FBI could do its investigation of the cybercrime and some sort of federal agency would look at the root causes of a cyberattack and make the appropriate changes to the way we do things. Mandia said something like that probably needs to exist.

"When you think about the conflict, you have air, land and sea and space and now cyber," he said. "But in cyber, the private sector is front and center. Any conflict in cyberspace, whether motivated by a criminal element or motivated by geopolitical conditions, it's going to involve both the government and the private sector. And that response, because it impacts both, you almost need a triage that both sides, both private and public sector, benefit from similar to the NTSB."

Mandia envisions a review board for significant incidents where intelligence is gathered and the nation finds a way to defend itself appropriately. Right now, the onus is on private companies to do all the investigations.

A Biden administration official told reporters during a background briefing Thursday that one reason the White House responded so strongly to the SolarWinds attack is because these kinds of hacks put an undue burden on private companies.

A federal review might help with one of the issues that has plagued cyberspace up to now: how to ensure software and hardware vendors disclose hacks when they discover them. Could a review board take the sting out of the reputation damage of admitting publicly you've been hacked? Would it give companies such as Volexity and Palo Alto Networks somewhere to go when they see a problem?

Ultimately, the goal is to connect the dots and respond in a way that makes us safer. And the impetus for all of this might be that tainted routine update. That's one of the key reasons SolarWinds decided to go public, Ramakrishna said.

"We went out and published the entire source code because what we wanted people to do, no matter the vendor, whether it could be a competitor of ours or not, is to check your software, make sure you don't have a situation like this, and if there is, clean it up," he said. "So while it was unfortunate that we were the subject of this attack, my hope is, by us learning from it, we can also help the broader community."

Even so, there are parts of this story that may sound familiar: missed opportunities, hints of a problem that were ignored, the failure of U.S. intelligence officials to connect the dots. Who would have thought a routine software update could launch a cyberattack of epic proportions?

"This was an intelligence collection operation meant to steal information, and it's not the last time that's going to happen," CrowdStrike's Meyers warned. "This is going to happen every day. ... And I think there's a lot that we all need to do to work together to stop this from happening."

NPR's Monika Evstatieva contributed to this report.

• cyber security
• russian hackers

The SolarWinds cyberattack: The hack, the victims, and what we know

Lawrence abrams.

• December 19, 2020

Since the SolarWinds supply chain attack was disclosed in December, there has been a whirlwind of news, technical details, and analysis released about the hack.

Because the amount of information that was released in such a short time is definitely overwhelming, we have published this as a roundup of SolarWinds news.

The information is distilled into a format that will hopefully explain the attack, who its victims are, and what we know to this point.

The SolarWinds supply chain attack

While we learned of SolarWinds' attack on December 13th, the first disclosure of its consequence was made on December 8th when leading cybersecurity firm FireEye revealed that it was hacked by a nation-state APT group . As part of this attack, the threat actors stole Red Team assessment tools that FireEye uses to probe its customers' security.

It was not known how the hackers gained access to FireEye's network until Sunday, December 13th, 2020, when Microsoft , FireEye , SolarWinds , and the U.S. government issued a coordinated report that SolarWinds had been hacked by state-sponsored threat actors believed to be part of the Russian S.V.R.

One of SolarWinds' customers who was breached in this attack is FireEye.

As part of the attack, the threat actors gained access to the SolarWinds Orion build system and added a backdoor to the legitimate SolarWinds.Orion.Core.BusinessLayer.dll DLL file. This DLL was then distributed to SolarWinds customers in a supply chain attack via an automatic update platform used to push out new software updates.

This DLL backdoor is known as Sunburst (FireEye) or Solorigate (Microsoft, and is loaded by the SolarWinds.BusinessLayerHost. exe program. Once loaded, it will connect back to the remote command & control server at a subdomain of avsvmcloud[.]com to receive "jobs," or tasks, to execute on the infected computer.

The backdoor's command control server's DNS name is created utilizing a domain generation algorithm (DGA) to create an encoded subdomain of avsvmcloud[.]com . FireEye states that the subdomain is created by "concatenating a victim userId with a reversible encoding of the victims local machine domain name," and then hashed. For example, a subdomain used in this attack is '1btcr12b62me0buden60ceudo1uv2f0i.appsync-api.us-east-2[.]avsvmcloud.com.'

It is unknown what tasks were executed, but it could be anything from giving remote access to the threat actors, downloading and installing further malware, or stealing data.

Microsoft published a technical writeup on Friday for those interested in the technical aspects of the Sunburst backdoor.

A report by Kim Zetter released Friday night indicates that the threat actors may have performed a dry run of the distribution method as early as October 2019. During this dry run, the DLL was distributed without the malicious Sunburst backdoor.

After the threat actors began distributing the backdoor in March 2020, researchers believe that the attackers have been silently sitting in some of the compromised networks for months while harvesting information or performing other malicious activity.

Zetter's report stated that FireEye eventually detected they were hacked after the threat actors registered a device to the company's multi-factor authentication (MFA) system using stolen credentials. After the system alerted the employee and the security team of this unknown device, FireEye realized that they had been compromised.

Additional malware discovered

After performing investigations of SolarWinds supply chain victims, researchers have begun to get a better idea of the different malware used in the attack.

According to CrowdStrike, a malware named SunSpot was first executed in the SolarWinds network to monitor for and automatically inject the Sunburst backdoor in the SolarWinds development builds. 

The Sunburst backdoor would then be transferred to victims via automatic updates for the SolarWinds Orion platform. Once executed, it would routinely connect to a remote command and control server for commands to execute on the infected device.

FireEye discovered that the Sunburst backdoor would drop a malware named  Teardrop , which is a previously unknown memory-only dropper and a post-exploitation tool used to deploy customized Cobalt Strike beacons.

Finally, Symantec discovered the RainDrop malware , which was also used to deploy Cobalt Strike beacons on other hosts in an already compromised network.

The hackers behind the SolarWinds attack

FireEye is currently tracking the threat actor behind this campaign as UNC2452 , while Washington-based cybersecurity firm Volexity has linked this activity to a hacking group known under the Dark Halo moniker.

Volexity says that Dark Halo actors have coordinated malicious campaigns between late 2019 and July 2020, targeting and successfully compromising the same US-based think tank three times in a row .

“In the initial incident, Volexity found multiple tools, backdoors, and malware implants that had allowed the attacker to remain undetected for several years,” the company said.

In the second attack, after being cast out from the victim’s network, Dark Halo leveraged a newly disclosed Microsoft Exchange server bug that helped them to circumvent Duo multi-factor authentication (MFA) defenses for unauthorized email access via the Outlook Web App (OWA) service.

During the third attack targeting the same think tank, the threat actor used the SolarWinds supply chain attack to deploy the same backdoor Dark Halo used to breach FireEye's networks and several U.S. government agencies.

Unconfirmed media reports have also cited sources linking the attacks to APT29 (aka Cozy Bear) , a state-sponsored hacking group associated with the Russian Foreign Intelligence Service (SVR).

Researchers, including FireEye, Microsoft, or Volexity, have not attributed these attacks to APT29 at this time.

Reminder: Neither FireEye (UNC 2452), Microsoft (untamed, but no public references tying to YTTRIUM), nor Volexity (Dark Halo) have linked SolarWinds event to APT29... — Joe Slowik (@jfslowik) December 17, 2020

The Russian Embassy in the USA reacted [ 1 , 2 ] to these media reports saying that they were an “unfounded attempt of the U.S. media to blame Russia for hacker attacks on U.S. governmental bodies.”

“ Russia does not conduct offensive operations in the cyber domain ,” the Embassy added.

While Russia continues to deny these attacks, Secretary of State Mike Pompeo stated in an interview released Friday night that it is “pretty clear” that Russia was behind that attack.

“This was a very significant effort, and I think it’s the case that now we can say pretty clearly that it was the Russians that engaged in this activity,” Pompeo told radio host Mark Levin.

Microsoft believes that the ultimate goal of these attacks was to gain access to victims' cloud assets after deploying the Sunburst/Solorigate backdoor on their local networks.

The victims of the attack

Researchers believe that the malicious DLL was pushed out to approximately 18,000 customers as part of this attack.

The threat actors, though, only targeted organizations that they perceived as 'high value,' so even though some of these customers may have received the DLL, it is unknown if they were actively targeted in further attacks.

The currently known list of organizations that were hit by the SolarWinds supply chain attack include:

• U.S. Department of the Treasury
• U.S. National Telecommunications and Information Administration (NTIA)
• U.S. Department of State
• The National Institutes of Health (NIH) (Part of the U.S. Department of Health)
• U.S. Department of Homeland Security (DHS)
• U.S. Department of Energy (DOE)
• U.S. National Nuclear Security Administration (NNSA)
• Some US states (Specific states are undisclosed)

Microsoft has also identified and notified more than 40 of its customers affected by this attack but has not disclosed their names. They state that 80% of the victims were from the U.S., and 44% were in the IT sector.

Based on the decoding of subdomains generated by the malware domain generation algorithm (DGA), many well-known companies may disclose targeted attacks at a later date.

What are security firms doing to protect victims

Since the cyberattack has been disclosed, security firms have been adding the malicious Sunburst backdoor binaries to their detections.

While Microsoft was already detecting and alerting customers of malicious SolarWinds binaries, they were not quarantining them out of concern it could affect an organization's network management services. On December 16th, at 8:00 AM PST, Microsoft Defender began quarantining detected binaries even if the process is running.

Microsoft, FireEye, and GoDaddy also collaborated to create a kill switch for the Sunburst backdoor distributed in the SolarWinds hack.

When the malicious binaries attempt to contact the command & control servers, they will perform DNS resolution to get the IP address. If this IP address is part of certain IP ranges, including ones owned by Microsoft, the backdoor will terminate and prevent itself from executing again.

To create the kill switch, GoDaddy created a wildcard DNS resolution so that any subdomain of avsvmcloud[.]com resolves to the IP address 20.140.0.1, which belongs to Microsoft and is on the malware's blocklist. This wildcard resolution is illustrated by a DNS lookup for a made-up subdomain, as shown below.

As this IP address is part of the malware's blocklist, when it connects to any subdomain of avsvmcloud[.]com, it will unload and no longer execute.

While this kill switch will disable Sunburst backdoor deployments connecting the command & control servers, FireEye has stated the threat actors may have deployed other backdoors.

"However, in the intrusions FireEye has seen, this actor moved quickly to establish additional persistent mechanisms to access to victim networks beyond the Sunburst backdoor. This killswitch will not remove the actor from victim networks where they have established other backdoors. However, it will make it more difficult to for the actor to leverage the previously distributed versions of Sunburst," FireEye warned about the kill switch," FireEye told BleepingComputer in a statement.

How to check if you were compromised

If you are a user of SolarWinds products, you should immediately consult their advisory and Frequently Asked Questions as it contains necessary information about upgrading to the latest 'clean' version of their software.

Microsoft has also published a list of nineteen malicious SolarWinds.Orion.Core.BusinessLayer.dll DLL files spotted in the wild.

This list, shown below, contains a file's SHA256 hash, the file version, and when it was first seen.

Finally, security researchers have released various tools that allow you to check if you were compromised or what credentials were stored in your SolarWinds Orion installation.

• SolarFlare Release: Password Dumper for SolarWinds Orion

SpearTip’s SolarWinds’ Orion Vulnerability Tool SunScreen – SPF 10

The source code for both projects is published to GitHub. You are strongly encouraged to review the source code, if available, of any program you plan to run on your network.

Security researcher Cory Kennedy has also released a python tool to help you find the Sunburst malware on your network.

This tool is called Sunburst hunter and can be downloaded from the project's GitHub page .

SolarWinds Orion abused in other supply chain attacks

During the investigation into the SolarWinds hack, Palo Alto Networks and Microsoft found an additional malware named SUPERNOVA distributed using the App_Web_logoimagehandler.ashx.b6031896.dll DLL file.

This malware is a backdoor that allowed the threat actors to send C# code that would be compiled and executed by the malware.

This malware is not believed to be related to the SolarWinds.Orion.Core.BusinessLayer.dll supply chain attack. It does, though, indicate that the SolarWinds Orion platform was used in two different attacks, and possibly by different groups, to distribute malware.

Last week, SolarWinds released an update advisory that advises all Orion Platform customers to upgrade to the latest versions to be protected from not only the SUNBURST vulnerability but the SUPERNOVA malware as well.

Additional reporting by Sergiu Gatlan and Ionut Ilascu .

Update 12/19/20: Added Cisco to the victim list. Update 12/27/20: Added information about second SUPERNOVA malware. Update 01/20/20: Added information about further malware

Related Articles:

Greece’s Land Registry agency breached in wave of 400 cyberattacks

SolarWinds fixes 8 critical bugs in access rights audit software

Why all accounts (even test accounts) need strong passwords

Polyfill.io, BootCDN, Bootcss, Staticfile attack traced to 1 operator

Polyfill claims it has been 'defamed', returns after domain shut down

• Cyberattack
• Supply Chain Attack
• Previous Article
• Next Article

Mike_Walsh - 3 years ago

Interesting, yet hardly surprising to note that the entire thing has been exclusively targeted solely against the US... (*shrug*)

Some-Other-Guy - 3 years ago

"While Russia continues to deny these attacks, Secretary of State Mike Pompeo stated in an interview released Friday night that it is “pretty clear” that Russia was behind that attack." ------------- That's great Maybe Pomeo should inform the President that all the fake intel indicating that the Presedential election was stolen actually came from hacks like this, and that Trump actually is the BIGGEST LOSER in political history After all, he needs accurate Intel, doesn't he? LoL If you have evidence that it was the Russians, then show me the evidence It's kinda hard to believe anything Pomeo say's at this point

Joepete - 3 years ago

Attributing a cyber attack can be a very complex process. The opportunity for "false flag" attribution is immense. Consider this: If Country B appears to be able to break into the infrastructure of Country C, who is to say that Country A did not break into B and launch the attacks from its infrastructure?

EmanuelJacobsson - 3 years ago

Dont feed the troll.

This just in..... Even Trump disputes Pompeo's allegation! https://www.cnbc.com/2020/12/19/trump-contradicts-pompeo-plays-down-alleged-russian-role-in-hack.html

igioz - 3 years ago

what about weak Solarwinds password?

What about Commie Joe winning the election ? https://www.newsweek.com/trump-tweets-solarwinds-hack-voter-fraud-election-loss-1556165 and why so few comments on the biggest hack ever ? Whats wrong with this picture ?

buddy215 - 3 years ago

The believers in false conspiracy theories and the elective ignorant fail to recognize or admit that voting machines are NOT connected to the Internet. Without the false conspiracy theorists believers and the elective ignorant voters....the Republican Party would shrink to totally irrelevant. .

Democrats (the promoters of false conspiracy theories) are blaming Russia without evidence and demanding Trump do something immediately to punish them for the attack Just read the news, starting with CNN to see what I mean A reasonable person would gather "EVIDENCE" to show who committed the attack before striking back and present the evidence for independent verification If the Russians were already in more than 18,000 critical computers and networks for more than 6 months, then why did Biden win the election? If we are going to speculate without evidence, then a biden win might rather point to China or even the Democrats themselves as the source of the attack Let's wait and see what the "EVIDENCE" says as to who did what instead of resorting to wild conspiracy theories Who exactly were the targets of the attack? Was one political party attacked more than the other? and what evidence do you have that it was the Russians?

Lawrence Abrams - 3 years ago

Comments will be closed if they continue to derail the topic of this article. If you want to discuss politics, there are plenty of threads in the speak easy forum.

I'd like to hear your input on the source Have you seen any actual "evidence" as to the source, and if so, can "we" see the evidence as well?

Please just go ahead and close the comment section, this isnt going anywhere anyways.

Dont feed the troll guys...

Comments have been disabled for this article.

You may also like:.

Telegram zero-day allowed sending malicious Android APKs as videos

Google rolls back decision to kill third-party cookies in Chrome

Malwarebytes Anti-Malware

BitDefender Uninstall Tool

Not a member yet? Register Now

Help us understand the problem. What is going on with this comment?

• Abusive or Harmful
• Inappropriate content
• Strong language

Read our posting guidelinese to learn what content is prohibited.

To revisit this article, visit My Profile, then View saved stories .

• The Big Story
• Newsletters
• Steven Levy's Plaintext Column
• WIRED Classics from the Archive
• WIRED Insider
• WIRED Consulting

If you buy something using links in our stories, we may earn a commission. Learn more.

The Untold Story of the Boldest Supply-Chain Hack Ever

Steven Adair wasn’t too rattled at first.

It was late 2019, and Adair, the president of the security firm Volexity, was investigating a digital security breach at an American think tank. The intrusion was nothing special. Adair figured he and his team would rout the attackers quickly and be done with the case—until they noticed something strange. A second group of hackers was active in the think tank’s network. They were going after email, making copies and sending them to an outside server. These intruders were much more skilled, and they were returning to the network several times a week to siphon correspondence from specific executives, policy wonks, and IT staff.

Adair and his colleagues dubbed the second gang of thieves “Dark Halo” and booted them from the network. But soon they were back. As it turned out, the hackers had planted a backdoor on the network three years earlier—malicious code that opened a secret portal, allowing them to enter or communicate with infected machines. Now, for the first time, they were using it. “We shut down one door, and they quickly went to the other,” Adair says.

His team spent a week kicking the attackers out again and getting rid of the backdoor. But in late June 2020, the hackers somehow returned. And they were back to grabbing email from the same accounts. The investigators spent days trying to figure out how they had slipped back in. Volexity zeroed in on one of the think tank’s servers—a machine running a piece of software that helped the organization’s system admins manage their computer network. That software was made by a company that was well known to IT teams around the world, but likely to draw blank stares from pretty much everyone else—an Austin, Texas, firm called SolarWinds.

Adair and his team figured the hackers must have embedded another backdoor on the victim’s server. But after considerable sleuthing, they couldn’t find one. So they kicked the intruders out again and, to be safe, disconnected the server from the internet. Adair hoped that was the end of it. But the incident nagged at him. For days he woke up around 2 am with a sinking feeling that the team had missed something huge.

They had . And they weren’t the only ones. Around the time Adair’s team was kicking Dark Halo out of the think tank’s network, the US Department of Justice was also wrestling with an intrusion —one involving a server running a trial version of the same SolarWinds software. According to sources with knowledge of the incident, the DOJ discovered suspicious traffic passing from the server to the internet in late May, so they asked one of the foremost security and digital forensics firms in the world—Mandiant—to help them investigate. They also engaged Microsoft, though it’s not clear why. (A Justice Department spokesperson confirmed that this incident and investigation took place but declined to say whether Mandiant and Microsoft were involved. Neither company chose to comment on the investigation.)

According to the sources familiar with the incident, investigators suspected the hackers had breached the Justice Department server directly, possibly by exploiting a vulnerability in the SolarWinds software. The Justice Department team contacted the company, even referencing a specific file that they believed might be related to the issue, according to the sources, but SolarWinds’ engineers were unable to find a vulnerability in their code. After weeks of back and forth the mystery was still unresolved, and the communication between investigators and SolarWinds stopped. (SolarWinds declined to comment on this episode.) The department, of course, had no idea about Volexity’s uncannily similar hack.

As summer turned to fall, behind closed doors, suspicions began to grow among people across government and the security industry that something major was afoot. But the government, which had spent years trying to improve its communication with outside security experts, suddenly wasn’t talking. Over the next few months, “people who normally were very chatty were hush-hush,” a former government worker says. There was a rising fear among select individuals that a devastating cyber operation was unfolding, he says, and no one had a handle on it.

In fact, the Justice Department and Volexity had stumbled onto one of the most sophisticated cyberespionage campaigns of the decade. The perpetrators had indeed hacked SolarWinds’ software. Using techniques that investigators had never seen before, the hackers gained access to thousands of the company’s customers. Among the infected were at least eight other federal agencies, including the US Department of Defense, Department of Homeland Security, and the Treasury Department, as well as top tech and security firms, including Intel, Cisco , and Palo Alto Networks —though none of them knew it yet. Even Microsoft and Mandiant were on the victims list.

After the Justice Department incident, the operation remained undiscovered for another six months. When investigators finally cracked it, they were blown away by the hack’s complexity and extreme premeditation. Two years on, however, the picture they’ve assembled—or at least what they’ve shared publicly—is still incomplete. A full accounting of the campaign’s impact on federal systems and what was stolen has never been provided to the public or to lawmakers on Capitol Hill. According to the former government source and others, many of the federal agencies that were affected didn’t maintain adequate network logs, and hence may not even know what all was taken. Worse: Some experts believe that SolarWinds was not the only vector—that other software makers were, or might still be, spreading malware. What follows is an account of the investigation that finally exposed the espionage operation—how it happened, and what we know. So far.

on November 10, 2020, an analyst at Mandiant named Henna Parviz responded to a routine security alert—the kind that got triggered anytime an employee enrolled a new phone in the firm’s multifactor authentication system. The system sent out one-time access codes to credentialed devices, allowing employees to sign in to the company’s virtual private network. But Parviz noticed something unusual about this Samsung device: It had no phone number associated with it.

She looked closely at the phone’s activity logs and saw another strange detail. The employee appeared to have used the phone to sign in to his VPN account from an IP address in Florida. But the person didn’t live in Florida, and he still had his old iPhone enrolled in the multifactor system. Then she noticed that the Samsung phone had been used to log in from the Florida IP address at the same time the employee had logged in with his iPhone from his home state. Mandiant had a problem.

The security team blocked the Samsung device, then spent a day investigating how the intruder had gotten into the network. They soon realized the issue transcended a single employee’s account. The attackers had pulled off a Golden SAML attack—a sophisticated technique for hijacking a company’s employee authentication system. They could seize control of a worker’s accounts, grant those accounts more privileges, even create new accounts with unlimited access. With this power, there was no telling how deep they had burrowed into the network.

On November 17, Scott Runnels and Eric Scales, senior members of Mandiant’s consulting division, quietly pulled together a top-tier investigative team of about 10, grabbing people from other projects without telling managers why, or even when the employees would return. Uncertain what the hunt would uncover, Runnels and Scales needed to control who knew about it. The group quickly realized that the hackers had been active for weeks but had evaded detection by “living off the land”—subverting administration tools already on the network to do their dirty deeds rather than bringing in their own. They also tried to avoid creating the patterns, in activity logs and elsewhere, that investigators usually look for.

But in trying to outsmart Mandiant, the thieves inadvertently left behind different fingerprints. Within a few days, investigators picked up the trail and began to understand where the intruders had been and what they had stolen.

On Friday morning, November 20, Kevin Mandia, Mandiant’s founder and CEO, clicked out of an all-hands meeting with 3,000 employees and noticed that his assistant had added a new meeting to his calendar. “Security brief” was all it said. Mandia, a 52-year-old former Air Force intelligence officer who still sports taper-cut military hair two decades after leaving service, was planning to get an early start on the weekend, but he dialed into the call anyway. He expected a quick update of some kind. Five minutes into the conversation, he knew his weekend was shot.

Many of the highest-profile hacks of the past two decades have been investigated by Mandia’s firm, which he launched in 2004. Acquired by FireEye in 2013, and again last year by Google, the company has threat hunters working on more than 1,000 cases annually, which have included breaches at Google, Sony, Colonial Pipeline, and others. In all that time, Mandiant itself had never suffered a serious hack. Now the hunters were the hunted.

The intruders, Mandia learned, had swiped tools his company uses to find vulnerabilities in its clients’ networks. They had also viewed sensitive information identifying its government customers. As his team described how the intruders had concealed their activity, Mandia flashed back to incidents from the early days of his career. From 1995 to 2013, while in the Air Force Office of Special Investigations and in the private sector, he had observed Russian threat actors continuously testing systems, disappearing as soon as investigators got a lock on them. Their persistence and stealth made them the toughest adversaries he’d ever faced. Now, hearing about the activity inside his own network, he “started getting pattern recognition,” he later told a conference audience. The day after getting the unsettling news of the breach, he reached out to the National Security Agency (NSA) and other government contacts.

While Mandia conferred with the government, Charles Carmakal, the CTO of Mandiant Consulting, contacted some old friends. Many of the hackers’ tactics were unfamiliar, and he wanted to see whether two former Mandiant colleagues, Christopher Glyer and Nick Carr, had seen them before. Glyer and Carr had spent years investigating large, sophisticated campaigns and had tracked the notorious hackers of the SVR—Russia’s foreign intelligence agency—extensively. Now the two worked for Microsoft, where they had access to data from many more hacking campaigns than they had at Mandiant.

Carmakal told them the bare minimum—that he wanted help identifying some activity Mandiant was seeing. Employees of the two companies often shared notes on investigations, so Glyer thought nothing of the request. That evening, he spent a few hours digging into the data Carmakal sent him, then tapped Carr to take over. Carr was a night owl, so they often tag-teamed, with Carr passing work back to Glyer in the morning.

The two didn’t see any of the familiar tactics of known hacking groups, but as they followed trails they realized whatever Mandiant was tracking was significant. “Every time you pulled on a thread, there was a bigger piece of yarn,” Glyer recalls. They could see that multiple victims were communicating with the hackers Carmakal had asked them to trace. For each victim, the attackers set up a dedicated command-and-control server and gave that machine a name that partly mimicked the name a real system on the victim’s network might have, so it wouldn’t draw suspicion. When Glyer and Carr saw a list of those names, they realized they could use it to identify new victims. And in the process, they unearthed what Carmakal hadn’t revealed to them—that Mandiant itself had been hacked.

It was a “holy shit” moment, recalls John Lambert, head of Microsoft Threat Intelligence. The attackers weren’t only looking to steal data. They were conducting counterintelligence against one of their biggest foes. “Who do customers speed-dial the most when an incident happens?” he says. “It’s Mandiant.”

As Carr and Glyer connected more dots, they realized they had seen signs of this hack before, in unsolved intrusions from months earlier. More and more, the exceptional skill and care the hackers took to hide their tracks was reminding them of the SVR.

back at mandiant, workers were frantically trying to address what to do about the tools the hackers had stolen that were designed to expose weak spots in clients’ defenses. Concerned that the intruders would use those products against Mandiant customers or distribute them on the dark web, Mandiant set one team to work devising a way to detect when they were being used out in the wild. Meanwhile, Runnels’ crew rushed to figure out how the hackers had slipped in undetected.

Because of the pandemic, the team was working from home, so they spent 18 hours a day connected through a conference call while they scoured logs and systems to map every step the hackers took. As days turned to weeks, they became familiar with the cadence of each other’s lives—the voices of children and partners in the background, the lulling sound of a snoring pit bull lying at Runnels’ feet. The work was so consuming that at one point Runnels took a call from a Mandiant executive while in the shower.

Runnels and Scales briefed Mandia daily. Each time the CEO asked the same question: How did the hackers get in? The investigators had no answer.

On December 8, when the detection tools were ready and the company felt it had enough information about the breach to go public, Mandiant broke its silence and released a blockbuster statement revealing that it had been hacked . It was sparse on details: Sophisticated hackers had stolen some of its security tools, but many of these were already public, and there was no evidence the attackers had used them. Carmakal, the CTO, worried that customers would lose confidence in the company. He was also anxious about how his colleagues would react to the news. “Are employees going to feel embarrassed?” he wondered. “Are people not going to want to be part of this team anymore?”

What Mandiant did not reveal was how the intruders got in or how long they had been in the company’s network. The firm says it still didn’t know. Those omissions created the impression that the breach was an isolated event with no other victims, and people wondered whether the company had made basic security errors that got it hacked. “We went out there and said that we got compromised by a top-tier adversary,” Carmakal says—something every victim claims. “We couldn’t show the proof yet.”

Mandiant isn’t clear about exactly when it made the first discovery that led it to the source of the breach. Runnels’ team fired off a barrage of hypotheses and spent weeks running down each one, only to turn up misses. They’d almost given up hope when they found a critical clue buried in traffic logs: Months earlier, a Mandiant server had communicated briefly with a mysterious system on the internet. And that server was running software from SolarWinds.

SolarWinds makes dozens of programs for IT administrators to monitor and manage their networks—helping them configure and patch a lot of systems at once, track performance of servers and applications, and analyze traffic. Mandiant was using one of the Texas company’s most popular products, a software suite called Orion. The software should have been communicating with SolarWinds’ network only to get occasional updates. Instead it was contacting an unknown system—likely the hackers’ command-and-control server.

Back in June, of course, Mandiant had been called in to help the Justice Department investigate an intrusion on a server running SolarWinds software. Why the pattern-matchers at one of the world’s preeminent security firms apparently didn’t recognize a similarity between the two cases is one of the lingering mysteries of the SolarWinds debacle. It’s likely that Runnels’ chosen few hadn’t worked on the Justice case, and internal secrecy prevented them from discovering the connection. (Mandiant declined to comment.)

Runnels’ team suspected the infiltrators had installed a backdoor on the Mandiant server, and they tasked Willi Ballenthin, a technical director on the team, and two others with finding it. The task before him was not a simple one. The Orion software suite consisted of more than 18,000 files and 14 gigabytes of code and data. Finding the rogue component responsible for the suspicious traffic, Ballenthin thought, would be like riffling through Moby-Dick for a specific sentence when you’d never read the book.

But they had been at it only 24 hours when they found the passage they’d been looking for: a single file that appeared to be responsible for the rogue traffic. Carmakal believes it was December 11 when they found it.

The file was a .dll, or dynamic-link library—code components shared by other programs. This .dll was large, containing about 46,000 lines of code that performed more than 4,000 legitimate actions, and—as they found after analyzing it for an hour—one illegitimate one.

The main job of the .dll was to tell SolarWinds about a customer’s Orion usage. But the hackers had embedded malicious code that made it transmit intelligence about the victim’s network to their command server instead. Ballenthin dubbed the rogue code “Sunburst”—a play on SolarWinds. They were ecstatic about the discovery. But now they had to figure out how the intruders had snuck it into the Orion .dll.

This was far from trivial. The Orion .dll file was signed with a SolarWinds digital certificate, which was supposed to verify that the file was legitimate company code. One possibility was that the attackers had stolen the digital certificate, created a corrupt version of the Orion file, signed the file to make it look authentic, then installed the corrupt .dll on Mandiant’s server. Or, more alarmingly, they might have breached SolarWinds’ network and altered the legitimate Orion .dll source code before SolarWinds compiled it—converting the code into software—and signed it. The second scenario seemed so far-fetched that the Mandiant crew didn’t really consider it—until an investigator downloaded an Orion software update from the SolarWinds website. The backdoor was in it.

The implication was staggering. The Orion software suite had about 33,000 customers, some of whom had started receiving the hacked software update in March. That meant some customers might have been compromised for eight months already. The Mandiant team was facing a textbook example of a software-supply-chain attack —the nefarious alteration of trusted software at its source. In a single stroke, attackers can infect thousands, potentially millions, of machines.

In 2017 hackers had sabotaged a software supply chain and delivered malware to more than 2 million users by compromising the computer security cleanup tool CCleaner . That same year, Russia distributed the malicious NotPetya worm in a software update to the Ukrainian equivalent of TurboTax, which then spread around the world. Not long after, Chinese hackers also used a software update to slip a backdoor to thousands of Asus customers . Even at this early stage in the investigation, the Mandiant team could tell that none of those other attacks would rival the SolarWinds campaign.

it was a Saturday morning, December 12, when Mandia called SolarWinds’ president and CEO on his cell phone. Kevin Thompson, a 14-year veteran of the Texas company, was stepping down as CEO at the end of the month. What he was about to hear from Mandia—that Orion was infected—was a hell of a way to wrap up his tenure. “We’re going public with this in 24 hours,” Mandia said. He promised to give SolarWinds a chance to publish an announcement first, but the timeline wasn’t negotiable. What Mandia didn’t mention was that he was under external pressure himself: A reporter had been tipped off about the backdoor and had contacted his company to confirm it. Mandia expected the story to break Sunday evening, and he wanted to get ahead of it.

Thompson started making calls, one of the first to Tim Brown, SolarWinds’ head of security architecture. Brown and his staff quickly confirmed the presence of the Sunburst backdoor in Orion software updates and figured out, with alarm, that it had been delivered to as many as 18,000 customers since the spring of 2020. (Not every Orion user had downloaded it.) Thompson and others spent most of Saturday frantically pulling together teams to oversee the technical, legal, and publicity challenges they faced. They also called the company’s outside legal counsel, DLA Piper, to oversee the investigation of the breach. Ron Plesco, an attorney at Piper and former prosecutor with forensic expertise, was in his backyard with friends when he got the call at around 10 pm.

Plesco beelined to his home office, arrayed with whiteboards, and started sketching out a plan. He set a timer for 20 hours, annoyed by what he felt was Mandia’s arbitrary deadline. A day was nowhere near enough to prepare affected customers. He worried that once SolarWinds went public, the attackers might do something destructive in customers’ networks before anyone could boot them out.

The practice of placing legal teams in charge of breach investigations is a controversial one. It puts cases under attorney-client privilege in a manner that can help companies fend off regulatory inquiries and fight discovery requests in lawsuits. Plesco says SolarWinds was, from the start, committed to transparency, publishing everything it could about the incident. (In interviews, the company was mostly forthcoming, but both it and Mandiant withheld some answers on the advice of legal counsel or per government request—Mandiant more so than SolarWinds. Also, SolarWinds recently settled a class action with shareholders over the breach but still faces a possible enforcement action from the Securities and Exchange Commission, making it less open than it might otherwise be about events.)

In addition to DLA Piper, SolarWinds brought on the security firm CrowdStrike, and as soon as Plesco learned this, he knew he wanted his old friend, Adam Meyers, on the case. The two had known each other for decades, ever since they’d worked on incident response for a defense contractor. Meyers was now the head of CrowdStrike’s threat intelligence team and rarely worked investigations. But when Plesco texted him at 1 am to say “I need your help,” he was all in.

Later that Sunday morning, Meyers jumped on a briefing call with Mandiant. On the call was a Microsoft employee, who told the group that in some cases, the hackers were systematically compromising Microsoft Office 365 email accounts and Azure cloud accounts. The hackers were also able to bypass multifactor authentication protocols. With every detail Meyers heard, the scope and complexity of the breach grew. Like others, he also suspected the SVR.

After the call, Meyers sat down in his living room. Mandiant had sent him the Sunburst code—the segment of the .dll file that contained the backdoor—so now he bent over his laptop and began picking it apart. He would remain in this huddled position for most of the next six weeks.

at solarwinds, shock, disbelief, and “controlled chaos” ruled those first days, says Tim Brown, the head of security architecture. Dozens of workers poured into the Austin office they hadn’t visited in months to set up war rooms. The hackers had compromised 71 SolarWinds email accounts—likely to monitor correspondence for any indication they’d been detected—so for the first few days, the teams communicated only by phone and outside accounts, until CrowdStrike cleared them to use their corporate email again.

Brown and his staff had to figure out how they had failed to prevent or detect the hack. Brown knew that whatever they found could cost him his job.

One of the team’s first tasks was to collect data and logs that might reveal the hackers’ activity. They quickly discovered that some logs they needed didn’t exist—SolarWinds didn’t track everything, and some logs had been wiped by the attackers or overwritten with new data as time passed. They also scrambled to see whether any of the company’s nearly 100 other products were compromised. (They only found evidence that Orion was hit.)

Around midmorning on Sunday, news of the hack began to leak. Reuters reported that whoever had struck Mandiant had also breached the Treasury Department. Then around 5 pm Eastern time, Washington Post reporter Ellen Nakashima tweeted that SolarWinds’ software was believed to be the source of the Mandiant breach. She added that the Commerce Department had also been hit. The severity of the campaign was growing by the minute, but SolarWinds was still several hours from publishing its announcement. The company was obsessing over every detail—a required filing to the Securities and Exchange Commission got so heavily lawyered that Thompson, the CEO, quipped at one point that adding a single comma would cost $20,000.

Around 8:30 that night, the company finally published a blog post announcing the compromise of its Orion software—and emailed customers with a preliminary fix. Mandiant and Microsoft followed with their own reports on the backdoor and the activity of the hackers once inside infected networks. Oddly, Mandiant didn’t identify itself as an Orion victim, nor did it explain how it discovered the backdoor in the first place. Reading Mandiant’s write-up, one would never know that the Orion compromise had anything to do with the announcement of its own breach five days earlier.

Monday morning, calls started cascading in to SolarWinds from journalists, federal lawmakers, customers, and government agencies in and outside the US, including president-elect Joe Biden’s transition team. Employees from across the company were pulled in to answer them, but the queue grew to more than 19,000 calls.

The US Cybersecurity and Infrastructure Security Agency wanted to know whether any research labs developing Covid vaccines had been hit. Foreign governments wanted lists of victims inside their borders. Industry groups for power and energy wanted to know whether nuclear facilities were breached.

As agencies scrambled to learn whether their networks used Orion software—many weren’t sure—CISA issued an emergency directive to federal agencies to disconnect their SolarWinds servers from the internet and hold off on installing any patch aimed at disabling the backdoor until the security agency approved it. The agency noted that it was up against a “patient, well-resourced, and focused adversary” and that removing them from networks would be “highly complex and challenging.” Adding to their problems, many of the federal agencies that had been compromised were lax about logging their network activity, which effectively gave cover to the hackers, according to the source familiar with the government’s response. The government “couldn’t tell how they got in and how far across the network they had gone,” the source says. It was also “really difficult to tell what they had taken.”

It should be noted that the Sunburst backdoor was useless to the hackers if a victim’s Orion server wasn’t connected to the internet. Luckily, for security reasons, most customers did not connect them—only 20 to 30 percent of all Orion servers were online, SolarWinds estimated. One reason to connect them was to send analytics to SolarWinds or to obtain software updates. According to standard practice, customers should have configured the servers to only communicate with SolarWinds, but many victims had failed to do this, including Mandiant and Microsoft. The Department of Homeland Security and other government agencies didn’t even put them behind firewalls, according to Chris Krebs, who at the time of the intrusions was in charge of CISA. Brown, SolarWinds’ security chief, notes that the hackers likely knew in advance whose servers were misconfigured.

But it soon became clear that although the attackers had infected thousands of servers, they had dug deep into only a tiny subset of those networks—about 100. The main goal appeared to be espionage.

The hackers handled their targets carefully. Once the Sunburst backdoor infected a victim’s Orion server, it remained inactive for 12 to 14 days to evade detection. Only then did it begin sending information about an infected system to the attackers’ command server. If the hackers decided the infected victim wasn’t of interest, they could disable Sunburst and move on. But if they liked what they saw, they installed a second backdoor, which came to be known as Teardrop. From then on, they used Teardrop instead of Sunburst. The breach of SolarWinds’ software was precious to the hackers—the technique they had employed to embed their backdoor in the code was unique, and they might have wanted to use it again in the future. But the more they used Sunburst, the more they risked exposing how they had compromised SolarWinds.

Through Teardrop, the hackers stole account credentials to get access to more sensitive systems and email. Many of the 100 victims that got Teardrop were technology companies—places such as Mimecast, a cloud-based service for securing email systems, or the antivirus firm Malwarebytes. Others were government agencies, defense contractors, and think tanks working on national security issues. The intruders even accessed Microsoft’s source code, though the company says they didn’t alter it.

victims might have made some missteps, but no one forgot where the breaches began. Anger against SolarWinds mounted quickly. A former employee claimed to reporters that he had warned SolarWinds executives in 2017 that their inattention to security made a breach inevitable. A researcher revealed that in 2018 someone had recklessly posted, in a public GitHub account, a password for an internal web page where SolarWinds software updates were temporarily stored. A bad actor could have used the password to upload malicious files to the update page, the researcher said (though this would not have allowed the Orion software itself to be compromised, and SolarWinds says that this password error was not a true threat). Far worse, two of the company’s primary investors—firms that owned about 75 percent of SolarWinds and held six board seats—sold $315 million in stock on December 7, six days before news of the hack broke, prompting an SEC investigation into whether they had known about the breach.

Government officials threatened to cancel their contracts with SolarWinds; lawmakers were talking about calling its executives into a hearing. The company hired Chris Krebs, CISA’s former head, who weeks earlier had been fired by President Donald Trump, to help navigate interactions with the government.

Meanwhile, Brown and his security team faced a mountain of work. The tainted Orion software was signed with the company’s digital certificate, which they now had to invalidate. But the same certificate had been used to sign many of the company’s other software products too. So the engineers had to recompile the source code for every affected product and sign those new programs with new certificates.

But they still didn’t know where the rogue code in Orion had come from. Malicious code could be lurking on their servers, which could embed a backdoor in any of the programs being compiled. So they ditched their old compilation process for a new one that allowed them to check the finished program for any unauthorized code. Brown says they were under so much stress to get the recompiled programs out to customers that he lost 25 pounds in three weeks.

While Brown’s team rebuilt the company’s products and CrowdStrike tried to figure out how the hackers got into SolarWinds’ network, SolarWinds brought on KPMG, an accounting firm with a computer forensics arm, to solve the mystery of how the hackers had slipped Sunburst into the Orion .dll file. David Cowen, who had more than 20 years of experience in digital forensics, led the KPMG team.

The infrastructure SolarWinds used to build its software was vast, and Cowen and his team worked with SolarWinds engineers through the holidays to solve the riddle. Finally, on January 5, he called Plesco, the DLA Piper attorney. A SolarWinds engineer had spotted something big: artifacts of an old virtual machine that had been active about a year earlier. That virtual machine—a set of software applications that takes the place of a physical computer—had been used to build the Orion software back in 2020. It was the critical puzzle piece they needed.

Forensic investigations are often a game of chance. If too much time has passed since a breach began, traces of a hacker’s activity can disappear. But sometimes the forensic gods are on your side and evidence that should be gone remains.

To build the Orion program, SolarWinds had used a software build-management tool called TeamCity, which acts like an orchestra conductor to turn source code into software. TeamCity spins up virtual machines—in this case about 100—to do its work. Ordinarily, the virtual machines are ephemeral and exist only as long as it takes to compile software. But if part of the build process fails for some reason, TeamCity creates a “memory dump”—a kind of snapshot—of the virtual machine where the failure occurred. The snapshot contains all of the virtual machine’s contents at the time of failure. That’s exactly what occurred during the February 2020 build. Ordinarily, SolarWinds engineers would delete these snapshots during post-build cleanup. But for some reason, they didn’t erase this one. If it hadn’t been for its improbable existence, Cowen says, “we would have nothing.”

In the snapshot, they found a malicious file that had been on the virtual machine. Investigators dubbed it “Sunspot.” The file had only 3,500 lines of code, but those lines turned out to be the key to understanding everything.

It was around 9 pm on January 5 when Cowen sent the file to Meyers at CrowdStrike. The CrowdStrike team got on a Zoom call with Cowen and Plesco, and Meyers put the Sunspot file into a decompiler, then shared his screen. Everyone grew quiet as the code scrolled down, its mysteries slowly revealed. This tiny little file, which should have disappeared, was responsible for injecting the backdoor into the Orion code and allowing the hackers to slip past the defenses of some of the most well-protected networks in the country.

Now the investigators could trace any activity related to Sunspot. They saw that the hackers had planted it on the build server on February 19 or 20. It lurked there until March, when SolarWinds developers began building an Orion software update through TeamCity, which created a fleet of virtual machines. Not knowing which virtual machine would compile the Orion .dll code, the hackers designed a tool that deployed Sunspot into each one.

At this point, the beauty and simplicity of the hack truly revealed itself. Once the .dll appeared on a virtual machine, Sunspot quickly and automatically renamed that legitimate file and gave its original name to the hackers’ rogue doppelgänger .dll. The latter was almost an exact replica of the legitimate file, except it contained Sunburst. The build system then grabbed the hackers’ .dll file and compiled it into the Orion software update. The operation was done in a matter of seconds.

Once the rogue .dll file was compiled, Sunspot restored the original name to the legitimate Orion file, then deleted itself from all of the virtual machines. It remained on the build server for months, however, to repeat the process the next two times Orion got built. But on June 4, the hackers abruptly shut down this part of their operation—removing Sunspot from the build server and erasing many of their tracks.

Cowen, Meyers, and the others couldn’t help but pause to admire the tradecraft. They’d never before seen a build process get compromised. “Sheer elegance,” Plesco called it. But then they realized something else: Nearly every other software maker in the world was vulnerable. Few had built-in defenses to prevent this type of attack. For all they knew, the hackers might have already infiltrated other popular software products. “It was this moment of fear among all of us,” Plesco says.

the next day, January 6—the same day as the insurrection on Capitol Hill—Plesco and Cowen hopped on a conference call with the FBI to brief them on their gut-churning discovery. The reaction, Plesco says, was palpable. “If you can sense a virtual jaw drop, I think that’s what occurred.”

A day later they briefed the NSA. At first there were just two people from the agency on the video call—faceless phone numbers with identities obscured. But as the investigators relayed how Sunspot compromised the Orion build, Plesco says, more than a dozen phone numbers popped up onscreen, as word of what they’d found “rippled through the NSA.”

But the NSA was about to get another shock. Days later, members of the agency joined a conference call with 50 to 100 staffers from the Homeland Security and Justice Departments to discuss the SolarWinds hack. The people on the call were stumped by one thing: Why, when things had been going so well for them, had the attackers suddenly removed Sunspot from the build environment on June 4?

The response from an FBI participant stunned everyone.

The man revealed matter-of-factly that, back in the spring of 2020, people at the agency had discovered some rogue traffic emanating from a server running Orion and contacted SolarWinds to discuss it. The man conjectured that the attackers, who were monitoring SolarWinds’ email accounts at the time, must have gotten spooked and deleted Sunspot out of fear that the company was about to find it.

Callers from the NSA and CISA were suddenly livid, according to a person on the line—because for the first time, they were learning that Justice had detected the hackers months earlier. The FBI guy “phrased it like it was no big deal,” the attendee recalls. The Justice Department told WIRED it had informed CISA of its incident, but at least some CISA people on the call were responding as if it was news to them that Justice had been close to discovering the attack—half a year before anyone else. An NSA official told WIRED that the agency was indeed “frustrated” to learn about the incident on the January call. For the attendee and others on the call who hadn’t been aware of the DOJ breach, it was especially surprising, because, the source notes, in the months after the intrusion, people had been “freaking out” behind closed doors, sensing that a significant foreign spy operation was underway; better communication among agencies might have helped uncover it sooner.

Instead, says the person with knowledge of the Justice investigation, that agency, as well as Microsoft and Mandiant, surmised that the attackers must have infected the DOJ server in an isolated attack. While investigating it in June and July, Mandiant had unknowingly downloaded and installed tainted versions of the Orion software to its own network. (CISA declined to comment on the matter.)

the discovery of the Sunspot code in January 2021 blew the investigation open. Knowing when the hackers deposited Sunspot on the build server allowed Meyers and his team to track their activity backward and forward from that time and reinforced their hunch that the SVR was behind the operation.

The SVR is a civilian intelligence agency, like the CIA, that conducts espionage outside the Russian Federation. Along with Russia’s military intelligence agency, the GRU, it hacked the US Democratic National Committee in 2015. But where the GRU tends to be noisy and aggressive—it publicly leaked information stolen from the DNC and Hilary Clinton’s presidential campaign—SVR hackers are more deft and quiet. Given various names by different security firms (APT29, Cozy Bear, the Dukes), SVR hackers are noted for their ability to remain undetected in networks for months or years. The group was very active between 2014 and 2016, Glyer says, but then seemed to go dark. Now he understood that they’d used that time to restrategize and develop new techniques, some of which they used in the SolarWinds campaign.

Investigators found that the intruders had first used an employee’s VPN account on January 30, 2019, a full year before the Orion code was compromised. The next day, they returned to siphon 129 source code repositories for various SolarWinds software products and grabbed customer information—presumably to see who used which products. They “knew where they were going, knew what they were doing,” Plesco says.

The hackers likely studied the source code and customer data to select their target. Orion was the perfect choice. The crown jewel of SolarWinds’ products, it accounted for about 45 percent of the company’s revenue and occupied a privileged place in customer networks—it connected to and communicated with a lot of other servers. The hackers could hijack those connections to jump to other systems without arousing suspicion.

Once they had the source code, the hackers disappeared from the SolarWinds network until March 12, when they returned and accessed the build environment. Then they went dark for six months. During that time they may have constructed a replica of the build environment to design and practice their attack, because when they returned on September 4, 2019, their movements showed expertise. The build environment was so complex that a newly hired engineer could take months to become proficient in it, but the hackers navigated it with agility. They also knew the Orion code so well that the doppelgänger .dll they created was stylistically indistinguishable from the legitimate SolarWinds file. They even improved on its code, making it cleaner and more efficient. Their work was so exceptional that investigators wondered whether an insider had helped the hackers, though they never found evidence of that.

Not long after the hackers returned, they dropped benign test code into an Orion software update, meant simply to see whether they could pull off their operation and escape notice. Then they sat back and waited. (SolarWinds wasn’t scheduled to release its next Orion software update for about five months.) During this time, they watched the email accounts of key executives and security staff for any sign their presence had been detected. Then, in February 2020, they dropped Sunspot into place.

On November 26, the intruders logged in to the SolarWinds VPN for the last time—while Mandiant was deep into its investigation. The hackers continued to monitor SolarWinds email accounts until December 12, the day Kevin Mandia called Kevin Thompson to report the backdoor. Nearly two years had passed since they had compromised SolarWinds.

steven adair, the Volexity CEO, says it was pure luck that, back in 2019, his team had stumbled on the attackers in a think tank’s network. They felt proud when their suspicion that SolarWinds was the source of the intrusion was finally confirmed. But Adair can’t help but rue his missed chance to halt the campaign earlier. “We were so close,” he says.

Mandiant’s Carmakal believes that if the hackers hadn’t compromised his employer, the operation might have gone undetected for much longer. Ultimately, he calls the SolarWinds hacking campaign “a hell of an expensive operation for very little yield”—at least in the case of its impact on Mandiant. “I believe we caught the attackers far earlier than they ever anticipated,” he says. “They were clearly shocked that we uncovered this … and then discovered SolarWinds’ supply chain attack.”

But given how little is still known publicly about the wider campaign, any conclusions about the success of the operation may be premature.

The US government has been fairly tight-lipped about what the hackers did inside its networks. News reports revealed that the hackers stole email, but how much correspondence was lost or what it contained has never been disclosed. And the hackers likely made off with more than email. From targeting the Departments of Homeland Security, Energy, and Justice, they could plausibly have accessed highly sensitive information—perhaps details on planned sanctions against Russia, US nuclear facilities and weapons stockpiles, the security of election systems, and other critical infrastructure. From the federal court’s electronic case-files system, they could have siphoned off sealed documents, including indictments, wiretap orders, and other nonpublic material. Given the logging deficiencies on government computers noted by one source, it’s possible the government still doesn’t have a full view of what was taken. From technology companies and security firms, they could have nabbed intelligence about software vulnerabilities.

More concerning: Among the 100 or so entities that the hackers focused on were other makers of widely used software products. Any one of those could potentially have become a vehicle for another supply chain attack of similar scale, targeting the customers of those companies. But few of those other companies have revealed what, if anything, the hackers did inside their networks. Why haven’t they gone public, as Mandiant and SolarWinds did? Is it to protect their reputations, or did the government ask them to keep quiet for national security reasons or to protect an investigation? Carmakal feels strongly that the SolarWinds hackers intended to compromise other software, and he said recently in a call with the press that his team had seen the hackers “poking around in source code and build environments for a number of other technology companies.”

What’s more, Microsoft’s John Lambert says that judging by the attackers’ tradecraft, he suspects the SolarWinds operation wasn’t their first supply chain hack. Some have even wondered whether SolarWinds itself got breached through a different company’s infected software. SolarWinds still doesn’t know how the hackers first got into its network or whether January 2019 was their first time—the company’s logs don’t go back far enough to determine.

Krebs, the former head of CISA, condemns the lack of transparency. “This was not a one-off attack by the SVR. This is a broader global-listening infrastructure and framework,” he says, “and the Orion platform was just one piece of that. There were absolutely other companies involved.” He says, however, that he doesn’t know specifics.

Krebs takes responsibility for the breach of government networks that happened on his watch. “I was the leader of CISA while this happened,” he says. “There were many people in positions of authority and responsibility that share the weight here of not detecting this.” He faults the Department of Homeland Security and other agencies for not putting their Orion servers behind firewalls. But as for detecting and halting the broader campaign, he notes that “CISA is really the last line of defense … and many other layers failed.”

The government has tried to address the risks of another Orion-style attack—through presidential directives , guidelines , initiatives , and other security-boosting actions . But it may take years for any of these measures to have impact. In 2021, President Biden issued an executive order calling on the Department of Homeland Security to set up a Cyber Safety Review Board to thoroughly assess “cyber incidents” that threaten national security. Its first priority: to investigate the SolarWinds campaign. But in 2022 the board focused on a different topic , and its second investigation will also not be about SolarWinds . Some have suggested the government wants to avoid a deep assessment of the campaign because it could expose industry and government failures in preventing the attack or detecting it earlier.

“SolarWinds was the largest intrusion into the federal government in the history of the US, and yet there was not so much as a report of what went wrong from the federal government,” says US representative Ritchie Torres, who in 2021 was vice-chair of the House Committee on Homeland Security. “It’s as inexcusable as it is inexplicable.”

At a recent conference, CISA and the US’s Cyber National Mission Force, a division of Cyber Command, revealed new details about their response to the campaign. They said that after investigators identified Mandiant’s Orion server as the source of that firm’s breach, they gleaned details from Mandiant’s server that allowed them to hunt down the attackers. The two government teams implied that they even penetrated a system belonging to the hackers. The investigators were able to collect 18 samples of malware belonging to the attackers—useful for hunting for their presence in infected networks.

Speaking to conference attendees, Eric Goldstein, the leader for cybersecurity at CISA, said the teams were confident that they had fully booted these intruders from US government networks.

But the source familiar with the government’s response to the campaign says it would have been very difficult to have such certainty. The source also said that around the time of Russia’s invasion of Ukraine last year, the prevailing fear was that the Russians might still be lurking in those networks, waiting to use that access to undermine the US and further their military efforts.

Meanwhile, software-supply-chain hacks are only getting more ominous. A recent report found that in the past three years, such attacks increased more than 700 percent.

This article appears in the June 2023 issue. Subscribe now .

Let us know what you think about this article. Submit a letter to the editor at [email protected] .

You Might Also Like …

Scoop : Joe Biden's campaign team were told he was leaving the race in an email telling them to check X

J.D. Vance left his Venmo public. Here's what it shows

If you were affected by the Crowdstrike outage, don't fall for these scams

Navigate election season with our WIRED Politics Lab newsletter and podcast

She made $10,000 a month defrauding apps like Uber and Instacart. Meet the queen of the rideshare mafia

Give your back a break: Here are the best office chairs we’ve tested

How SolarWinds Responded to the 2020 SUNBURST Cyberattack

Brian Kenny:

In the time it takes me to read this introduction, two firms will be hacked. Every 39 seconds, 24 hours a day, 352 days a year, there's a data breach. 120 million individuals in the US had their personal information exposed in data breaches in the past year, and one in three became victims of identity theft, which makes it easy to see why 42 percent of consumers have low or no trust that companies will keep their personal information safe. And 59 percent think that companies care more about profiting from their customer data than protecting it.

More than 80 percent of US firms say they have been hacked at some point and almost always as a result of human error. Data breaches, it seems, are inevitable, but how we respond to them can make or break customer relationships. Today on Cold Call , we welcome Professor Frank Nagle to discuss the case, “SolarWinds Confronts SUNBURST.” I'm your host, Brian Kenny, and you're listening to Cold Call on the HBR Podcast Network. Frank Nagle's research looks at the future of work, the economics of IT and digital transformation. He is a first-time visitor to Cold Call . Frank, thanks for joining me today.

Frank Nagle:

It's great to be here. Thanks for having me.

I do think this is a highly relatable case as somebody who has had their personal information stolen a bunch of times. I think everybody has probably experienced that in one way or another. So, why don't we just dive right in? I'm going to ask you to start by telling us what Jason Bliss, the protagonist in this case, what does Jason learn as the case opens, and what's your cold call to start the discussion in class?

Sure, Brian. So at the opening of the case, Jason Bliss, who is SolarWinds' general counsel at the time, has just learned from the company's outgoing CEO, who's leaving a few weeks later, that the company experienced a substantial, significant cybersecurity breach that allowed attackers direct access into the IT environments of potentially all of SolarWinds' 275,000 customers. At that point, they don't know how big it is, but that's what he's just learned. And so the way I think about opening this case in class, I usually set the stage. I talk a little bit about SolarWinds, the company, and then give the famous Mike Tyson quote, "Everyone has a plan until they get punched in the mouth." And that's essentially what happens here, is they had a pretty decent plan and they get walloped in the mouth in a way they weren't expecting.

And so I start the first cold call by asking the students, was SolarWinds really unlucky and this was just a fluke type of thing? Or were they unprepared or at least underprepared and this was more their fault? And that usually opens up a nice discussion because there's good arguments on both sides in this instance.

Yeah, and it's a pretty dramatic case too. I love cases that start with this kind of a narrative where there's an incident that's happening and unfolding right in front of you. Why did you decide to write about this case? Why was it important to you and the kinds of things you think about as a scholar?

Before becoming a scholar and returning to the academic world, I actually worked in cybersecurity for almost a decade at various companies, including the last one was Mandiant, which actually the CEO of that company that was later bought by FireEye ends up being a player in this case. So my old boss is a player in this case. And having that background, now I study things more related to open source software, but still a little bit about cybersecurity. There's not a whole lot of teaching cases on cybersecurity. There's a couple and they focus on different aspects. But in particular, this SolarWinds attack here was one of the most complex attacks that anybody has ever seen. And so bringing that into the classroom to show students, you talked about having your data stolen, which I think lots of people can relate to, and that's one type of attack. The customer data is stolen or financial data is stolen or something like that. But this type of supply chain attack that we see in SolarWinds is something that had very rarely been seen at that point, and this was the most extreme example of it. So we thought it would be a good teaching case to bring into the classroom to give this extreme version of what unfortunately is becoming something that's even more common today.

That's a great segue into the next question I have, which is just, what is SolarWinds' business? Who are their customers and how do they work with them?

What SolarWinds does is they make these small IT management tools. So if you're a network engineer or running a company's IT infrastructure, you are likely going to use their tools to really help you see into the network yourself, to understand what's going on on your network, who's doing what, and where's the traffic flowing, and things like that. And so they have lots of these different types of tools that help you do that. And so what's interesting about them is these tools are easy to use and they solve specific use cases. And so often they'd have different customers within the same company. One small IT group over here may buy some of their tools for a couple thousand dollars. A different IT group may buy a different IT tool of theirs for a few other thousand dollars. And so that ends up actually playing a role later on because they had lots of customers within the same company or within the same governmental organization.

Yeah, it was interesting. As I was reading the case, I could see you were sprinkling all these little details throughout, and they all came to fruition as you read further on the case because it makes the whole thing much more complicated for SolarWinds as they're trying to figure out what's the scope of this situation we're dealing with, including their sales strategy. Can you talk a little bit about the way they approach sales?

You know, their sales strategy was an outgrowth of their startup days, and it was mostly inside sales. Once they have somebody that's bought one of their tools, they try to upsell them and get them on other tools, but they didn't really have this kind of, again, centralized sales force that was going after companies as a whole. And that comes back later because when this big breach happens, they don't know who the right people at their customers are to talk to. There's 15 different people that they're connected to in a given organization and they don't know where to start. And obviously they have to try with all of them and contact all of them, and that slows down their response.

One of the many lessons that I think comes out of this case is that you really do need to know who your contact people are at your customer sites.

That's right.

And then to add to the drama, they were going through a leadership transition. They were just about to leave from one CEO and welcome another. Can you talk about that dynamic?

Yeah. I mean, this is one of the things we see in cybersecurity events. Inevitably, the attack happens at the worst possible time and that's the way it goes. So their outgoing CEO Kevin Thompson had been there for 10 years, had done a great job, really helped with this growth trajectory. He wasn't one of the founders. He came in after the founders decided it was time to bring in a more professional CEO. And so he had announced his pending retirement. The board had already found and identified a new CEO who was going to come on board on January 1st. The attack or SolarWinds discovers the attack or is told about the attack in December 12, so literally in the midst of the holiday season, which that always happens, too.

Friday afternoon at 4:00 before a holiday.

Exactly. Right. Exactly. And B, it's not like the new CEO has just gotten on board, but obviously the transition is in progress. The old CEO is winding things down and the new CEO has yet to start winding things up because he's not an employee yet. He's not even at the company. And so that is a pretty tricky time. And that's actually why often we have protagonists being CEOs, but Jason Bliss, who's again, the general counsel, ends up being the protagonist because of the CEO transition. And we actually have a small B case where the new CEO ends up being the protagonist.

We'll talk a little bit about that. I think that dynamic is really, really interesting. But maybe we can pull the lens back a little bit and just talk... I mean, I teased in the intro about cyber attacks and the frequency with which they occur. Those numbers are probably a moving target, but maybe you can talk a little bit about what's the financial impact of these things? And more specifically, what was the nature of the attack at SolarWinds?

At the high level of this kind of attacks, it really depends on the company and depends on the type of breach. But a 2023 IBM report said that the average breach cost that they saw in their work was about $5 million US dollars. Obviously there's a huge range there. But then some numbers from an IC3 report in 2022 said that in the US, cyber attacks cost companies $10 billion. These numbers aren't on the order of GDP, they're not trillions of dollars, but they're certainly growing and have been growing for many years. What's particularly concerning is that the attacks are becoming more prevalent and more sophisticated at the same time, and that's what we see in the SolarWinds attack. So this what became known as the SUNBURST attack was highly sophisticated. There were estimates that there were over 1,000 professional Russian hackers, state backed, government sponsored hackers that went into this. And that's just something that's very hard to prepare for. I mean, again, SolarWinds had a pretty good incident response plan for things like malware or ransomware or if customer data leaked out to the public or something like that, they knew what to do. They didn't have quite a plan for what happens if 1,000 government backed hackers come after you. The attack is what we call in the industry a supply chain attack. So SolarWinds itself wasn't actually the end target of we'll call them the bad guys and who was was all of SolarWinds' customers. So again, SolarWinds had a clientele list. In the case we talk about, they had I think 80 percent of the Fortune 500, most of the US governments, all the different departments of the government, many foreign governments as well. And so you can imagine how a company that's a keystone like that and has all these useful customers to an attacker might be a target of attack itself. And so a supply chain attack is when we see the attackers come in and breach, in this case, one of SolarWinds' products. Insert a backdoor into the SolarWinds product that then SolarWinds distributes to its customers. And of course, its customers trust it because it's coming from a known entity. And so when we see these supply chain attacks, they can happen at different points in the software development process. And where it happened in the SolarWinds one was in what's called the build process.

You have coders who code and they write their software, and then it all gets compiled and turned into what we end up using is the executable piece of software at the end through the build process. And so it was during that build process that the attackers got in and were able to stick the bad code in. And so if you looked at the raw source code that the developers wrote, it all looked good. There were no backdoors, no issues there. And if you looked at the end result, it was something that looked like exactly what it was supposed to coming from SolarWinds because they hit that very last step in the chain, in the software development lifecycle during this build process, and that's where they got it in. And so essentially what they did was put in a backdoor into this. And so if you installed SolarWinds' Orion product at a certain time, certain version numbers, then the Russian attackers... I think it hasn't been fully proven it was the Russians. Yes. The FBI has pointed the finger at the Russians, so we'll take them at their word. And so a very complicated attack that really ended up with all of SolarWinds' customers at least with that version of that software had a backdoor into their environment that the attackers could take advantage of.

Super sophisticated and really alarming given the nature of what SolarWinds does, which if they have insight and a view across their customer's IT infrastructure, then you might think that the bad actors would have the same.

Exactly the same. That's right. That's right.

So take us inside the incident and tell us, these things unfold obviously in real time. Anybody who's ever managed a crisis of any kind knows that you get a certain amount of information in the beginning, it may or may not all be right, and then you've got to act, so you can't wait. But at the same time, you're not sure what you exactly should do because you're still waiting for more information. So it sounded like that was the dynamic here.

Oh yes, absolutely. And they actually had even more information than you would normally have in this from the get-go, because who discovered this breach originally was, again, my old employer Mandiant, which was a division of FireEye. They found this in their own environment because they were a SolarWinds customer, and they investigated it and figured out what the malware was doing, how the backdoor worked and things like that before they brought it to SolarWinds.

When they came to SolarWinds a day or two after they had discovered it, they had a whole lot of information that they could give to SolarWinds, which made SolarWinds' job much easier because then they knew what they had to fix and they could identify what versions had that. And so they were able to move fairly rapidly, but that was partially because of how they found out about it. I will say that that may be surprising to some folks in the audience that they didn't find out about it themselves. They were told about it by a customer. That's actually I think something like 70 to 80 percent of these types of events you find out either from a customer or a law enforcement agency discovers it and lets you know that you have a problem in your network. And so often that's the first way of finding out is you get a phone call from somebody that's outside your organization that says, "You have a problem." And that's how this turned out.

And so for them at this early stage, they had a bunch of information about what it was, and then they had to go look at their software version history to understand when did this first happen. And then they could say, "All right, everybody before that version is okay. Everybody after that version potentially has a problem." And they quickly identified that of those 275,000 customers, there's about 18,000 of them at least potentially have that software, that version that has the backdoor.

So then the problem becomes that because of this distributed sales model and also the fact that the way that they sell their software is not software as a service, but instead is on-premise, so you as a customer buy this software and put it in your environment and manage it yourself, they don't have clear insight into exactly what version everybody is running. So they're kind of doing some guesswork at that point, which, of course, makes things a little bit fuzzier.

And at the same time, they're obligated, I think, by law to inform customers about what's happening. And one of the criticisms that we often hear and these things after the fact is, why did you wait so long to tell me? Why were you sitting on this for 48 or 72 hours or whatever the number might be? How did they manage that?

As you pointed out, this attack happened in late 2020, so only about three years ago. And even since then, back then there was really a patchwork of legal legislation and mostly at the state level in the US. And so it was a little bit unclear exactly what they had to do and who they had to report to. Today, the laws are a little bit clearer and there's more effort to nationalizing the rules around privacy breaches and cyber breaches and things like that. And so that said, at the moment they knew, because they were a public company, one of the main things they had to do was issue what's called an 8-K statement. And so this is a statement required by the SEC when a material event happens to a company. Doesn't have to be a cyber breach, could be loss of a major customer or a natural disaster wipes out a manufacturing plant or something like that. They have four business days to issue this public statement saying, "Here's what we know and here's what happened and here's the likely impact on our business."

So that was really what they organized around at first was making sure they could make that public statement as required by the SEC and use that as their anchor for talking with their customers. And so they get the phone call. I want to say it's on Friday night I believe is when they...

Of course. Of course, it is.

Of course. Exactly. Friday at 4:00 PM. They have four business days, but they decide to be a bit more aggressive and their goal is to release it before the markets open on Monday morning. So markets are already closed, which actually is a pretty good thing for them. They have some time over the weekend to get things together and figure out what's going on. But at the same time, they also know there's a ticking clock because they get word that The Washington Pos t has wind of this and is going to release a story as well. FireEye itself is going to release something because its customers are affected as well. And so there's really a ticking clock here. And as a result, they have even less information and less details.

This was all happening during COVID, is that right?

That's right. That's right.

We'll throw another wrench into the thing. So how do you organize a team in essentially a war room type environment? Because this is all hands on deck, I would imagine.

How do you do that when everybody's scattered?

That's the thing, right? This was December 2020, so no vaccines yet. We're still well into COVID, work from home. Almost the whole company's been working from home. They're lucky enough that they have an auditorium in their building. So they stick most of the people in the auditorium with what they end up calling a “Tiger Team,” which is essentially the department heads that are organizing at the top level to manage the whole operation, working in a smaller boardroom in the building.

So let's go back to the dual CEOs. We've got Thompson, who is obviously on the scene, he's the current CEO, and then we've got Ramakrishna, who has been appointed CEO. How does he plug into this?

Ramakrishna, lucky for SolarWinds, actually had a cyber security background. His most recent job was the CEO of Pulse Secure, which is a VPN company. But he's not a part of the company yet, and so they can't tell him anything. And so they end up calling him during his birthday party, which of course, again, worst timing ever. And they say, "There's a big problem and you need to know some stuff. So we need to get you in the loop somehow." Meanwhile, the current CEO and the board chair basically say to him, "Look, this is not what you signed up for. So if you want to bail, we get it." But he says, "No, I get it. Whatever this is that I don't know the details about is going to be really bad." And so he takes it in stride and says, "Let's get going." And so the easiest way ends up actually being for them to appoint him to the board before his turn as CEO starts taking over so that he can then start accessing information. So within the first few days, that's what they do. Ramakrishna becomes a board member, even though he's not CEO yet. And he told us in our interviews that what he did was just for the first two days, he just listened to soak up information about not only the event, but also the company, to get a better sense of what the company stood for and how they prioritize things. And then on the third day, he starts offering advice. And so he brings that background from cybersecurity experience at other companies and starts helping deal with external relations and dealing with some of the customers and helping soothe some nerves by saying, "Look, we're working on it and we're going to help our customers first." And that was kind of a joint decision because SolarWinds really consider themselves a very customer oriented company. I know lots of companies would say that, but they're very customer focused. And so they make these decisions to prioritize the customers and help them get fixed. And that ends up being both Thompson and Ramakrishna's priorities. And so from the discussions with them, it really sounds like they were on the same wavelength, which is good because you can certainly imagine contentious CEOs.

Thompson was leaving on good terms, so it wasn't like he was being forced out or anything like that. So that helped as well. But again, complicated because your top level leadership is one's on the way out and one's on the way in.

A huge trial by fire, not just in terms of his subject matter expertise, but his character and the fact that he didn't walk out or turn and run away from this. He ran into it, which is pretty amazing, and says a lot to your employees too as they see you even before you're officially on the job.

I think that's right. He definitely had the faith of the employees before he even showed up on day one.

I've got to believe that's true. You talk about the customers and customer first approach that they took. Not all of their customers are the same in the sense that they've got obviously private firms that they do business with, but they've got a lot of government customers. I would imagine that the stakes there are almost even higher if they've got DOD or you name whatever department. The secondary impact of this could have been really, really significant. Did they deal differently with those sets of customers?

Yeah. That was, again, this customer-centric focus, right? They wanted to try and talk to every single customer they had, but they had to prioritize. And knowing, again, from an early stage because of the work that FireEye had done and because of some law enforcement discussions, they knew that this was likely a state attacker and that it was likely that the government customers were the real targets. And so it's possible or likely that many of their customers had infected versions of the software, but that they weren't really the targets of what was happening here. So they end up prioritizing the government clients, both the US and other countries as well, because there was some evidence that it wasn't just US customers that were the focus. And so they set up this tiered customer response approach. And so because of the technical details, they're able to at least narrow down, we think this set of 18,000 customers are the most likely ones to have this infected version.

And then of that subset, focusing on government and other marquee or high potential customers that they believe that are the true targets of this attack. They focus on them and helping them figure out which version are you running first of all. Because if they're not actually running an infected version, then they're okay. But if they are, then help them shut it down and update it and patch it and get it fixed to get the bad guys out of the network as soon as possible.

They state that 18,000 number in the 8-K when they released that on Monday morning, but knowing that that was an overstatement of how many customers were actually not only affected, but likely being breached. So you can think about this in concentric circles. There's 18,000 that might have the infected software. Of that, some subset, probably thousands, have the infected software. And then of that, some subset that ends up only being about a hundred are actively being exploited by the attackers.

Okay, but I'm sure that didn't do much to assuage all of their customers who thought, "I may not be in the 18,000, but how do I know my information is safe?"

And that's right. I think that that was a substantial concern. They shift their customer service organization to be basically a panic room and dealing with trying to talk to as many of the customers as possible in a way that they wanted to the extent that they could avoid having just an automated email blast, which when you have almost 300,000 customers is hard to do. And so they did a little bit of automated, but then they really set up this big operation to try and touch and talk to every single customer that they had.

Obviously some had more priority based on whether or not they were in the affected group and who they were, but they tried to touch every customer and actually say, "Hey, look, we know you're running X version. That's not affected. We appreciate that you're concerned about this, and here are some things you can do and update to the latest version anyways and we'll help you do that." And I think one of the things that came out of this was that customer approach really ended up saving them at least from the customer side, because that year or the following year, 96 percent of their customers still re-upped their contracts and stayed there. And that's against a baseline of 98 percent. So normal year, 98 percent would renew, and that year was only 96 percent. They only lost a few handful of customers as a result of this, and I think that that customer focus throughout all of this really saved them from that perspective.

Yeah, no doubt. Why don't we fast-forward a little bit now to the B case as you mentioned before?

This was a little bit where they took a look back and said, "What worked well here? What didn't?" Can you give some of the highlights of that?

The B case takes place I think it's roughly a month or so later. At that point, Ramakrishna is now CEO and he's having to spend a whole lot of his time doing something he wasn't really anticipating when he signed up for the job. But I think they make two main pieces of their customer response, of their industry response of how they're trying to shape this going forward. One is what they call the Orion Assistance Program where they're really helping customers that have this affected software update, patch. So they really make sure everybody's got the latest version and doesn't have affected software. So that's part one, which as you can imagine is the early stage. And then in the moderate stage, the middle stage, they introduced this concept of secure by design. And so they really revamped their whole development process such that security is baked in from the beginning. It's only a few years ago, but this really helped the industry understand that we can't just latch on security at the end, which was very much the way that software development was done for a long time. The shift of focus on secure by design is something that Ramakrishna helps introduce and really re-orients the way that the company makes its products and bakes in security from the very beginning so that this super complicated type of breach can't happen or is less likely to happen. And also more simple run-of-the-mill bugs in the software that might cause a problem down the road are also less likely to happen.

And then he goes on the road, and actually with my former boss, the CEO of FireEye, but also the head of CISA, which is the government agency in charge of managing these types of events. They are speaking at conferences and all sorts of things about this secure by design principle and to really proselytize to the industry as a whole that we're in a new world and we need to all really be much more focused on security from the ground up, not just layering it on later.

And the bad actors, as we know, are always out there trying to think about the next vulnerability that they can exploit somewhere down the line. So it just never ends.

Exactly. Unfortunately, it's a never-ending game of cat and mouse. It's a fascinating world to be involved in, but the bad guys are always going to think of some new way to get in. When I worked in the industry, my main job was red teaming, so it was doing the breaking in to check defenses and test defenses. And the tough thing is is that as a bad guy or as a pretend bad guy, in that case, you only need to find one way in. You need to find just one weakness that lets you in the door and then you can go from there. As the defender and the good guys, you really have to try to cover every single base. And I think this is one thing too that we've seen as a shift over time in the industry. When I started, it was all about protect the perimeter, keep the bad guys out, don't let them in. Nowadays, what most places have embraced is a concept called zero trust, where we assume that we're breached and have a lot more internal defenses and trust. There is no trust. We verify every single thing that's happening inside the organization as well. So this is why all of us that are listening to this have to deal with more two-factor authentication or entering your password more times or VPN-ing for everything or things like this.

No matter how irritating that is.

It's necessary even if it's slightly annoying.

You did mention to me that there's been some late-breaking developments that aren't part of the case obviously with the SEC and charges brought against the SolarWinds chief information security officer, which seems like, I don't know, a game changer in some ways. And I'm wondering if you can talk a little bit about what the implications are for people in that role.

And so just a few weeks ago, the details are still unfolding, but the SEC brought new charges against the company and against the chief information security officer in particular for not being fully transparent and for knowing some things that they didn't tell the public or tell their customers or tell their shareholders about. Now, I'm not here to judge on that. I don't know what the truth is. I'm sure we will find out. But indeed, I think we're seeing more of these types of charges being brought against either CISOs or VPs of security, people that are running the security operation at companies.

About a year ago, a similar type of charge was brought against folks at Uber. There I think it was a little more open and shut that there was clearly some things that they knew that they didn't tell the public and things like that. But more broadly speaking, it's concerning because we're only going to see more and more cyber attacks. And so knowing that qualified people are leaving these positions because of fear of this gray space and ending up on the wrong side of it and ending up in jail is a broader concern. So I think what can we do about that, regulatory clarity so that it's much clearer for folks what they need to do and when they need to do. Is it four business days? Is it four days? Is it 48 hours? These types of things. And consistency and clarity will go a long way in helping CISOs feel secure in that they're doing the right thing and doing what they're supposed to do.

I think we all want the best people in those roles because really it impacts everybody. It could potentially impact everybody.

It does. Absolutely, right? Cybersecurity matters for everyone. Even if you don't know anything about it, you're still maybe affected by it.

Frank, this has been a great conversation. I've got one more question for you and that is simply, if you want our listeners to remember one thing about the SolarWinds case, what would it be?

For those that are in positions of not even cybersecurity positions, but just in managerial positions, having a plan is super important because it allows you to act faster when these types of things happen. And I think in SolarWinds case, if they didn't have this pretty good incident response plan in place, everything would've been much worse.

And for everybody listening, change your passwords.

Regularly. That's right.

Frank, thanks for joining me on Cold Call .

Thanks so much for having me. It was great to be here.

If you enjoy Cold Call , you might like our other podcasts, After Hours , Climate Rising , Deep Purpose , Idea Cast , Managing the Future of Work , Skydeck , and Women at Work . Find them on Apple, Spotify, or wherever you listen, and if you could take a minute to rate and review us, we'd be grateful. If you have any suggestions or just want to say hello, we want to hear from you. Email us at [email protected] . Thanks again for joining us. I'm your host, Brian Kenny, and you've been listening to Cold Call , an official podcast of Harvard Business School and part of the HBR Podcast Network.

  Read more

  Close

• 25 Jun 2024
• Research & Ideas

Rapport: The Hidden Advantage That Women Managers Bring to Teams

• 11 Jun 2024
• In Practice

The Harvard Business School Faculty Summer Reader 2024

How transparency sped innovation in a $13 billion wireless sector.

• 24 Jan 2024

Why Boeing’s Problems with the 737 MAX Began More Than 25 Years Ago

• 27 Jun 2016

These Management Practices, Like Certain Technologies, Boost Company Performance

• Crisis Management
• Information Management
• Crime and Corruption
• Information Technology
• North America

Sign up for our weekly newsletter

• Work & Careers
• Life & Arts

The great hack attack: SolarWinds breach exposes big gaps in cyber security

To read this article for free register now.

Once registered, you can:

• Read free articles
• Get our Editor's Digest and other newsletters
• Follow topics and set up personalised events
• Access Alphaville: our popular markets and finance blog
• Global news & analysis
• Expert opinion
• Special features
• FirstFT newsletter
• Videos & Podcasts
• Android & iOS app
• FT Edit app
• 10 gift articles per month

Explore more offers.

Standard digital.

• FT Digital Edition

Premium Digital

Print + premium digital, ft professional, weekend print + standard digital, weekend print + premium digital.

Then $75 per month. Complete digital access to quality FT journalism. Cancel anytime during your trial.

• Global news & analysis
• Exclusive FT analysis
• FT App on Android & iOS
• FirstFT: the day's biggest stories
• 20+ curated newsletters
• Follow topics & set alerts with myFT
• FT Videos & Podcasts
• 20 monthly gift articles to share
• Lex: FT's flagship investment column
• 15+ Premium newsletters by leading experts
• FT Digital Edition: our digitised print edition
• Weekday Print Edition
• Videos & Podcasts
• Premium newsletters
• 10 additional gift articles per month
• FT Weekend Print delivery
• Everything in Standard Digital
• Everything in Premium Digital

Today's FT newspaper for easy reading on any device. This does not include ft.com or FT App access.

• 10 monthly gift articles to share
• Everything in Print
• Make and share highlights
• FT Workspace
• Markets data widget
• Subscription Manager
• Workflow integrations
• Occasional readers go free
• Volume discount

Essential digital access to quality FT journalism on any device. Pay a year upfront and save 20%.

Terms & Conditions apply

Explore our full range of subscriptions.

Why the ft.

See why over a million readers pay to read the Financial Times.

Judge Guts SEC Case Against SolarWinds Over Cyber Practices

By Cassandre Coyer and Jeff Stone

A US federal judge dismissed much of the Securities and Exchange Commission ’s lawsuit against SolarWinds Corp. that alleged the software provider misled investors about its cybersecurity practices and the significance of a major data breach that spilled into the US government.

Thursday’s ruling was seen as a blow to the SEC’s aggressive efforts to regulate the cybersecurity practices of publicly traded companies, actions that had created significant angst among the private sector and from security practitioners.

But the agency’s case wasn’t entirely dismissed. US District Judge Paul Engelmayer of the Southern District of New York allowed the SEC to move forward with a claim that SolarWinds committed securities fraud with a statement about the Austin-based company’s cyber preparedness. However, in his decision Thursday, Engelmayer threw out the SEC’s allegations that other statements and filings were misleading, and its claims that the company had mininized the scope and severity of a major hack that was disclosed in December 2020.

Judge Engelmayer also dismissed at least some claims against SolarWinds’ Chief Information Officer Timothy Brown, whom the SEC accused of intentionally failing to disclose the company’s expansive security vulnerabilities in From 8-K filings during the months leading up to and after the Russian intrusion.

But Engelmayer found that SolarWinds’ executives and Brown’s bosses were ultimately the parties responsible for crafting and signing the disclosures, not Brown himself.

The SEC’s complaint failed to claim that “the officers who approved the cybersecurity risk disclosure understood it was misleading,” he said. “These executives, not Brown, appear to have had ultimate authority over the company’s risk disclosure.”

Engelmayer upheld claims over Brown’s role in the company’s allegedly misleading security statement about SolarWinds’ practices before the hacking disclosures. The rest of the claims against Brown over his public statements in company-approved press releases, blog posts, podcasts and the disclosures made in the Forms S-1 and 8-Ks were dismissed.

“I think the SolarWinds case is a bellwether action,” said Jennifer Lee, a partner at Jenner & Block and a former SEC official. “I absolutely think that the SEC will be looking at this decision, thinking about how it might want to refine its theories, and also consider just how it wants to move forward in terms of enforcement actions.”

Michael Borgia, a partner at Davis Wright Tremaine LLP , said, “I think it’s a bit of a slap down certainly.”

“I do not think this spells a more sort of reticent, timid SEC in the cyber enforcement space,” he said. “I think they’ll dust themselves off and keep going because clearly they think this is a significant priority.”

Gerry Stegmaier , a partner at Reed Smith LLP , said any sighs of relief over the decision may be premature. “The SEC continues to remain very active in looking at cybersecurity generally and companies’ incident response,” he said, in a statement.

Russian hackers breached SolarWinds by inserting malicious code into a software update that was sent to its customers. The hackers then used the malware as a backdoor for further intrusions on a relatively small number of them, including dozens of companies and at least nine government agencies. The breach was revealed in December 2020.

“We are pleased that Judge Engelmayer has largely granted our motion to dismiss the SEC’s claims. We look forward to the next stage, where we will have the opportunity for the first time to present our own evidence and to demonstrate why the remaining claim is factually inaccurate,” a SolarWinds spokesperson said in a statement. “We are also grateful for the support we have received thus far across the industry, from our customers, from cybersecurity professionals, and from veteran government officials who echoed our concerns, with which the court agreed.”

An SEC spokesperson declined to comment.

(Updates with quote from Gerry Stegmaier in 11th paragraph. A previous version of this story mispelled Judge Paul Engelmayer’s name.)

To contact the reporters on this story: Cassandre Coyer in Arlington at [email protected] ; Jeff Stone in New York at [email protected]

To contact the editors responsible for this story: Andrew Martin at [email protected]

Adam M. Taylor

© 2024 Bloomberg L.P. All rights reserved. Used with permission.

Learn more about Bloomberg Law or Log In to keep reading:

Learn about bloomberg law.

AI-powered legal analytics, workflow tools and premium legal & business news.

Already a subscriber?

Log in to keep reading or access research tools.

• [email protected]
• Call 417-881-8333

Cyber Case Study: SolarWinds Supply Chain Cyberattack

In the final month of 2020, it was revealed that foreign hackers had orchestrated a supply chain cyberattack throughout the past year in an effort to compromise several federal agencies and private organizations. The cybercriminals first infiltrated the digital infrastructure of SolarWinds—a Texas-based technology company—before using that infrastructure to gain access to sensitive data from a range of government departments and organizations via malware-ridden software updates. The incident ultimately exploit- ed numerous SolarWinds customers and led to millions of dollars in total losses.

The attack has been dubbed as one of the largest and most sophisticated cyber incidents in U.S. history, motivating many organizations to take a closer look at secur- ity risks stemming from their supply chains and software providers. In hindsight, there are various cybersecurity lessons that organizations can learn by reviewing the details of the SolarWinds incident.

Read on for everything your organization needs to know.

The Details

The incident first began in September of 2019, when foreign cybercriminals were able to gain unauthorized access to SolarWinds’ digital infrastructure. Although it’s unclear exactly how the infrastructure was infiltrated, IT experts have confirmed that the hackers likely leveraged highly advanced digital skills to accomplish this feat. From there, the cybercriminals utilized the final months of 2019 to test whether they could inject a form of malware called Sunburst into SolarWinds’ software. During this time, the hackers remained undetected within the company’s digital infrastructure.

On Feb. 20, 2020, the cybercriminals officially administered Sunburst into SolarWinds’ flagship software product, Orion. Just over one month later, SolarWinds—unaware that the hackers had weaponized its product with malware—began sending out Orion software updates to customers. By installing these updates, impacted customers unknowingly introduced the harmful malware to their own technology. As a result, this malware provided the cybercriminals with a hidden entry point—also known as a digital back- door—to all affected customers’ networks. Although the hackers did not appear to use this backdoor to compromise customers’ sensitive data, it certainly gave them the ability to do so.

In total, more than 18,000 SolarWinds customers downloaded the malware and were at risk of potentially having their records exposed during the attack. Because SolarWinds built a reputation as a top U.S. technology company, many of the victimized customers were high-profile organizations and federal agencies. These customers included Microsoft, Intel, Cisco and Deloitte, as well as the Pentagon and the U.S. Departments of Homeland Security, Justice, State, Commerce and Treasury.

Despite the malware incident occurring in early 2020, the hackers’ activity went undiscovered for several more months, allowing them continued access to customers’ sensitive data. In December of 2020, FireEye— a cybersecurity firm and SolarWinds customer—detected the malware within its network and traced it back to SolarWinds. On Dec. 11, 2020, FireEye informed Solar- Winds of the incident. Days later, SolarWinds reported the attack to the U.S. Securities and Exchange Commission (SEC).

Upon investigating the incident, the federal government confirmed that the cybercriminals responsible were likely associated with APT29—which is a Russian hacking group. IT experts then helped SolarWinds and its impacted customers implement a “kill switch” to control the malware and effectively close the digital backdoor that the hackers had created.

Because the incident was relatively recent, its overall impact has yet to be seen. As of now, the following consequences resulted from this large-scale attack:

Recovery costs

Both SolarWinds and its impacted customers are expected to incur a combined total of more than $90 million in recovery expenses related to the incident. These costs include investigating the attack, informing all affected parties, removing the malware from every infected network, recovering compromised data and implementing updated cybersecurity protocols to prevent future incidents. Since the attack impacted federal agencies, these costs have the potential to trickle down to U.S. taxpayers as well.

Reputational damages

Considering SolarWinds maintained a trusted and respected reputation prior to the attack, the technology company received significant criticism from customers and the public for its cybersecurity shortcomings after the incident occurred. In particular, SolarWinds was scrutinized for failing to detect the cybercriminals’ initial activity within its network and remaining unaware that Orion had been injected with malware until FireEye’s eventual discovery months later. Further, while the method hackers used to infiltrate SolarWinds’ network is unknown, it was soon discovered that a handful of the company’s employees possessed weak passwords leading up to the incident (one employee’s password was “solarwinds123”)— paving the way for additional security criticism. Amid this scrutiny, SolarWinds’ stock price fell by 40% the week following the incident.

Legal ramifications

In January of 2021—one month after the details of the incident became public—disgruntled shareholders filed a class-action lawsuit against SolarWinds for its cyber-security failures during the attack. Several months later, the SEC announced plans to investigate whether SolarWinds’ affected customers accurately estimated the impact of the incident within their financial records. As time goes on and additional damages come to light, it’s certainly possible that both SolarWinds and its customers could encounter more lawsuits and regulatory fines related to the incident.

Lessons Learned

There are several cybersecurity takeaways from the SolarWinds attack. Specifically, the incident emphasized these critical lessons:

Supply chain exposures shouldn’t be ignored.

Above all, this attack showcased how critical

it is for organizations to evaluate and address security concerns within their supply chains, including IT and software providers. Even if an organization follows proper cyber policies and procedures internally, a compromised supplier could still end up threatening its security and digital assets. Supply chain exposures can stem from various avenues— including vendors with access to organizational networks, third parties with inadequate data storage measures and suppliers with poor overall cybersecurity practices.

While it’s not possible to totally eliminate supply chain risks, there are several steps organizations can take to help reduce these exposures and prevent costly attacks, such as:

• Incorporating cyber risk management into vendor contracts—This can include requiring vendors to obtain cyber insur- ance, having them issue timely notifica- tions regarding cyber incidents and establishing clear expectations regarding the destruction of data following the termination of contracts.
• Minimizing access that third parties have to organizational data—Once a vendor or supplier has been selected, it’s crucial to work with them to address any existing vulnerabilities and cybersecurity gaps. Moving forward, suppliers’ access to sensitive data should be restricted on an as-needed basis.
• Monitoring suppliers’ compliance with supply chain risk management proce- dures—This may entail adopting   a “one strike and you’re out” policy with suppliers that experience cyber incidents or fail to meet applicable compliance guidelines.

Third parties must prioritize cybersecurity. As organizations begin to more closely evaluate their supply chain exposures, it’s increasingly vital for third-party vendors themselves to adopt effective cybersecurity measures. In particular, suppliers need to recognize that cybercriminals may target them in order to compromise their larger clients and take steps to prevent such incidents from occurring. After all, failing to do so could not only result in cybersecurity vulnerabilities but also contribute to reduced client trust and lost business. By upholding proper digital practices, third-party vendors can show their clients that they take security seriously, boost their overall reliability and— in some cases—secure additional contracts.

Access controls can offer a strong defense. Although it’s unknown whether SolarWinds’ access control protocols or password blunders contributed to the incident, IT experts attest that bolstering these cybersecurity elements can play a major role in defending against hackers and subsequent attacks.

Valuable    access     control    and    password tactics include the following:

• Instructing employees to develop compli- cated and unique passwords for their accounts in addition to changing these passwords on a routine schedule
• Implementing multifactor authentication measures that require employees to verify their identities in several ways (e.g., entering a password and answering a security question)
• Limiting employees’ digital access solely to the technology, networks and data they need to perform their job responsibilities
• Segmenting different workplace net- works to prevent all networks from being compromised if a single employee’s credentials are exploited

Effective security and threat detection software is critical .

This incident emphasizes the importance of having appropriate security and threat detection software in place. This software can be used to better identify suspicious digital activity and reduce dwell time—which refers to how long it takes to detect cybercriminals’ presence after their initial network infiltration. Although this software may seem like an expensive investment, it’s well worth it to help continuously monitor security threats, catch perpetrators before it’s too late and minimize the impacts of potentially devastating cyber incidents. Necessary software to consider includes network monitoring systems, antivirus programs, endpoint detection products and patch management tools. Also, it’s valuable to conduct routine penetration testing to determine whether this software possesses any security gaps or ongoing vulnerabilities. If such testing reveals any problems, these issues should be addressed immediately.

Proper coverage can provide much-needed protection.

Finally, the SolarWinds incident made it clear

that no organization—not even a major technology company—is immune to cyber-related losses. That’s why it’s crucial to ensure adequate protection against potential cyber incidents by securing proper coverage. Make sure your organization works with a trusted insurance professional when navigating these coverage decisions.

For additional risk management guidance and insurance solutions, contact us today .

Our Commercial Risk Team

© 2021 Zywave, Inc. All rights reserved.

Search for news articles by category.

Service Options

Manage your account quickly and easily.

• Account Login
• Report a Claim
• Make a Policy Change
• Request a Certificate
• Request an Auto ID Card
• Pay Your Bill
• Review Your Policy

Related Topics

• Application Security
• Cybersecurity Careers
• Cloud Security
• Cyberattacks & Data Breaches
• Cybersecurity Analytics
• Cybersecurity Operations
• Data Privacy
• Endpoint Security
• ICS/OT Security
• Identity & Access Mgmt Security
• Insider Threats
• Mobile Security
• Physical Security
• Remote Workforce
• Threat Intelligence
• Vulnerabilities & Threats
• Middle East & Africa
• Upcoming Events
• Newsletters
• Whitepapers
• Partner Perspectives:
• > Microsoft

Sizable Chunk of SEC Charges Against SolarWinds Tossed Out of Court Sizable Chunk of SEC Charges Against SolarWinds Tossed Out of Court

Judge dismisses claims against SolarWinds for actions taken after its systems had been breached, but allows the case to proceed for alleged misstatements prior to the incident.

July 18, 2024

A judge has dismissed a major portion of the Securities and Exchange Commission (SEC) litigation against SolarWinds and its chief information security officer (CISO), Tim Brown, ruling that they cannot be held liable for statements and filings made after the breach of the company's flagship Orion product.

However, the SEC can proceed with its charge against SolarWinds and Brown for misrepresentations made about the company's cybersecurity posture leading up to the cyberattack, according to the ruling from US District Court Judge Paul A. Engelmayer released on July 18. Court filings refer to the cyber incident as "Sunburst."

The ruling is in response to SolarWinds' motion to dismiss the SEC lawsuit filed in January of this year.

SolarWinds Information-Sharing "Vindicated"

Legal and cybersecurity experts say the ruling is a positive move toward providing guidance to other publicly traded companies on how to deal with cybersecurity incident disclosure regulations.

"For public companies rushing both to investigate an incident and make a materiality disclosure, the court's opinion allows the totality of the disclosure to prevail over the nitty-gritty details," says cyber attorney Beth Burgin Waller of Woods, Rogers, Vandeventer, Black PLC. "This decision vindicates SolarWinds' information sharing with the cybersecurity community post-incident."

While the ruling removes many of the charges against SolarWinds and Brown , the SEC will be allowed to pursue action for statements and other claims made about the cybersecurity posture of the company prior to its compromise. Disclosures and statements made about the company's security posture prior to the breach are "viably pled as materially false and misleading in numerous aspects," the judge wrote.

After joining SolarWinds in 2017, Brown internally highlighted deficits in the company's defenses while delivering more rosy assessments to customers, the ruling explained. Notably, the SolarWinds "Security Statement" falsely claimed compliance with the National Institute of Standards and Technology (NIST) Cybersecurity Framework.

A SolarWinds spokesperson said the company was "pleased" with the ruling in a statement.

"We look forward to the next stage, where we will have the opportunity for the first time to present our own evidence and to demonstrate why the remaining claim is factually inaccurate," the statement said. "We are also grateful for the support we have received thus far across the industry, from our customers, from cybersecurity professionals, and from veteran government officials who echoed our concerns, with which the court agreed."

CISO Hot Takes

Jessica Sica, CISO with Weave, was especially encouraged by the court's decision to toss out internal communications evidence among SolarWinds employees.

"Internally, you need to be able to discuss the state of security — for better or for worse — and not have that get out as if you weren’t doing your job," Sica says. "The SEC keeping that portion in could have led to more companies having a sort of 'don’t ask, don’t tell' policy on security, and that would make things much worse."

The court ruling also loosens some constraints on CISOs, according to Fred Kwong, Ph.D., vice president, and CISO of DeVry University.

"Holding CISOs personally liable, especially those CISOs that do not hold a position on the executive committee, is deeply flawed and would have set a precedent that would be counterproductive and weaken the security posture of organizations," Kwong says. "While not out of the woods, I'm happy to see that the court has dismissed most of the charges, especially those post-Sunburst."

Regardless of the ultimate outcome of the SEC's action against SolarWinds and Brown, Sica urges fellow CISOs to continue to be transparent.

"I think this doesn’t change the fact that you need to be honest about your security posture, and that’s a good thing," Sica says. "If you are promising publicly that you are doing it."

About the Author(s)

Becky Bracken, Senior Editor, Dark Reading

You May Also Like

The Role of Identity Access Management in a Zero Trust Framework

The Rise of AI-Powered Malware and Application Security Best Practices

CISO Perspectives: How to make AI an Accelerator, Not a Blocker

Securing Your Cloud Assets

Black Hat USA - Aug 3-8 - The Premier Technical Cybersecurity Conference - Learn More

Black Hat Europe - December 9-12 - Learn More

SecTor - Canada's IT Security Conference Oct 22-24 - Learn More

Editor's Choice

2024 InformationWeek US IT Salary Report

The Rise of the No-Code Economy

How Enterprises are Attacking the Cybersecurity Problem

Proven Success Factors for Endpoint Security

Intel 471 Breach Report

Boston Beer Company Transforms OT Security & Reduces Costs

5 Critical Controls for World-Class OT Cybersecurity

Cisco Panoptica for Simplified Cloud-Native Application Security

ESG E-Book: Taking a Holistic Approach to Securing Cloud-Native Application Development

The Future of Cloud Security: Attack Paths & Graph-based Technology

• Skip to main content
• Skip to primary sidebar
• Skip to footer

April 23, 2021

What Else We’ve Learned About the SolarWinds Data Breach

In January, we covered a massive supply-chain data breach known as the SolarWinds attack. To get a broad overview of the incident, how the malicious agents carried out the hack, and the known victims, please read our coverage . Over the past four months, there have been new developments in the story that warrant a follow-up. Here, we go over these updates and discuss the potential for lasting fallout.

A brief synopsis

In December 2020, cybersecurity firm FireEye reported a significant flaw in the SolarWinds Orion database management software suite. When the dust settled, experts found that over 18,000 organizations had inadvertently installed a backdoor for an Advanced Persistent Threat (APT) group, likely Russian in origin. These state-sponsored actors infiltrated major corporations and high-level United States governmental agencies alike. Officials believe it to be the most widespread digital espionage campaign ever carried out against the United States. So, what have we found out since then?

More sophisticated than initially thought

From the very beginning, cybersec professionals knew the culprits were sophisticated and that the program’s scope was enormous. As it turns out, however, initial estimates seemed to have underestimated it. According to a recent analysis by RiskIQ, the infrastructure used by the threat actors was at least 56% larger than originally thought [1] .

This implies the state hackers had access to significantly more computing power and probably targeted even more organizations than the known 18,000 victims. The same report also concluded that the use of United States-based infrastructure during the initial attack stage prevented the National Security Agency (NSA) from noticing the situation due to stricter laws against domestic surveillance.

Russians officially blamed

United States intelligence agencies have always blamed Russia for the attack, but it turned into more than an accusation when President Joe Biden and the United States formally sanctioned the adversarial country on March 15 [2] . Provisions of the sanctions include:

• Forbidding U.S. banks from buying bonds from or lending money to Russia’s national financial institutions after June 14.
• Expelling 10 Russian diplomats accused of being intelligence agents from the United States.
• Sanctioning six technology companies in Russia accused of supporting intelligence agencies.

The sanctions significantly ratchet up tensions between the two nations and mark a major departure from standard espionage protocol. Previously, the United States and other countries assumed cyber espionage campaigns were always underway from their enemies, and their enemies were under similar assumptions. This meant that there was an implicit understanding that everyone is spying on everyone else, and nobody felt real consequences for it. The sanctions set a new precedent that could result in escalation rather than diplomacy. Although, Russia pulled back troops from the Ukrainian border after the sanctions [3] , so perhaps the message landed as intended. Only time will tell what ramifications this act has, but hopefully, it doesn’t increase the divide between the two largest nuclear powers.

Concurrent Chinese involvement

Although analysts blame Russia for the initial breach, it appears like Chinese state hackers also took advantage of the situation [4] . According to a report by Secureworks, some malicious agents used tactics similar to those employed by the Chinese APT, SPIRAL [5] . Furthermore, during the intrusion, the group accidentally revealed its IP, which originated from China. So, while sanctions only targeted Russia, there is evidence that China played a role too.

Of course, as we talked about in the original SolarWinds blog, it’s exceedingly difficult to analyze blame with a hundred percent certainty. State-sponsored digital espionage groups are adept at covering their tracks and obfuscating origins. And, while the United States government seems positive the Russians were the main culprits, hard evidence of this assertion hasn’t been made public. Not to mention the United States government has been wrong about some pretty bold claims before. We may never know the full truth.

Congress grills Microsoft

Interestingly, the company in the hottest water over the whole snafu isn’t SolarWinds; it’s Microsoft. Probably due to its high-profile nature, the U.S. Congress set its sights on the tech behemoth [6] . This is because, after the breach’s first stage, the hackers exploited Microsoft products and stole sensitive emails and other data from thousands of organizations.

Microsoft itself had its source code exposed to the hackers. Since source code is the lifeblood of a tech company, it shows exactly how all-encompassing the breach was. It also proves a crucial point; no matter how secure a system is, nothing can be completely safe from ill-intentioned cyberspies with the backing of an entire country’s resources. So, although House members assuredly loved grandstanding about the holes in Microsoft’s security, the truth is more complex and nuanced.

White House ramps down recovery efforts

This brings us to the conclusion of the saga. On April 19, the White House announced that several national agencies such as the FBI, CISA, and NSA would soon begin ramping down their efforts regarding SolarWinds. Combined with the Russian sanctions, it signals that the U.S. Government considers the incident largely settled. China appears unlikely to receive any formal retaliation. Hopefully, the most significant data breach of our times serves as a lesson for the future of cybersecurity. Undoubtedly similar incidents will occur in the future, but perhaps mitigation policies will improve, and potential damages will be reduced.

Security is a personal responsibility

If there’s one takeaway everyone should have about SolarWinds, it’s that relying on Big Tech’s security policies is a mistake. People should do a bit of research to find redundant cybersecurity methods for their sensitive data.

You can protect your confidential files by ditching cloud drives like Dropbox, OneDrive, and Google Drive and switch to AXEL Go. AXEL Go utilizes our decentralized, distributed files sharing network backed by blockchain and the InterPlanetary File System. This ensures your documents aren’t stored in one place with a single point of failure.

Additionally, every file you transfer via the AXEL Network gets “digitally shredded” and distributed to scattered server nodes. This means even if a malicious agent compromised a server, they wouldn’t have access to the complete file. Documents are only reconfigured for the initial user and any recipients. This system, combined with military-grade encryption, provides multiple layers of security for AXEL Go users.

You can try AXEL Go Premium with all features unlocked free for 14-days. Sign up today and see how AXEL can improve your workflow and harden your organization’s cybersecurity.

[1] “SolarWinds: Advancing the Story”, RiskIq.com, April 22, 2021, https://community.riskiq.com/article/9a515637

[2] Morgan Chalfant, Maggie Miller, “Biden administration sanctions Russia for SolarWinds hack, election interference”, April 15, 2021, https://thehill.com/homenews/administration/548367-biden-administration-unveils-sweeping-sanctions-on-russia?rl=1

[3] “Russia to pull troops back from near Ukraine”, BBC, April 22, 2021, https://www.bbc.com/news/world-europe-56842763

[4] Dan Goodin, “Chinese hackers targeted SolarWinds customers in parallel with Russian op”, Ars Technica, March 8, 2021, https://arstechnica.com/gadgets/2021/03/chinese-hackers-targeted-solarwinds-customers-in-parallel-with-russian-op/

[5] Counter Threat Unit Research Team, “SUPERNOVA Web Shell Deployment Linked to SPIRAL Threat Group”, Secureworks.com, March 8, 2021, https://www.secureworks.com/blog/supernova-web-shell-deployment-linked-to-spiral-threat-group

[6] Frank Bajak, “SolarWinds hacking campaign puts Microsoft in the hot seat”, The Associated Press, April 17, 2021, https://apnews.com/article/business-technology-government-and-politics-f51e53523312b87121146de8fd7c0020

SolarWinds and the SEC lawsuit

• Medium Text

• SolarWinds Corporation Follow

The SolarWinds Orion attack — fodder for analysis

The sec solarwinds complaint in a nutshell, painting a rosy picture, ciso as security officer, marketing officer, or both, the heart of the complaint, false and real lessons, diligence and negligence remain key touchstones, diligent organizations will debate and disagree but improve.

Sign up here.

John Bandler is a lawyer, consultant, author, and adjunct professor at Elisabeth Haub School of Law at Pace University. He helps protect organizations from cybercrime, improve cybersecurity, and better manage information assets. His latest book is “Policies and Procedures for Your Organization” (2024). His firm, based in New York, is Bandler Law Firm PLLC, and he can be reached at [email protected].

Read Next / Editor's Picks

Industry Insight

Luc Cohen, Susan Heavey

David Thomas

Mike Scarcella, David Thomas

Karen Sloan

Cyber Case Study: SolarWinds Supply Chain Cyberattack

by Kelli Young | Oct 18, 2021 | Case Study , Cyber Liability Insurance

In the final month of 2020, it was revealed that foreign hackers had orchestrated a supply chain cyberattack throughout the past year in an effort to compromise several federal agencies and private organizations. The cybercriminals first infiltrated the digital infrastructure of SolarWinds—a Texas-based technology company—before using that infrastructure to gain access to sensitive data from a range of government departments and organizations via malware-ridden soft-ware updates. The incident ultimately exploited numerous SolarWinds customers and led to millions of dollars in total losses.

The attack has been dubbed as one of the largest and most sophisticated cyber incidents in U.S. history, motivating many organizations to take a closer look at security risks stemming from their supply chains and software providers. In hindsight, there are various cybersecurity lessons that organizations can learn by reviewing the details of the SolarWinds incident. Here’s what your organization needs to know.

The Details of the SolarWinds Supply Chain Cyberattack

The incident first began in September of 2019, when foreign cybercriminals were able to gain unauthorized access to SolarWinds’ digital infrastructure. Although it’s unclear exactly how the infrastructure was infiltrated, IT experts have confirmed that the hackers likely leveraged highly advanced digital skills to accomplish this feat. From there, the cybercriminals utilized the final months of 2019 to test whether they could inject a form of malware called Sunburst into SolarWinds’ software. During this time, the hackers remained undetected within the company’s digital infrastructure.

On Feb. 20, 2020, the cybercriminals officially administered Sunburst into SolarWinds’ flag-ship software product, Orion. Just over one month later, SolarWinds—unaware that the hackers had weaponized its product with malware—began sending out Orion software updates to customers. By installing these updates, impacted customers unknowingly introduced the harmful malware to their own technology. As a result, this malware provided the cybercriminals with a hidden entry point—also known as a digital back-door—to all affected customers’ networks. Although the hackers did not appear to use this backdoor to compromise customers’ sensitive data, it certainly gave them the ability to do so.

In total, more than 18,000 SolarWinds customers downloaded the malware and were at risk of potentially having their records exposed during the attack. Because SolarWinds built a reputation as a top U.S. technology company, many of the victimized customers were high-profile organizations and federal agencies. These customers included Microsoft, Intel, Cisco and Deloitte, as well as the Pentagon and the U.S. Departments of Homeland Security, Justice, State, Commerce and Treasury.

Despite the malware incident occurring in early 2020, the hackers’ activity went undiscovered for several more months, allowing them continued access to customers’ sensitive data. In December of 2020, FireEye— a cybersecurity firm and SolarWinds customer—detected the malware within its network and traced it back to SolarWinds. On Dec. 11, 2020, FireEye informed Solar-Winds of the incident. Days later, SolarWinds reported the attack to the U.S. Securities and Exchange Commission (SEC).

Upon investigating the incident, the federal government confirmed that the cybercriminals responsible were likely associated with APT29—which is a Russian hacking group. IT experts then helped SolarWinds and its impacted customers implement a “kill switch” to control the malware and effectively close the digital backdoor that the hackers had created.

The Impact of the SolarWinds Supply Chain Cyberattack

Because the incident was relatively recent, its overall impact has yet to be seen. As of now, the following consequences resulted from this large-scale attack:

Recovery costs Both SolarWinds and its impacted customers are expected to incur a combined total of more than $90 million in recovery expenses related to the incid e nt . These costs include investigating the attack, informing all affected parties, removing the malware from every infected network, recovering compromised data and implementing updated cybersecurity protocols to prevent future incidents. Since the attack impacted federal agencies, these costs have the potential to trickle down to U.S. taxpayers as well.

Reputational damages Considering SolarWinds maintained a trusted and respected reputation prior to the attack, the technology company received significant criticism from customers and the public for its cybersecurity shortcomings after the incident occurred. In particular, SolarWinds was scrutinized for failing to detect the cybercriminals’ initial activity within its network and remaining unaware that Orion had been injected with malware until FireEye’s eventual discovery months later. Further, while the method hackers used to infiltrate SolarWinds’ network is unknown, it was soon discovered that a handful of the company’s employees possessed weak passwords leading up to the incident (one employee’s password was “solarwinds123”)—paving the way for additional security criticism. Amid this scrutiny, SolarWinds’ stock price fell by 40% the week following the incident .

Legal ramifications In January of 2021—one month after the details of the incident became public—disgruntled shareholders filed a class-action lawsuit against SolarWinds for its cybersecurity failures during the attack. Several months later, the SEC announced plans to investigate whether SolarWinds’ affected customers accurately estimated the impact of the incident within their financial records. As time goes on and additional damages come to light, it’s certainly possible that both SolarWinds and its customers could encounter more lawsuits and regulatory fines related to the incident.

Lessons Learned

There are several cybersecurity takeaways from the SolarWinds Supply Chain Cyberattack. Specifically, the incident emphasized these critical lessons:

Supply chain exposures shouldn’t be ignored. Above all, this attack showcased how critical it is for organizations to evaluate and address security concerns within their supply chains, including IT and software providers. Even if an organization follows proper cyber policies and procedures internally, a compromised supplier could still end up threatening its security and digital assets. Supply chain exposures can stem from various avenues—including vendors with access to organizational networks, third parties with inadequate data storage measures and suppliers with poor overall cybersecurity practices.

While it’s not possible to totally eliminate supply chain risks, there are several steps organizations can take to help reduce these exposures and prevent costly attacks, such as:

• Incorporating cyber risk management into vendor contracts—This can include requiring vendors to obtain cyber insurance, having them issue timely notifications regarding cyber incidents and establishing clear expectations regarding the destruction of data following the termination of contracts.
• Minimizing access that third parties have to organizational data—Once a vendor or supplier has been selected, it’s crucial to work with them to address any existing vulnerabilities and cybersecurity gaps. Moving forward, suppliers’ access to sensitive data should be restricted on an as-needed basis.
• Monitoring suppliers’ compliance with supply chain risk management procedures—This may entail adopting a “one strike and you’re out” policy with suppliers that experience cyber incidents or fail to meet applicable compliance guidelines.

Third parties must prioritize cybersecurity. As organizations begin to more closely evaluate their supply chain exposures, it’s increasingly vital for third-party vendors themselves to adopt effective cybersecurity measures. In particular, suppliers need to recognize that cybercriminals may target them in order to compromise their larger clients and take steps to prevent such incidents from occurring. After all, failing to do so could not only result in cybersecurity vulnerabilities, but also contribute to reduced client trust and lost business. By upholding proper digital practices, third-party vendors can show their clients that they take security seriously, boost their overall reliability and—in some cases—secure additional contracts.

Access controls can offer a strong defense. Although it’s unknown whether SolarWinds’ access control protocols or password blunders contributed to the incident, IT experts attest that bolstering these cybersecurity elements can play a major role in defending against hackers and subsequent attacks.

Valuable access control and password tactics include the following:

• Instructing employees to develop complicated and unique passwords for their accounts in addition to changing these passwords on a routine schedule
• Implementing multifactor authentication measures that require employees to verify their identities in several ways (e.g., entering a password and answering a security question)
• Limiting employees’ digital access solely to the technology, networks and data they need to perform their job responsibilities
• Segmenting different workplace networks to prevent all networks from being compromised if a single employee’s credentials are exploited

Effective security and threat detection software is critical. This incident emphasizes the importance of having appropriate security and threat detection software in place. This software can be used to better identify suspicious digital activity and reduce dwell time—which refers to how long it takes to detect cybercriminals’ presence after their initial network infiltration. Although this software may seem like an expensive investment, it’s well worth it to help continuously monitor security threats, catch perpetrators before it’s too late and minimize the impacts of potentially devastating cyber incidents. Necessary software to consider includes network monitoring systems, antivirus programs, endpoint detection products and patch management tools. Also, it’s valuable to conduct routine penetration testing to determine whether this software possesses any security gaps or ongoing vulnerabilities. If such testing reveals any problems, these issues should be addressed immediately.

Proper coverage can provide much- needed protection. Finally, the SolarWinds Supply Chain Cyberattack made it clear that no organization—not even a major technology company—is immune to cyber-related losses. That’s why it’s crucial to ensure adequate protection against potential cyber incidents by securing proper coverage. Make sure your organization works with a trusted insurance professional when navigating these coverage decisions

We are here to help.

If you’d like additional information and resources, we’re here to help you analyze your needs and make the right coverage decisions to protect your operations from unnecessary risk. You can download a free copy of our  eBook , or if you’re ready make Cyber Liability Insurance a part of your insurance portfolio,  Request a Proposal  or download and get started on our  Cyber & Data Breach Insurance Application  and we’ll get to work for you.

Recent Posts

• Cyber Case Study: Colonial Pipeline Ransomware Attack
• Understanding the Difference Between Life Insurance and Annuities
• Cyber Solutions: Cloud Security Management Explained
• Live Well Work Well – July 2024
• 6 Reasons Why Life Insurance Matters

U.S. Government Accountability Office

SolarWinds Cyberattack Demands Significant Federal and Private-Sector Response (infographic)

The cybersecurity breach of SolarWinds’ software is one of the most widespread and sophisticated hacking campaigns ever conducted against the federal government and private sector. In today’s WatchBlog post, we look at this breach and the ongoing federal government and private-sector response. This information is based on publicly disclosed information from federal and private industry sources. We here at GAO are currently conducting a comprehensive review of the breach with plans to issue a public report later this year.

Beginning in September 2019, a campaign of cyberattacks, now identified to be perpetrated by the Russian Foreign Intelligence Service (hereafter referred to as the threat actor), breached the computing networks at SolarWinds—a Texas-based network management software company. The threat actor first conducted a “dry run,” injecting test code into SolarWinds’ network management and monitoring suite of products called Orion. Then, beginning in February 2020, the threat actor injected trojanized (hidden) code into a file that was later included in SolarWinds’ Orion software updates. SolarWinds released the software updates to its customers not realizing that the updates were compromised. The trojanized code had provided the threat actor with a “backdoor”—a program that can give an intruder remote access to an infected computer. According to cybersecurity researchers, the threat actor was then able to remotely exploit the networks and systems of SolarWinds’ customers who had downloaded the compromised software updates using a sophisticated computing infrastructure.

Since SolarWinds is widely used in the federal government to monitor network activity on federal systems, this incident allowed the threat actor to breach infected agency information systems. SolarWinds estimates that nearly 18,000 of its customers received a compromised software update. Of those, the threat actor targeted a smaller subset of high-value customers, including the federal government, to exploit for the primary purpose of espionage.

Discovery and response: What now?

FireEye—a cybersecurity professional services firm—stated that in November 2020 it had detected an intrusion to its systems and later informed SolarWinds of the compromise of the Orion platform. In addition, in coordination with FireEye, Microsoft reported the threat actor was able to compromise some of Microsoft’s cloud platforms. The compromise allowed the threat actor to gain unauthorized network access. Microsoft informed several federal agencies that their unclassified systems had been breached and took steps with other industry partners to redirect the malicious network traffic away from the domain used by the threat actor to render the malicious code ineffective and prevent further compromise. 

In response to this breach, on December 13, 2020, the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) released an emergency directive outlining required mitigations for federal agencies to prevent further exploitation of federal information systems. On December 16, the White House’s National Security Council activated the Cyber Unified Coordination Group, who is responsible for coordinating the government-wide response to the incident. This group includes officials from the Office of the Director of National Intelligence, FBI, and CISA, with support from the National Security Agency.

GAO and Congressional monitoring will continue

Congress held multiple hearings to gather and report information on the timeline of events related to the SolarWinds hack, and larger issues such as IT supply chain security (meaning the security of information and communications technology products and services), threat actor capability and motivation, and future federal actions and improvements.

Although our examination of SolarWinds is ongoing, we have previously reported on IT supply chain risks and major cybersecurity challenges . We continue to emphasize that the federal government needs to move with greater urgency to improve the nation's cybersecurity as the country faces grave and rapidly evolving threats. Ensuring the cybersecurity of the nation has been on our High Risk List since 1997.

A detailed timeline of federal government and private sector activities to remediate the breach is illustrated in the graphic below.

• Comments on GAO’s WatchBlog? Contact  [email protected] .

GAO Contacts

Related Posts

Priority Recommendation Round Up—Actions that Can Bring the Best Rewards

What are the Biggest Challenges to Federal Cybersecurity? (High Risk Update)

The U.S. Now Has a National Cybersecurity Strategy, but Is It as Strong as It Could Be?

Related products, information technology: federal agencies need to take urgent action to manage supply chain risks, product number, high-risk series: federal government needs to urgently pursue critical actions to address major cybersecurity challenges.

GAO's mission is to provide Congress with fact-based, nonpartisan information that can help improve federal government performance and ensure accountability for the benefit of the American people. GAO launched its WatchBlog in January, 2014, as part of its continuing effort to reach its audiences—Congress and the American people—where they are currently looking for information.

The blog format allows GAO to provide a little more context about its work than it can offer on its other social media platforms. Posts will tie GAO work to current events and the news; show how GAO’s work is affecting agencies or legislation; highlight reports, testimonies, and issue areas where GAO does work; and provide information about GAO itself, among other things.

Please send any feedback on GAO's WatchBlog to [email protected] .

• Artificial Intelligence
• Generative AI
• Business Operations
• IT Leadership
• Application Security
• Business Continuity
• Cloud Security
• Critical Infrastructure
• Identity and Access Management
• Network Security
• Physical Security
• Risk Management
• Security Infrastructure
• Vulnerabilities
• Software Development
• Enterprise Buyer’s Guides
• United States
• United Kingdom
• Newsletters
• Foundry Careers
• Terms of Service
• Privacy Policy
• Cookie Policy
• Member Preferences
• About AdChoices
• E-commerce Links
• Your California Privacy Rights

Our Network

• Computerworld
• Network World

SolarWinds attack explained: And why it was so hard to detect

A group believed to be russia's cozy bear gained access to government and other systems through a compromised update to solarwinds' orion software. most organizations aren't prepared for this sort of software supply chain attack..

The recent breach of major cybersecurity company FireEye by nation-state hackers was part of a much larger attack that was carried out through malicious updates to a popular network monitoring product and impacted major government organizations and companies. The incident highlights the severe impact software supply chain attacks can have and the unfortunate fact that most organizations are woefully unprepared to prevent and detect such threats.

A hacker group believed to be affiliated with the Russian government gained access to computer systems belonging to multiple US government departments including the US Treasury and Commerce in a long campaign that is believed to have started in March. The news triggered an emergency meeting of the US National Security Council on Saturday.

The attack involved hackers compromising the infrastructure of SolarWinds, a company that produces a network and applications monitoring platform called Orion, and then using that access to produce and distribute trojanized updates to the software’s users. On a page on its website that was taken down after news broke out, SolarWinds stated that its customers included 425 of the US Fortune 500, the top ten US telecommunications companies, the top five US accounting firms, all branches of the US Military, the Pentagon, the State Department, as well as hundreds of universities and colleges worldwide.

The SolarWinds software supply chain attack also allowed hackers to access the network of US cybersecurity firm FireEye, a breach that was announced last week . Even though FireEye did not name the group of attackers responsible, the Washington Post reports it is APT29 or Cozy Bear, the hacking arm of Russia’s foreign intelligence service, the SVR.

“FireEye has detected this activity at multiple entities worldwide,” the company said in an advisory Sunday. “The victims have included government, consulting, technology, telecom and extractive entities in North America, Europe, Asia and the Middle East. We anticipate there are additional victims in other countries and verticals. FireEye has notified all entities we are aware of being affected.”

The malicious Orion updates

The software builds for Orion versions 2019.4 HF 5 through 2020.2.1 that were released between March 2020 and June 2020 might have contained a trojanized component. However, FireEye noted in its analysis that each of the attacks required meticulous planning and manual interaction by the attackers.

The attackers managed to modify an Orion platform plug-in called SolarWinds.Orion.Core.BusinessLayer.dll that is distributed as part of Orion platform updates. The trojanized component is digitally signed and contains a backdoor that communicates with third-party servers controlled by the attackers. FireEye tracks this component as SUNBURST and has released open-source detection rules for it on GitHub.

“After an initial dormant period of up to two weeks, it retrieves and executes commands, called ‘Jobs,’ that include the ability to transfer files, execute files, profile the system, reboot the machine, and disable system services,” the FireEye analysts said. “The malware masquerades its network traffic as the Orion Improvement Program (OIP) protocol and stores reconnaissance results within legitimate plugin configuration files allowing it to blend in with legitimate SolarWinds activity. The backdoor uses multiple obfuscated blocklists to identify forensic and anti-virus tools running as processes, services, and drivers.”

The attackers kept their malware footprint very low, preferring to steal and use credentials to perform lateral movement through the network and establish legitimate remote access. The backdoor was used to deliver a lightweight malware dropper that has never been seen before and which FireEye has dubbed TEARDROP. This dropper loads directly in memory and does not leave traces on the disk. Researchers believe it was used to deploy a customized version of the Cobalt Strike BEACON payload. Cobalt Strike is a commercial penetration testing framework and post-exploitation agent designed for red teams that has also been adopted and used by hackers and sophisticated cybercriminal groups.

To avoid detection, attackers used temporary file replacement techniques to remotely execute their tools. This means they modified a legitimate utility on the targeted system with their malicious one, executed it, and then replaced it back with the legitimate one. A similar technique involved the temporary modification of system scheduled tasks by updating a legitimate task to execute a malicious tool and then reverting the task back to its original configuration.

“Defenders can examine logs for SMB sessions that show access to legitimate directories and follow a delete-create-execute-delete-create pattern in a short amount of time,” the FireEye researchers said. “Additionally, defenders can monitor existing scheduled tasks for temporary updates, using frequency analysis to identify anomalous modification of tasks. Tasks can also be monitored to watch for legitimate Windows tasks executing new or unknown binaries.”

This is some of the best operational security exhibited by a threat actor that FireEye has ever observed, being focused on detection evasion and leveraging existing trust relationships. However, the company’s researchers believe these attacks can be detected through persistent defense and have described multiple detection techniques in their advisory.

SolarWinds advises customers to upgrade to Orion Platform version 2020.2.1 HF 1 as soon as possible to ensure they are running a clean version of the product. The company also plans to release a new hotfix 2020.2.1 HF 2 on Tuesday that will replace the compromised component and make additional security enhancements.

The US Department of Homeland Security has also issued an emergency directive to government organizations to check their networks for the presence of the trojanized component and report back.

No easy solution

Software supply-chain attacks are not a new development and security experts have been warning for many years that they are some of the hardest type of threats to prevent because they take advantage of trust relationships between vendors and customers and machine-to-machine communication channels, such as software update mechanisms that are inherently trusted by users.

Back in 2012, researchers discovered that the attackers behind the Flame cyberespionage malware used a cryptographic attack against the MD5 file hashing protocol to make their malware appear as if it was legitimately signed by Microsoft and distribute it through the Windows Update mechanism to targets. That wasn’t an attack where the software developer itself, Microsoft, was compromised, but the attackers exploited a vulnerability in the Windows Update file checking demonstrating that software update mechanisms can be exploited to great effect.

In 2017, security researchers from Kaspersky Lab uncovered a software supply-chain attack by an APT group dubbed Winnti that involved breaking into the infrastructure of NetSarang, a company that makes server management software, which allowed them to distribute trojanized versions of the product that were digitally signed with the company’s legitimate certificate. That same group of attackers later broke into the development infrastructure of Avast subsidiary CCleaner and distributed trojanized versions of the program to over 2.2 million users. Last year, attackers hijacked the update infrastructure of computer manufacturer ASUSTeK Computer and distributed malicious versions of the ASUS Live Update Utility to users.

“I don’t know of any organization that incorporates what a supply chain attack would look like in their environment from a threat modeling perspective,” David Kennedy, former NSA hacker and founder of security consulting firm TrustedSec, tells CSO. “When you look at what happened with SolarWinds, it’s a prime example of where an attacker could literally select any target that has their product deployed, which is a large number of companies from around the world, and most organizations would have no ability to incorporate that into how they would respond from a detection and prevention perspective. This is not a discussion that’s happening in security today.”

While software that is deployed in organizations might undergo security reviews to understand if their developers have good security practices in the sense of patching product vulnerabilities that might get exploited, organizations don’t think about how that software could impact their infrastructure if its update mechanism is compromised, Kennedy says. “It’s something that we’re still very immature on and there’s no easy solution for it, because companies need software to run their organizations, they need technology to expand their presence and remain competitive, and the organizations that are providing this software don’t think about this as a threat model either.”

Kennedy believes it should start with software developers thinking more about how to protect their code integrity at all times but also to think of ways to minimize risks to customers when architecting their products.

“A lot of times you know when you’re building software, you think of a threat model from outside in, but you don’t always think from inside out,” he said. “That’s an area a lot of people need to be looking at: How do we design our architecture infrastructure to be more resilient to these types of attacks? Would there be ways for us to stop a lot of these attacks by minimizing the infrastructure in the [product] architecture? For example, keeping SolarWinds Orion in its own island that allows communications for it to function properly, but that’s it. It’s good security practice in general to create as much complexity as possible for an adversary so that even if they’re successful and the code you’re running has been compromised, it’s much harder for them to get access to the objectives that they need.”

Companies, as users of software, should also start thinking about applying zero-trust networking principles and role-based access controls not just to users, but also to applications and servers. Just as not every user or device should be able to access any application or server on the network, not every server or application should be able to talk to other servers and applications on the network. When deploying any new software or technology into their networks, companies should ask themselves what could happen if that product gets compromised because of a malicious update and try to put controls in place that would minimize the impact as much as possible.

It’s likely that the number of software supply-chain attacks will increase in the future, especially as other attackers see how successful and wide ranging they can be. The number of ransomware attacks against organizations exploded after the WannaCry and NotPetya attacks of 2017 because they showed to attackers that enterprise networks are not as resilient as they thought against such attacks. Since then many cybercrime groups have adopted sophisticated techniques that often put them on par with nation-state cyberespionage actors .

Ransomware gangs have also understood the value of exploiting the supply chain and have started hacking into managed services providers to exploit their access into their customers’ networks . NotPetya itself had a supply chain component because the ransomware worm was initially launched through the backdoored software update servers of an accounting software called M.E.Doc that is popular in Eastern Europe.

Both organized crime and other nation-state groups are looking at this attack right now as “Wow, this is a really successful campaign,” Kennedy said. From a ransomware perspective, if they simultaneously hit all the organizations that had SolarWinds Orion installed, they could have encrypted a large percentage of the world’s infrastructure and made off with enough money that they wouldn’t have ever had to work again. “They probably know their sophistication level will need to be increased a bit for these types of attacks, but it’s not something that is too far of a stretch, given the progression we’re seeing from ransomware groups and how much money they’re investing in development. So, I definitely think that we can see this with other types of groups [not just nation states] for sure.”

Related content

Ics malware frostygoop disrupted heating in ukraine, remains threat to ot worldwide, crowdstrike failure: what you need to know, google abandons plans to drop third-party cookies in chrome, wiz shocks the tech world as it rejects google’s $23 billion bid, from our editors straight to your inbox.

Lucian Constantin writes about information security, privacy, and data protection for CSO.

More from this author

Chinese apt group daggerfly revamps malware toolkit with new backdoors, attackers abuse url protection services to hide phishing links in emails, apt exploits windows zero-day to launch zombie ie attack, 39 hardware vulnerabilities: a guide to the threats, python github token leak shows binary files can burn developers too, md5 attack puts radius networks everywhere at risk, new intel cpu side-channel attack indirector can leak sensitive data, cisco patches actively exploited zero-day flaw in nexus switches, most popular authors.

Show me more

Nhis may be your biggest — and most neglected — security hole.

Early IT takeaways from the CrowdStrike outage

Key considerations for adopting a platform approach to cybersecurity

CSO Executive Session India with Dr Susil Kumar Meher, Head Health IT, AIIMS (New Delhi)

CSO Executive Session India with Charanjit Bhatia, Head of Cybersecurity, COE, Bata Brands

CSO Executive Sessions: Data protection in Malaysia

Cybersecurity Insights for Tech Leaders: Addressing Dynamic Threats and AI Risks with Resilience

Sponsored Links

• Get Cisco UCS X-Series Chassis and Fabric Interconnects offer.

• Share full article

Advertisement

Supported by

Why Was SolarWinds So Vulnerable to a Hack?

It’s the economy, stupid.

By Bruce Schneier

Mr. Schneier is a security technologist and the author of 14 books, including “Click Here to Kill Everybody: Security and Survival in a Hyper-Connected World.” He is a fellow at the Belfer Center at the Harvard Kennedy School and a fellow at the Berkman Klein Center for Internet and Society at Harvard University.

Early in 2020, cyberspace attackers apparently working for the Russian government compromised a piece of widely used network management software made by a company called SolarWinds. The hack gave the attackers access to the computer networks of some 18,000 of SolarWinds’s customers, including U.S. government agencies such as the Homeland Security Department and State Department, American nuclear research labs, government contractors, IT companies and nongovernmental agencies around the world.

It was a huge attack, with major implications for U.S. national security. The Senate Intelligence Committee is scheduled to hold a hearing on the breach on Tuesday. Who is at fault?

The U.S. government deserves considerable blame, of course, for its inadequate cyberdefense. But to see the problem only as a technical shortcoming is to miss the bigger picture. The modern market economy, which aggressively rewards corporations for short-term profits and aggressive cost-cutting, is also part of the problem: Its incentive structure all but ensures that successful tech companies will end up selling unsecure products and services.

Like all for-profit corporations, SolarWinds aims to increase shareholder value by minimizing costs and maximizing profit. The company is owned in large part by Silver Lake and Thoma Bravo, private-equity firms known for extreme cost-cutting.

SolarWinds certainly seems to have underspent on security. The company outsourced much of its software engineering to cheaper programmers overseas, even though that typically increases the risk of security vulnerabilities. For a while, in 2019, the update server’s password for SolarWinds’s network management software was reported to be “solarwinds123.” Russian hackers were able to breach SolarWinds’s own email system and lurk there for months. Chinese hackers appear to have exploited a separate vulnerability in the company’s products to break into U.S. government computers. A cybersecurity adviser for the company said that he quit after his recommendations to strengthen security were ignored.

There is no good reason to underspend on security other than to save money — especially when your clients include government agencies around the world and when the technology experts that you pay to advise you are telling you to do more.

We are having trouble retrieving the article content.

Please enable JavaScript in your browser settings.

Thank you for your patience while we verify access. If you are in Reader mode please exit and  log into  your Times account, or  subscribe  for all of The Times.

Thank you for your patience while we verify access.

Already a subscriber?  Log in .

Want all of The Times?  Subscribe .

SolarWinds data breach lawsuit takeaways for CISOs

A lawsuit alleges solarwinds did not take adequate actions to prevent a data breach. security executives offer insights into this complex case.

The SolarWinds data breach of 2020 was one of the most widespread and sophisticated hacking campaigns to be conducted against the federal government and private sector.

 As early as January 2019, the Russian Foreign Intelligence Service breached the computing networks at SolarWinds — a Texas-based network management software company. Since the company’s software, SolarWinds Orion, was widely used in the federal government to monitor network activity and manage network devices on federal systems, the incident allowed the threat actor to breach several federal agencies’ networks.

After the SolarWinds attack became public in late 2020, the value of SolarWinds stock on the public market decreased in one week, from almost $25 per share to less than $15 per share, a decline of approximately 40%.

In the aftermath of the loss in share value, a class of SolarWinds shareholders sued the company, its executives and its investors for violations of the Exchange Act, which prohibits public corporations and their leaders from knowingly making misrepresentations or omissions that cause financial harm.

In late March, a Texas judge dismissed claims that former SolarWinds Chief Executive Officer (CEO) Kevin Thompson was personally liable for deceiving investors about the state of the company’s cybersecurity and allowed the class-action lawsuit to proceed.  

The lawsuit names Thompson, Chief Financial Officer J. Barton Kalsu, Chief Information Security Officer (CISO) Tim Brown and private equity firms Thomas Bravo and Siler Lake Technology Management as defendants. The suit alleges the company lied and materially misled investors about security practices leading up to the breach. Furthermore, the complaint claims each defendant was directly involved in the day-to-day operations at the highest levels and therefore privy to confidential information about business operations and oversight of internal controls. By omitting what they knew about the breach and employing poor security practices, the suit alleges SolarWinds executives were reckless and participated in a “fraudulent scheme.”

The lawsuit is a stark reminder of the damaging consequences that a data breach can have on the organization — from financial loss, to reputational damage, operational downtime, loss of customer trust, and legal action.

Security talked to security experts to gauge their feelings on how this lawsuit will have ramifications for both security leaders and organizations going forward.

Security : What are the implications of the lawsuit for SolarWinds?

Casey Ellis (Ellis), Founder and CTO, Bugcrowd: For SolarWinds, it sounds like the case will force them to lay out their cybersecurity and operational security controls as a matter of public court record for the purpose of having them judged as being sufficient or deficient. [They’ll also need to] deal with the court of public opinion.

Phil Neray, Vice President of Cyber Defense Strategy, CardinalOps: The key questions about the merit of the lawsuit revolve around (1) When did SolarWinds management learn their build environment had been compromised? If it was after they issued their standard 10K and 10Q boilerplate statements about cybersecurity risks, then it seems they should not be at fault for issuing false or misleading statements; (2) Did the company exhibit standards of due care in their day-to-day cybersecurity practices? For example, if we learn they did not properly segment their networks, or used weak password policies, or did not implement sufficient monitoring to detect suspicious or unauthorized activities in their security operations center (SOC), then the lawsuit may reveal additional grounds for shareholder lawsuits.

Casey Bisson, Head of Product and Developer Relations,  BluBracket : The wake of the SolarWinds incident is playing out like a low-key version of Unsafe at Any Speed: The Designed-In Dangers of the American Automobile, the 1960s book that raised public understanding of highway safety issues and upended the automobile industry.

The lawsuit is hastening the ultimate shift left: incorporation of security into business goals. Executives that had treated security as a barrier to minimize in the pursuit of business goals have realized it’s a real factor to contend with, and everybody is now taking lessons from those that had made security a business requirement.

Archie Agarwal, Founder and CEO,  ThreatModeler : This lawsuit will test the boundaries of liability for executives and majority shareholders.

John Bambenek, Principal Threat Hunter,  Netenrich : The biggest item is that the judge did not dismiss the part of the lawsuit on the CISO’s **personal** liability for the breach. Often in breaches, there are lawsuits; however, when executives are held personally responsible by the court, it will have an enormous impact on the psyches of other CISOs. Ultimately, it will depend on how this case is resolved in the end. But, any time an executive might be held personally reliable gets executives everywhere to pay attention. 

John Hellickson, Field CISO and Executive Advisor,  Coalfire : I believe it’s too early to tell what the implications will be, but many security leaders in the industry will be watching how this suit progresses. 

Security : How will this lawsuit impact other organizations and cybersecurity?

Ellis: The Texan ruling coincides with last month’s updated release of Cybersecurity Risk Management, Strategy, Governance, and Incident Disclosure by Public Companies rules by the SEC , in which a recommendation is made which would require public companies to report the level of cybersecurity expertise on their board. To me, the emerging pattern is one that pushes cybersecurity risk management into overall corporate risk management and governance, as opposed to its traditional place within a technology silo. Ultimately this is good news for the user as it adds urgency to doing cybersecurity well - and minimizing user risk in the process.

Neray: The implications, in that case, would be quite broad for other organizations, because it would put management teams on notice that they need to be providing CISOs with sufficient budget and resources — as well as management level authority and prioritization — to implement best practice controls for their organizations.

Bisson : Liability for software security has been evolving slowly relative to the rapid growth in the importance of software and technology to business, but companies should look to the evolution of automobile and product safety liability to see where cybersecurity liability might go next. Just as Unsafe at Any Speed led to consumer advocacy organizations, industry groups, and government regulation to establish and raise standards for automotive safety, the same is happening in cybersecurity. The Whitehouse’s 2021 landmark executive order on cybersecurity is just one example.

Agarwal:  There’s an uncomfortable point at which investment in a company’s growth phase occurs: sales and marketing take off with 100+% growth goals. Software delivery gets pulled into the same rapid pace with new and exciting prospect demands. Security investment will always lag sales or delivery goals in this scenario. Lawsuits such as this make the cost of sale[s growth] explicit, bringing to light this often-invisible externality. As organizations race to meet customer demands for a new feature or product line, it’s wise for them to threat model how these changes will affect the motivation, opportunity, and impact of adversaries targeting them or their customers. 

At its core, this breach comes down to the complexity of today’s developer pipelines. With the continued move to more aggressive DevOps that include many different components such as source code, open-source packages, and APIs, it is very rare that one person or team understands the threat landscape of the entire application, system, or appliance. Organizations need to better understand how their systems work and what type of threats the architecture may be prone to. One way to better understand this is through a formal process that asks that hard question of “What if?”

Bambenek: Hopefully, this will make CISOs get serious about ensuring basic security hygiene is in place…such as not using silly default passwords. 

Hellickson: I believe this will result in legal teams reviewing their own public statements and balancing those statements with security assessments performed by third parties to reduce the liability of overpromising statements on the robustness of their security posture to both the public and their clients. Additionally, I would expect organizations to expand their internal public media training beyond the typical senior executives who get trained on making public statements, to those in security positions of authority, especially considering cybersecurity is often a top three risk at any given organization.

My hope is that we’ll also see CEOs take note by moving the CISO to a direct report, enabling the CISO to have a true seat in the C-Suite, which is often covered by the CIO today.

Security : What are the lessons learned for organizations and security leaders?  

Ellis: The lesson for security leaders and organizations is that we’re entering a season of accountability for doing the basics well, and that it’s as good a catalyst as any to revisit the kinds of issues that contributed to the SolarWinds breach, remediate or mitigate when necessary, and introduce controls to avoid them in the future.

Bisson : Companies of all sizes learned about the risk of supply chain attacks as attackers used SolarWinds to gain entry into the most sensitive networks in government and industry. But we’ve also learned about the importance of strong passwords, the risks of secrets in code, and the reality that small security mistakes can have huge implications.

Forward-leaning companies recognize that security is a process, not a product — but some of those processes can be automated. [For example], automated scanning and enforcement of access permissions and activity might have identified the long-running remote access of the software workflow by external threat actors.

Companies that automate the basics demonstrate to their teams the priority of security, giving them structure and space to see and address more significant issues.

Agarwal:  It’s long been understood that the CISO’s head could roll when a breach showed the firm negligent. Security maturity models, such as the BSIMM, show the [steps] organizations are expected to take to avoid this situation. In laying out a minimum standard, those models predicted that there would be personal liability for executives that failed to meet minimum standards. An interest in cyber insurance and what kind of underwriting might support that insurance has emerged in a variety of spaces — CSPs seeking “shared outcome,” highly regulated industries seeking to add “security” to “compliance,” and others.

We see all the forces motivating personal liability (and insurance) converging: customers beginning to understand the impact and prevalence of attacks on their digital life and assets, the measurability of a firm’s security initiative maturity and posture, and a distinct focus by attackers on those whose security investments dramatically lag their software and company’s growth.

There are claims that investors “aimed to keep costs low to eventually sell the company at a profit.” These claims, along with the suspect timing of executive stock sales ahead of disclosing the breach, will be adjudicated in the courts, but leaders should take note of how these stories affect perceptions of the cybersecurity industry generally.

Hellickson: There are several lessons from the breach and associated lawsuit that come to mind, particularly around speaking openly about one’s security program. CISOs often joke that their title is more akin to the ‘Chief Scapegoat Officer’ of the organization if the worst-case scenario does occur while on their watch. This lawsuit specifically naming the VP of Security Architecture as a defendant and considering he did not have the CISO at the time of the breach, likely has many security professionals in leadership roles recollecting previous statements they may have recently made about their company’s own security posture, even if they don’t have the CISO title. When security leaders speak publicly about the state of security at their company, they’ve always had to consider their role as spokespersons for the organization, and usually refrain from ‘touting’ a robust state of their security posture. This lawsuit underscores the need to choose one’s words carefully, particularly if they are in a leadership position where people external to the organization can treat them as statements of fact.

Additionally, when organizations undergo cost-cutting efforts, the security organization often gets similar requests to cut costs and do more with less. It is up to the security executive to be as open and honest about the current threat landscape and known risks with the executive team and their Board of Directors and leverage their own expertise to advise on the impacts of making such cuts. When such requests to security leaders are made, depending on how deep the impact to the budget may be, I often encourage leaders to perform a zero-based budget exercise and classify all current security operating costs and future needs into three buckets:

• Critical operations / keep the lights on.
• Discretionary but necessary (a gray area).
• Discretionary items the security leader has already cut as part of the initiative.

For items in the second category, I recommend the CISO engage the executive team and/or risk committee to discuss and justify why those items were not chosen to be cut and defer any further cuts to be decided by those groups. What they will generally find is that if the respected security leader does not support the cuts, no one else will want to be on record for making the decision to cut further. Now, this may limit the longevity of that security leader’s employment at the given organization for not cutting as deep, but one could argue it’s what’s best for the organization while limiting personal liability when done appropriately.

A spokesperson for SolarWinds provided the following statement to Security :

“We disagree strongly with the claims made by the plaintiff and look forward to having the opportunity to present the true facts as this process continues beyond its current very early stage.”

Share This Story

Maria Henriquez is a former Associate Editor of Security . She covered topics including cybersecurity and physical security, risk management and more.

Restricted Content

You must have JavaScript enabled to enjoy a limited number of articles over the next 30 days.

Related Articles

Class Action Lawsuit Filed Against Target for Data Breach

Lessons learned: Six COVID-19 takeaways for CISOs from a CISO

Dunkin' Donuts settles data breach lawsuit

Sign-up to receive top management & result-driven techniques in the industry., join over 20,000+ industry leaders who receive our premium content..

Copyright ©2024. All Rights Reserved BNP Media.

Design, CMS, Hosting & Web Development :: ePublishing

Special Features

Vendor voice.

Judge mostly drags SEC's lawsuit against SolarWinds into the recycling bin

Russia-invaded software biz 'grateful for the support we have received'.

A judge has mostly thrown out a lawsuit brought by America's financial watchdog that accused SolarWinds and its chief infosec officer of misleading investors about its computer security practices and the backdooring of its Orion product.

In a Thursday ruling [PDF], US federal district Judge Paul Engelmayer dismissed all of the so-called "post-SUNBURST" claims the SEC levied against SolarWinds. That is to say, all the claims against SolarWinds for what followed the 2019-2020 SUNBURST attack.

SUNBURST is the code-name for some technologically top-notch backdoor malware Russian spies planted in the IT network monitoring software suite Orion after the snoops gained access to SolarWinds' internal infrastructure.

Orion is used by some 18,000 orgs including Microsoft and US government departments of State, Treasury, Homeland Security, and Commerce, making this a classic supply-chain attack. Infect a product a lot of valuable targets use so that when they come to deploy that compromised code in their networks, now you have remote-control access to those systems.

In its lawsuit, the SEC alleged SolarWinds and CISO Timothy Brown underhandedly played down the scope and severity of the cyberattack to the world, which included investors. Following a motion by SolarWinds to have those allegations binned, Judge Engelmayer rejected those particular claims in his 107-page opinion. 

"These do not plausibly plead actionable deficiencies in the company's reporting of the cybersecurity hack," Engelmayer wrote. "They impermissibly rely on hindsight and speculation."

The judge also tossed out the SEC's claims relating to SolarWinds' internal accounting and disclosure controls and procedures. 

• SolarWinds says SEC sucks: Watchdog 'lacks competence' to regulate cybersecurity
• SolarWinds charged after SEC says biz knew IT was leaky ahead of SUNBURST attack
• SolarWinds slams SEC lawsuit against it as 'unprecedented' victim blaming
• Uber ex-CSO Joe Sullivan: We need security leaders running to work, not giving up

Engelmayer did, however, sustain the regulator's claims of securities fraud based on SolarWinds' pre -SUNBURST statement about the security of its Orion product. Those allegations being:

The SEC contends SolarWinds hid the fact that its products and practices had porous cybersecurity. The SEC contends that the company's hype misled the investing public to believe that SolarWinds' central software product had minimal vulnerability to cyberattacks. 

Other statements and filings made by SolarWinds supported the SEC's claims regarding the developer's "porous" security, the judge noted. These charges will proceed, and there's no word on whether the SEC will appeal the ruling.

A SEC spokesperson declined to comment on the judge's opinion. SolarWinds, however, applauded the decision.

"We are pleased that Judge Engelmeyer has largely granted our motion to dismiss the SEC's claims," a SolarWinds spokesperson told The Register . "We look forward to the next stage, where we will have the opportunity for the first time to present our own evidence and to demonstrate why the remaining claim is factually inaccurate."

The spokesperson also said the company is "grateful for the support we have received thus far across the industry, from our customers, from cybersecurity professionals, and from veteran government officials who echoed our concerns, with which the court agreed." ®

• US Securities and Exchange Commission

Narrower topics

• Advanced persistent threat
• Application Delivery Controller
• Authentication
• Common Vulnerability Scoring System
• Cybersecurity
• Cybersecurity and Infrastructure Security Agency
• Cybersecurity Information Sharing Act
• Data Breach
• Data Protection
• Digital certificate
• Identity Theft
• Incident response
• Kenna Security
• Palo Alto Networks
• Quantum key distribution
• Remote Access Trojan
• RSA Conference
• Surveillance
• Trusted Platform Module
• Vulnerability

Broader topics

• Federal government of the United States

Send us news

Other stories you might like

Google reportedly in talks to buy infosec outfit wiz for $23b, forget security – google's recaptcha v2 is exploiting users for profit, frostygoop malware shut off heat to 600 ukraine apartment buildings, accelerate migration and go beyond virtualisation to cloud native.

DarkGate, the Swiss Army knife of malware, sees boom after rival Qbot crushed

Kaspersky gives us customers six months of free updates as a parting gift, crowdstrike's falcon sensor also linked to linux kernel panics and crashes, cybercriminals quickly exploit crowdstrike chaos, kaspersky challenges us government to put up or shut up about kremlin ties, merged exabeam and logrhythm cut jobs, face lawsuit, mozilla thunderbird finally gets system tray notifications, kaspersky culls staff, closes doors in us amid biden's ban.

• Advertise with us

Our Websites

• The Next Platform
• Blocks and Files

Your Privacy

• Cookies Policy
• Privacy Policy
• Ts & Cs

Copyright. All rights reserved © 1998–2024

SolarWinds Patches 8 Critical Flaws in Access Rights Manager Software

SolarWinds has addressed a set of critical security flaws impacting its Access Rights Manager (ARM) software that could be exploited to access sensitive information or execute arbitrary code.

Of the 13 vulnerabilities, eight are rated Critical in severity and carry a CVSS score of 9.6 out of 10.0. The remaining five weaknesses have been rated High in severity, with four of them having a CVSS score of 7.6 and one scoring 8.3.

The most severe of the flaws are listed below -

• CVE-2024-23472 - SolarWinds ARM Directory Traversal Arbitrary File Deletion and Information Disclosure Vulnerability
• CVE-2024-28074 - SolarWinds ARM Internal Deserialization Remote Code Execution Vulnerability
• CVE-2024-23469 - Solarwinds ARM Exposed Dangerous Method Remote Code Execution Vulnerability
• CVE-2024-23475 - Solarwinds ARM Traversal and Information Disclosure Vulnerability
• CVE-2024-23467 - Solarwinds ARM Traversal Remote Code Execution Vulnerability
• CVE-2024-23466 - Solarwinds ARM Directory Traversal Remote Code Execution Vulnerability
• CVE-2024-23470 - Solarwinds ARM UserScriptHumster Exposed Dangerous Method Remote Command Execution Vulnerability
• CVE-2024-23471 - Solarwinds ARM CreateFile Directory Traversal Remote Code Execution Vulnerability

Successful exploitation of the aforementioned vulnerabilities could allow an attacker to read and delete files and execute code with elevated privileges.

The shortcomings have been addressed in version 2024.3 released on July 17, 2024, following responsible disclosure as part of the Trend Micro Zero Day Initiative (ZDI).

The development comes after the U.S. Cybersecurity and Infrastructure Security Agency (CISA) placed a high-severity path traversal flaw in SolarWinds Serv-U Path (CVE-2024-28995, CVSS score: 8.6) to its Known Exploited Vulnerabilities (KEV) catalog following reports of active exploitation in the wild.

The network security company was the victim of a major supply chain attack in 2020 after the update mechanism associated with its Orion network management platform was compromised by Russian APT29 hackers to distribute malicious code to downstream customers as part of a high-profile cyber espionage campaign.

The breach prompted the U.S. Securities and Exchange Commission (SEC) to file a lawsuit against SolarWinds and its chief information security officer (CISO) last October alleging the company failed to disclose adequate material information to investors regarding cybersecurity risks.

However, much of the claims pertaining to the lawsuit were thrown out by the U.S. District Court for the Southern District of New York (SDNY) on July 18, stating "these do not plausibly plead actionable deficiencies in the company's reporting of the cybersecurity hack" and that they "impermissibly rely on hindsight and speculation."

Cybersecurity Webinars

Learn how to turn your developers into security champions.

Struggling with developer resistance to security guidelines? Discover how Security Champions can change that dynamic. Register now.

Explore All-in-One Solutions with Industry Experts

Guard your business like a Fortune 500 with a fraction of the resources. Find out why All-in-One solutions are a game-changer.

Leveraging AI as a Tool in Threat Management

7 Resources to Inform Your Next Hunt for Malicious Infrastructure

Exploitability is the Missing Puzzle Piece of SCA (Software Composition Analysis)

9 Customer Service Chatbots Ranked For Risk Exposure

Get the latest news, expert insights, exclusive resources, and strategies from industry leaders – all for free.