Nobody can be more tight-lipped than the U.S. federal government, especially when the topic stands to embarrass the country’s leaders. But let’s be clear – the 2020 SolarWinds cyberattack is the worst cyberattack in our nation’s history. It is literally “beyond measure” at this point. In fact, the government may never know how severely the attack compromised our nation’s digital infrastructure. So why isn’t the historic incident bigger news?

The Attack

The SolarWinds attack is believed to have begun around October 2019. Initially, Austin-based SolarWinds was blamed for a vulnerability in its network. Later however, it was found that other, potentially more effective channels were used (e.g. Microsoft resellers).

The cyberattack began after Russian hackers compromised the software signature of SolarWind’s Orion network monitoring software, possibly via an “inside job” that exposed the company’s private key. This let the attackers use SolarWinds’ certificate to sign Orion software updates. With the cert in hand, from March through June 2020, the hackers then distributed malware (SUNBURST/SUPERNOVA) as an Orion software update (in builds 2019.4 HF 5 through 2020.2.1).

More specifically, a digitally-signed component of the Orion software framework, SolarWinds.Orion.Core.BusinessLayer.dll, was injected into the update channel. The .dll contained a backdoor (see source code example to the right) that let the hackers communicate via HTTP to third-party servers. About 18,000 SolarWinds customers downloaded the infected updates.

Once the malware-infected update was installed on the target system, it remained dormant for a few weeks, hiding quietly before mimicking SolarWinds API communications to masquerade as Orion Improvement Program software.

The malware took care to avoid detection. For instance, exfiltrated data was stored alongside valid Orion plugin data making it more difficult to recognize. Most importantly, the malware had administrative permissions, granting it total control over the compromised system. Microsoft explained the criticality of the situation:

“An intruder using administrative permissions acquired through an on-premises compromise to gain access to an organization’s trusted SAML token- signing certificate. This enables them to forge SAML tokens that impersonate any of the organization’s existing users and accounts, including highly privileged accounts.”

The malware payload was the full package. It included the ability to execute commands (jobs) that let it transfer files to the server, execute files on the server, profile the system, disable selected system services, and reboot the machine.

It is not known how the private key used to sign Orion updates was obtained. However, in November 2019, around the time of the first attack, SolarWinds was notified that its ftp credentials, with a stupidly easy password, were sitting quietly in an obscure library on GitHub. It is believed the credentials would allow the hacker to upload malicious code into Orion’s update library.

Why the attack was not immediately discovered

One of the most surprising, and detrimental, aspects of the attack was the length of time it went unnoticed. For a variety of reasons, the malware remained undetected on SolarWinds Orion servers for several months.

Subterfuge under cover

The SolarWinds attacks were always conducted from servers inside the United States (e.g. GoDaddy). In some cases, the attackers even used servers in the city’s where the target company resided in order to avoid suspicion. Since the attacks originated in the USA, the NSA, who cannot legally engage in domestic surveillance, were unable to detect the activity.

Warning systems fail

The U.S. Cyber Command places sensors inside foreign networks to detect attacks as they happen. For a yet-to-be-disclosed reason, those sensors never fired. Clearly there were no failover sensors in place nor were sophisticated U.S. Artificial Intelligence mechanisms able to detect the anomalous activity.

There are a few ways the hackers could avoid the sensors. It was long known that the SVR, Russia’s foreign intelligence agency, had gained access to the NetWitness Investigator Tool which the United States uses to detect and close Russian back doors in software systems. After gaining access, the Russians did not disable the system but rather, modified it so Russian hackers could continue to evade detection. Russia’s reaction to this type of software has been clearly demonstrated – turn the opponent’s systems against themselves. It is likely the Cyber Command sensors were discovered by the Russians and reprogrammed to trick Americans into complacency.

Cyber Command sensors weren’t the only defensive failure. Homeland Security’s malware detection system failed to detect the attack. The Einstein detection system, used to detect malware in government systems, failed to detect the intrusion. The malware was likely not recognized because the attack came from a trusted source (SolarWinds’ Orion software update system).

The Americans turned their focus elsewhere

The footprint of the Russia attack was large and wide and should have been easily noticed. However, the U.S. had bigger battles to tackle. Authorities sought to avoid a repeat of the 2016 election attack and turned their attention to preventing a similar attack on the 2020 election. With attention diverted elsewhere, the software supply chain was an easy target.

SolarWinds history of insecure software and lax security practices

SolarWinds quickly claimed they were a “victim of a highly-sophisticated, complex and targeted cyberattack”. However, according to current and former employees, SolarWinds, the software that acted as the conduit for the attacks, had a history of lackluster security. Employees say security was not a priority for the company who instead, focused on software enhancements and cost efficiencies.

According to its customers, SolarWinds requires installation directories be exempted from virus-scanning software. Any compromised files within their directory structure would go unnoticed (unless SolarWinds themselves monitored the stack for intrusions).

SolarWinds’ poor security practices are not new. A former cybersecurity adviser had warned the company that their security policies were poor. He says he left the company when his recommendations were ignored. In fact, it was only in 2017, after a European privacy law threatened penalties against the company, they hired their first CIO and VP of security architecture.

Even SolarWinds response to the 2020 Russian attack was pitifully slow. In fact, it was five days after FireEye announced the vulnerability in SolarWinds Orion software that the company removed the infected software from their distribution network.

An inside job?

Shortly after the attack, it was discovered that foreign contractors worked on critical infrastructure software. SolarWinds used engineers in Eastern Europe, Czech Republic, Poland, and Belarus, places where Russian intelligence operations have a strong foothold. Even SolarWinds employees were unaware of foreign involvement.

Attackers use of Microsoft resellers rather than Microsoft itself

It is believed the SolarWinds channel was not the only attack vector. In addition to the SolarWinds channel, attackers used Microsoft resellers to attack through Microsoft Office 365 software.

Resellers have access to software distribution channels via unique arrangements with Microsoft. This makes resellers a powerful distribution point – one that is likely not as secure as Microsoft’s official channel. An attack against Target in 2013 used a similar method. The Target attack was believed to have been initiated by an Eastern European country.

The SolarWinds attack was brilliant

Failure of important sensors, lax SolarWinds security practices, and inattentive guardians at the gate may not have mattered. This was a very sophisticated attack. Domain resolution was jacked to resolve IP addresses to hosts commonly used by the target allowing the malware traffic to blend in seamlessly with legitimate traffic. And C2 (Command and Control) IP addresses used to communicate back to the attackers were always within the target’s country – and the IP addresses were rotated frequently. A different IP address was used for each of the hacker’s tasks.

Once the network was compromised, the credentials were disposed of and different credentials used for each lateral movement within the network. Files for legitimate utilities were replaced with compromised files, then restored back to the legitimate file after the payload was executed. Similarly, scheduled tasks were manipulated to execute the hacker’s tools, then returned to the original scheduled task to avoid detection. Backdoors were removed once legitimate access was obtained. The malware even checked to see if the infected systems was in networks belonging to SolarWinds and gracefully uninstalled itself to avoid detection.

FireEye called the attack the “best operation security” they had ever seen in a cyberattack.

The discovery of the attack

The SolarWinds attack was not discovered by SolarWinds, the NSA, or the USA’s Cyber Command – it was discovered by a private security company – FireEye. Senator Mark Warner noted:

“If FireEye had not come forward, I’m not sure we would be fully aware of it to this day.”

Domestic servers inside the United States were used for the attack. It was on these servers that FireEye discovered remnants of the attack. Hackers covered their tracks well on target systems they penetrated but like many hacking missions, the attackers were less careful on the machines they used for the attacks.

The intent of the SolarWinds cyberattack

Investigators believe the attack was a Russian state-sponsored attack. This belief is held in part because the attack was so sophisticated. To penetrate and install malware on tens of thousands of systems and remain undetected for several months is no small feat. For instance, the C2 (Command and Control) server (avsvmcloud[.]com) contained a killswitch that could be used to turn the malware against itself. The killswitch triggering mechanism was brilliant. The IP address behind the C2 domain was used as a signal to the malware – if set to a specific value, the malware would deactivate itself and initiate a cleaning process.

The targets

The US believes only Russia could mount an attack this sophisticated. Still, one month later and American officials still don’t know what Russia was trying to accomplish.

The U.S. government was the primary objective. The Treasury Department, State Department, Commerce Department, Energy Department, and the Pentagon were all compromised. The Defense Department claims the attacks against it were unsuccessful.

Research institutions and large, critical corporations were also targeted. Microsoft (who initially said it had not been breached, only to discover later that it had), Cisco, Intel, and Nvidia were all compromised.

A somewhat odd target were the energy grids. Russia almost certainly had malware already planted in electric grid systems (these systems are notoriously insecure). Why they were attacked again is not clear. Possibly to embed redundant or improved Command and Control (C2) points deeper into the infrastructure.

The objective

Or course it is impossible to know Russia’s true intention but several theories have been offered. The objective could be as simple as destabilizing the Biden administration before it even takes office. This could be especially important to leverage future nuclear talks. Russian may have been trying to diminish confidence in the U.S. military and infrastructure security to gain leverage in nuclear arms talks with Biden. Suzanne Spaulding, former senior cyberofficial at the Homeland Security Department suggested such.

“We still don’t know what Russia’s strategic objectives were. But we should be concerned that part of this may go beyond reconnaissance. Their goal may be to put themselves in a position to have leverage over the new administration, like holding a gun to our head to deter us from acting to counter Putin.”

But these explanations would presume the attack was discovered. It is more likely that backdoors were installed to ensure control of America’s critical infrastructure.

What the government says happened in the SolarWinds attack vs. what really happened

To be clear, the Russia attack against SolarWinds was the ultimate embarrassment for U.S. officials. The breadth of the attack, the time it went undiscovered, and the degree of control the hackers gained over the targeted systems is unparalleled. The government has every reason to deflect the story and little reason to come clean.

Government says SolarWinds is to blame

SolarWinds was the first to be blamed but later was it found that “other channels” were used too. Microsoft first hinted that their systems were penetrated then confirmed that resellers were compromised. Obsidian Security noted,

“They targeted the weakest points in the supply chain and through our most trusted relationships.”

The Microsoft compromise could be a much bigger problem. If Microsoft distribution channels were compromised, potentially every Windows-based computer system in the country could be targeted.

Government says hackers were unable to access classified systems containing sensitive plans

The government was quick to claim the attackers did not penetrate classified systems. However, at the same time officials admitted they did not have a clear picture of what might have been stolen.

Hackers owned the entire target network. They had access to any system they wanted and they had little reason to voluntarily stop at classified systems. We can only hope that 2FA or other protection mechanisms protected those critical systems. If the attackers were able to access classified systems – this would easily be the biggest embarrassment to U.S. security in the history of the country. The United States has every reason to deny their classified systems were penetrated and little reason to admit they were compromised.

Government says hackers merely exfiltrated data

The government softened the consequences from the attack, saying the hackers merely exfiltrated data. But the malware had administrative permissions across the entire target network. This gives them much more authority than the permissions required to just extract data. In fact, the malware could disable system services, reboot machines, execute files, and change system configurations. The hackers maintained this level of permissions on every machine within the targeted network that was controlled or monitored by SolarWinds Orion software. In most companies, this includes every piece of hardware in their company.

Government says the hackers have been shut down

In past Russian intrusions, America has had a tough time kicking the hackers out of the systems. In many cases, the hackers would be expunged only to return using another door. The SolarWinds attackers surely planted backdoors – many of them – in systems across the government and big business networks. That means backdoors exist on every system inside the target’s network – including those inevitable machines that companies have forgotten about and left running, unattended, is some dusty rack in the back of the server room.

Have the keys to the kingdom been lost?

We can dream a myriad of responses to the attack. Maybe the U.S. offers to replace all systems (monetarily) and maintains the compromised systems in some sort of honeypot configuration until they’ve figured out the protective/defensive mechanism that will be effective. But that’s not financially feasible nor has the government demonstrated the prowess to operate such a honeypot.

It is likely that all backdoors will never be discovered which means there is only one way forward – build new, better detection and protection frameworks, shut off all compromised systems, and begin anew. But even this solution is cost prohibitive. The 2019-2020 coronavirus pandemic makes this all but impossible.

Until a solution can be found, Russia possesses the keys to the kingdom – and they will likely hold those keys for many years to come. The only recourse for America will be a full-scale cyberattack against the perpetrators – a tit-for-tat response that ensures the U.S. possesses the same level of control over their attackers – and hope Russia does not use their newfound keys to counter.

Additional information

Sunburst process filename checks

Processes that Sunburst malware looks for. Processes were hashed to avoid detection.

• © 2021 GitHub, Inc.