Posted on Leave a comment

The Russian SolarWinds cybersecurity attack is worse than we think. In fact, the U.S. may have lost the keys to the kingdom forever.

SolarWinds building

Nobody can be more tight-lipped than the U.S. federal government, especially when the topic stands to embarrass the country’s leaders. But let’s be clear – the 2020 SolarWinds cyberattack is the worst cyberattack in our nation’s history. It is literally “beyond measure” at this point. In fact, the government may never know how severely the attack compromised our nation’s digital infrastructure. So why isn’t the historic incident bigger news?

The Attack

The SolarWinds attack is believed to have begun around October 2019. Initially, Austin-based SolarWinds was blamed for a vulnerability in its network. Later however, it was found that other, potentially more effective channels were used (e.g. Microsoft resellers).

Code injected into InventoryManager.cs which initiates the Sunburst backdoor.

The cyberattack began after Russian hackers compromised the software signature of SolarWind’s Orion network monitoring software, possibly via an “inside job” that exposed the company’s private key. This let the attackers use SolarWinds’ certificate to sign Orion software updates. With the cert in hand, from March through June 2020, the hackers then distributed malware (SUNBURST/SUPERNOVA) as an Orion software update (in builds 2019.4 HF 5 through 2020.2.1).

More specifically, a digitally-signed component of the Orion software framework, SolarWinds.Orion.Core.BusinessLayer.dll, was injected into the update channel. The .dll contained a backdoor (see source code example to the right) that let the hackers communicate via HTTP to third-party servers. About 18,000 SolarWinds customers downloaded the infected updates.

Once the malware-infected update was installed on the target system, it remained dormant for a few weeks, hiding quietly before mimicking SolarWinds API communications to masquerade as Orion Improvement Program software.

The malware took care to avoid detection. For instance, exfiltrated data was stored alongside valid Orion plugin data making it more difficult to recognize. Most importantly, the malware had administrative permissions, granting it total control over the compromised system. Microsoft explained the criticality of the situation:

Email to SolarWinds notifying a FTP password leaked on GitHub

“An intruder using administrative permissions acquired through an on-premises compromise to gain access to an organization’s trusted SAML token- signing certificate. This enables them to forge SAML tokens that impersonate any of the organization’s existing users and accounts, including highly privileged accounts.”

The malware payload was the full package. It included the ability to execute commands (jobs) that let it transfer files to the server, execute files on the server, profile the system, disable selected system services, and reboot the machine.

It is not known how the private key used to sign Orion updates was obtained. However, in November 2019, around the time of the first attack, SolarWinds was notified that its ftp credentials, with a stupidly easy password, were sitting quietly in an obscure library on GitHub. It is believed the credentials would allow the hacker to upload malicious code into Orion’s update library.

Why the attack was not immediately discovered

One of the most surprising, and detrimental, aspects of the attack was the length of time it went unnoticed. For a variety of reasons, the malware remained undetected on SolarWinds Orion servers for several months.

Subterfuge under cover

The SolarWinds attacks were always conducted from servers inside the United States (e.g. GoDaddy). In some cases, the attackers even used servers in the city’s where the target company resided in order to avoid suspicion. Since the attacks originated in the USA, the NSA, who cannot legally engage in domestic surveillance, were unable to detect the activity.

Warning systems fail

The U.S. Cyber Command places sensors inside foreign networks to detect attacks as they happen. For a yet-to-be-disclosed reason, those sensors never fired. Clearly there were no failover sensors in place nor were sophisticated U.S. Artificial Intelligence mechanisms able to detect the anomalous activity.

SVR Emblem.svg
Seal of Russia’s SVR RF

There are a few ways the hackers could avoid the sensors. It was long known that the SVR, Russia’s foreign intelligence agency, had gained access to the NetWitness Investigator Tool which the United States uses to detect and close Russian back doors in software systems. After gaining access, the Russians did not disable the system but rather, modified it so Russian hackers could continue to evade detection. Russia’s reaction to this type of software has been clearly demonstrated – turn the opponent’s systems against themselves. It is likely the Cyber Command sensors were discovered by the Russians and reprogrammed to trick Americans into complacency.

Cyber Command sensors weren’t the only defensive failure. Homeland Security’s malware detection system failed to detect the attack. The Einstein detection system, used to detect malware in government systems, failed to detect the intrusion. The malware was likely not recognized because the attack came from a trusted source (SolarWinds’ Orion software update system).

The Americans turned their focus elsewhere

The footprint of the Russia attack was large and wide and should have been easily noticed. However, the U.S. had bigger battles to tackle. Authorities sought to avoid a repeat of the 2016 election attack and turned their attention to preventing a similar attack on the 2020 election. With attention diverted elsewhere, the software supply chain was an easy target.

SolarWinds history of insecure software and lax security practices

Fireeye logo

SolarWinds quickly claimed they were a “victim of a highly-sophisticated, complex and targeted cyberattack”. However, according to current and former employees, SolarWinds, the software that acted as the conduit for the attacks, had a history of lackluster security. Employees say security was not a priority for the company who instead, focused on software enhancements and cost efficiencies.

According to its customers, SolarWinds requires installation directories be exempted from virus-scanning software. Any compromised files within their directory structure would go unnoticed (unless SolarWinds themselves monitored the stack for intrusions).

SolarWinds’ poor security practices are not new. A former cybersecurity adviser had warned the company that their security policies were poor. He says he left the company when his recommendations were ignored. In fact, it was only in 2017, after a European privacy law threatened penalties against the company, they hired their first CIO and VP of security architecture.

Even SolarWinds response to the 2020 Russian attack was pitifully slow. In fact, it was five days after FireEye announced the vulnerability in SolarWinds Orion software that the company removed the infected software from their distribution network.

An inside job?

Shortly after the attack, it was discovered that foreign contractors worked on critical infrastructure software. SolarWinds used engineers in Eastern Europe, Czech Republic, Poland, and Belarus, places where Russian intelligence operations have a strong foothold. Even SolarWinds employees were unaware of foreign involvement.

Attackers use of Microsoft resellers rather than Microsoft itself

new microsoft365 logo horiz c gray rgb

It is believed the SolarWinds channel was not the only attack vector. In addition to the SolarWinds channel, attackers used Microsoft resellers to attack through Microsoft Office 365 software.

Resellers have access to software distribution channels via unique arrangements with Microsoft. This makes resellers a powerful distribution point – one that is likely not as secure as Microsoft’s official channel. An attack against Target in 2013 used a similar method. The Target attack was believed to have been initiated by an Eastern European country.

The SolarWinds attack was brilliant

Failure of important sensors, lax SolarWinds security practices, and inattentive guardians at the gate may not have mattered. This was a very sophisticated attack. Domain resolution was jacked to resolve IP addresses to hosts commonly used by the target allowing the malware traffic to blend in seamlessly with legitimate traffic. And C2 (Command and Control) IP addresses used to communicate back to the attackers were always within the target’s country – and the IP addresses were rotated frequently. A different IP address was used for each of the hacker’s tasks.

Once the network was compromised, the credentials were disposed of and different credentials used for each lateral movement within the network. Files for legitimate utilities were replaced with compromised files, then restored back to the legitimate file after the payload was executed. Similarly, scheduled tasks were manipulated to execute the hacker’s tools, then returned to the original scheduled task to avoid detection. Backdoors were removed once legitimate access was obtained. The malware even checked to see if the infected systems was in networks belonging to SolarWinds and gracefully uninstalled itself to avoid detection.

FireEye called the attack the “best operation security” they had ever seen in a cyberattack.

The discovery of the attack

The SolarWinds attack was not discovered by SolarWinds, the NSA, or the USA’s Cyber Command – it was discovered by a private security company – FireEye. Senator Mark Warner noted:

“If FireEye had not come forward, I’m not sure we would be fully aware of it to this day.”

Domestic servers inside the United States were used for the attack. It was on these servers that FireEye discovered remnants of the attack. Hackers covered their tracks well on target systems they penetrated but like many hacking missions, the attackers were less careful on the machines they used for the attacks.

The intent of the SolarWinds cyberattack

Investigators believe the attack was a Russian state-sponsored attack. This belief is held in part because the attack was so sophisticated. To penetrate and install malware on tens of thousands of systems and remain undetected for several months is no small feat. For instance, the C2 (Command and Control) server (avsvmcloud[.]com) contained a killswitch that could be used to turn the malware against itself. The killswitch triggering mechanism was brilliant. The IP address behind the C2 domain was used as a signal to the malware – if set to a specific value, the malware would deactivate itself and initiate a cleaning process.

The targets

The US believes only Russia could mount an attack this sophisticated. Still, one month later and American officials still don’t know what Russia was trying to accomplish.

electronicdesign 26314 powergrid 695555248

The U.S. government was the primary objective. The Treasury Department, State Department, Commerce Department, Energy Department, and the Pentagon were all compromised. The Defense Department claims the attacks against it were unsuccessful.

Research institutions and large, critical corporations were also targeted. Microsoft (who initially said it had not been breached, only to discover later that it had), Cisco, Intel, and Nvidia were all compromised.

A somewhat odd target were the energy grids. Russia almost certainly had malware already planted in electric grid systems (these systems are notoriously insecure). Why they were attacked again is not clear. Possibly to embed redundant or improved Command and Control (C2) points deeper into the infrastructure.

The objective

Or course it is impossible to know Russia’s true intention but several theories have been offered. The objective could be as simple as destabilizing the Biden administration before it even takes office. This could be especially important to leverage future nuclear talks. Russian may have been trying to diminish confidence in the U.S. military and infrastructure security to gain leverage in nuclear arms talks with Biden. Suzanne Spaulding, former senior cyberofficial at the Homeland Security Department suggested such.

“We still don’t know what Russia’s strategic objectives were. But we should be concerned that part of this may go beyond reconnaissance. Their goal may be to put themselves in a position to have leverage over the new administration, like holding a gun to our head to deter us from acting to counter Putin.”

But these explanations would presume the attack was discovered. It is more likely that backdoors were installed to ensure control of America’s critical infrastructure.

What the government says happened in the SolarWinds attack vs. what really happened

To be clear, the Russia attack against SolarWinds was the ultimate embarrassment for U.S. officials. The breadth of the attack, the time it went undiscovered, and the degree of control the hackers gained over the targeted systems is unparalleled. The government has every reason to deflect the story and little reason to come clean.

Government says SolarWinds is to blame

SolarWinds was the first to be blamed but later was it found that “other channels” were used too. Microsoft first hinted that their systems were penetrated then confirmed that resellers were compromised. Obsidian Security noted,

“They targeted the weakest points in the supply chain and through our most trusted relationships.”

The Microsoft compromise could be a much bigger problem. If Microsoft distribution channels were compromised, potentially every Windows-based computer system in the country could be targeted.

Government says hackers were unable to access classified systems containing sensitive plans

The government was quick to claim the attackers did not penetrate classified systems. However, at the same time officials admitted they did not have a clear picture of what might have been stolen.

Hackers owned the entire target network. They had access to any system they wanted and they had little reason to voluntarily stop at classified systems. We can only hope that 2FA or other protection mechanisms protected those critical systems. If the attackers were able to access classified systems – this would easily be the biggest embarrassment to U.S. security in the history of the country. The United States has every reason to deny their classified systems were penetrated and little reason to admit they were compromised.

Government says hackers merely exfiltrated data

The government softened the consequences from the attack, saying the hackers merely exfiltrated data. But the malware had administrative permissions across the entire target network. This gives them much more authority than the permissions required to just extract data. In fact, the malware could disable system services, reboot machines, execute files, and change system configurations. The hackers maintained this level of permissions on every machine within the targeted network that was controlled or monitored by SolarWinds Orion software. In most companies, this includes every piece of hardware in their company.

Government says the hackers have been shut down

In past Russian intrusions, America has had a tough time kicking the hackers out of the systems. In many cases, the hackers would be expunged only to return using another door. The SolarWinds attackers surely planted backdoors – many of them – in systems across the government and big business networks. That means backdoors exist on every system inside the target’s network – including those inevitable machines that companies have forgotten about and left running, unattended, is some dusty rack in the back of the server room.

Have the keys to the kingdom been lost?

We can dream a myriad of responses to the attack. Maybe the U.S. offers to replace all systems (monetarily) and maintains the compromised systems in some sort of honeypot configuration until they’ve figured out the protective/defensive mechanism that will be effective. But that’s not financially feasible nor has the government demonstrated the prowess to operate such a honeypot.

It is likely that all backdoors will never be discovered which means there is only one way forward – build new, better detection and protection frameworks, shut off all compromised systems, and begin anew. But even this solution is cost prohibitive. The 2019-2020 coronavirus pandemic makes this all but impossible.

Until a solution can be found, Russia possesses the keys to the kingdom – and they will likely hold those keys for many years to come. The only recourse for America will be a full-scale cyberattack against the perpetrators – a tit-for-tat response that ensures the U.S. possesses the same level of control over their attackers – and hope Russia does not use their newfound keys to counter.

SunBurst initialize
The entry point of the Sunburst backdoor, the Initialize() function.

Additional information

Sunburst process filename checks

Processes that Sunburst malware looks for. Processes were hashed to avoid detection.

Process filename checks (137):
2597124982561782591uL// apimonitor-x64
2600364143812063535uL// apimonitor-x86
13464308873961738403uL// autopsy64
4821863173800309721uL// autopsy
12969190449276002545uL// autoruns64
3320026265773918739uL// autoruns
12094027092655598256uL// autorunsc64
10657751674541025650uL// autorunsc
11913842725949116895uL// binaryninja
292198192373389586uL// cff explorer
12790084614253405985uL// cutter
5219431737322569038uL// de4dot
15535773470978271326uL// debugview
7810436520414958497uL// diskmon
13316211011159594063uL// dnsd
13825071784440082496uL// dnspy
14480775929210717493uL// dotpeek32
14482658293117931546uL// dotpeek64
8473756179280619170uL// dumpcap
12027963942392743532uL// fakedns
576626207276463000uL// fakenet
7412338704062093516uL// ffdec
682250828679635420uL// fiddler
13014156621614176974uL// fileinsight
18150909006539876521uL// floss
10336842116636872171uL// gdb
13260224381505715848uL// hiew32
8709004393777297355uL// idaq64
14256853800858727521uL// idaq
8129411991672431889uL// idr
15997665423159927228uL// ildasm
10829648878147112121uL// ilspy
9149947745824492274uL// jd-gui
3656637464651387014uL// lordpe
4501656691368064027uL// ollydbg
10296494671777307979uL// pdfstreamdumper
14630721578341374856uL// pe-bear
9531326785919727076uL// peid
6461429591783621719uL// pe-sieve32
6508141243778577344uL// pe-sieve64
10235971842993272939uL// pestudio
2478231962306073784uL// peview
9903758755917170407uL// pexplorer
14710585101020280896uL// ppee
14710585101020280896uL// ppee
13611814135072561278uL// procdump64
2032008861530788751uL// processhacker
27407921587843457uL// procexp64
6491986958834001955uL// procexp
2128122064571842954uL// procmon
10484659978517092504uL// prodiscoverbasic
10463926208560207521uL// r2agent
7080175711202577138uL// rabin2
8697424601205169055uL// radare2
16130138450758310172uL// ramcapture
18294908219222222902uL// regmon
3588624367609827560uL// resourcehacker
9555688264681862794uL// retdec-ar-extractor
5415426428750045503uL// retdec-bin2llvmir
3642525650883269872uL// retdec-bin2pat
13135068273077306806uL// retdec-config
3769837838875367802uL// retdec-fileinfo
191060519014405309uL// retdec-getsig
1682585410644922036uL// retdec-idr2pat
7878537243757499832uL// retdec-llvmir2hll
13799353263187722717uL// retdec-macho-extractor
1367627386496056834uL// retdec-pat2yara
12574535824074203265uL// retdec-stacofin
16990567851129491937uL// retdec-unpacker
8994091295115840290uL// retdec-yarac
13876356431472225791uL// rundotnetdll
14968320160131875803uL// sbiesvc
14868920869169964081uL// scdbg
106672141413120087uL// scylla_x64
79089792725215063uL// scylla_x86
14111374107076822891uL// sysmon
17351543633914244545uL// tcpdump
7516148236133302073uL// tcpvcon
15114163911481793350uL// tcpview
15457732070353984570uL// vboxservice
16292685861617888592uL// win32_remote
10374841591685794123uL// win64_remotex64
3045986759481489935uL// windbg
17109238199226571972uL// windump
5945487981219695001uL// winhex
8052533790968282297uL// winobj
17574002783607647274uL// wireshark
3341747963119755850uL// x32dbg
14193859431895170587uL// x64dbg
17439059603042731363uL// xwforensics64
17683972236092287897uL// xwforensics
3660705254426876796uL// avgsvc
12709986806548166638uL// avgui
3890794756780010537uL// avgsvca
3890769468012566366uL// avgsvcx
14095938998438966337uL// avgwdsvcx
11109294216876344399uL// avgadminclientservice
1368907909245890092uL// afwserv
11818825521849580123uL// avastui
13029357933491444455uL// aswidsagenta
16423314183614230717uL// bccavsvc
2532538262737333146uL// psanhost
4454255944391929578uL// psuaservice
6088115528707848728uL// psuamain
13611051401579634621uL// avp
18147627057830191163uL// avpui
17633734304611248415uL// ksde
13581776705111912829uL// ksdeui
7175363135479931834uL// tanium
13599785766252827703uL// taniumdetectengine
Driver filenames (17):
17097380490166623672uL// cybkerneltracker.sys
15194901817027173566uL// atrsdfw.sys
12718416789200275332uL// eaw.sys
18392881921099771407uL// rvsavd.sys
3626142665768487764uL// dgdmk.sys
12343334044036541897uL// sentinelmonitor.sys
397780960855462669uL// hexisfsmonitor.sys
6943102301517884811uL// groundling32.sys
13544031715334011032uL// groundling64.sys
11801746708619571308uL// safe-agent.sys
18159703063075866524uL// crexecprev.sys
835151375515278827uL// psepfilter.sys
16570804352575357627uL// cve.sys
1614465773938842903uL// brfilter.sys
12679195163651834776uL// brcow_x_x_x_x.sys
2717025511528702475uL// lragentmf.sys
17984632978012874803uL // libwamf.sys
Services (8):
Windows Defendermsmpeng
Windows Defender Advanced Threat Protectionmssense
Carbon Black cavpcb
Crowdstrike csfalconservicecsfalconcontainer
Eset ekrneguiekrnepfw
F-Secure fsgk32fsma32fssm32fnrb32fsauafsorspfsav32fsdevconfsgk32stfswebuid
  • © 2021 GitHub, Inc.