U.S. Cyber Experts Scramble to Assess the Scope of the ‘Hack of a Decade’

U.S. government cyber experts are working furiously in secure offices around the globe, sifting through computer traffic to figure out which federal systems have been penetrated in the sweeping cyber-spying attack that the FBI warned this week is “significant and ongoing.” Suspected Russian hackers have broken into sensitive U.S. government computer networks from the Pentagon to the Department of Energy, as well as top U.S. private businesses, rummaging around in them and likely reading emails and gathering data.

The Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) called the attack, which started in March or possibly earlier, “a grave risk” to the U.S. government. Experts from both the government and top U.S. private firms compromised in the attack are taking whole sections of their computer networks offline or quarantining them for a deeper forensic dive to figure out what was copied or taken, and if the hackers left any malware code behind.

The hackers exploited a little-known but widely used software program called Orion made by cyber company SolarWinds, whose client list includes the Office of the U.S. President, the Pentagon, NASA, NSA, all five branches of the U.S. military and most of the Fortune 500 companies, including the top ten U.S. communications companies.

The Austin, Texas-based company removed its client list from its website after reporting the hack may have affected some 18,000 customers. The company says it has been “advised that the nature of this attack indicates that it may have been conducted by an outside nation state” and is urging clients to update their systems to remove the threat. The company did not immediately respond to request for comment. CISA referred to the attackers as “a patient, well-resourced, and focused adversary” adding that the Orion software vulnerability wasn’t the only way it attacked, but declining to share further details.

Since it was first reported by Reuters on Sunday, the known extent of the hack has grown wider each day. So far, government agencies, including the Departments of Commerce and Energy, are among those confirmed to be infected, though an Energy spokesperson said the hack only compromised “business networks” and didn’t reach the National Nuclear Security Administration. On Thursday, Microsoft reported that it has found the malware uploaded to some of its systems.

“You’ve got to assume that they’re still in right now and they’re gonna stay in,” says cyber expert Suzanne Spaulding, who led the Homeland Security office that is now known as CISA during the Obama Administration. “So you pretty quickly have to start looking at what they could have gotten access to.”

President Donald Trump has so far been silent on what appears to be one of the worst cyber attacks on the United States in decades. The White House has offered little public comment beyond confirming via Twitter that an investigation is underway, but Secretary of State Mike Pompeo called it “very significant effort” in an interview Friday, adding “we can say pretty clearly that it was the Russians that engaged in this activity. A senior administration official, speaking anonymously to describe the ongoing investigation, told TIME that the list of compromised U.S. government agencies will likely include the Pentagon, as well as other agencies that used the software.

The official also confirmed to TIME that the attackers are believed to be Russian state operatives, reported by The Washington Post to be the infamous hacking collective known as APT29, or Cozy Bear. Russia’s Ambassador to Washington Anatoly I. Antonov denounced those reports as “unfounded attempts of the U.S. media to blame Russia for hacking attacks on the U.S. government bodies,” in remarks this week to a Georgetown University virtual forum. The diplomat, who complained that he can’t get a meeting with top government officials or lawmakers to discuss a host of accusations against Moscow, offered to arrange a meeting between Russian intelligence, FBI and CIA officials to get to the bottom of it.

The Orion software is used to monitor and manage computer systems — the kind of program an in-house IT tech might use to remotely fix an employee’s computer when it has a meltdown. The hackers essentially piggybacked on the Orion program’s software updates, so when users updated their systems, they unwittingly uploaded malware that gave the hackers a backdoor, explains Dmitri Alperovitch, former co-founder and CTO of cybersecurity firm CrowdStrike. His former cybersecurity company was hired by the Democratic National Committee when its emails were hacked in 2016, which the FBI later attributed to Russia.

“When the update came down, it basically was a master key for the Russians to get access to every single one of these systems,” Alperovitch says, calling it a “hack of a decade” and “one of the most significant cyber espionage operations in history,” though it apparently did not penetrate classified systems, he says, citing conversations with industry insiders.

The FBI is leading an investigation to track the hackers, while CISA, which became a household name this year after disputing Trump’s allegations of election fraud, resulting in Trump’s firing its director, is sending out new warnings on what they’ve found, and how government and private companies should respond to fix it.

The spying attack was first discovered by cybersecurity firm FireEye, which reported being hacked by a “highly sophisticated threat actor” that it says inserted malware onto its own network via the SolarWinds Orion software, and then stole some of the cybersecurity company’s own “ethical” hacking tools used to probe FireEye clients’ networks for vulnerabilities.

Aaron Hughes, former U.S. deputy assistant secretary of defense for cyber policy, says the sophistication of targeting a contractor in the government’s supply chain to get to its clients immediately points to an actor like Russia or China. “Impersonating users, and using inherently trusted mechanisms to conduct what appear to be trusted communication…That’s not an insignificant task,” says Hughes, who is now a cyber expert with the Center for Strategic and International Studies (CSIS). “The level of effort and stealth that goes into instituting that sort of computer-to-computer communication requires the skill set and breadth of a nation state.”

There’s a ranking in terms of urgency that governs what cyber techs are combing their networks for: first, they make sure the enemy hasn’t accessed critical U.S. weapons systems, like the launch codes for nuclear bombs. Then they look at things like control of electricity or water supplies, triaging a Malthusian hierarchy of threats, says Spaulding, who is a senior Homeland Security advisor with CSIS. After you figure out what the adversary accessed, you play out the worst-case scenario of what they might do with what they were able to reach, she adds.

And the game is still on. Hackers often “store” things they want to copy within the system they’ve targeted, so they can download it in a way that won’t arouse suspicion, copying the data a few kilobytes at a time. So there may be packets of information they’ve “marked” to move that are still untouched, and the race is on to cut the hackers off, and then rebuild. “You can’t just go in and kind of surgically remove the adversary,” Spaulding explains. “You have to rebuild a completely new infrastructure that isn’t connected to the old infrastructure.”

Though the operatives were active while Americans were voting in the presidential election this fall, Spaulding doesn’t think the suspected Russian hackers were able to affect 2020 voting, because of “such overwhelming use of paper ballots and the ability to audit. I think it would have been detected.”

Russian government hackers have penetrated sensitive systems before, as they did in 2016, when Russian military intelligence officers were indicted for crimes including hacking the computers of the Democratic National Committee primarily through phishing emails.

Russian hackers were also blamed for a 2015 phishing attack that seized control of the Pentagon Joint Staff’s unclassified email systems. Retired Chairman of the Joint Chiefs Gen. Martin Dempsey told CBS News that within an hour of accessing the system, the hackers seized Dempsey’s computer credentials as well as hundreds of other senior officers, and the only way to stop it was to take the system down. Cyber expert Hughes, who was serving as the Pentagon’s cyber defense deputy assistant secretary at the time, recalls that his team had “to isolate and shut down parts of the unclassified network to identify and then eradicate the threat actor.”

Ironically, it was a failure to update software that enabled what’s still seen as the worst cyberattack on the U.S. government, when Chinese hackers stole the personnel files of 4.2 million government employees, as reported by the U.S. Office of Personnel Management in 2015, including the real names of intelligence officers serving in covert positions around the world.

Responding to news of the SolarWinds hack, the incoming Biden Administration released a statement Thursday vowing to make dealing with “what appears to be a massive cybersecurity breach affecting potentially thousands of victims …. a top priority from the moment we take office.”

Alperovitch, who is now the chairman of the Silverado Policy Accelerator, a Washington, D.C.-based cyber think tank, warned U.S. officials or lawmakers against overreacting to what he calls run-of-the-mill spying, as opposed to an act of war. “This is a case where we should say good for them, shame on us for letting them,” he says. “We need to use this as a wake-up call to reorganize our government to better deal with this massive intelligence failure and massive failure of cybersecurity.”

This article has been updated to add comment from Secretary of State Mike Pompeo in a Dec. 18 radio interview.