The White House was prepared to call out the military to guard the vote from a Russian hack
Riverside County District Attorney Michael Hestrin was at his desk on June 7, 2016, when the calls started coming in. It was the day of the California presidential primary, and upset voters wanted the county’s top prosecutor to know that they had been prevented from casting their ballots. “There were people calling our office and filing complaints that they had tried to vote and that their registration had been changed unbeknownst to them,” says Hestrin. Soon there were more than 20 reports of trouble, and Hestrin, a 19-year veteran of the office and a graduate of Stanford Law School, dispatched investigators to county polling places to see what was going on.
At first what they found was reassuring. Everyone who had been blocked from voting had been offered a provisional ballot, and most had cast their votes that way. But as the investigators dug deeper, things looked less innocuous. In the days after the vote, more people started coming forward to say they’d also had problems with their voter registration on primary day. In at least half a dozen cases, Hestrin and his investigators concluded, the changes had been made by hackers who had used private information, like Social Security or driver’s-license numbers, to access the central voter-registration database for the entire state of California.
There the trail went cold. The California secretary of state’s office told Hestrin’s investigators that the state’s system hadn’t recorded the Internet addresses of the computers that had made the changes, so there was no way to learn the identity of the hackers. Hestrin could go no further, but that wasn’t the end of it. The lingering mystery of the voter-registration changes bred doubt among members of both parties. Local Republicans publicly alleged that Democrats were ignoring the issue and privately accused them of trying to suppress the GOP vote. Democrats thought Republicans were making up an excuse for their losses at the county polls. “That was a big concern,” says Hestrin, an elected Republican. “People should still have faith in our election systems.”
It was only months later that it dawned on investigators in D.C. that undermining voters’ faith may have been the point of the Riverside County hack all along. In the months following the California primaries, the feds discovered that Russian hackers had broken into more than 20 state and local election systems and attempted to alter voter registration in several of them. Looking back at the events in Riverside County, cybersecurity officials at the White House wondered whether it had been a test run by the Russians. “It looked like a cyberattacker testing what kind of chaos they could unleash on Election Day,” says one former federal cybersecurity official who looked into the case. “There was no forensic evidence, so we may never know for sure, but the intelligence told us the Russians were bragging about doing just that.”
It is easy to forget, in the constant flurry of news, that the abiding goal of the Russian operation against the 2016 presidential election was, in the words of the U.S. intelligence community, “to undermine public faith in the U.S. democratic process.” What unfolded from early spring 2016 through the close of polls on Nov. 8 in states and counties across America was an aggressive attack on the credibility of our elections and a largely unseen and futile attempt by the federal government to counter it. The FBI, the Department of Homeland Security (DHS) and U.S. intelligence services worked to identify the hackers and determine how widespread their malicious influence operation was. The feds struggled to help states protect their ballot machines and voter-registration rolls, only to become suspected of election meddling themselves amid mounting partisanship. In the end, realizing there was little they could do to stop what they feared might be a final Russian attack on the vote, the feds worked up an extraordinary plan to limit the damage on Election Day and in the days after.
The previously undisclosed 15-page plan, produced by President Obama’s cybersecurity officials and obtained by TIME, shows just how worried Washington was. It deferred to states in most cases of a cyberincident on Election Day. But in a severe attack “likely to result in demonstrable impact to election infrastructure,” it provided for “enhanced procedures” in response. The plan allowed for the deployment of “armed federal law enforcement agents” to polling places if hackers managed to halt voting. In a crisis, it also foresaw the deployment of “Active and Reserve military forces” and members of the National Guard “upon a request from a federal agency and the direction of the Secretary of Defense or the President.” For three days after the election, a special interagency effort would be tasked with addressing “any postelection cyberincidents,” including “planted stories calling into question the results.”
On Nov. 1, the White House went so far as to war-game an Election Day attack. Over the course of five hours, the National Security Council ran a fictionalized sequence of events to rehearse how federal agencies would communicate and respond in a real attack. Some of the scenarios dealt with actual vote meddling, while others focused on disinformation efforts to undermine the election. As the nightmare scenarios unfolded–from voters turned away to violence at polling places–the team went over what actions each agency would take and what the legal constraints were on what they could do.
As it happened, Nov. 8 came and went with no final, spectacular attack on the integrity of the election. But the Russian effort may nonetheless be working, helped wittingly or otherwise by Donald Trump. Most Americans believe that their own votes will be correctly counted, but their faith that elections are honest is collapsing. In 2009, 59% of Americans had confidence in the honesty of elections, while 40% did not, according to Gallup. By 2015, those numbers had flipped, and just before the November vote, amid Trump’s repeated talk of rigged elections and the widespread coverage of Russian hacking, Gallup found that only 30% of Americans had confidence in the honesty of our elections, while 69% did not.
The diminished faith may deepen. Recent revelations and testimony have shown that the Russian operation targeting state and local voting systems was broader and more intrusive than previously thought. They have also shown that our election systems remain vulnerable to different kinds of attack designed to undermine not the vote count itself but America’s faith in the result. Which is why the story of how officials scrambled to secure the 2016 vote only to become mired in partisan suspicion is important. Because the question of U.S. vulnerability to election meddling is less about the past votes than it is about the next ones.
RUSSIA’S DANGEROUS NEW GAME
About three weeks after the Riverside County hack, a Russian agent signed on to the voter-registration website of one of Illinois’s 109 election jurisdictions, each of which has its own voting system. But instead of entering his personal information in one of the fields for names and addresses, the hacker uploaded a string of malicious prewritten code, executing a classic hack known as SQL injection. With that, the hacker opened a back door to all 15 million files on past and current voters in the state since 2006. And for nearly three weeks, no one knew he was there.
Such intrusions weren’t entirely new. Russia had been probing U.S. state and local electoral systems for years. In 2008, Moscow hacked the campaigns of both Obama and John McCain. Then, in 2014, the Russians became more brazen. “Previously, when you discovered the Russians somewhere, they disappeared like ghosts–poof!” says Michael Daniel, former White House cybersecurity coordinator. “After 2014, you’d find them in networks, and they’d stay, almost like they were taunting us. They became much more aggressive.”
Election 2016 was a step well beyond that. After the Illinois hack and a similar one at about the same time in Arizona, “we realized we were playing a different game,” Daniel says. The Russians weren’t just stealing information for the purposes of collecting intelligence as they had been in previous election cycles. Instead, Daniel’s team concluded, they were showing a possible intent to meddle with the vote.
Illinois discovered the intrusion on July 12, when the hackers triggered an alarm by trying to download the whole file of 15 million voters. Illinois officials took the system offline and found that about 90,000 files had been stolen, more than 75,000 of which included personal data like driver’s-license numbers and the last four digits of the voters’ Social Security numbers. When Illinois reported the news to the FBI in late July, the bureau dispatched a tactical Cyber Action Team to the state capital, Springfield, where the computers are kept.
Fortunately for the feds, Illinois officials had kept a full backup of all the data on the system from before the SQL attack, so the FBI was able to track what the hackers had done. Bureau agents found that while they were inside, the hackers had attempted to alter and delete information in the voter rolls. In particular, they had tried to change voters’ names and addresses. As far as they could tell, none of the efforts had been successful. Most important, Illinois had recorded the IP addresses of the attackers. Those digital fingerprints and the techniques the hackers had used, combined with the intelligence reporting on Russian plans, convinced the feds that the attackers were a group, known as Fancy Bear, that operates as an arm of Russian military intelligence (GRU).
At first, says a former senior White House official, that revelation “was terrifying.” For a week or so starting in late July, the feds faced the prospect that Russia might be planning to physically hack into the voting machines and fiddle with the vote count. The urgent need: to figure out if Moscow could actually swing the election. As it turned out, the White House had on staff one of the country’s leading experts in voting-machine manipulation, professor Ed Felten of Princeton, who was serving as deputy to the U.S. chief technology officer. Felten had famously been the first academic to obtain a Diebold voting machine and publish a public study showing it could be compromised.
With colleagues from the National Institute of Standards and Technology who had written the standards for electronic-voter-machine security, Felten and Daniel concluded that hacking voting machines was technically possible. “In many places in the U.S., there are touchscreen voting machines, which are vulnerable to manipulation by someone who gets access ahead of time,” Felten says. The cyberteam began worrying that Russia might try to compromise a poll worker and gain access to touchscreen machines before the election. But doing that in a way that could alter the outcome of the election was very hard. First, the attackers would have to know which districts could affect the outcome. Then they’d have to change just enough votes to ensure victory without switching so many that it would draw attention.
That didn’t mean all was well. The whole point of the election wasn’t just to count ballots; it was for the U.S. to reach consensus that the democratic will of the people had been freely and fairly expressed. Hacking the consensus was much easier. “We concluded that Russia could erode the confidence of millions of voters and undermine our ability to conduct free and fair elections,” says Anthony Ferrante, former director for cyberincident response at the National Security Council, who ran the frontline efforts to combat the Russian operation.
Since May, U.S. spy hunters had seen evidence that Russia’s military intelligence might try to damage the expected winner, Hillary Clinton. The intel was incomplete but pointed in the same direction: an initial report of a bragging GRU official that month was followed by other intelligence reports indicating a widespread willingness to interfere. In the wake of the Illinois intrusion and on the basis of the intelligence it had received, the White House team by mid-August believed there were three main ways Russian President Vladimir Putin could undermine the integrity of the vote.
The first and most disruptive thing Russia might do: subtly alter the voter rolls. Deleting records would draw too much attention, but running a program against registration files that would, for instance, flip the second letter in every voter’s address could go unnoticed. Then, on Election Day, every voter in a swing county would have to vote by provisional ballot, giving the impression of chaos and allowing a propagandist who wanted to call into question the vote to do so after the fact.
Another possibility involved the propaganda value of fiddling with a voting machine. Says Daniel: “We worried, Could [a hacker] document an intrusion into a [single] voting machine and then say, ‘Here’s the YouTube video. We did this a hundred thousand times across the United States,’ even though they had never done anything like that?” That would sow doubt about every machine in the country and would also undermine the final vote’s credibility.
Lastly, the Russians could interfere with the election reporting system. The actual vote tally is decentralized and extremely slow: local officials count and validate their results, and state secretaries, election boards or other state officials sign off on the total tabulations, and only then is the official vote certified. That decentralization is the system’s strength. But on election night, nearly all reporting across television, the Internet and news wires relied on the Associated Press. Altering the data reported by the AP, or just taking down the AP system with a sustained attack, could cause chaos.
Knowing all this, Ferrante began working up an emergency plan for what to do on Election Day, and the day after, if the Russians attacked the vote. Drawing on election experts at the Justice Department, the FBI and DHS, Ferrante scrambled to figure out what powers the federal government had, legally, to push back.
But it turned out the credibility of the vote would come into question well before Election Day. And rather than the Russians, it would be the government of the U.S. that would become suspected, by some Americans at least, of subverting the vote.
THE ENEMY WITHIN
From the first report of Russian hacking in mid-June, Donald Trump denied Moscow’s involvement, improbably accusing the Democratic National Committee of hacking itself “as a way to distract from the many issues facing their deeply flawed candidate and failed party leader.” As the story accelerated with the dump of stolen emails right before the Democratic National Convention, Trump doubled down on his counterclaims. On Aug. 1 in Columbus, Ohio, he said, “I’m afraid the election is going to be rigged.”
Which may partly explain why the atmosphere was so tense when Secretary of Homeland Security Jeh Johnson convened a conference call on Aug. 15 with representatives of election officials from every state across the country. On Aug. 3, Johnson, an Obama appointee, had said he was considering declaring elections part of the U.S. critical infrastructure, along with things like the banking and electrical systems. That designation would give the federal government access to state-level voter information and would open regular lines of communication with local election officials. On the Aug. 15 call, Johnson said DHS stood ready to help the states by conducting vulnerability scans, providing what he said was “actionable information” about threats and delivering cybertools to help protect election systems from intrusion. But some of the states were less concerned about outsiders than they were about federal overreach, according to Johnson and several participants in the call. States didn’t know what being declared critical infrastructure meant and were suspicious for partisan reasons: Were Democratic officials in Washington preparing to take control of the nation’s polling places? The call grew contentious as participants felt the feds were encroaching on the constitutional role of states to run elections. “We secretaries of state were faced with an issue where there were perhaps foreign actors trying to get into our databases,” says Arizona secretary of state Michele Reagan, a Republican and one of the first victims of the Russian hack. “And their answer was, ‘Let’s just take over the election infrastructure, which goes against the Constitution and our state law.’”
The atmosphere of mistrust of the feds was hampering the White House’s ability to respond to the Russian attack too. Obama was already worried about the possibility of an escalating cyberwar with Moscow if he retaliated for the ongoing Russian hack, senior White House officials privately said at the time. With Trump fueling antigovernment suspicion, Obama was even less inclined to take strong measures against Moscow, in part because of the danger of seeming political. Coming out hard against Russia, which was widely believed to favor Trump, Obama thought, would make it look as if the White House were trying to help get Clinton elected. “It was pervasive in the discussions,” says a former senior White House official, “because some state officials were questioning whether some of our actions were advancing the interests of the Democratic Party.”
Even as they were trying to communicate the dangers, the feds were seeing more evidence of just how expansive the Russian intrusions were. Three days after the Johnson call, on Aug. 18, the FBI sent out a flash alert to all the states including the digital fingerprints of the hackers they had gathered in Illinois and Arizona. By mid-August, Daniel’s group had concluded that the GRU had infiltrated the electoral systems of Florida and New Mexico. In Tennessee, hackers had reached into the state’s campaign-finance system. Soon the number of states probed by the Russians had crossed half of all states, and it was clear the Russians had tried to hack everyone; the only question was how successful they had been.
The emerging picture wasn’t pretty. “In some cases we saw them try to get in and they failed,” says Daniel. “In some cases we saw them get a little way in and then get stuck. And in other cases they got a little bit further and were doing these kinds of testing.” What was most frightening was that they knew they were seeing only Russia’s clumsiest efforts. Moscow’s state-sponsored hackers are among the most skilled cyberactors in the world. The feds had to assume there were other intrusions they weren’t seeing. The fact that they didn’t see intrusions in some states, says one official, “just means we didn’t find them.”
THE PLAN AND THE LAST STAND
Paralyzed by politics at home, Obama tried to blunt the threat directly abroad. In a now famous one-on-one meeting with Putin in early September in Hangzhou, China, Obama told him to “cut it out” or face unspecified consequences. The confrontation was memorialized in a photo of the two men staring icily at each other.
For a while it looked as if the warning might work. “The intelligence community basically told us that [they were] not seeing [the Russians] continuing to go down that road,” says a former senior White House official. And in the U.S., DHS scanned voting systems remotely across the country and found and patched vulnerabilities. Some states also accepted visits by DHS cybersecurity teams that checked for vulnerabilities in person. But relations between the states and the feds remained chilly.
Then, in October, the attacks resumed. The GRU launched an operation against a software company, VR Systems, that provided election software and devices to at least eight states, according to a report by the Intercept. The intruders used the information to craft a convincing-looking email that served a spear-phishing campaign against the electoral officials across the country.
With just weeks to go until the vote, the White House cybersecurity team realized there was little it could do to stop a Russian attempt to undermine the credibility of the vote on Election Day, so it shifted into damage-control mode. In late October, the White House distributed its 15-page plan to deal with an Election Day attack to the top cybersecurity officials across the federal government. Daniel says he briefed Obama’s chief of staff, Denis McDonough, on the plan; it is not clear if President Obama himself was informed of it or any of its details. It started by saying that “in almost all potential cases of malicious cyberactivity impacting election infrastructure” the feds would defer to state and local governments. But it also authorized robust federal action as well. If there were a “significant cyberincident” that would result in a “demonstrable impact to election infrastructure,” DHS, FBI and the office of the Director of National Intelligence would “activate enhanced procedures and allocate their resources.”
The Justice Department’s election-crimes unit and civil rights divisions were on standby, as were parts of DHS and the Secret Service. The FBI could dispatch any of its Cyber Task Forces from their 56 field offices to “facilitate joint information sharing, incident response, law enforcement and intelligence actions.” Four FBI Cyber Action Teams were on standby “if cyberinvestigative techniques are needed to rapidly respond to a call for assistance,” but the plan indicated that they could not “self-deploy” without FBI higher-ups’ approval. Several Obama Administration officials said the plan and the powers it envisioned tapping in a crisis were similar to those available in cases of natural disasters.
Heavier forces waited in the wings. The White House plan included the possibility of deploying active and reserve components of the military. “The Department of Defense may support civil authorities in response to cyberincidents based upon a request from a federal agency, and the direction of the Secretary of Defense or the President,” the plan said. Two people familiar with it say the idea was to make the Pentagon’s cyberexperts available to mitigate and investigate an attack.
At 6 a.m. on Election Day 2016, Ferrante opened the door to the “second Situation Room,” a carbon copy of the President’s secure West Wing conference room a stone’s throw away in the Eisenhower Executive Office Building. On a secure video teleconference system, the team dialed into the FBI and DHS command posts that were running the Election Day response. They were joined by election-crimes coordinators from the Justice Department and cyberintelligence agents in the office of the Director of National Intelligence. Russia experts at CIA, NSA and other intelligence-community agencies were standing by on the classified Joint Worldwide Intelligence Communications System (JWICS) email system.
Over the course of the day, reports came in that made the group think it might be seeing a repeat of Riverside County, or worse. In Colorado, the election voter database went down for 30 minutes. In Utah, lines formed in what had become an unlikely battleground thanks to the independent candidacy of Evan McMullin. At one point, sensitive intelligence came in that needed to be run to ground. But ultimately the level of disruption was no greater than in any normal national election, and all in all, the vote went off smoothly. As the polls closed, and the election was called for Donald Trump, some on the White House cyberteam celebrated the fact that there had been no disruptive attack.
FROM BAD TO WORSE
On Nov. 25, amid talk of possible challenges to the vote in Wisconsin, Pennsylvania and Michigan, the Obama White House released a statement saying, “We stand behind our election results, which accurately reflect the will of the American people [and] believe our elections were free and fair from a cybersecurity perspective.” But even as the calls for recounts faded, doubts about the security of the election system spread.
A senior intelligence official tells TIME that while the cybersecurity officials at the FBI, DHS and the White House may have been scrambling to secure the vote throughout the fall, the counterintelligence operation at the bureau aimed at uncovering whether the Russian operation was trying to aid Trump only really began in earnest once the election was over. Given the focus of FBI cyber and counterintelligence officials on Hillary Clinton’s emails, this looks like a spectacular blunder in retrospect. The Russia counterintelligence probe “never got any intelligence legs until after the election,” the senior official says, “because I don’t think anybody believed Trump would win, so nobody really put a lot of stock into the Russian attempts [to help him].”
At the same time, some division remained over who the real threat was to America’s electoral system remained. Georgia was the only state that didn’t accept some form of assistance from the federal assistance, according to officials familiar with the matter. But as the Georgians looked for intruders themselves, they found a DHS employee scanning their system on Nov. 15. DHS looked into the matter and said it was an employee of the federal training facility in Glynco, Ga., confirming that job applicants in fact had licenses to be armed guards, which are on the same system as the voter data. The DHS inspector general, John Roth, looked into the matter and concluded in June that DHS employees did not conduct any unauthorized scans of the Georgia election system.
If some state officials remain angry about the perceived threat of federal overreach, many also appear to be in denial about the extent of the risk from overseas. The Russians succeeded in compromising more than 20 state systems, according to Jeanette Manfra, a senior Homeland Security official. But calls by TIME to the offices of election officials in every state revealed only two, Arizona’s and Illinois’s, that know or are willing to admit that they were hacked. And when TIME asked Illinois to confirm that the Russian intruder had tried to alter data fields, state officials at first denied that had happened, but after checking with their technical team, confirmed that it had.
Partisan suspicion now runs in both directions. On May 11, President Trump announced the creation of an election-integrity commission. Nominally it was supposed to “study vulnerabilities in voting systems,” but it got off to a rocky start, requesting vast amounts of personal information on voters from every state. That led Vanita Gupta, head of the civil rights division in the Obama Justice Department to allege that the true goal of the Trump panel is to “lay the groundwork for voter suppression” by making it more difficult for typically pro-Democrat minorities to vote.
In fact, Trump’s commission asks only for voter information that states can legally provide. But it is arousing the same kind of partisan suspicions that Obama’s efforts to work with the states did last fall. In Arizona, secretary of state Reagan says she has declined to hand over the personal information Trump’s panel wants. As a result, she says, her competitor in next year’s Republican primary is attacking her for not supporting Trump. “I can remember when all the states were like, ‘Heck no, we’re not supporting the federal government’s election intrusions.’”
The result may be that safeguarding future elections is only going to get harder. Meanwhile, House Republicans are trying to defund the $8 million Congress provides annually to the Election Assistance Commission (EAC), which guides states in running safe and reliable elections. Republicans say that the agency is ineffective and unnecessary and that its work can be done instead by the famously weak Federal Election Commission. Democrats argue that gutting the tiny agency is reckless, especially right now. Democratic Senator Amy Klobuchar has introduced a bill that would refund the EAC and provide $325 million to the states to improve election infrastructure, expand voting opportunities and strengthen cyberprotections. The bill has no support among Republicans.
In Riverside County, election security remains front and center. Next month the county will hold votes for water district boards of directors and in November will have general elections on community services, libraries and schools. The county registrar, Rebecca Spencer, says she has been working with a local state assemblywoman to get emails and texts sent when a voter changes any part of online voter information. DA Hestrin sponsored the bill that will make that happen, but worries about continuing voter doubts. “People’s faith in the system is a fragile thing,” he says. “Once people lose faith that the elections are fair and honest, then our entire system of government is in jeopardy.”
–With reporting by JACK BREWSTER and EMMA TALKOFF/WASHINGTON