How OPM Bilked a Security Contractor That Confirmed a Major Hack

On June 4, 2015, officials from the Office of Personnel Management (OPM) announced that the agency’s computer systems had been compromised in “two separate but related cybersecurity incidents.” Unidentified hackers had stolen the personal information of more than 20 million federal employees, including extremely sensitive dossiers compiled during security clearance investigations. The potentially catastrophic situation, at the time, was being called a “cyber 9/11.”

The House Oversight and Government Reform Committee conducted a yearlong review of the breach and released its report on Wednesday, in conjunction with an event at the right-leaning American Enterprise Institute on “lessons learned,” hosted by Committee Chairman Jason Chaffetz (R-Utah). It noted that the OPM data breach was “preventable” and that “OPM leadership failed to heed repeated recommendations … failed to sufficiently respond to growing threats of sophisticated cyber attackers … [and] failed to prioritize resources for cyber security.” Even more damning, the committee concluded that “OPM misled the public on the extent of the damage of the breach and made false statements to Congress.”

Indeed, certain details of the attack are still unclear. The report reveals that investigators from US-CERT — the United States Computer Emergency Readiness Team, which is part of the Department of Homeland Security’s National Cybersecurity and Communications Integration Center — discovered the first intrusion, by an individual referred to in the report as “Hacker X1,” on March 20, 2014. They monitored Hacker X1 and developed a plan to lock the intruder out of the system over a three-day weekend in May 2014, in an operation called “Big Bang.”

However, while everyone was focused on catching Hacker X1, a second intruder, which the report dubs “Hacker X2,” quietly slipped inside. On April 15, 2015, according to the report, OPM deployed a diagnostic tool by the California-based cybersecurity company Cylance. When it “lit up like a Christmas tree,” it was immediately obvious that, somewhere along the line, something had gone very, very wrong.

But there’s more to the story. A week later, on April 21, a security contractor named Ben Cotton was invited to OPM headquarters to give a product demo for his company, CyTech Services. Yet Cotton, a retired U.S. Army Special Forces operator, was never informed about any breach. Furthermore, he saw no clues that the agency was in the midst of a potentially crippling national security crisis and felt no sense of urgency among OPM staff. It was “business as usual,” Cotton says.

Although he arrived at OPM headquarters at 9:30 a.m., it was 4 p.m. by the time Cotton says he made it through the various security protocols, had his equipment screened, received a visitor’s pass, and finally got what he needed from OPM’s technical department.

“Of course, 4 p.m. in Washington, D.C. — my escort tapped me on the shoulder and said, ‘Hey, would you mind coming back tomorrow? I’ve got a bus to catch,’” Cotton recalls.

Before he left, Cotton launched his company’s digital forensics platform CyFIR, short for CyTech Forensics and Incident Response, that would identify every process running on the OPM network. To Cotton, this was still nothing more than a routine product demo. He’d look at the results in the morning.

After another protracted security procedure to get into the building the next day, Cotton stood before a roomful of OPM officials and opened up CyFIR’s analysis screen. The processes were split up into three categories: known-good, known-bad, and unknown.

Cotton says he found “some adware in the known malware bucket, nothing really major.” But then something in the unknown set of running processes caught his eye. There were three McAfee anti-virus files in there, all with “the right install pass and the right subdirectories off of that.” They followed the proper McAfee naming conventions. So, why were they not in the “known-good” category, where they belonged?

“I said, ‘What kind of anti-virus protection are you using?’ They said, ‘We’re not using McAfee in any of our systems.’ I said, ‘Look, I think you’ve got a problem here.’” (According to a statement later made by OPM’s then-chief information officer, Donna Seymour, OPM was in fact testing Cotton to see if his “tool set would also discover what we had already discovered [with the help of Cylance].” Cotton maintains he was given no indication of this.

“I knew what those databases contained — my own information was in there,” Cotton says. “I knew immediately what the impact would be for anyone who ever held a security clearance.”

An emergency meeting was called, and things “went from zero to 120 in a real short time frame.” At that point, Cotton says a high-ranking OPM official told him, “‘Hey, we need an emergency purchase order for CyFIR across the entire environment.’”

Three hours later, the FBI and US-CERT showed up. The FBI began heading down what Cotton calls a “very traditional incident response route” and says he and his team supported the agents with real-time analysis and evidence collection. Within a “day or two,” the response team knew what the total scope of the breach was. Signs pointed in an expected direction; Cotton says he found a “piece of Chinese malware that had been in there for over a year.”

The OPM hack was entirely preventable, argues the House report. It blames OPM, which had no professional IT security staff until 2013, for ignoring repeated warnings about the vulnerability of its digital systems.

But the agency argues that lessons have been learned. “The cybersecurity report issued today by the Republican members of the House Oversight and Government Reform Committee (HOGR) on the cyber intrusions at the U.S. Office of Personnel Management (OPM) does not fully reflect where this agency stands today,” OPM acting Director Beth Cobert said in a blog post published early Wednesday morning. “The cybersecurity incidents at OPM provided a catalyst for accelerated change within our organization. Throughout this agency, management has embraced cybersecurity as a top priority. I am proud of the way the team at OPM rose to the challenge and appreciate the collaborative spirit with which our partners across government worked — and continue to work — side by side with us each and every day.”

Ben Cotton is no longer one of those partners. When he was asked by OPM for help containing the breach, he “realized right away that it was a significant problem for the country” and didn’t bother waiting for a formal contract.

“A handshake was good enough for me,” Cotton says.

The House report notes CyTech, Cotton’s company, “began expanding its services to OPM,” which included six CyFIR appliances, 15,000 CyFIR licenses, six training vouchers, and 1,040 engineering support hours, plus a CyTech expert to “provide incident response and forensic support.” OPM used the CyFIR tool through early June 2015. The total cost: $818,000.

“Documents show CyTech’s role in providing forensic support was significant,” the report says.

It never got paid.

Photo credit: Joe Raedle/Getty Images