The Graveyard of Empires and Big Data
The Pentagon's secret plan to crowdsource intelligence from Afghan civilians turned out to be brilliant — too brilliant.
In February 2010, two months after Lee’s red balloon contest, Dugan did something that no DARPA director had done since the Vietnam War: She traveled to a war zone to see what the agency might be able to contribute. Gen. Michael Oates, the head of the Joint Improvised Explosive Device Defeat Organization, the Pentagon’s bomb-fighting agency, invited Dugan on a three-day tour of Afghanistan. Military personnel expressed surprise to see her. “You’re from DARPA,” she recalled their general reaction. “We call you when we have three- to five-year problems.”
When Dugan returned to Washington she assembled the office directors and their deputies and gave them a month to come up with ideas for technologies DARPA could contribute immediately to the war in Afghanistan. By April, DARPA had identified about a dozen projects that could have an immediate effect on the war, and then Dugan narrowed those down to a final list. The technologies ranged from a blast gauge that would go in soldiers’ helmets to detect exposure to possible blast waves from IEDs to an imaging sensor, called the High Altitude LIDAR Operations Experiment, which could be used to create three-dimensional maps of Afghanistan.
Dugan’s priority, however, was a new program based on Lee’s big data work, called Nexus 7, which would help predict insurgency in Afghanistan. In August, Dugan met with the chairman of the Joint Chiefs of Staff and laid out DARPA’s plan for Afghanistan. The data-mining project, her briefing noted, would “sequester [a] team of the Nation’s leading researchers in large scale computation techniques and social science.” She called Nexus 7, a data-mining program named after a humanoid robot in the movie Blade Runner, “the potentially big win.”
In the summer of 2010, Nexus 7 was launched with Randy Garrett, a former NSA official involved in big data, as its head. At the National Security Agency, Garrett had been working on creating a cloud that would allow analysts to search through real-time data as it was vacuumed up by intelligence agencies. This cloud would include “essentially every kind of data there is,” Garrett said, from the millions of calls a day around the world the NSA intercepts, to various forms of internet traffic, from emails to Skype calls. Afghanistan, after 10 years of war, was one of the NSA’s top targets for cell phone interception.
Garrett’s goal with Nexus 7 was to “actually build this big data aggregated environment, a cloud, and then see how you would use it,” according to a scientist who was involved in creating the DARPA program. Nexus 7 was a direct carryover of work started at the NSA, according to the scientist.
Other key members of the Nexus 7 team came from Sandy Pentland’s Human Dynamics Lab at MIT, drawing on the same ideas that drove his team’s win with the balloon contest and applying them to an entire society. Pentland described his contribution as informal, providing more of an intellectual framework than nuts-and-bolts work. At the core of Nexus 7 was Peter Lee, who created the program and ran it from his office. “Nexus 7 turned out to be a bunch of desks, laptops, and secure computers literally in the hallway outside of my office,” Lee said. “It was just a zoo.”
The direct relationship between the NSA and DARPA was one of the hallmarks of Nexus 7, but it was also the most problematic, because working with data from the NSA required navigating a maze of legal and statutory requirements that often prevent sharing and aggregating data among government agencies. As for why DARPA wanted the NSA data, L. Neale Cosby, a retired Army officer who consulted on the program, invoked the bank robber Willie Sutton: “Why rob a bank? Because that’s where the money is.” The NSA was the bank; it had all the data.
Nexus 7 was unabashedly interested in data of all kinds. It was particularly interested in using patterns of daily life, including the costs of transportation and exotic vegetables, to make predictions about insurgencies in Afghanistan. “We were really using the latest research in quasi-experimental design, in machine learning, and data mining literally on hundreds of intelligence feeds to make inferences about what would happen next,” Lee said.
The program was kept secret to avoid any controversy. DARPA workers one office down had little idea what was going on when the group of young computer scientists set up shop in the agency’s headquarters. “We had to have cover stories to tell people if various Beltway people came to visit me in my office and they were walking through this pandemonium,” Lee said. In budget documents, Nexus 7 was obliquely described as a program that combined data analysis and forecasting with social network analysis.
According to Dugan, Nexus 7 started making its “first discoveries”— or meaningful predictions — just 82 days into the operation. But the program soon met resistance. Gen. Stanley McChrystal, the head of the International Security Assistance Force in Afghanistan when Nexus 7 started, was interested in the data-driven work promoted by DARPA. But in 2010 he was forced to resign after a Rolling Stone magazine profile depicted him and his staff as mocking senior White House leaders. Gen. David Petraeus returned to Afghanistan to take over, but he was not enthusiastic about Nexus 7. A disastrous meeting between Petraeus and Dugan in Afghanistan almost brought it to a halt. DARPA’s proposal for algorithms did not sit well with a general who believed he wrote the book, metaphorically and literally, on counterinsurgency.
At that point, however, Nexus 7 had support from Gen. James Cartwright, the vice chairman of the Joint Chiefs of Staff and a technology enthusiast, who had given an official green light to the deployment of the program and its personnel. (Dugan said Cartwright’s response to Nexus 7’s first discoveries was “Go and go faster.”)
DARPA would eventually deploy more than 100 people across Afghanistan, working on Nexus 7 and other technology programs. Peter Lee, the creator of Nexus 7, was not among them. He abruptly left DARPA after less than a year to become the head of research at Microsoft. On the day he left for Seattle in September 2010 to start his new job, the Nexus 7 team, some members as young as their mid-20s, was departing for Afghanistan. “I should have been with them,” he said.
“It was the first operational deployment from DARPA since the Vietnam War,” Dugan later recounted. The program also became Dugan’s top priority as she shuttled back and forth to Afghanistan with Cartwright. In congressional testimony in 2011, Dugan did not use the Nexus 7 name, but simply described “a 90-day Skunk Works activity” that involved scientists and counterinsurgency experts working on “crowdsourcing and social-networking technologies.”
DARPA’s “army of technogeeks,” as Dugan affectionately referred to them, was young and had no military experience, and the culture shock soon became apparent. Military officials in Kabul were reluctant to share intelligence with computer scientists just out of graduate school, and the intelligence they did provide was not nice and neat, like consumer data. Once in Afghanistan, the analysts began to gather as much intelligence as they could: phone records from the NSA, radar feeds from the military, and intelligence reports. But much of the data that came into Nexus 7 were qualitative, rather than quantitative, which were not easy to plug into a computer program. Even when the data were quantitative, like from radar, they rarely covered the exact same place over time.
By late 2010, DARPA was touting Nexus 7’s successes within the Pentagon, but it was not clear what, if anything, it had accomplished. As members of the team worked on a base crunching numbers from military and intelligence data feeds, another team of contractors, the Synergy Strike Force, was working in the provinces of Afghanistan, swapping beer for data and using crowdsourcing techniques honed in the red balloon hunt.