Total Recall

A decade ago, a Pentagon research project called "Total Information Awareness" sparked a mass panic because of its seemingly Orwellian interest in categorizing and mining every aspect of our digital lives. It was "the supersnoop’s dream," declared William Safire of the New York Times, a "computerized dossier on your private life from commercial sources, [combined with] every piece of information that government has about you…."

If this sounds reminiscent of the current uproar over NSA surveillance, you’re paying attention. That’s because the NSA monitoring tools are very similar to — and, in many cases are directly based on — the technology that Total Information Awareness (TIA) tried to use.

The story of that convergence starts on the morning of Feb. 2, 2002, when retired Admiral John Poindexter drove to the headquarters of the National Security Agency at Ft. Meade, Maryland, and sat down with the agency’s deputy director, an NSA veteran named Bill Black. Poindexter, a former White House national security adviser, was now running the TIA program at the Defense Advanced Research Projects Agency, the organization that tackles some of the hardest engineering and technology challenges in the Pentagon. Poindexter thought TIA was an innovative new way to stop terrorist attacks, and he wanted the NSA to help him test it. 

The idea, he explained to Black, was to give U.S. intelligence analysts access to the vast universe of electronic information stored in private databases that might be useful for detecting the next plot. Data such as phone call records, emails, and Internet searches. Poindexter wanted to build what he called a "system of systems" that would access all this raw information, sort and analyze it, and hopefully find indications of terrorist plotting. 

The NSA was the biggest collector of electronic data in the government, and Poindexter thought the NSA would be a natural partner in his endeavor. But what he didn’t know was that under secret orders from President George W. Bush, the NSA was already building its own version of Total Information Awareness. Fewer than 100 people at the NSA knew that for the past few months, the agency had been monitoring the phone calls and other electronic communications of Americans, and that it was obtaining copies of domestic phone call records and looking at them for potential clues about terrorist attacks. 

Poindexter left Ft. Meade that day with no firm commitment from Black that the NSA would assist in his research. And TIA didn’t last long. Although Poindexter’s work wasn’t classified, the press soon caught wind of his grand data-mining ambitions, and Poindexter was held up as the poster boy for intrusive government surveillance. "I think it’s fair to say that in the country’s history there has never been proposed a program with something this far reaching in terms of surveillance capacity," said Sen. Ron Wyden at the time. "And my sense is that the country just does not want to unleash a bunch of virtual bloodhounds to go sniffing into the medical, financial and travel records of law-abiding Americans."

TIA was officially shut down in 2003, and Poindexter left the government. But this wasn’t the end of his grand vision.  

In a secret negotiation, members of Congress, some of whom had been among Poindexter’s critics, reached an agreement to keep TIA research going, and to fund it from the classified portion of the military budget, the so-called "black budget." TIA’s research components were given new cover names, and the program was moved under the control of the very agency that Poindexter had originally wanted to help him — the NSA. There, Poindexter’s ideas were incorporated into NSA’s surveillance activities, the latest glimpses of which we have seen in the past two weeks. 

The NSA went on to build its own total information awareness system. What was once an idea in Poindexter’s head is now a fully realized global surveillance apparatus, capable of gathering unprecedented amounts of digital information for near real-time analysis, or to be stored for future investigations, perhaps years from now. It is not precisely the system that Poindexter imagined, but what became clear to him looking back, as he explained to me in a series of interviews for my book, The Watchers, was that the NSA was doing many of the things he had proposed all along. Poindexter was demonized for even suggesting that the government get access to vast troves of private information. The NSA now does this routinely, and under the law.  

There are several key respects in which the NSA’s system today mirrors that which Poindexter had proposed more than a decade ago. 

Access to many categories of private information

TIA envisioned, as its name suggested, access to the total universe of electronic information that might be useful for investigating terrorists. It placed particular emphasis on phone records, e-mails, Internet searches, travel records, and financial transactions — because in order to plan attacks, terrorists need to communicate, conduct research, move around, and make purchases. 

This NSA is using the same body of information to investigate terrorists. Under a court order, the NSA obtains copies of all domestic phone call records, as well as records of international calls into and out of the United States. The agency also reportedly taps into undersea cables that carry Internet traffic. Using the PRISM system, it can read emails and see Internet searches, as well as forms of electronic messages that didn’t even exist when TIA was proposed, such as Facebook messages. The agency has also obtained credit card receipt transactions and records from Internet service providers. The NSA has teams of analysts that work at the National Counterterrorism Center, where airline travel records are monitored and watch lists are compiled. And soon, the NSA will get to tap into a rich repository of financial information compiled by the Treasury Department that’s used for investigating financial crimes and tracking terrorist money flows.  

Use of "virtual" databases

Rather than trying to make copies of private databases and hold them in a government facility, TIA proposed a kind of federated or "virtual" database. The system would effectively reach out and touch the private databases themselves, or systems that were set up attached to them, working with the information at, or close to, the source and siphoning off what it needed for analysis.  

This is what’s happening with the NSA’s Internet mining tool known as PRISM. The NSA doesn’t have "direct access" to company servers, but obtains information on an as-needed basis using a technology that some have described as a drop box: The company deposits the information NSA wants in the box, and NSA takes it. Notably, Google has said that it doesn’t use this kind of setup. But the company does comply with lawful orders for information without turning over the entirety of its data or letting the NSA jack into its central servers. The same goes for other companies that have provided information to the NSA, reportedly as many as 50. 

(Lack of) privacy protection  

At the core of TIA was a device, then yet-to-be-invented, that Poindexter called a "privacy appliance." It would strip all data of personally identifying information — such as names and addresses — and give each data point a unique, encrypted designation. A TIA analyst would see how pieces of information fit together, but he would need a court order if he wanted to unlock the privacy appliance and see the names associated with that data. 

NSA abandoned this privacy research when it took over Poindexter’s programs in 2003, and a privacy appliance as sophisticated as what was hoped for in TIA still doesn’t exist. However, the NSA’s database of phone call records, known as Mainway, now have some privacy controls, according to intelligence employees who have used the system. The database does not contain any names, nor is the NSA collecting geolocation data that could pinpoint a user on a map, according to administration officials. When an analyst comes upon a phone number associated with a U.S. citizen or legal resident, a black ‘X’ mark appears over the number, says one former defense intelligence employee. Administration officials have said publicly that the databases only can be queried as part of a terrorism investigation, and that it has been accessed about 300 times last year. 

Much less is known about how PRISM protects the privacy and identities of U.S. persons, whose communications the NSA cannot target without a warrant. Analysts are allowed to target a set of foreign communications — such as emails — if PRISM determines with 51 percent confidence that they are indeed foreign. Technically, it is very difficult to determine whether an email was actually sent by a foreigner and not a US person, experts say. And a 51 percent confidence rating is not a high threshold. This practically guarantees that US persons’ information is swept up by PRISM. The NSA has not disclosed the procedures it uses to separate that information from the foreigners’ data. 

Use of broad searches 

Poindexter believed that in order to find the proverbial "needle in the haystack," analysts needed to be able to look at a lot of haystacks. TIA would cast a wide net searching among mostly innocent and innocuous communications for those that merited further scrutiny. 

NSA attempts to do just that with PRISM. It is meant to filter out potentially meaningful signals from an ocean of noise. Gen. Keith Alexander, the NSA director, has said that in the vast majority of terrorist attacks that the United States was able to stop, this kind of analysis was essential. (With the Mainway database, officials say their searches are more targeted, and are begun based on a specific phone number of a known or suspected terrorist.) 

Reliance on court orders 

Poindexter never envisioned giving the intelligence community unsupervised access to private data. He thought that courts should play a role, because this would legitimize the government’s monitoring of private data and provide a check on potential abuse–at least in theory. He pointed to the longstanding practice of issuing intelligence surveillance warrants under the Foreign Intelligence Surveillance Court as a model. 

The NSA has come to rely on that court, which is now issuing broad orders for information that, prior to the 9/11 attacks, would have been unimaginable. The court has sanctioned the copying of all phone records in the United States. It also reviews the government’s Internet surveillance methods in an attempt to ensure that they don’t unreasonably scoop up Americans’ data too. This is far from perfect science. On at least one occasion, the court has found that these procedures were unconstitutional. We still don’t know how they were changed to make them legal. 

Amended privacy laws

Poindexter also hoped to ignite a national debate on whether to change privacy laws to reflect both advances in technology and the difficult task of finding an enemy — such as terrorist networks — that don’t announce their presence.

He got his wish. In 2007 and 2008, after some of the NSA’s secret programs were exposed, Congress debated changes to the Foreign Intelligence Surveillance Act. There were many public, heated exchanges. Ultimately, lawmakers voted to give the NSA more authority to search broadly for potential terrorists, and to do so without individualized warrants that name the specific person and place the agency wants to search.  

* * * 

The key question still unanswered is whether any of this surveillance actually prevents terrorist attacks. Poindexter couldn’t say for sure that TIA would, though he believed that his early research showed promising signs. Officials now say that the NSA’s surveillance activities have helped stop dozens of attacks. If the details of those plots are released, we’ll all be able to decide for ourselves. Perhaps then can finally decide whether building Total Information Awareness was a good idea.