Oracle Announces It’ll Use NSA Hand-Me-Down Tech

Oracle CEO Larry Ellison announced this week that his company will sell "in memory" databases, promising customers the power to analyze huge amounts of information at speeds 100 times faster than systems they’re using now. This is news for Oracle. But the company is not the first denizen of big data to use this super-fast technology. That credit goes to the National Security Agency.

Beginning in 2004, the NSA took an important turn in its insatiable quest to store and manipulate huge amounts of data. The agency started storing intercepted e-mail, phone, and other communications traffic with in-memory databases, which were built using random access memory, or RAM. Up until that point, the NSA had used disk-based storage. Every piece of data the NSA collected was like a book, and the database on which it was stored was like  a bookshelf. Whenever an analyst wanted that book, computers had to mechanically retrieve it. And at the scale the NSA was operating — looking at hundreds of thousands, sometimes millions of books at once, and cross-referencing them — analysis of big data sets could take hours, or even days.

In-memory databases promised to change all that, because they stored and retrieved data at vastly greater speeds than a traditional system. In 2001, a group of researchers in Washington State had shown that an in-memory database could retrieve 30,000 individual records in one second; it took a traditional machine, using disk-based storage, 16 seconds. More impressive still, it took the in-memory system 2.5 seconds to "write," or store all those records in its memory. The traditional machine took almost an hour to perform that task.

In-memory databases were the secret ingredient that turned the NSA from a relatively slow user of big data into a high-speed information cruncher. The databases gave the NSA the ability to store and retrieve huge numbers of communications practically in real-time. This was an essential for an agency trying to spot the electronic data trials of terrorists at the time they made them, not days later.

In-memory databases are fairly common today. RAM also costs much less than it did in 2004. Oracle is actually somewhat late in selling the databases, trailing behind its competitor, SAP. (That’s surprising since the Oracle founders got their big break thanks to a secret intelligence project for the CIA.)  

But in 2004, in-memory databases were largely untested in a real-world environment, and they were too expensive for most companies. But fortunately for the NSA, it had practically limitless resources.

Most budget figures are classified, but we have some frame of reference for what the NSA spent, and continues to spend, on data analysis. In fiscal year 2013, according to documents leaked by former NSA contractor Edward Snowden, the NSA spent $429 million on research and development of new technologies, which in-memory was back in 2004. Today, these databases are integrated into the NSA’s operations, so they’re funded by the agency’s much bigger data analysis budget.  In fiscal 2013, the agency spent $1.5 billion in that area. The NSA’s total budget has increased 53 percent since 2004, so if we presume that all areas of the budget increased at roughly the same pace, the agency was still spending nearly $800 million. This is an imperfect estimate, but it makes the essential point that the NSA spends extraordinary amounts of money trying to store and retrieve data.

I write about this critical turning point in NSA’s data mission in my book, The Watchers. It’s a case study in how a huge intelligence agency was able to help create a viable commercial market for an expensive and little-used technology. Ellison should send the geeks at Ft. Meade a thank you note.