Advances in Data Storage Have Implications for Civil Liberties

http://thecaucus.blogs.nytimes.com/2012/08/14/advances-in-data-storage-have-implications-for-government-surveillance/?hp

Data Storage Could Expand Reach of Surveillance

A wave of worry about a software program called TrapWire, designed to detect terrorists casing possible targets, appears to be unjustified, as I wrote in Tuesday’s Times.

Based on stolen corporate e-mails posted by WikiLeaks, some reports hugely exaggerated the program’s sweep and capabilities; the New York Police Department, for instance, says that contrary to claims on the Web, it has never used TrapWire.

But the bogus flap over one particular surveillance product should not eclipse the very real issues lurking behind it. Government at every level is experimenting with sophisticated surveillance equipment whose capabilities are improving as rapidly as every other kind of electronic technology.

The Police Department itself, for example, just last week unveiled a new “domain awareness” system, developed with Microsoft, that links 3,000 cameras, 2,600 radiation detectors and dozens of license plate readers in six locations and mounted on cars. If officers spot a suspicious package in range of a video camera, for example, they will be able to quickly track who put it there. If a terrorist suspect’s tag number is known, the network will scan passing cars to find it.

Paul J. Browne, the Police Department’s chief spokesman, said that under its privacy policy, the department will discard images after 30 days unless the images are part of an active investigation. But certainly the technology to capture and store such data is no longer a limiting factor.

Not so long ago, even the most aggressive government surveillance had to be selective: the cost of data storage was too high and the capacity too low to keep everything.

Not anymore. John Villasenor, an electrical engineer at the University of California, Los Angeles, studied the plummeting cost of computer data storage and reached an astonishing conclusion: It will soon be technically feasible and affordable to record and store everything that can be recorded about what everyone in a country says or does.

And there is plenty of data to store. The average person today leaves an electronic trail unimaginable 20 years ago — visiting Web sites, sending e-mails and text messages, using credit cards, passing before a proliferating network of public and private video cameras and carrying a cellphone that reports a person’s location every minute of the day.

Mr. Villasenor, also a senior fellow at the Brookings Institution, estimates that to store the audio from telephone calls made by an average person in the course of a year would require about 3.3 gigabytes and cost just 17 cents to store, a price that is expected to fall to 2 cents by 2015. Tracking a person’s movements for a year, collected from their cellphone, would take so little space as to carry a trivial cost. Storing video takes far more space, but the price is dropping so steadily that storing millions of hours of material will not be a problem soon.

“It’s so cheap that you can afford to throw away 99.9 percent without looking at it,” says Mr. Villasenor, who explores the possibilities in a Brookings paper, “Recording Everything: Digital Storage as an Enabler of Authoritarian Governments.”

And a government sleuth would, of course, be able to efficiently find anything of interest in the data because of the parallel revolution in search technology.

It is hard to exaggerate how dramatic the change has been. In the 1960s, the National Security Agency used rail cars to store magnetic tapes containing audio recordings and other material that the agency had collected but had never managed to examine, said James Bamford, an author of three books on the agency.

In those days, the agency used the I.B.M. 350 disk storage unit, bigger than a full-size refrigerator but with a capacity of 4.4 megabytes of data. Today, some flash drives that are small enough to put on a keychain hold a terabyte of data, about 227,000 times as much.

That kind of convenience and economy is hard to resist, whether you are a parent storing photos of your children — or a government trying to keep an eye out for threats.