Open Data: Compare and Contrast

Soon after I posted on the very open and collaborative project at the U.S. Geological Survey, Wired had a post on the other side of the coin, Officials Hoard Valuable Databases Funded by Taxpayers

Government agencies across the country are sitting on gigabytes of valuable digital data that could be mashed, mixed and re-organized in crafty ways by Web 2.0 entrepreneurs and public interest groups engaged in everything from government oversight, to providing practical information to Americans.

Yet, despite federal and state public records laws designed to make the data accessible, many agencies are fighting more ferociously than ever to keep data created with public funds out of public hands. In their battles to withhold information, bureaucrats are citing everything from copyright and trade secret privileges to privacy and national security concerns. -Read more on Wired.

It's the clash of the titans. In one example from Wired, a nonprofit requested anonymous information about patient diagnoses, along with the physicians' identifying number, the procedures performed and amounts billed to Medicare to help match patients to experienced doctors for specific procedures and to determine if practitioners were over-billing Medicare or receiving government funds for high-risk procedures they didn't have the experience to perform. The nonprofit wanted to make this information available to consumers via their website. But the U.S. Department of Health and Human Services--parent of Medicare--denied the request on grounds that it violated doctor privacy by disclosing physician incomes.

And therein lies the rub for more than a few datasets: individual privacy and personally identifiable information (PII). Information that had been gathered painstakingly using paper and data cards and maps stored in City Hall are now much easier to put together and much easier to access. And data gathered for one purpose might be very useful for another, unrelated purpose. One that the data owner did not necessarily agree to. 

Government under the Privacy Act has an obligation to protect citizens' privacy. The Privacy Act states in part:

No agency shall disclose any record which is contained in a system of records by any means of communication to any person, or to another agency, except pursuant to a written request by, or with the prior written consent of, the individual to whom the record pertains... (See the Department of Justice for the Privacy Act of 1974).

We are experiencing a huge shift in data accessibility. 

Example of Big Shift #1: When we were buying our house in the early 1990's, I had to go downtown during "business hours" to research the amount of money that the current owners paid. [They were claiming that they were going to lose money on the transaction, and the data showed otherwise!] Now, anyone can go to WashingtonPost.com and see the price we paid. 

Example of Big Change #2: A few years ago, we used to call people to get directions to their houses. (Make a right at the second light there's a 7-11, then you'll drive by the house with the gnomes...) Now you give me a number and street, I can easily get specific directions--and even a picture--of your nearby Starbucks or your house. From my GPS enabled cellphone.

So what?, you say. Well, the Supreme Court, privacy advocates, and just plain people who don't want everyone to know how much money they make are concerned about making private information--or information that they consider private--easily available and combinable online. The barriers of time, location, and, even, expertise have fallen away. It changes the entire game.

In many, many cases, the PII can be scrubbed from the data set. But in some subset of cases, when data sets are combined, PII is exposed.  

This is the true clash of the titans--transparency versus privacy. The public versus the personal. Increasingly, it is government between these two giants, trying to balance and move forward (okay, and to be honest, sometimes playing one against the other to meet other, less pure objectives).

We live in interesting times. There are real problems to solve. We got some work cut out for us.