Last fall I had high hopes that New York City would loosen the shackles that agencies too often held tightly around “their” data sets. The city’s BigApps competition had just been announced, the new Data Mine website was launched with many data sets I never imagined would see the light of day, and the city (i.e., the Mayor’s office and his agencies) seemed to be jumping on the open data bandwagon.
These days, I’m less optimistic about NYC’s #opendata efforts. Sure, there are bright spots (DOT, MTA, some aspects of City Planning’s Bytes of the Big Apple). But for the past several months I’ve been hearing rumors that Data Mine will be updated “soon” and “any week now”. So far, nothing new on the site — data is still from 2009. I’ve also been hearing that Data Mine will be updated when the next BigApps competition is announced. Maybe that’ll happen, but even if a new BigApps prompts the city to update Data Mine, that’s problematic – I explain below.
When Mayor Bloomberg announced BigApps, he made a big deal of how the city would be “providing information to New Yorkers as fast and in as many ways as possible” and of helping entreprenuers use city data to “increase accessibility and transparency in City government, generate jobs, and improve the quality of life for New Yorkers.”
And since then, the city’s own information technology agency indicated it would usher in a sea change in how city agencies made data publicly accessible. This was somewhat buried in DoITT’s “30 Day Report” (issued Feb. 2010), but page 29 featured a section titled “Open Data/Transparent Information Architecture”. It said [PDF],
In 2010, DoITT will work to establish citywide policies around “open data.” These efforts will align with Mayoral initiatives of openness and transparency, and further improve access to information by creating citywide standards that are practical and feasible. As a start, City agencies should be required to make available, to the greatest extent possible, all public‐facing data in usable electronic formats for publication in the NYC DataMine. This mandate would apply to all public data that is not subject to a valid restriction, such as public safety or personal information. City data is by and large the property of the people it serves, and DoITT will be at the forefront of continuing to make it available in as many ways as possible. [Emphasis added.]
Note that this policy was meant to be only a beginning, and that DoITT would be “at the forefront” of aggressively making public data widely available.
… vs. action
The Data Mine website was launched in October 2009. Most of the data sets at the site had a vintage of 2009 (and some were substantially older — for example, NYC Economic Development Corporation provides geographic data sets that are “based on PLUTO 2005” [PLUTO is the city’s tax parcel data]).
The Data Mine website itself claims that it will be “… refreshed when new data becomes available.” The data update frequency for many data sets on Data Mine is listed as daily (such as detailed school information from Dept of Education and traffic and parking data from Dept of Transportation), monthly (recycling rates from Dept of Sanitation), or quarterly (most of the geographic data from the Parks Department). Others are listed as “annually” or “as required”, but the “as required” data sets include NYC landmarks and historic districts (several of which have been updated since Fall 2009) and 311 data.
Even though some of these data updates are already publicly available directly from the individual agencies, Data Mine — as the city’s portal to public data access — hasn’t kept up. And it appears that Data Mine is really just an adjunct to the city’s BigApps competition, which is focused primarily on application development (and the resulting economic development from these apps), not so much on transparency and open data access.
For example, testimony from DoITT’s commissioner at a recent City Council hearing for Intro 029 (a bill requiring city agencies to provide formalized open access to their data) was revealing. Among other things, she explained that the Mayor’s office would wait till the next iteration of the BigApps competition before updating the Data Mine website with new data sets. (Note that this is the same Commissioner who issued DoITT’s 30-day report cited above.)
The commissioner’s presentation starts about 9 minutes into the clip below. Here’s her testimony [PDF]. Another disconcerting point she made in her comments was that the Mayor’s office wanted to put a priority on data that they believed had value to the public (rather than posting data regardless of how the public might use it or value it).
Linking Data Mine to BigApps has at least two problems. The first is: Why wait? Some agencies are already taking steps on their own to publish data and update it regularly (such as City Planning and Transportation). I don’t see any reason to delay updates to Data Mine. Otherwise the site is stale, and sends the wrong message.
In this era, it’s a no-brainer to make data widely and easily available, given all the amazing things people are doing with public data (helping reduce costs, promote economic development, enhance quality of life, improve government efficiency, etc). As one blogger put it, “there’s really no reason for the city to spend the time to ‘discuss’ when the city could spend the time to ‘do’.”
The other problem is that we shouldn’t have to rely on a competition to make data publicly available.
Remember that when the state’s Freedom of Information Law (FOIL) was first enacted (in the mid-1970s), “apps” didn’t exist. It was all about accountability. The public had a right to know what its government knew — and to have easy access to that information so we could evaluate legislative, executive, and regulatory decision making.
Actually, the “legislative declaration” to FOIL in New York State makes a bolder statement: that public awareness of government actions is essential to maintain “a free society”. FOIL also emphasizes that people will (hopefully) understand and participate more fully in government when they know fully what their government is up to.
So apps are cool and powerful, but open government and open data goes much deeper than the latest iPhone app to find the best parking spots. The more the city ties public data access to app development and competitions like BigApps, the more they veer away from facilitating the public’s fundamental right to know.
App competitions also take attention away from the vibrant community of nonprofits, neighborhood planning groups, Community Boards, and others who want to improve quality of life in the city and steer a progressive course when it comes to local development and citywide policies. Not to mention the mainstream media and bloggers. These players may not be developing apps, but they’re doing good work in other ways. Information access for these groups and individuals is vital. Some city agencies are smart and know how to work strategically with these groups to move good policies forward. But too often agencies hunker down and get defensive, and don’t want anyone to have access to data.
Clay Johnson, former director of Sunlight Labs, also makes this point at his “InfoVegan” blog. And even some BigApps competitors noted the downsides of relying on a competition to made public data accessible.
A better approach
I think the city’s open data efforts would be greatly enhanced by:
- passing the City Council’s Intro 029;
- opening up more data (things like property data that are still restricted by a license and access fees);
- redesigning Data Mine as a pointer to existing agency data repositories; and
- ensuring that public data sets are refreshed as often as practical.
Of course, we can’t place our faith in just putting the data out there. It still takes people making policies and actual improving things. It still takes an educated public to take action, etc. But having more data, as long as it’s not in closed formats and is widely accessible, is a good thing.
(Disclaimer: my viewpoints on this blog are my own, not necessarily my employer’s.)