This week I submitted our recommendations [PDF] for data sets to include in New York City’s “Big Apps” competition. September 1 was the deadline for data ideas, and also for responses to the RFP to run the competition itself.
I was a bit surprised by the city’s announcment in June about Big Apps, because I know painfully well how tight-fisted the city’s agencies can be with their data. (For example, I have helped sue city officials to turn over data, and have written more FOIL request letters than I can count.) Some agencies have opened up in recent years, but many still cling to outmoded thinking about “their” data and wanting to control it and prevent people from accessing it.
But as I thought more about BigApps, it seems to be as much about leveraging economic development as it is about access to data. The announcement itself talks about transparency & open government, but this is in the context of “encourag[ing] the public to develop applications” rather than data access as a good thing on its own. In Washington, DC — the model for New York’s Big Apps competition — the data was made available first, and the competition came second. New York is doing it the other way around.
So I submitted my comments hoping for the best, but realizing that important questions still remain:
- Will the city provide full data sets, or just excerpts for prototype apps?
- Will all agencies participate, or just some? (The city’s news release talked about 80 data sets from just 32 agencies, but that’s just the tip of the iceberg – see below.)
- Will data be available just for app developers, or for anyone (or any entity)?
- If no one writes an app for a specific data set, will the city continue making that data set accessible?
- What about data that might be used to criticize government efforts (eg, inadequate buildings inspections or complaints that haven’t been addressed)?
Also, the city is just one player (though a big one) in the game. Many state and federal agencies have already opened up their data. But others, such as the Metropolitan Transit Authority – a public authority (not under the mayor’s control, or anyone else’s, some might say), are still fighting open access to their key data sets. In order to give the public a complete data picture, we need to make sure all levels of government become transparent.
Nonetheless, it’s great to see the growing interest in opening up access to city data. The city’s BigApps announcement has re-focused attention on the issue, there are several sites now dedicated to it, and the City Council is considering legislation along these lines. And New York is just the latest city to move in this direction, following Washington DC, Boston, and San Francisco (and perhaps others I’m not aware of). It’s easy to say that open data initiatives should’ve/could’ve been implemented much sooner (for example, my BigApps submission includes a summary of a 1999 white paper I co-authored laying out a framework for public access to municipal data). But the advent of all the impressive open source technologies and mobile apps have captured the public’s imagination so that even hardened city officials can’t ignore the push in this direction. So hopefully some good will come of it.
My submission referenced a 2001 inventory of NYC data systems [PDF] as an example of virtually all city agencies maintaining computerized data which can & should be publicly accessible. Most of these (in 2001) were mainframe apps and have now likely been subsumed in other systems or initiatives (such as 311). But given the glacial pace of (technological) change in municipal government, I wouldn’t be surprised if some of these systems are still around. And the data in them certainly still exists. And, by and large, the regulatory programs that created them in the first place are still around.
Aside from its age, the 2001 inventory should provide a helpful road map for what data we might reasonably ask the city to make available. Hopefully people will use this information constructively and persuasively toward that end.
Btw, my BigApps recommendations didn’t focus so much on GIS data as it did on other interesting data sets. New York definitely needs to do a better job about access to its GIS data — many agencies don’t publicize or make it obvious how to access their GIS data. But the two main repositories of GIS data sets (DoITT and the Dept of City Planning) already do a pretty good job about making people aware of downloadable GIS data and providing access to it.
But I did focus part of my comments on the untenable license restrictions and fees charged by City Planning for its “MapPLUTO” data — GIS layers combining tax parcel boundaries with detailed property information.
True, the Planning Dept puts time and effort into aggregating several sources of property data into PLUTO. But they’ve already done so much to remove fees for other GIS layers such as the street grid (the LION file) and district boundaries — why stop there? Unfortunately they have monopoly control over the data, so even we pay the fees, but it flies in the face of the Mayor’s Big Apps announcement, so hopefully Big Apps will finally force City Planning to see the light when it comes to PLUTO.