Opening up access to NYC data?

This week I submitted our recommendations [PDF] for data sets to include in New York City’s “Big Apps” competition.  September 1 was the deadline for data ideas, and also for responses to the RFP to run the competition itself.

I was a bit surprised by the city’s announcment in June about Big Apps, because I know painfully well how tight-fisted the city’s agencies can be with their data.  (For example, I have helped sue city officials to turn over data, and have written more FOIL request letters than I can count.)  Some agencies have opened up in recent years, but many still cling to outmoded thinking about “their” data and wanting to control it and prevent people from accessing it.

But as I thought more about BigApps, it seems to be as much about leveraging economic development as it is about access to data.  The announcement itself talks about transparency & open government, but this is in the context of “encourag[ing] the public to develop applications” rather than data access as a good thing on its own.  In Washington, DC — the model for New York’s Big Apps competition — the data was made available first, and the competition came second.  New York is doing it the other way around.

So I submitted my comments hoping for the best, but realizing that important questions still remain:

  • Will the city provide full data sets, or just excerpts for prototype apps?
  • Will all agencies participate, or just some? (The city’s news release talked about 80 data sets from just 32 agencies, but that’s just the tip of the iceberg – see below.)
  • Will data be available just for app developers, or for anyone (or any entity)?
  • If no one writes an app for a specific data set, will the city continue making that data set accessible?
  • What about data that might be used to criticize government efforts (eg, inadequate buildings inspections or complaints that haven’t been addressed)?

Also, the city is just one player (though a big one) in the game.  Many state and federal agencies have already opened up their data.  But others, such as the Metropolitan Transit Authority – a public authority (not under the mayor’s control, or anyone else’s, some might say), are still fighting open access to their key data sets.  In order to give the public a complete data picture, we need to make sure all levels of government become transparent.

Nonetheless, it’s great to see the growing interest in opening up access to city data.  The city’s BigApps announcement has re-focused attention on the issue, there are several sites now dedicated to it, and the City Council is considering legislation along these lines.  And New York is just the latest city to move in this direction, following Washington DC, Boston, and San Francisco (and perhaps others I’m not aware of).  It’s easy to say that open data initiatives should’ve/could’ve been implemented much sooner (for example, my BigApps submission includes a summary of a 1999 white paper I co-authored laying out a framework for public access to municipal data).  But the advent of all the impressive open source technologies and mobile apps have captured the public’s imagination so that even hardened city officials can’t ignore the push in this direction.  So hopefully some good will come of it.

My submission referenced a 2001 inventory of NYC data systems [PDF] as an example of virtually all city agencies maintaining computerized data which can & should be publicly accessible.  Most of these (in 2001) were mainframe apps and have now likely been subsumed in other systems or initiatives (such as 311).  But given the glacial pace of (technological) change in municipal government, I wouldn’t be surprised if some of these systems are still around.  And the data in them certainly still exists.  And, by and large, the regulatory programs that created them in the first place are still around. 

nyc2001datainventory

Aside from its age, the 2001 inventory should provide a helpful road map for what data we might reasonably ask the city to make available.  Hopefully people will use this information constructively and persuasively toward that end.

Btw, my BigApps recommendations didn’t focus so much on GIS data as it did on other interesting data sets.  New York definitely needs to do a better job about access to its GIS data — many agencies don’t publicize or make it obvious how to access their GIS data.  But the two main repositories of GIS data sets (DoITT and the Dept of City Planning) already do a pretty good job about making people aware of downloadable GIS data and providing access to it. 

But I did focus part of my comments on the untenable license restrictions and fees charged by City Planning for its “MapPLUTO” data — GIS layers combining tax parcel boundaries with detailed property information. 

mappluto

True, the Planning Dept puts time and effort into aggregating several sources of property data into PLUTO.  But they’ve already done so much to remove fees for other GIS layers such as the street grid (the LION file) and district boundaries — why stop there?  Unfortunately they have monopoly control over the data, so even we pay the fees, but it flies in the face of the Mayor’s Big Apps announcement, so hopefully Big Apps will finally force City Planning to see the light when it comes to PLUTO.

About these ads

7 Responses

  1. I wasn’t aware that this 2001 directory was available anywhere online, thanks for posting it. The Open Gov NYC group has been working with Sunlight Labs through transparencycorps.org to digitize the public data directory, but until now, we just had access to the 1993 document. It’ll be interesting to compare the two. The goal was to first digitize the directory, update it piece by piece as much as possible and provide that for the National Data Catalog. I’m guessing you only have this document as a scanned PDF, not in any machine readable form? In any case, it’d be great if you could help the group out with this effort. I’ve got a blog post that covers some of this work and frames it in the larger context – http://topplabs.org/civichacker/2009/08/help-open-the-big-apple/

    • Thanks for your note. I’ll take a look at the Open Gov NYC group site.

      I only have the 2001 inventory in hard copy and the scanned PDF version. It was part of a series of meetings hosted by the New York Area Data Council in 2002 on the topic of “data integration” within New York City agencies and on a regional basis.

      Your comment reminds me, though, of another similar effort in 1997 that also might be a helpful starting point for a public data directory. The Center for Urban Research created a regional compendium of statistical data sets called “FYI : a comprehensive guide to statistical sources for the New York City Metropolitan Area : 1997″. Here’s the cover image and a screenshot. We have a couple of extra copies at our Center, and the NY Public Library has a reference copy (http://www.nypl.org/research/sibl/govt/govlocal.htm). But here’s a link to an Access database with the underlying entries: FYI database.

  2. [...] Romalewski, a pioneer of web-based GIS and community mapping projects, is also skeptical of the contest. He notes that it offers no explicit guarantee that any datasets will be fully [...]

  3. [...] Romalewski, a pioneer of web-based GIS and community mapping projects, is also skeptical of the contest. He notes that it offers no explicit guarantee that any datasets will be fully [...]

  4. [...] two related initiatives, a new “DataMine” website and the NYC BigApps competition.  I blogged about both items last month, and though I was skeptical then, I’m quite impressed [...]

  5. [...] to the city’s BigApps competition, which is focused primarily on application development (and the resulting economic development from these apps), not so much on transparency and open data [...]

  6. [...] not easily, publicly accessible.  My blog post when BigApps was first announced in 2009 has a listing of some key data files that still haven’t seen the light of [...]

Comments are closed.

Follow

Get every new post delivered to your Inbox.

Join 1,512 other followers

%d bloggers like this: