I noticed today that NYC’s new OpenData site (on the Socrata platform) has made some modest improvements since I blogged about it earlier this month, and since several people have responded to comments from Socrata’s CEO.
In particular, many of the files listed in the Socrata/OpenData site as “GIS” files or “shapefiles” are now actually available for download as shapefiles. You have to dig a bit to find the download option — it’s not available via the button. You have to click the button, and then scroll down to the “Attachments” section of the About page. But in many cases, you’ll now find a zipped file containing a GIS shapefile. Small — but important — victory!
The back story
When the OpenData site first launched, I was very concerned because there was no option to actually download most geospatial data sets — you could only access them as spreadsheets or web services via an API. That’s not very helpful for people who want to work with the actual data using geographic information systems. And it was a step backward, since many agencies already provide the GIS data for download, and earlier versions of the OpenData site had made the data available for direct download.
It also seemed like it was extra work for the agencies and for us — extra work to convert the data from GIS format into spreadsheets, for example, and then extra work for the public to try to convert the data back into GIS format once they had downloaded a spreadsheet from the OpenData site. Seems pretty silly.
It also seemed like it was an example of DoITT not understanding the needs of the public — which includes Community Boards, urban planning students, journalists, and many others who routinely use GIS to analyze and visualize data. Spreadsheets and APIs are nice for app developers — and the “tech community” broadly speaking — but what about the rest of us?
More public access to data, not less
If the city adds the shapefiles as a download option, that’s providing more open access to data, not less. But by not offering GIS data along with the other formats, the Socrata system seems to be limiting access. I’d hope that NYC would be as open and flexible and accommodating as possible when it comes to accessing public data. Socrata’s CEO seems to argue that with the Socrata platform it’s too hard to do that. If he’s right, maybe we should just stick with a tried and true approach — NYC agency websites already provide direct download of GIS data along with many other formats.
But I know that we can do better. In fact, Chicago’s open data portal (also powered by Socrata) has offered many GIS datasets for direct download from Day 1. Actually, Chicago has 159 datasets tagged as “GIS” files, while New York only has 69 – what’s up with that, NYC? I thought NYC was the best in everything when it comes to open data?
Still more to be done
Alas, even though we’re talking about a victory here, we can’t pop open the champagne quite yet. Several of NYC’s data sets via the Socrata site aren’t as current as what you can already get from agency websites. For example:
- zoning is current as of August 2011, but you can download more current data (September 2011) from the ever-improving Planning Department’s Bytes of the Big Apple website;
- building footprints are older (September 2010) than what you can download from DoITT’s GIS site itself (click through DoITT’s online agreement and you’ll get a buildings database from March 2011); and
Also, some data sets described on the Socrata/OpenData site as “shapefiles” are still not available in GIS format. Some examples:
- NYC’s landmarks data. The OpenData site describes this data as a “point shapefile … for use in Geographic Information Systems (GIS).” But it’s only available from the OpenData site as a spreadsheet (or similar format) or via an API.
- Waterfront Access Plans. The OpenData site describes this file as a “polygon shapefile of parklands on the water’s edge in New York City … for mapping all open spaces on the water’s edge in New York City.” But like the landmarks data, it’s only available as a spreadsheet or via an API. False advertising, if you ask me. But if you go to the source (the City Planning Department), the shapefile is there for all to access. So why is the Socrata/OpenData site any better? I’m still wondering that myself.
And the Socrata/OpenData site still doesn’t provide the kind of meaningful data descriptions (or metadata) that you’ll get from agency websites such as Bytes of the Big Apple or Dept of Finance — data descriptions that are absolutely essential for the public to understand whether the information from NYC OpenData is worth accessing.
But hope springs eternal — someone listened to our concerns about lack of actual geospatial data downloads, maybe they’ll also listen when it comes to everything else. Fingers crossed!