NYC neighborhood changes mapped with aerial imagery, historic land use data

Our team at the CUNY Graduate Center has enhanced the OASISnyc.net mapping site with new data and features to visualize neighborhood change across the city. On the eve of the 10th anniversary of 9/11, the updates help provide context for the transformation taking place in lower Manhattan, as well as in other key areas of the city.

I’ve only included some of the highlights below. Our OASIS wiki has more details plus screenshots and other examples.

2010 Aerial Imagery

We’ve added new aerial imagery, thanks to the NYS GIS Clearinghouse. Now you can view overhead images from 2010 (as well as 1996, 2004, 2006, and 2008) throughout New York City and Long Island. (The 1996 imagery is from NYC DoITT, 2004 is from USGS, and the other years are from the NYS GIS Clearinghouse).

For example, you can see what the World Trade Center site looked like from above in 1996, and then in 2006, and more recently in 2010. The overhead images show clearly how the building footprints are reflected in the memorial plaza fountains now under construction.

WTC 1996

OASIS_WTC_1996.png

WTC 2006

OASIS_WTC_2006.png

WTC 2010

OASIS_WTC_2010.png

Visualize Aerial Photo Changes like a Timelapse Movie

We’ve also changed the way you can view the imagery over time. Now you can move the aerial timeline slider across years to transition from one year to the next, creating the effect of a timelapse movie within the OASIS map.

aerialtimelineslider200.png

You can move the slider as slow or as fast as you’d like.

A good example of the new timelapse feature is Shea Stadium (now Citi Field) in Queens. The images below illustrate the transition in recent years.

Shea Stadium (2006)

Shea_2006.png

Shea Stadium (2006-08)

Shea_06-08.png

Citi Field almost done (2008-10)

Shea_08-10.png

Citi Field (2010)

Shea_2010.png

Land Use Changes Citywide

Since 2010, OASIS has provided the ability to display historical land use patterns (for 2003 through 2009). This gives you the ability to easily see how patterns have changed in key areas of the city.

In lower Manhattan, the area around the World Trade Center site has changed substantially in the past 10 years. Of course reconstruction is underway at the WTC site itself, but the surrounding community has become much more residential. The land use maps below from OASIS visualize some of these changes (yellow and orange are residential properties, brown is vacant, and light red is commercial).

North Battery Park City & TriBeCa vacant land (and WTC empty): 2003

LMLU_WTCBPC03sm.png

Residential towers built, WTC site in redevelopment

LMLU_WTCBPC10sm.png

The maps below highlight the changes from commercial office buildings to residential towers, such as the block between Broad and Hanover streets & Wall and Exchange streets — especially the JP Morgan Building at 15 Broad St and the National City Bank Building at 55 Wall St.

Financial District commercial property circa 2003

LMComm03.png

Replaced by residential by 2010

LMResidential10.png

More Community Data

The latest example of linking mapped information from grassroots groups is the layer of skate parks in the city by longtime OASIS partner Open Road of NY. (The OASIS community mapping effort is based on Google’s new Fusion Tables service; more info here.)

Here’s the link to the map on OASIS, and the original Google map from Open Road.

We’ve also added the locations of stalled development sites across the city (based on a map from Crain’s New York Business), and the city’s hurricane evacuation centers (more on that here).

Advertisements

Mapping Hurricane Irene in NYC (plus some thoughts on the city’s digital response to the storm)

A disaster, natural or otherwise, always creates an opportunity to demonstrate the power of maps. Hurricane Irene did not disappoint. In New York City, which hadn’t seen a hurricane of this magnitude in decades, there were at least a half dozen websites with interactive maps related to the storm (plus at least one PDF map – more on that below) that were used extensively and were tweeted about extensively. My team at the CUNY Graduate Center was in the mix with our OASISnyc.net site, and I was watching with keen interest as more maps kept coming online as Irene kept coming closer. I thought I’d share some observations below about how Irene was mapped in New York.

I think I kept good track of the various maps that were deployed, but I’m sure my list and descriptions are incomplete so please chime in if I’ve missed anyone or mischaracterized any of the efforts.

The Context

Hurricane maps are nothing new, but usually the maps show the path of a hurricane while it’s happening or analyze its impact after the storm has past. This time, for New York City, the more interesting and useful maps were focused primarily on the possibility of evacuation, and the potential impact of the storm on New York’s shores.

(That said, the damage from Irene continues north of NYC, and several important mapping efforts are helping with the recovery effort there. For example, follow tweets from @DonMeltz and @watershedpost in upstate New York, or @jarlathond in Vermont.)

The interest in these maps was also perhaps more intense than in earlier situations. First, New Yorkers almost never evacuate for anything (at least on a scale of hundreds of thousands of residents), so the idea that so many people from only certain areas of the city needed to move to higher ground meant that everyone wanted/needed to know: am I in the evacuation zone? And that meant maps.

Second, online interest in this storm in particular was high. Other storms have hit since Twitter and Facebook have been around, but not in the New York area and not at this scale. One writer for GigaOm who had lived through hurricanes on the Gulf Coast wrote that she was “overwhelmed” by the “overall hoopla surrounding Irene online.” For her, it replaced TV as a key source of news (I agree, I barely checked TV news throughout the storm. Twitter and weather-related websites provided all the information I needed, and the news from these sources was more up-to-date.) And because so many New Yorkers were online and hungry for information about evacuations and storm impacts, online maps were critically important.

Will I Need to Evacuate?

Mayor Bloomberg and other officials started talking about the possibility of evacuation on Wednesday (8/24). That night, my wife reminded me that our flagship mapping site OASISnyc.net included a layer of “coastal storm impact zones”.

Actually, we’ve had that data online since 2007, when it was a Map of the Day on Gothamist. It shows areas at greatest risk of storm surges from a hurricane (and, as it turns out, those areas closely match the boundaries of the city’s evacuation zones – see screenshot below). I had also received a couple of emails that night from other groups wanting to map the evacuation zones and were worried that the city’s mapping resources weren’t up to the task.

So I wrote a blog post about how using OASIS could help people see if they were in harm’s way if the storm hit the city. I published the post the next morning (Thursday, 8/25).

That same morning, Gothamist posted an item about the potential for evacuation, and they embedded the city’s evacuation zone map. I was the first to add a comment on the Gothamist piece (via our @oasisnycmaps Twitter account), and I included a link to my blog post and to the maps.

PDF maps: blessing, curse, or both?

Let’s look at the city’s evacuation zone map [PDF – see image at right]. It’s a PDF file. It shows all the city’s streets in black ink, in an 8.5″ x 11″ layout, overlain on color shaded areas (muted green, yellow, and brown) corresponding to the 3 evacuation zones A, B, and C. And it has the evacuation center locations labeled on the map.

So it puts a lot of information into one map, which is challenging on its own. But trying to view that as a PDF online can be especially problematic. People who expected something better complained — it was described (perhaps too harshly) as “terrible” and “useless” in the Gothamist comments. People said it was hard to read, took too long to download, it didn’t work well on mobile phones, etc. And quickly after I posted my comment at Gothamist, several people were thankful that they could access OASIS as an alternative to the city’s map.

Distributing a PDF map in a situation like this has pros and cons. On the one hand, it has flexibility. The PDF format can be viewed in any web browser, or can be downloaded to your computer and viewed there, and can be printed out to share with someone who doesn’t have Internet access. And lots of people on Twitter were appreciative. On the other hand, it’s not something that can be easily updated, and it’s not what the growing population of digitally savvy New Yorkers would expect or desire. NYC has been touting itself as the most digital city on the planet, and all they could do was put out a PDF? People were underwhelmed.

To be fair, the city also had an online “Hurricane Evacuation Zone Finder.” You’d type in an address and it would display a zoomed in zone map of your location. But that provided little context, and it wasn’t as user-friendly as the public was expecting. For a long time this type of web service would’ve been considered state of the art. But these days, I think a lot of people were wondering if New York couldn’t do any better.

Luckily the city had posted a dataset in GIS format representing the city’s evacuation zone boundaries. It was available on Datamine, and anyone could download it for free and use it without restriction. So when people asked me if I had the evacuation zones in a format that could be mapped, I just pointed them to Datamine.

(In a mix of optimism and revisionist history, New York City’s Panglossian chief digital officer was quoted saying that As always, we support and encourage developers to develop civic applications using public data” (emphasis added), in reference to other groups that were using the evacuation zone map in their websites. I chuckled when I read this. If you’ve been around this business for more than just a year or two, you’d know that it hasn’t “always” been this way. It’s terrific that at least some of the city’s data is openly available now. But let’s keep it in perspective, and also remember that there are still important public datasets the city is not making easily available to developers or others.)

NYC.gov goes down

By Thursday afternoon, online interest in NYC’s impending evacuation announcement was so intense that not only did the city’s zone finder application go down, but even the city’s website — in particular, its homepage — was inaccessible.

Although the city will certainly congratulate itself for using social media to get the word out (and I agree they did a good job in this area), it’s not good that the city that strives to be the nation’s premier digital city could not even serve up its homepage at the exact moment when everyone was relying on that web page for information on what was happening next. And with a situation as complex as an approaching hurricane, 140 character tweets are just not enough. I can’t imagine it’s easy to withstand several million hits in a day, but I and a lot of others expected better.

(After complaints from the community, at least one civic activist posted links to hurricane resources on his own and shared that via Twitter.)

The city was left apologizing for no web access and pointing people to its PDF map (at this point hosted on Tumblr and elsewhere). Mayor Bloomberg posted the PDF on his website, but that’s the least he could do. Simply taking a PDF and putting it on another website? Doesn’t take much to pull that off.

More Maps Come Online

In the meantime, more maps appeared. WNYC was next. John Keefe, the public radio station’s Senior Executive News Producer, mashed up the city’s evacuation zone data with Google Maps, and put a simple, easy to use interface together. The map didn’t include evacuation centers at first, but it was clean, effective, and … in the absence of the city’s online resources … it worked.

In fact, several people noted the irony. When @nycgov tweeted that the city’s hurricane zone finder was down “due to high traffic”, a Google representative quickly tweeted back that “WNYC’s map is based on NYC OEM data and is running fine.”

John has developed a successful system creating news-oriented maps in short order, and his hurricane map was the latest example. And it was embeddable, so sites such as Gothamist that originally embedded the city’s PDF map, quickly replaced the PDF with WNYC’s interactive map. People were happy.

By Friday morning, the city was still having difficulty providing online access to its web page and its hurricane evacuation zone finder app, so more mapping sites stepped up. ESRI published an interactive map of the evacuation zones and evacuation centers using their relatively new ArcGIS.com online platform. The map looked great, and included the evacuation centers that WNYC’s map was missing.

But the ESRI map didn’t have the slimmed down, focused look and feel of WNYC’s site. It included ArcGIS.com options such as geographic feature editing that maybe weren’t needed for this situation. (That’s just a quibble. Though at one point I clicked “Edit” and it seemed like I was about to delete all of Zone A!)

One nice thing about the WNYC site is that is uses Google’s Fusion Tables service on the backend, which makes it easy to set up geographic data and then overlay that data on a Google map or any other modern, online mapping site. At the CUNY Graduate Center we’ve started to use Fusion Tables to integrate community-oriented mapped information into the OASISnyc site. By Friday morning we were able to use Fusion Tables to display the city’s evacuation centers on OASIS’s maps. The OASIS site provides a wealth of information such as subway and bus routes, schools, public housing sites, etc. so it provided a way (hopefully an easy way) to locate evacuation sites in relation to these other locations.

By Friday, Google had also stepped in with a mapping service of its own, a customized version of its crisis mapping application.

Originally Google’s map omitted the city’s evacuation zones or centers, but it did include several other layers of data related to potential storm impacts (like the storm surge map at OASIS). The federal weather and environmental agencies such as NOAA and FEMA have consistently done a great job of providing free, online access to observation and modeling data about storms, and Google put this information to use.

Regional Maps

On Friday our team at the CUNY Graduate Center also made two enhancements to our mapping applications to make it easy for a wide range of people to find out if they might be hardest hit by Irene. First, we reconfigured the OASIS maps so the storm surge layer could load quick. We created a pre-cached tiled layer instead of a dynamic layer and also set up the map page so that most of the dynamic layers were turned off by default. This made the map page load quicker, and made the storm surge layer load instantaneously (our site had bogged down a bit on Thursday due to increased traffic — site usage almost tripled to 9,000 pageviews almost solely from my comment at Gothamist with a link to OASISnyc.net — so quick loading was key).

We also incorporated the storm surge layer to an interactive mapping site we maintain with the Long Island Index focused on Nassau and Suffolk counties. It seemed that the storm might have a greater impact on Long Island. The storm surge data we used for OASIS was statewide in scope (it was created by NY SEMO), so we coordinated with our partners at the Index and updated the site Friday afternoon.

Newsday included a link to the LI Index mapping site, and usage soared over the weekend.

Understandably, an organization such as WNYC would limit its map to the city’s 5 boroughs. But there weren’t similar maps for any other part of the tri-state region.

Even though mandatory evacuations had been called for much of Long Island’s south shore, the best data available on those areas were lists (some in PDF format) of affected addresses and affected streets. Given the surge in usage of the LI Index mapping site, I like to think that we helped meet a key need.

Mandatory Evacuation and More Maps

During the day on Friday, Mayor Bloomberg announced the city’s mandatory evacuation plans. The scramble was on to see if you were in Zone A!

Not to be outdone by WNYC, Google, or anyone else, the New York Times launched its version of an interactive evacuation zone map late in the day Friday.

Like WNYC’s version, the NY Times map was focused and easy to use. But it was also limited to NYC, despite the Times’s readership outside the 5 boroughs who had also been required to evacuate.

By then, WNYC and Google had also added the locations of evacuation centers to its maps.

Lessons Learned?

So what to make of all these maps?

I think the first thing is that they were all generally helpful. When the nation’s premier digital city was incapable of providing digital information in a timely, useful way, others stepped in and succeeded.

These efforts, however, suffered to some extent from inconsistencies and lack of coordination.

For example, different mapping sites displayed different kinds of information in ways that may have been confusing to the person on the street.

Google and OASIS posted storm surge zones and the city (and WNYC, ESRI, and the Times – and eventually Google too) posted evacuation zones. Ultimately what most people wanted to know was if they lived in evacuation Zone A. The storm surge areas were important in terms of anticipating where the storm would do the most damage, but perhaps a more pressing issue was the evacuation.

But this difference in approaches underscores the lack of coordination among the various mapping entities. It was as if everyone just wanted to get *their* map online.

We’re as guilty of that as anyone. I know top staff at OEM and I easily could’ve contacted them to coordinate the OASIS layer with their’s. But it was somewhat frantic at the time, and the communication didn’t happen. I’d say WNYC was the most earnest in this regard, since they probably just saw a hole that needed to be filled – the city was talking about evacuation, but the city’s evacuation map was sorely lacking or not online.

But once WNYC went online, as far as I know there was little coordination among them, us, ESRI, the NY Times, Google, etc. I think you could reasonably ask — since WNYC’s map worked perfectly well, and provided the information about evacuation zones — why have essentially the same map from ESRI, Google, and the NY Times. Were these groups talking with each other? For the media outlets (WNYC and the Times), was it just a competition thing?

I do know that when the city’s GIS community was more cohesive, this probably would’ve been coordinated a bit more, perhaps through GISMO. Not that the lack of cohesion is a bad thing necessarily. And not to fault GISMO or other coordinating groups. But I wonder if better information could’ve been provided to the public in a better way if all of us making the maps were in communication.

For example, for at least a day WNYC’s map lacked the evacuation center locations. I added the locations to OASIS using Fusion Tables. Then WNYC added the locations to its map, also using Fusion Tables. We easily could’ve shared the backend data, but WNYC never contacted us to discuss it. I sent a tweet to @jkeefe about it, but didn’t hear back. It was important to keep the evacuation center data up-to-date and consistent because the city changed the locations of 4 centers before Irene hit. Keeping the maps in sync would’ve minimized any confusion for the public.

Overall, I think the biggest takeaway is that the Mayor’s office and NYC agencies – especially DoITT (since they’re responsible for coordinating the city’s technology resources) – need to engage better with mapping/data/online communities in a much more open, collaborative way.

Despite the city’s talk of apps and open data, there’s still very much a closed approach on the city’s part when it comes to public/private partnerships. True, the city has developed partnerships with local startup tech companies. But the city’s nonprofit and academic communities, along with established private entities, have much to share and have proven they have the technological resources to do as good if not a better job than the city providing essential information online.

In terms of mapping Hurricane Irene in NYC, NGOs filled a big void. The city should not only recognize that effort, but cultivate it and help sustain it so that it works more smoothly and effectively next time.

Coastal storm impact risk mapped in NYC

UPDATED (Monday, Oct. 29, 2012 1pm)

If you need to locate any of NYC’s 76 hurricane evacuation shelters, you can use the OASIS mapping site.

You can also use NYC.gov to find out the latest with Hurricane Sandy.


UPDATED (8/25/11 9am): We’ve added a temporary map layer on OASIS showing the locations of NYC’s hurricane evacuation centers. Here’s the link: http://bit.ly/oBsUY8 . It’s easy to use:

  • Hover your mouse over each one to highlight it (the site details will also be highlighted in the panel on the right).
  • Click on a map marker to bring the site details up to the top of the list.
  • Double-click on a site in the list and the map will zoom in right to that location.

You can type in your address above the map to see if you’re in an area that’s at risk of storm impacts, and how close you are to an evacuation center. The OASIS map automatically also shows any nearby subway stations. And you can add any other layers from the Legend list to the right of the map.

For more information about Hurricane Irene and what you should do to prepare, visit NYC’s website.

UPDATED (8/24/11 12 noon): We’ve expanded the map layer showing hurricane impact zones throughout the downstate region. It now shows potential impact zones on Long Island’s south shore, as well as some areas along the coast of the Long Island Sound.

ORIGINAL POST

For the past several years the www.OASISnyc.net mapping site has displayed a map of “coastal storm impact zones” in New York City (in addition to the wealth of other mapped data included with OASIS). This coming weekend, it seems like the coastal storm map may be especially useful with city officials bracing for the risk of Hurricane Irene bringing 72 mph winds or more to the NYC region by Sunday.

In 2005, the New York State Office of Emergency Management developed a map of areas that would be at greatest risk of hurricane impacts based on wind speed and other factors. The shorthand for the map is a “SLOSH” map, because the zones of impact are based on NOAA’s “Sea, Lake and Overland Surge from Hurricanes” (SLOSH) model projections of vertical surge heights associated with category 1 – 4 storms.

In 2006, we downloaded the SLOSH data from the NYS GIS Clearinghouse website (metadata here), and used the following color shading to represent the zones:

  • A Category 1 storm (impact areas shown on the map in yellow) means winds of 74-95 mph.
  • A Category 2 storm (impact areas shown on the map in light orange) means winds of 96-110 mph.
  • A Category 3 storm (impact areas shown on the map in dark orange) means winds of 111-130 mph.
  • A Category 4 storm (impact areas shown on the map in red) means winds of 131 mph or more.

Here’s the map:

Through the OASIS website, you can easily enter your address and find out if you’re in an area that might be at greatest risk of the hurricane, depending on how severe the winds are by the time Irene moves up the coast. But the real power of OASIS’s maps is that you can do much more. For example, you can:

  • customize the OASIS map to show transit routes, schools, public housing, and libraries in or near the zones;
  • use OASIS’s mapping tools to see areas near the zones that have been recently developed (using the 1996-2010 aerial image timeline tool);
  • zoom in to see individual property boundaries and click on each one to determine ownership, zoning, and land use characteristics; and
  • find out which elected officials represent the area, as well as the local Community Board (good sources for planning and safety resources).

In the images below, I’ve used the aerial timeline slider and the dynamic transparency tool to show recent housing development in an area at risk of impact from a Category 1 storm, along South Beach on Staten Island (at the OASIS website, you can click and drag the name of any map layer in the Legend to the “Transparency Control” box at the bottom of the legend and make it more or less transparent, so the layers underneath can shine through):

Sparse housing near the beach (1996)

Dense housing development (2010)

The zones mapped on OASIS closely mirror the city’s hurricane evacuation zones (mapped here [PDF] and described here). The city also provides a Hurricane Evacuation Zone Finder where you can enter an address and get useful safety tips depending on your zone.

The message: hopefully Irene passes us by, but if not: be prepared. Better safe than sorry.

Mapping the Cityscape exhibit

This summer the Center for Architecture in New York is all about maps.  One of the main exhibits at the Center, “Mapping the Cityscape”, features a dozen or so wall-mounted 8-foot-high maps of Manhattan — different representations and views from 1609 to the present.  Several panel discussions are accompanying the exhibit, including two this week (“Mapping Manhattan” and “Mapping Risk“).  Originally scheduled for the month of July, by popular demand the exhibit will be open through the end of August (Aug. 27th).

The exhibit came out of a panel discussion in May at the Center, organized by Abby Suckle of CultureNOW, with a broad group of participants: Matt Knutzen of the NY Public Library Map Division, John Tauranac of subway mapping fame, Laura Kurgan of Columbia University, and me discussing the OASISnyc mapping site — along with respondents from Google and the Wall Street Journal.  The panel covered lots of ground, from the Library’s invaluable collection of historical maps (now being digitized and geo-referenced) to the evolution of transit mapping in New York to the OASISnyc online mapping site (parks, open space, and much much more) to Columbia’s “Million Dollar Blocks” project to CultureNOW’s mobile maps to the latest from Google and others.  The hall was packed, the audience had lots of questions, and apparently they wanted more — hence the decision by the Center to transform the panel discussion into a summer-long exhibit.

Here are some photos of what you’ll see when you visit:

OASIS maps – each “slice” of the two maps above highlights the different types of mapped data you can display and analyze at the OASIS website.

Million Dollar Blocks – Columbia University’s Spatial Design Lab project that mapped the impact of prison policies on local neighborhoods in New York.

Mannahatta – the Wildlife Conservation Society’s take on what Manhattan likely looked like in 1609; an amazing project, rich with insights, analysis, and visual power.

NY Public Library – historical maps brought to life.

Maps of ecological patterns, historical maps overlain with current geography, transit mapping …

… and land use patterns, cultural icons, and more.

The exhibit opening in early July was packed — Center director Rick Bell & exhibit curator Abby Suckle talk to the crowd about the event:

If you’re interested in how maps — and the very definition of mapping and understanding/visualizing spatial relationships — are changing through the latest interactive technologies,  then this exhibit is for you.  Or if you’re interested in the history of visualizing New York City through maps, then this exhibit is for you.  Either way, please stop by, check out the maps, and attend one of the public programs.  It’ll be a cartographically illuminating experience.

(Exhibit logo from Center for Architecture website.  All photos: Steven Romalewski)

NYC Data Mine data: an object lesson in #opendata challenges

Earlier this week I posted an item about stagnation at NYC Data Mine as well as my thoughts more generally on the city’s #opendata policies and practices.  Today I discuss another challenge regarding open data: data quality and poor metadata.

Background

We recently updated the OASIS community mapping website with several data sets: community gardens, subways, bus routes, bike lanes, and more.  We also updated the map layer representing New York City park areas.  That might seem straightforward for a website focused on open space.  And we were using information from New York’s Data Mine website, which is intended to promote ease of access and use of the city’s publicly available data sets.

But adding the latest parks data to OASIS was much more complicated than it needed to be.  This post describes why, how we got around the complications, and offers suggestions for improvement going forward.  I also include links below to my updated versions of the parks (and playgrounds) data.

Data can be messy, no question about it, but the hassles with the parks data provide an example of the challenges that remain for cities to embrace #opendata, and for the public (or even app developers, for that matter) to seamlessly make use of public data sets.

The context

2006 was the last year that we requested a GIS data set of park properties from the NYC Dept of Parks and Recreation (DPR) for use on the OASIS website.  The Parks Department is a partner in the OASIS project, and was one of the project’s founding organizations.  The agency sees the value in mapping open space resources beyond just city data, integrating (as OASIS does) a wealth of data layers to provide a comprehensive picture of open space issues in any given neighborhood and citywide.

For various reasons (project transitions, mapping application updates, other priorities) our team at the CUNY Graduate Center hadn’t implemented major data upgrades to the OASIS website till recently.  Even last year (2009) when the city’s Data Mine was launched, we didn’t update the parks data on OASIS – the earlier parks data we were using seemed to be more comprehensive.

Therefore, I never looked closely at the parks data from Data Mine until this summer, when we started planning for a major data update on OASIS. When I took a close look at the parks data, I was frustrated and disappointed.

It’s important to note that I don’t fault the Parks Department, per se, for the difficulties I encountered.  I think the problem has to do with a disconnect that often exists between data creators and data users, with little being done by the city itself to mediate.  The Data Mine concept is a good start.  But for a meaningful open data effort, data should be vetted before it’s published, helpful metadata has to be included with each data file, and a dialogue should be fostered to help agencies understand how others seek to use their data in order to create an opportunity to learn from each other.  The parks data just happens to be one set of files that I’ve focused on, but these problems aren’t unique to parks, as others have pointed out (including me — I feel like I’ve been on a tear lately, blogging about data sets with great potential but that need lots of work before they’re re-used).

Data Mine disappointments

Data Mine has three types of “parks” data.  One is the geographic data from the Parks Department (in particular, the “Map of Parks” file, downloaded as “DPR_Parks_001”).  The second is the “raw” data from DPR.  The third is another “OPEN_SPACE” geographic dataset from the city’s Dept of Information Technology and Telecommunications (DoITT).

With all those options, how can you go wrong?  Here’s how:

The DPR geographic data is a great visual depiction of the park areas.  But, in GIS parlance, the dataset contains almost no attributes.  In other words, each park area in the dataset is identified only by its park ID (such as “M010”, which happens to be the code for Central Park – see www.nycgovparks.org/parks/M010).  No name or other information is provided except for an undefined “category code”.

Even the park ID is hard to identify – there’s a “GIS_Propnu” field and a “OMP_PropID” field that each contain values in the “M010” format. For the most part the values in both fields are identical.  But there are 43 records where these fields don’t match — see below.  I have no idea why they don’t match, but it turns out that the “OMP_PropID” values work in the DPR URL scheme, but the GIS_Propnu values don’t. So I went with the OMP data.

And the values in the “cat_code” field (1, 2, and 4) are not explained.  I’ve even talked with DPR employees who use GIS data, and they weren’t familiar with the details of the category codes.

Then I looked at the “raw” data.  I assumed the raw data would include a file to link park IDs with park names.  Nope.  The DPR “raw” data includes lists of many different types of park features (directories of barbecuing areas, beaches, dog runs, nature centers, playgrounds, etc).  But there’s no overall list of actual parks.  And of the 21 “raw” data files related to park properties from DPR, none of them provides park IDs.  They all include names and other attributes (generalized address info, website URLs, etc), but no IDs.

The closest we get from DPR’s “raw” data is a list of “capital projects” which includes names and park IDs, but they look like this:

So you need to do some text parsing to extract just the park IDs.  And this wouldn’t even provide a complete list.  The capital projects file includes 580 unique park names – far short of the 1,956 features included in the park geography file – as well as more than 330 entries with blank names.

After exhausting the options with the DPR data, I turned to the DoITT data.  Aha, the “OPEN SPACE” geographic data has park names and park IDs!  (It’s described as a “Planimetric basemap polygon layer containing open space features, such as parks, courts, tracks, cemetery outlines, etc.”)

But looking a bit closer, here’s why the DoITT file isn’t very helpful:

  • According to its metadata, it hasn’t been updated since 2006 – no better than the data that we already had on OASIS.
  • This file includes 1,600 unique names, but this includes areas not covered by DPR such as cemeteries, so therefore it’s not a complete list that will match the “DPR_Parks_001” geographic file.
  • Also, the naming conventions don’t really follow conventions.  For example, “Greenstreets” is spelled 8 different ways, there’s a mix of abbreviations and annotation (qualifiers in parentheses, etc)., extra (leading) spaces, misspellings, inconsistent spellings, etc.

  • There are more than 2,700 unique park numbers, but this includes park IDs that are blank (304 times) or IDs that don’t match the DPR list (for example, 158 records have “unset” listed in the Park ID field).

So as far as I can tell, it’s impossible to use the Data Mine data alone to link park names to map geography.  Maybe someone could match the data by hand (creating a map that labels the park areas by ID, and then comparing that with a DPR map with park name labels, and then manually entering those park names in DPR’s GIS file of park IDs).  But this would be so prone to error it wouldn’t be worth the trouble, and it also undermines the idea of providing “machine readable data” from Data Mine in order to automate how we access and analyze the information.

Since I didn’t find what I needed on Data Mine, I reached out directly to DPR for a file that links park IDs and park names.  The response was: a) wait for Data Mine to be updated; or b) if I can’t wait, then DPR needs to check with its public relations office before giving me the file.  Well, so much for openness and transparency.  Sigh.

How did BigApps developers handle this?

This made me wonder how the BigApps competitors could have created their applications, several of whom submitted apps that displayed maps of park locations showing park names.  I asked a couple of them how they did it.  One of them didn’t answer me directly, but instead suggested that I could “hire some free student labor to go through by hand for two days” to link the IDs/names manually.  Not very helpful.  Another BigApps project used the DoITT list of IDs/park names.  But for the reasons discussed above, that’s inadequate for our purposes.

On OASIS, a current list of park IDs and names is essential.  First, we want to display the latest and most accurate information for our visitors. Second, we not only display the park names on the map, but we use the park IDs to create a park-specific URL that sends an OASIS visitor to the DPR website to access the wealth of info DPR maintains about each park.

The workaround

I figured the Parks Department must have better data itself, so I looked online to see what I could find.  In the “Explore Your Park” section of DPR’s website, they have lists of parks by borough.  Each park is displayed by name as a link, and the underlying URL includes the park ID (for example, the URL for Claremont Park in the Bronx is http://www.nycgovparks.org/parks/X008).

Here’s the root URL for the parks lists: http://www.nycgovparks.org/sub_your_park/park_list/full_park_list.html?boro=X (just change the last letter for each borough – X is the Bronx, B is Brooklyn, M is Manhattan, Q is Queens, and R is Staten Island).

So I scraped these pages and stripped out the extra HTML code, leaving just the park names and IDs in order to create my own crosswalk table.  Then I joined the names with the DPR geography file using the IDs.  Not pretty, but more comprehensive, accurate, and up-to-date than relying on the problematic Data Mine data.

The DPR website with park names and IDs also includes playgrounds.  So I also joined the scraped list to the Data Mine layer for playgrounds (“DPR_playgrounds_001”).

I added the following fields to each file based on the DPR website data: PARKID, PARKNAME, NameMain, NameSuffix (some names had text in parentheses that I separated out to this field), and Borough.  The other fields were in the original shapefile from Data Mine.

Here are the GIS files (ESRI shapefiles) for parks and playgrounds; use them as you wish:

Data Mine improvements?

After I did my screenscraping work, I found a couple of tools that were created to streamline access to Data Mine files.  One developer created a service that converts files in Excel (XLS) format or some other format not easily “consumable” by applications or web services.  His tool is called elev.at — more info here and here.  But even this effort to fix one of Data Mine’s problems wouldn’t have helped with the parks data — converting from Excel to XML would’ve improved the format but not the data quality itself.

The Data Mine files may be good enough for someone throwing together a quick mobile app to enter a competition.  But the city’s data – and apps created with the city’s data – should be better than that.  We should expect that city data is reliable, current, and easily accessible.  My experience with parks data from Data Mine reminds me that the city still has work to do to meet this goal.

Presumably the Parks Department itself has a better system.  But this obviously didn’t make it into Data Mine.  Hopefully this will be fixed the next time Data Mine is updated (if that ever happens – more on that in my earlier post).

OASISnyc.net map updates

Today our team at the CUNY Graduate Center updated the www.OASISnyc.net mapping site with lots of new data.  There’s more to come by summer’s end, but here’s the latest:

  • The biggest change is that we’ve added the latest community garden inventory from GrowNYC.  Over a year in the making, it comes just in time to provide valuable context for the proposed new rules for gardens being considered by the city.  More info at the OASIS wiki.
  • We’ve integrated the latest subway and bus data that I’ve blogged about earlier (here and here), and also added bike routes via the NYC Dept of Transportation (one of the latest city agencies embracing #opendata).

(Lots of bike routes in Brooklyn.)

  • A side note to the subway routes is that we’ve responded to user requests and added the ability to turn on/off the subway routes separately from the streets and rail lines.  (May not seem like a lot, but hopefully this will make it easier to view an already busy map.)

  • We also have gotten around to updating New York City parks and playgrounds data based on the Datamine release from last year.  I’ll blog more about this separately, since it was a much more involved process than it should have been, given what I now realize are some major limitations to the city’s effort via Datamine and BigApps to provide open access to data.
  • The latest zoning districts, scenic districts, and historic houses are also added to OASIS.
  • The search options for community gardens, Stew-Map turfs, and the NY/NJ Harbor’s Comprehensive Restoration Plan data have been updated.
  • We’re starting to use the OASIS wiki again for tutorials and examples of how OASIS can benefit local groups. For examples, see http://oasisnyc.gc.cuny.edu/index.php/Featured_Maps
  • Finally, we’ve started to incorporate historic maps of Manhattan in partnership with the NY Public Library.  I blogged about this earlier, and we have a post on our wiki with more info.

We’ll have more updates soon.  Historic land use parcel by parcel citywide (with an accompanying statistical analysis), more data (such as parcels & open space) for northern New Jersey, more Google Earth/KML links, etc.

Stay tuned!

@NYPLMaps & OASIS provide context for 18th century ship find

The www.OASISnyc.net mapping team has been working with the great folks at New York Public Library’s Map Division to integrate digitized historic maps aligned to the city’s current street grid.  But as we were working with Map Division staff to incorporate their maps, an amazing find at the World Trade Center construction site prompted us to speed up our work — earlier this month, construction workers unearthed an 18th century ship, largely intact, that likely hadn’t been disturbed for over 200 years.

Now you can display some key maps of lower Manhattan from the from the 18th and 19th centuries, view them in relation to the current street grid, and compare them to each other using OASIS’s dynamic transparency tool.  We added a brief tutorial at the OASIS wiki.

Now you can fade between current property maps …

… the 1775 Montresor map …

… the 1817 Poppleton map …

… and more.

We’ve also added the Viele map from 1874, and more are on their way.  This is all due to the groundbreaking NYPL “Map Rectifier” project.