• SR_spatial tweets

Coastal storm impact risk mapped on Long Island

UPDATED (Sunday, Oct. 28, 2012 11am)

In anticipation of Hurricane Sandy, you can use the Long Island Index mapping site to view the areas that would be at greatest risk from the storm.

You can also use Nassau County‘s or Suffolk County‘s websites for more information.

ORIGINAL POST (August 2011)

The latest tracking information for Hurricane Irene (as of Friday morning, 8/26) shows that the storm is likely going to pass east of New York City and make a head on collision with Long Island.  Newsday is reporting that it will hit western Suffolk County’s south shore on Saturday with “tropical-storm-force winds” and then ramp up to 110 mph winds by Sunday.  Yikes!

To help prepare for the storm, our team at the CUNY Graduate Center in collaboration with the Long Island Index has updated the Index’s mapping website with areas at greatest risk of hurricane-level storms.

Here’s the map:

This news release [PDF] provides more information.  The yellow-to-red shaded areas are “coastal storm impact zones.”  In 2005, the New York State Office of Emergency Management developed a map of areas that would be at greatest risk of hurricane impacts based on wind speed and other factors.

The shorthand for the map is a “SLOSH” map, because the zones of impact are based on NOAA’s “Sea, Lake and Overland Surge from Hurricanes” (SLOSH) model projections of vertical surge heights associated with category 1 – 4 storms.  In order to map this information at the Index site, we downloaded the SLOSH data from the NYS GIS Clearinghouse website (metadata here) and used the following color shading to represent the zones:

  • A Category 1 storm (impact areas shown on the map in yellow) means winds of 74-95 mph.
  • A Category 2 storm (impact areas shown on the map in light orange) means winds of 96-110 mph.
  • A Category 3 storm (impact areas shown on the map in dark orange) means winds of 111-130 mph.
  • A Category 4 storm (impact areas shown on the map in red) means winds of 131 mph or more.
If you’re in an area highlighted on the map, be sure to contact your local officials and follow media reports about the hurricane’s progress. Nassau County’s Office of Emergency Management has posted a map of evacuation routes if it comes to that:
Hopefully Irene passes us by, but if not: be prepared.  Better safe than sorry.

Coastal storm impact risk mapped in NYC

UPDATED (Monday, Oct. 29, 2012 1pm)

If you need to locate any of NYC’s 76 hurricane evacuation shelters, you can use the OASIS mapping site.

You can also use NYC.gov to find out the latest with Hurricane Sandy.

UPDATED (8/25/11 9am): We’ve added a temporary map layer on OASIS showing the locations of NYC’s hurricane evacuation centers. Here’s the link: http://bit.ly/oBsUY8 . It’s easy to use:

  • Hover your mouse over each one to highlight it (the site details will also be highlighted in the panel on the right).
  • Click on a map marker to bring the site details up to the top of the list.
  • Double-click on a site in the list and the map will zoom in right to that location.

You can type in your address above the map to see if you’re in an area that’s at risk of storm impacts, and how close you are to an evacuation center. The OASIS map automatically also shows any nearby subway stations. And you can add any other layers from the Legend list to the right of the map.

For more information about Hurricane Irene and what you should do to prepare, visit NYC’s website.

UPDATED (8/24/11 12 noon): We’ve expanded the map layer showing hurricane impact zones throughout the downstate region. It now shows potential impact zones on Long Island’s south shore, as well as some areas along the coast of the Long Island Sound.


For the past several years the www.OASISnyc.net mapping site has displayed a map of “coastal storm impact zones” in New York City (in addition to the wealth of other mapped data included with OASIS). This coming weekend, it seems like the coastal storm map may be especially useful with city officials bracing for the risk of Hurricane Irene bringing 72 mph winds or more to the NYC region by Sunday.

In 2005, the New York State Office of Emergency Management developed a map of areas that would be at greatest risk of hurricane impacts based on wind speed and other factors. The shorthand for the map is a “SLOSH” map, because the zones of impact are based on NOAA’s “Sea, Lake and Overland Surge from Hurricanes” (SLOSH) model projections of vertical surge heights associated with category 1 – 4 storms.

In 2006, we downloaded the SLOSH data from the NYS GIS Clearinghouse website (metadata here), and used the following color shading to represent the zones:

  • A Category 1 storm (impact areas shown on the map in yellow) means winds of 74-95 mph.
  • A Category 2 storm (impact areas shown on the map in light orange) means winds of 96-110 mph.
  • A Category 3 storm (impact areas shown on the map in dark orange) means winds of 111-130 mph.
  • A Category 4 storm (impact areas shown on the map in red) means winds of 131 mph or more.

Here’s the map:

Through the OASIS website, you can easily enter your address and find out if you’re in an area that might be at greatest risk of the hurricane, depending on how severe the winds are by the time Irene moves up the coast. But the real power of OASIS’s maps is that you can do much more. For example, you can:

  • customize the OASIS map to show transit routes, schools, public housing, and libraries in or near the zones;
  • use OASIS’s mapping tools to see areas near the zones that have been recently developed (using the 1996-2010 aerial image timeline tool);
  • zoom in to see individual property boundaries and click on each one to determine ownership, zoning, and land use characteristics; and
  • find out which elected officials represent the area, as well as the local Community Board (good sources for planning and safety resources).

In the images below, I’ve used the aerial timeline slider and the dynamic transparency tool to show recent housing development in an area at risk of impact from a Category 1 storm, along South Beach on Staten Island (at the OASIS website, you can click and drag the name of any map layer in the Legend to the “Transparency Control” box at the bottom of the legend and make it more or less transparent, so the layers underneath can shine through):

Sparse housing near the beach (1996)

Dense housing development (2010)

The zones mapped on OASIS closely mirror the city’s hurricane evacuation zones (mapped here [PDF] and described here). The city also provides a Hurricane Evacuation Zone Finder where you can enter an address and get useful safety tips depending on your zone.

The message: hopefully Irene passes us by, but if not: be prepared. Better safe than sorry.

Innovative map comparisons – Census change in 15 cities

Our team at the Center for Urban Research (at the CUNY Graduate Center) has updated our interactive maps showing race/ethnicity patterns from 2000 and 2010 in major cities across the US. We’ve enhanced the maps in several ways:

  1. Added more cities. We now have 15 major urban regions mapped across the US (Atlanta, Baltimore, Boston, Charlotte, Chicago, Detroit, Houston, Los Angeles, Miami, New York, Orlando, Philadelphia, Phoenix, San Francisco, and Washington D.C.).
  2. The maps now have three ways of comparing 2000 and 2010 racial patterns:
  3. We color-coded the population change data in the popup window. Population increase is shown in green; decrease is shown in red. See image below.

Here’s our news release with more info.

Btw, we’ve also updated our static maps to show New York City Council districts, to begin to get a sense of how demographic changes will shape upcoming redistricting efforts at the local level.  Here’s the link:www.urbanresearchmaps.org/plurality/nyccouncil.htm (For the static maps, you can view 2000-2010 demographic change with the vertical slider bar, but you can’t zoom in/out, etc.)

An initial version of the maps launched in June with the vertical bar technique, integrating it with interactive, online maps for the first time. Our Center crafted the maps so you could not only drag the bar left and right but also zoom in and out, click on the map to obtain detailed block-level population counts, and change the underlying basemap from a street view to an aerial image (via OpenLayers use of Microsoft’s Bing maps tiles), while also changing the transparency of the thematic Census patterns.

The latest iteration of CUNY’s Census maps continues to use the vertical slider but now incorporates this technique with two more comparison options. Each approach serves different purposes:

  1. The vertical slider bar provides a “before (2000) and after (2010)” visualization of change, either regionally or at the scale of a city neighborhood.
  2. The side-by-side comparison is ideal for lingering over a given area, especially at the local level, taking the time to absorb the differences in demographic patterns mapped with 2000 Census data on the left and 2010 on the right. We incorporated this approach specifically at the suggestion of the great interactive team at the Chicago Tribune, who have created some similar Census maps.
  3. The single-map 2010/2000 overlay is especially helpful for revealing the increase in diversity over a given area.

For example, you can zoom to Atlanta, GA on the single-map overlay and see the city’s predominantly Black population in 2000 surrounded by suburban Census blocks shaded dark blue, denoting a White population of 90% or more (see images below). As you transition the map from 2000 to 2010, the dark blue in the suburbs fades to a lighter shade (indicating a more mixed population demographically) coupled with more Census blocks shaded green, purple, and orange – each corresponding to communities that are now predominantly (even if only by a few percentage points) Hispanic, Asian, or Black respectively. This pattern is replicated in many of the urban regions featured at the website.

Atlanta & suburbs in 2000

Race/ethnicity change in Atlanta by 2010

Eventually we’ll be moving all this from pre-rendered tiles to vector tiles. CUR’s application architect Dave Burgoon contributed code he developed to TileStache to enable TileStache to produce AMF-based output for use in Flash-based interactive mapping applications. This will give us flexibility in mapping as many Census variables as needed, and also providing complete geographic coverage (hopefully down to the block level) nationwide. That’s the plan, anyway! Stay tuned.


Funding for much of the Center’s recent work on Census issues has been provided by the Building Resilient Regions Project of the John D. and Catherine T. MacArthur Foundation, the Hagedorn Foundation, as well as support from the CUNY Graduate Center and the City University of New York.

Several people provided feedback and helpful editorial suggestions on earlier versions of the maps and narrative. Though the materials at this site were prepared by the Center for Urban Research, those invdividuals improved our work. We greatly appreciate their contributions.

Mapping the Cityscape exhibit

This summer the Center for Architecture in New York is all about maps.  One of the main exhibits at the Center, “Mapping the Cityscape”, features a dozen or so wall-mounted 8-foot-high maps of Manhattan — different representations and views from 1609 to the present.  Several panel discussions are accompanying the exhibit, including two this week (“Mapping Manhattan” and “Mapping Risk“).  Originally scheduled for the month of July, by popular demand the exhibit will be open through the end of August (Aug. 27th).

The exhibit came out of a panel discussion in May at the Center, organized by Abby Suckle of CultureNOW, with a broad group of participants: Matt Knutzen of the NY Public Library Map Division, John Tauranac of subway mapping fame, Laura Kurgan of Columbia University, and me discussing the OASISnyc mapping site — along with respondents from Google and the Wall Street Journal.  The panel covered lots of ground, from the Library’s invaluable collection of historical maps (now being digitized and geo-referenced) to the evolution of transit mapping in New York to the OASISnyc online mapping site (parks, open space, and much much more) to Columbia’s “Million Dollar Blocks” project to CultureNOW’s mobile maps to the latest from Google and others.  The hall was packed, the audience had lots of questions, and apparently they wanted more — hence the decision by the Center to transform the panel discussion into a summer-long exhibit.

Here are some photos of what you’ll see when you visit:

OASIS maps – each “slice” of the two maps above highlights the different types of mapped data you can display and analyze at the OASIS website.

Million Dollar Blocks – Columbia University’s Spatial Design Lab project that mapped the impact of prison policies on local neighborhoods in New York.

Mannahatta – the Wildlife Conservation Society’s take on what Manhattan likely looked like in 1609; an amazing project, rich with insights, analysis, and visual power.

NY Public Library – historical maps brought to life.

Maps of ecological patterns, historical maps overlain with current geography, transit mapping …

… and land use patterns, cultural icons, and more.

The exhibit opening in early July was packed — Center director Rick Bell & exhibit curator Abby Suckle talk to the crowd about the event:

If you’re interested in how maps — and the very definition of mapping and understanding/visualizing spatial relationships — are changing through the latest interactive technologies,  then this exhibit is for you.  Or if you’re interested in the history of visualizing New York City through maps, then this exhibit is for you.  Either way, please stop by, check out the maps, and attend one of the public programs.  It’ll be a cartographically illuminating experience.

(Exhibit logo from Center for Architecture website.  All photos: Steven Romalewski)

Slippy maps, meet before-and-after jQuery slider (introductions by OpenLayers)

Our team at the Center for Urban Research (at the CUNY Graduate Center) has launched a set of maps showing race/ethnicity patterns from 2000 and 2010 in major cities across the US.  The maps combine several mapping/web technologies that offer a new way of visualizing population change.  This post explains how we did it.

(And by popular demand, we’ve also included a map of Congressman Anthony Weiner’s district in relation to demographic change — you may have heard of him and his Twitter travails recently?)

Race/Ethnicity Change

Briefly, the maps show race/ethnicity change from 2000 to 2010 at the local level throughout major urban regions across the U.S.  So far we include New York City, Los Angeles, Boston, Chicago, Houston, and San Francisco.  (Others are coming soon.)

For our methodology and data analysis (and static maps), we provide that here.  For the mapping and web techniques, see below.


So far we’ve received a pretty good response to our maps.  Here are some tweets posted recently:

  • @dancow (web journalist for ProPublica): Cool before/after map from CUNY’s urban research center showing NYC ethnic changes at the block level, from 2000-10.
  • @mericson (deputy graphics editor at NY Times): Nice block-level maps by @SR_spatial & CUNY Urban Research Center showing racial/ethnic change in NYC from 2000 to 2010.
  • @kelsosCorner (former Washington Post cartographer): Digging new 2010 Census plurality maps of NYC.
  • @albertsun (graphics editor at Wall St Journal): Coolest census map I’ve seen yet.
  • @PJoice (HUD employee; tweets are his own): This is the coolest map I have ever seen. Nice work by @SR_spatial and CUNY!
  • @MapLarge: I like how you can use the slider or move the map! Great Visualization!

Technical overview

The map uses the “before and after” technique that media websites have used for images of natural disasters.  We enhanced this technique by integrating it with interactive maps using OpenLayers, the open source mapping framework.  Now the slider works with two sets of overlapping, but perfectly aligned, maps from 2000 and 2010.

As it turns out, we didn’t set out to create an interactive version of these maps. In fact, we originally created static maps, but everyone we showed them to for feedback wanted the ability to zoom in/out and click on the map for more info.  So we developed the OpenLayers version. (And when I say “we”, that mainly means David Burgoon, CUR’s application architect, who I can’t say enough good things about.  I made the maps, and CUR’s Joe Pereira of the CUNY Data Service created the data sets, but Dave brought it all to life.)

OpenLayers enables us to introduce interactivity into the before-and-after images. Maps like these (to our knowledge) have not been available before — where you can move a slider back and forth while also zooming in/out and clicking on individual Census blocks for detailed information. You can also change the transparency of the thematic map layer, and switch between a street view and aerial view basemap.

It involved a good amount of work to integrate the slider technique with OpenLayers and also have two overlapping map instances working in tandem. The two maps need to appear as one, and this involves painstaking effort to ensure that the pixels on your screen are translated accurately to latitude/longitude coordinates in each of the separate but related interactive map instances, and the maps pan together seamlessly as you drag the slider left or right or move the map and it crosses the slider.


In order to create the application, we used a mix of software applications, technologies, and techniques, summarized below:

  • We used the statistical software package SPSS to extract the Census block-level data for both years (see our methodology), allocate the 2000 data to 2010 blocks using the Census Bureau’s block equivalency files, and calculate the race/ethnicity plurality for each block.
  • We exported these SPSS files in DBF format and used ESRI’s ArcGIS Desktop to join the DBFs with 2010 TIGER Census block shapefiles.
  • ArcGIS Desktop was also used to create the choropleth maps (based on color schemes from ColorBrewer.org);
  • The map layouts were published as temporary web map services using ESRI’s ArcGIS Server. We used these to create pre-cached tiles (.PNG files) for the 2000 and 2010 maps, corresponding to zoom levels 4 through 10 using the now-standard Google-Microsoft map scales for online web maps. (Our application accesses the choropleth tiles as PNGs directly from the cache created by ArcGIS Server, rather than accessing the ArcGIS web map service in order to assemble the tiles. The latter approach would be too slow and would undermine the transition as you dragged the slider across the map.)
  • The slider technique was adapted from the jQuery plugin by www.catchmyfame.com.
  • OpenLayers provides all the map navigation and serving the maps themselves, modified with customized JavaScript code.
  • The basemap shown beneath the color-shaded map tiles is provided by Microsoft’s Bing map service. The street map and aerial image tiles from Bing are accessed directly via OpenLayers, rather than using the Bing API. This is a key reason we used Bing for these maps; if we used Google Maps as a basemap, we were limited to accessing Google Maps via Google’s API, which would have slowed map drawing times and undermined the slider effect.
  • For geocoding we use the Yahoo! Placefinder API.
  • Some browsers are not able to handle the before/after slider effect smoothly. In particular, Firefox and Safari perform poorly; the slider transition between one map to the other is not smooth. Microsoft’s Internet Explorer is adequate, but Google’s Chrome browser is best.

Data sources/issues

We used block-level data from the Census Bureau’s 100% population counts from the 2000 and 2010 decennial censuses (from Table P2 in the “PL-94-171” files for 2000 and 2010).

The Census Bureau’s block geography changed between 2000 and 2010 — new blocks were created, blocks were merged, and block boundaries were modified in many places. In order to compare population data from 2000 and 2010 using a common set of blocks, we used the Census Bureau’s block relationship file to allocate the 2000 population counts to 2010 geography.

When you’re viewing the map, it is best to use the maps and block-level data to understand trends over a larger area, even over several blocks. Be careful when viewing a specific block on its own. It covers a small area, and the Census Bureau may have made errors.


Funding for much of the Center’s recent work on Census issues has been provided by the Building Resilient Regions Project of the John D. and Catherine T. MacArthur Foundation, the Hagedorn Foundation, as well as support from the CUNY Graduate Center and the City University of New York.

Several people provided feedback and helpful editorial suggestions on earlier versions of the maps and narrative. Though the materials at this site were prepared by the Center for Urban Research, those invdividuals improved our work. We greatly appreciate their contributions.

@NYPLMaps & OASIS provide context for 18th century ship find

The www.OASISnyc.net mapping team has been working with the great folks at New York Public Library’s Map Division to integrate digitized historic maps aligned to the city’s current street grid.  But as we were working with Map Division staff to incorporate their maps, an amazing find at the World Trade Center construction site prompted us to speed up our work — earlier this month, construction workers unearthed an 18th century ship, largely intact, that likely hadn’t been disturbed for over 200 years.

Now you can display some key maps of lower Manhattan from the from the 18th and 19th centuries, view them in relation to the current street grid, and compare them to each other using OASIS’s dynamic transparency tool.  We added a brief tutorial at the OASIS wiki.

Now you can fade between current property maps …

… the 1775 Montresor map …

… the 1817 Poppleton map …

… and more.

We’ve also added the Viele map from 1874, and more are on their way.  This is all due to the groundbreaking NYPL “Map Rectifier” project.

MTA subway data in GIS format

UPDATE Oct. 5, 2015: See this post for updates to the GIS files described below, including the new Hudson Yards subway station and 7 line extension.

As promised, I’ve created an updated GIS data set of subway routes in New York based on MTA’s latest GTFS data, which I’ve posted below for anyone to use.  (I’ve also converted the latest MTA GTFS data to GIS format for NYC Transit bus routes, following up on my earlier post this spring.)  Scroll to the end of this post for the links.

I hope my effort provides a template for creating a map layer for apps and/or printed maps that approximates the line symbology on MTA’s map, but improves on this map in several ways — the GIS version is spatially precise, scalable, and may even look better than what Google uses for its transit layer in New York City.  The images below show the map zoomed out and zoomed in, and the post below explains how I did it.

Going forward, hopefully the MTA itself will provide subway route GIS data in the format I’ve described below (or something similar) alongside the GTFS data.  Any feedback or suggestions for improvement will be much appreciated.


After reviewing the GTFS data files in detail, it became clear that GTFS is not necessarily applicable for displaying transit lines on a map and/or analyzing the spatial patterns with GIS.  At least that seems a fair characterization regarding the GTFS version of the city’s subway system.

Even though the GTFS files include a “shapes.txt” file for subways (the spec says this is for “drawing lines on a map to represent a transit organization’s routes“), this is only helpful for basic line representations.  This seems to work fine for bus routes.  But for subways, either the GTFS structure or MTA’s implementation of it poses challenges for creating a map layer of subway lines.

New York’s subway system includes local and express routes that are composed of inbound and outbound trips along the same line, some of which may start or end at different stations.  And there are “skip-stop” trains and “trunk” lines (where multiple routes run on the same set of tracks, such as the E, F, M, and R in Queens from Jackson Heights to Forest Hills).  But the GTFS “shapes” data from MTA only provide a partial representation of this complexity.

To attach route IDs to the “shapes” file, shapes.txt needs to be linked with trips.txt based on shape_id.  But doing so causes the 1, G, and latest version of the M line to drop out, because the trips.txt file does not include any trip entries for these routes.  Also, for some reason, it results in the inclusion of lines that aren’t used anymore (at least for passenger trips, as far as I know).

This is just an early iteration of MTA publishing its GTFS data, so I’m not surprised these limitations exist.  Until these issues are fixed, we have to rely on workarounds.  For example, the MTA has provided a separate shapefile for the 1 and the G (see earlier discussions at the MTA Developer Resources listserv).  According to MTA,

We [MTA] do not have shape data for these lines because of changes in their station configurations have occurred since we lost the staff member who had created the data. We have not had funding to replace him and update the data from 2008. We can provide the data next week with a hand-done solution, and/or better data at some later time, when we are able to acquire the staffing to do so.

This undermines the idea of using GTFS as a “feed” (as its name implies) for automatically displaying subway lines on a map, but hopefully the process will be more seamless as the issues are worked out.

But the and lines are not really missing from the GTFS data.  The GTFS “shapes” file on its own (without filtering it based on the “trips” file) includes line segments for virtually the entire subway system.  It’s just a question of being creative with combining the trip_id and shape_id fields from the trips.txt file to extract the appropriate geometry for the routes in question.  For example, the correct G shape is certainly there; it’s denoted by the “G..N05R” or “G..S05R” shape IDs.  It just so happens that there are no records in the trips.txt file with these shape_id values.  But it’s easy enough to create a new “route” field in the shapes file and populate it with a combination of values from the “routes.txt” file and manual entries for the lines that don’t seem to exist (such as the 1 and G).

Other issues with the shapes.txt file after filtering with the trips.txt file are that the line terminates at 57th Street/7th Avenue and the GTFS data includes shapes that show the running to Jamaica-179th St, the running to New Lots Ave on the 3/4 line, the extending to New Lots, and the running to New Lots.  As far as I can tell, these routes do not exist for the riding public.  Perhaps these are artifacts of older routing schemes, but it makes for an inadequate solution for mapping.  I’m curious how the automated routing and scheduling apps deal with this.


UPDATE 7/20/10

After I wrote the preceding paragraph, I did quite a bit more digging into the subway GTFS data for a post at my blog about subway stations.  I realized that my points above about the and lines were wrong.  There are, in fact, stops that these trains make at the stations I mention above.  There aren’t many of them, but they exist. My post about station data explains this more fully.  So for these routes at least, the shapes.txt file is ok.


A bigger problem, though, is that the latest version of the line is missing from the shapes data, and can’t be created from the approach described above for the 1 and G.  The images below highlight the challenge – the area in question is circled in blue on both images.  In the old map, the M runs along what is now the J-Z line, and the orange F-V and B-D lines cross the J-M-Z line.  Unfortunately this old routing is what the latest GTFS geometry follows.

But with the latest service changes, the new M line comes in from Brooklyn and then goes north to meet the B-D line. In the latest shapes.txt file, there is no such geometry for the M.  The geometry follows the old J-M-Z line with no obvious shape that follows the M’s new northward jog to meet the B-D line.

Therefore, I created a new line segment for the M, combining segments from the old M and V lines (shape_id values of “M..N89R” and “V..N01R”), along with an arc connecting the two, using ESRI’s ArcGIS editing tools.


Once I had the updated set of “shapes” from GTFS, my goal was to somehow convert this data into a GIS version of the MTA’s subway lines in a way that could be replicated (and perhaps integrated back into GTFS format) and also easily symbolized to show separate lines along trunk routes.

As far as I know, to the extent anyone had a GIS dataset of subway routes prior to GTFS (such as this one we had created for the OASIS website by digitizing the MTA’s subway map), the only way to display separate trunk lines was to manually edit the geometry of the GIS line segments along a trunk route by clipping the line and moving it parallel to the trunk line, so it would show up as a distinct line symbol.  Obviously this has problems — the manual work involved is tedious, imprecise, hard to replicate, and it doesn’t scale well — it might work at a certain zoom level, but then zooming in would show the parallel lines farther apart and zooming out would show them merged together — as illustrated by the images below from the NYC Citymap website, going from a wide zoom to a closer zoom:

(The Citymap site is just one example; you can see a similar situation on OASISnyc.net — as you zoom in on the map at this link, you’ll see the and lines become farther apart.)

Divisions and Lines

I remembered that NYC Transit uses “division” and “line” designations that might be helpful in distinguishing the segments. The divisions are a throwback to when the subway system was really three separate systems — the IRT, IND, and BMT. But the line designations are based on more or less current track arrangements (and you can see some of these on the current subway map – see excerpt below).

For example, the movie “The Taking of Pelham One Two Three” refers to the train running on the Pelham Line, leaving the Pelham Bay Park station at 1:23.  Wikipedia has lots of information about the line designations, such as the IND 6th Avenue line or the BMT Nassau Street line.

But how to assign these to the shapes.txt file? The line IDs/names are not included as part of GTFS, and I’ve not seen this information provided anywhere else (publicly anyway).

Station entrance/exit data provides the missing link

Then on July 1 the MTA released a file listing subway entrances and exits with latitude/longitude for each one (the file was updated July 7 to fix some issues in the earlier data). Useful in its own right, the file includes the station name for each entrance/exit along with its division and line. Neat! The entrance/exit points don’t necessarily overlap or intersect the line shapes, so I wouldn’t be able to automatically assign the divisions and lines to the shapes using GIS, but there are only 37 unique lines based on the entrance/exit data so it wouldn’t be that hard or time-consuming to do it manually.

My approach was to create a thematic map of the entrances color-coded by line designation, overlay the GTFS shapes file of subway routes, and then edit the shapes file by splitting the segments where each set of color-coded entrances ended and adding the corresponding line attribute to these new segments.  The image below illustrates the approach.

In other words, instead of a single shape representing the  line, I created six non-overlapping segments to represent the entire 2 train route (along the 7th Ave-Bway, Clark Street, Eastern Parkway, Lenox, Nostrand, and White Plains lines).  I used the ArcGIS “Split Tool” quite a bit, and ended up with a shapefile with 80 unique shapes (including the AirTrain — which is included in GTFS but isn’t managed by MTA so likely doesn’t have an MTA “line” designation).  The attributes from the new file look like this:

This was a manual process based on visual inspection of the line segments, so I’m sure error has crept in.  Also, the way I did it, I allowed for some exceptions.  I didn’t rigorously create new segments, for example, along what appears to be a trunk line in Manhattan where the IND 6th Avenue and 8th Avenue lines meet at the West 4th Street station. And I probably didn’t handle lines travelling over bridges or through tunnels as well as I could have.  And the 5 route along the IRT White Plains line extends from Nereid Avenue to 138th Street/Grand Concourse, but the #5 in the Bronx that just runs during rush hour goes from East 180th Street to Nereid Ave (so on my map the dashed line symbology extends too far south).

Overall, though, I think it works well — it’s pretty good for a first pass.

ArcGIS caused import hassles …

Btw, I should point out that though ArcGIS’s editing tools were great for splitting and re-combining the line segments, ArcGIS misinterpreted important fields when importing the GTFS text files.  Fields that were text (such as “route_id” in the trips.txt file) were imported as numeric, preventing an accurate join.  I needed to use another program (I used SPSS) to save the trips.txt file as a DBF which preserved the text format of the field.  (I had tried using Excel to convert from TXT to CSV and also to XLS, but that also forced the text field to convert to numeric.)

… but ArcGIS provided invaluable cartography tools

Next step was to create the symbology.  I relied on two ArcGIS features to display multiple lines along a trunk route as discrete line symbols: the “cartographic line symbol” feature, and “symbol levels”.  The cartographic line symbol component of ArcGIS’s Symbol Property Editor, among other things, enables you to attach an offset value to the line symbol.  See screen shot below.  The great thing here is that the offset is relative depending on the zoom level of the map — as you zoom in or out the line symbols do not merge together or move further apart, thereby solving the problem parallel copies of line segments.

The Cartographic Line Symbol tool also allows you to create a dashed line symbol, which I used for the Rockaway Park Shuttle and the rush hour extension of the 5 train in the Bronx.

The “symbol levels” feature enabled me to ensure that line segments weren’t inadvertently masked by others along the same geometry.  For example, simply offsetting the and routes from the and  routes running along the IND 8th Avenue line may result in two parallel orange lines, rather than a blue and an orange line.  Setting a priority symbol level ensures that the blue and orange lines will run in parallel.

The resulting trunk line symbology looks good whether you’re zoomed out …

… or zoomed in close:

Soon we’ll have the updated lines and symbology on the OASIS mapping site.

In order to recreate the map symbology, I’ve preserved the color scheme in an ESRI layer (.lyr) file, linked below along with the actual shapefile.  If you’re using a GIS that doesn’t use layer files you’ll need to redo the symbology, but at least you can use the attributes to do so.

(The layer file includes subway route labels that were inspired by an approach provided by ESRI’s New York City office that we first applied for the OASIS site — using the subway route icons a la the MTA subway map. I’ve streamlined it a bit here, and it’s easy to modify further either with ArcGIS or another GIS package. I’ve included a basic MXD file that preserves the labeling.  The MXD uses ESRI’s Maplex labeling engine, but the labels will work with ESRI’s standard labeling engine as well.)

The one missing component to this data is a layer of transfers between subway stations.  I know this has been discussed on the MTA Developers Resource list, but this will have to wait till a more robust data set is available (or I or others have the time to put one together).

Links to the data

Here’s the GIS subway data in shapefile format (zipped):

If you use the data and layer file (which I hope you do), please let me know how it works out.  I’m not including any kind of Creative Commons licensing, but I’d appreciate it if you could reference the “CUNY Mapping Service at the Center for Urban Research” if you use the data and especially the layer symbology in any printed maps or online applications.  Thanks!

Also, here are the post-June 27 service change bus routes in shapefile format (zipped):


Better than Google Maps cartography?

I definitely wanted to compare my GIS version of the subway GTFS data with Google Maps, which presumably uses the GTFS data not only for transit directions but also for the basemap itself.   Two things surprised me.  One was that, as of today (July 7) almost two weeks after the MTA’s service changes took effect, Google Maps still shows old subway routes and station information.

The map below, for example, still displays the old  line (see the 23rd St/6th Avenue station) and the discontinued line (see the 23rd St and 28th St Broadway line stations).

Also, the subway lines on Google Maps were choppy and not as smooth as the GTFS-derived GIS lines. The images below compare the two in lower Manhattan.

I don’t think it’s nitpicking to point out the difference. One important aspect of the MTA’s GTFS data from a cartographic perspective is the high-quality route geometry.  It makes it that much more useful not only for good map development, but also for spatial analysis and alignment with other NYC GIS data layers.  Kudos to MTA for providing it.  I’m surprised Google apparently doesn’t use GTFS for their basemap (hopefully they’ll correct me if I’m wrong).

Going forward

For my purposes (and I think I’m far from alone here), I’m more interested in displaying the subway lines in a map layout than developing an application that provides routing and scheduling. Whether or not I use the data for spatial analysis, I’d like to have a subway layer for use in a GIS or any other application that needs the symbology of MTA’s printed map but is more spatially precise than MTA’s map and not as fine-grained as individual trips.

The GTFS format is great for all the web and mobile applications that are being developed.  But for the purposes of local planning work by Community Boards, students, the media, public officials, etc — we want to see the subway lines on a map and analyze them spatially — visualizing and understanding the relationships of nearby land usesdemographics, etc as well as the ability to monitor maintenance and operations trends, determine who represents each line when service changes are being proposed, and more.  So hopefully MTA will see fit to provide subway route data in a systematic way so we can integrate it easily into our maps.

It’s likely that NYC Transit maintains its subway line/route data in a similar structure as I’ve described above in GIS format, either for planning/modeling purposes or for other mapping needs.  Ideally it’s in a format that allows for an automated, rules-driven way of displaying the routes by division/line so changes are handled as seamlessly as possible.  In other words, it would be great if MTA could provide the subway data in a way that doesn’t require the additional staff resources that are involved in coverting the scheduling/routing data to GTFS format.  I’m not expecting anything as simple as “just hitting the export button,” but hopefully something close :).  And since subway routing doesn’t change very often (certainly not as frequently as schedules), this should be much less of a burden on the agency than the work involved in providing the GTFS data.

I look forward to continuing the dialog.

Thoughts about Google’s Styled Maps announcement

At the Google I/O developer event this week, an impressive item was included in the announcement about the latest Javascript Maps API (v3).  Now you can use Javascript to change the style of Google’s base map, in effect creating your own CSS for Google Maps.

Yesterday’s Google Geo Developers Blog highlighted the new feature and capabilities, including a link to a style wizard – see screen shot below – so you can see in real-time what each change looks like (the wizard is kind of like ColorBrewer, without the helpful suggestions for good vs. bad color schemes and symbology).  Note that the Style Wizard won’t work in IE.

Google Maps Style Wizard

This step for Google Maps is big, no doubt, but I’m ambivalent.  On the one hand, it opens up myriad possibilities for using Google’s map API.  Before, some geoweb developers (our team at CUNY included) tended not to use the Maps API because each and every map looked the same.  It got to be boring, and from a cartographic perspective it was basically an acknowledgment that you took the easy way out.  And often the Google map style got in the way of other layers you wanted to add to your map.

As Google’s own blog puts it,

No matter which [Google] Maps API site you are on, every map looks the same. If you want your map to stand out from the crowd, your options are limited to customizing the markers and controls, and if your brand has a particular colour scheme that is reflected on your site, Google Maps may not sit well with it.

EveryBlock’s Paul Smith wrote about these limitations two years ago at his post about “Google Maps Fatigue”.  For our part at CUNY, if we wanted a basemap with our own style that didn’t conflict with other thematic layers and choropleth overlays, either we rolled our own.  Or if we used Google Maps, we used the Flash API so we’d have the ability to desaturate its color (following the New York Times and others) — our Census 2010 Hard to Count mapping site used this technique. 

On the other hand, I’m sure more of us will now use Google Maps as a basemap, but will that further undermine the diversity of locally-crafted and customized online cartography?  With the advent of Google map styles, our team is now looking closely at GMaps for our map applications.  I think we’ll likely only use Google’s basemap for nationwide applications where it’d be too difficult to assemble all the base map layers at that scale.  But … maybe we’ll use it beyond that.

There are still major issues, of course.  Modifying Google’s map styles is one thing, but it’s still the Google basemap with the Google logo.  Using Google’s basemap whether we style it ourselves or not still means we’re stuck with the underlying data — with all its errors, inconsistencies, outdated place names, etc.  Google is trying to fix the errors, and errors are bound to exist, but until the basemap gets to a high level of quality and consistency, in effect you’re endorsing the errors when you use their basemap.  And though we can use the GMap style options to turn on/off place names, for example, this is nowhere near as robust as setting layer-specific filters that display or omit only *certain* place names, or using different road label types for different types of roads, etc.

As I understand it, CloudMade gives you much more control over map content as well as style.  You need to use OpenStreetMap, but perhaps the openness of OSM is more to your liking than Google Maps.  This post at Programmable Web highlights some of the differences between GMaps map styles and CloudMade’s editor features.

Some critics also point out that just making a map look different doesn’t necessarily mean it looks good, or better than the standard Google look-and-feel.  The examples provided via Google’s Geo Developer blog are intriguing …

… but this one via the Javascript Maps v3 documentation is, well, not so much:

Perhaps one cartographic benefit in all this is that it will help the mashup neogeographers of the world realize that there are good and bad map styles — cartography matters.  Web map developers will need to make choices about symbology, it won’t just be on a silver platter from Google.

What do you think?

GIS and Census participation

It’s been too long since my last blog post. Have been quite busy with work, and even though Twitter is a microblogging service, sending a tweet now and then really isn’t an excuse to keep up my actual blog.

One of the projects keeping me (very) busy is our work to help boost participation in the 2010 Census. I thought I’d write about some of our interactive mapping and participation rate analysis along these lines.

In August I described how our team at CUNY’s Center for Urban Research was creating metro-scale maps showing where hard-to-count communities were located so local census advocates could target their outreach. Then in late January we launched our interactive version of those maps at www.CensusHardToCountMaps.org. Originally we designed the site to show hard-to-count areas, but these are only where it was expected there’d be low census participation. Then, on March 23, the Census Bureau started publishing the actual participation rates on a local and national basis. So a week later we updated our site to emphasize the latest participation rates (this link zooms in to Manhattan showing tract-level participation overlaid on a map of hard-to-count tracts).

Though the Census Bureau’s Take 10 map (and a related Google Earth application) display the daily participation rates nationwide, we decided to provide several features that the Census Bureau doesn’t. At our site you can:

  • type in a county and highlight the tracts below a certain participation rate (you can enter whatever rate you want);
  • sort the resulting list so you can see at a glance the highest and lowest performing tracts (this also will be highlighted on the map so you can see how concentrated they are); and
  • compare the 2010 rate map with the 2000 rate map (click the “More…” tab and check the box for “Participation Rate in 2000″).

(Of course, you can also click on any spot on the map to display the latest participation rate for that area — state, county, or tract — depending on how close in or out you’ve zoomed.)

These are the types of data analysis and spatial visualization tools that were requested by census advocates, so they can use the maps to focus on areas that need their help the most.

In order to provide some context for the interactive map, our Center also posted an analysis of the first week’s participation rate. It was a combination of basic statistical analysis and mapping. We examined the correlation between participation rate and hard-to-count scores at the tract level nationwide, and not surprisingly found that rates tended to be lower in hard to count areas. This should help bolster the work of groups who’ve been working in these communities, confirming that they’re focused on areas that need support the most if we want to achieve a 100% count.

We also examined county-level statistics on race/ethnicity using the Census Bureau’s latest population estimates from 2008. (The American Community Survey would provide a richer set of characteristics to examine, but any data from areas with less than 20,000 population are suppressed in the ACS — and this accounts for about 1,300 of the nation’s 3,200 counties.)

The county-level data indicate that race/ethnicity is strongly correlated with census participation (at least in the first week), with participation rates tending to be higher in counties with a greater percentage of whites while the rates tended to be lower in counties with a greater percentage of blacks and Latinos. Because we didn’t have other socio-economic data to evaluate, we weren’t able to disentangle the effects of other characteristics such as low educational attainment, poverty, housing conditions, etc. that may have a stronger correlation while cutting across racial and ethnic categories. An opportunity for further research. As a next step we may also examine county-level unemployment rates and participation rates, as well as evaluating how well the first week’s analysis holds up as time goes on.

Friday (April 2) we added another feature, information about the areas that will receive a second census questionnaire.  (The Funders Census Initiative sent out a news advisory highlighting this service on April 5.)  Now when you click on the www.CensusHardToCountMaps.org map or type in your street address, you’ll see a popup window that (among other things) tells you if households in your tract will be receiving replacement Census forms. We think this will help minimize confusion over people receiving another census form (even if they’ve already mailed their’s in!). This is a “just in case” thing from the Bureau — mailing another form to households in historically low response areas, and mailing another form to households in moderately low response areas who haven’t yet sent their’s in. But the geographic scope of the “blanket” and “target” replacement mailing areas are pretty extensive in most cities (see maps at CUR’s website), so lots of people may be confused. Our mapping site provides a simple way of clearing the air.

We’ve also mapped those second mailing areas. When you visit www.CensusHardToCountMaps.org, select the “More…” tab and zoom in to your area of interest. For example, here’s Boston, MA. Click either or both check boxes in the “April 2010 Replacement Questionnaires” section to map the tracts receiving replacement census forms.

Our hard-to-count mapping site still has its original functionality — such as visualizing the demographic characteristics that will make it difficult to achieve a complete Census count; overlaying ZIP Codes, mail return rates from 2000, recent foreclosure risks by tract; and seeing who’s tweeting about the Census in your area.

But we’re also planning for next steps, thinking of the mapping application as a platform for future Census-related efforts (tracking how successful census advocates were, displaying the 2010 results, enabling the general public to get involved in a meaningful way in the redistricting process). Any ideas? We’d love to hear them.

“Neo”, “Paleo”, “Geo”, what?

Just a quick note that the online journal V1 Magazine today posted an interview with me about the CUNY Mapping Service’s online mapping work.  I’m honored that Matt Ball asked to talk with me about our work — he has interviewed some pretty impressive people over the years and it’s a thrill to have our humble contributions included in the list.

We mainly talked about our experiences developing some big online mapping applications — the Long Island Index interactive map, the new version of OASIS, and an upcoming nationwide application for the 2010 Census.

I’m glad one of Matt’s takeaways from the interview is to think of our work as cutting through the debate/discussion/tempest in a teapot that’s been taking place lately about whether you’re a “neo” or “paleo” or some other type of geographer.  I can’t say that we have any definitive answers or better answers than anyone else.  But if our work helps clarify things or maybe even point the discussion in a new direction, that’d be a contribution I’m glad to make.

Btw, my slide presentation from the 2009 GeoWeb conference provides some background on how we decided on different GIS technologies/techniques for the Long Island Index mapping site.  And an earlier blog post discusses our cartographic decision-making for the new OASIS website.