• SR_spatial tweets

Coastal storm impact risk mapped on Long Island

UPDATED (Sunday, Oct. 28, 2012 11am)

In anticipation of Hurricane Sandy, you can use the Long Island Index mapping site to view the areas that would be at greatest risk from the storm.

You can also use Nassau County‘s or Suffolk County‘s websites for more information.


ORIGINAL POST (August 2011)

The latest tracking information for Hurricane Irene (as of Friday morning, 8/26) shows that the storm is likely going to pass east of New York City and make a head on collision with Long Island.  Newsday is reporting that it will hit western Suffolk County’s south shore on Saturday with “tropical-storm-force winds” and then ramp up to 110 mph winds by Sunday.  Yikes!

To help prepare for the storm, our team at the CUNY Graduate Center in collaboration with the Long Island Index has updated the Index’s mapping website with areas at greatest risk of hurricane-level storms.

Here’s the map:

This news release [PDF] provides more information.  The yellow-to-red shaded areas are “coastal storm impact zones.”  In 2005, the New York State Office of Emergency Management developed a map of areas that would be at greatest risk of hurricane impacts based on wind speed and other factors.

The shorthand for the map is a “SLOSH” map, because the zones of impact are based on NOAA’s “Sea, Lake and Overland Surge from Hurricanes” (SLOSH) model projections of vertical surge heights associated with category 1 – 4 storms.  In order to map this information at the Index site, we downloaded the SLOSH data from the NYS GIS Clearinghouse website (metadata here) and used the following color shading to represent the zones:

  • A Category 1 storm (impact areas shown on the map in yellow) means winds of 74-95 mph.
  • A Category 2 storm (impact areas shown on the map in light orange) means winds of 96-110 mph.
  • A Category 3 storm (impact areas shown on the map in dark orange) means winds of 111-130 mph.
  • A Category 4 storm (impact areas shown on the map in red) means winds of 131 mph or more.
If you’re in an area highlighted on the map, be sure to contact your local officials and follow media reports about the hurricane’s progress. Nassau County’s Office of Emergency Management has posted a map of evacuation routes if it comes to that:
Hopefully Irene passes us by, but if not: be prepared.  Better safe than sorry.

Coastal storm impact risk mapped in NYC

UPDATED (Monday, Oct. 29, 2012 1pm)

If you need to locate any of NYC’s 76 hurricane evacuation shelters, you can use the OASIS mapping site.

You can also use NYC.gov to find out the latest with Hurricane Sandy.


UPDATED (8/25/11 9am): We’ve added a temporary map layer on OASIS showing the locations of NYC’s hurricane evacuation centers. Here’s the link: http://bit.ly/oBsUY8 . It’s easy to use:

  • Hover your mouse over each one to highlight it (the site details will also be highlighted in the panel on the right).
  • Click on a map marker to bring the site details up to the top of the list.
  • Double-click on a site in the list and the map will zoom in right to that location.

You can type in your address above the map to see if you’re in an area that’s at risk of storm impacts, and how close you are to an evacuation center. The OASIS map automatically also shows any nearby subway stations. And you can add any other layers from the Legend list to the right of the map.

For more information about Hurricane Irene and what you should do to prepare, visit NYC’s website.

UPDATED (8/24/11 12 noon): We’ve expanded the map layer showing hurricane impact zones throughout the downstate region. It now shows potential impact zones on Long Island’s south shore, as well as some areas along the coast of the Long Island Sound.

ORIGINAL POST

For the past several years the www.OASISnyc.net mapping site has displayed a map of “coastal storm impact zones” in New York City (in addition to the wealth of other mapped data included with OASIS). This coming weekend, it seems like the coastal storm map may be especially useful with city officials bracing for the risk of Hurricane Irene bringing 72 mph winds or more to the NYC region by Sunday.

In 2005, the New York State Office of Emergency Management developed a map of areas that would be at greatest risk of hurricane impacts based on wind speed and other factors. The shorthand for the map is a “SLOSH” map, because the zones of impact are based on NOAA’s “Sea, Lake and Overland Surge from Hurricanes” (SLOSH) model projections of vertical surge heights associated with category 1 – 4 storms.

In 2006, we downloaded the SLOSH data from the NYS GIS Clearinghouse website (metadata here), and used the following color shading to represent the zones:

  • A Category 1 storm (impact areas shown on the map in yellow) means winds of 74-95 mph.
  • A Category 2 storm (impact areas shown on the map in light orange) means winds of 96-110 mph.
  • A Category 3 storm (impact areas shown on the map in dark orange) means winds of 111-130 mph.
  • A Category 4 storm (impact areas shown on the map in red) means winds of 131 mph or more.

Here’s the map:

Through the OASIS website, you can easily enter your address and find out if you’re in an area that might be at greatest risk of the hurricane, depending on how severe the winds are by the time Irene moves up the coast. But the real power of OASIS’s maps is that you can do much more. For example, you can:

  • customize the OASIS map to show transit routes, schools, public housing, and libraries in or near the zones;
  • use OASIS’s mapping tools to see areas near the zones that have been recently developed (using the 1996-2010 aerial image timeline tool);
  • zoom in to see individual property boundaries and click on each one to determine ownership, zoning, and land use characteristics; and
  • find out which elected officials represent the area, as well as the local Community Board (good sources for planning and safety resources).

In the images below, I’ve used the aerial timeline slider and the dynamic transparency tool to show recent housing development in an area at risk of impact from a Category 1 storm, along South Beach on Staten Island (at the OASIS website, you can click and drag the name of any map layer in the Legend to the “Transparency Control” box at the bottom of the legend and make it more or less transparent, so the layers underneath can shine through):

Sparse housing near the beach (1996)

Dense housing development (2010)

The zones mapped on OASIS closely mirror the city’s hurricane evacuation zones (mapped here [PDF] and described here). The city also provides a Hurricane Evacuation Zone Finder where you can enter an address and get useful safety tips depending on your zone.

The message: hopefully Irene passes us by, but if not: be prepared. Better safe than sorry.

Innovative map comparisons – Census change in 15 cities

Our team at the Center for Urban Research (at the CUNY Graduate Center) has updated our interactive maps showing race/ethnicity patterns from 2000 and 2010 in major cities across the US. We’ve enhanced the maps in several ways:

  1. Added more cities. We now have 15 major urban regions mapped across the US (Atlanta, Baltimore, Boston, Charlotte, Chicago, Detroit, Houston, Los Angeles, Miami, New York, Orlando, Philadelphia, Phoenix, San Francisco, and Washington D.C.).
  2. The maps now have three ways of comparing 2000 and 2010 racial patterns:
  3. We color-coded the population change data in the popup window. Population increase is shown in green; decrease is shown in red. See image below.

Here’s our news release with more info.

Btw, we’ve also updated our static maps to show New York City Council districts, to begin to get a sense of how demographic changes will shape upcoming redistricting efforts at the local level.  Here’s the link:www.urbanresearchmaps.org/plurality/nyccouncil.htm (For the static maps, you can view 2000-2010 demographic change with the vertical slider bar, but you can’t zoom in/out, etc.)

An initial version of the maps launched in June with the vertical bar technique, integrating it with interactive, online maps for the first time. Our Center crafted the maps so you could not only drag the bar left and right but also zoom in and out, click on the map to obtain detailed block-level population counts, and change the underlying basemap from a street view to an aerial image (via OpenLayers use of Microsoft’s Bing maps tiles), while also changing the transparency of the thematic Census patterns.

The latest iteration of CUNY’s Census maps continues to use the vertical slider but now incorporates this technique with two more comparison options. Each approach serves different purposes:

  1. The vertical slider bar provides a “before (2000) and after (2010)” visualization of change, either regionally or at the scale of a city neighborhood.
  2. The side-by-side comparison is ideal for lingering over a given area, especially at the local level, taking the time to absorb the differences in demographic patterns mapped with 2000 Census data on the left and 2010 on the right. We incorporated this approach specifically at the suggestion of the great interactive team at the Chicago Tribune, who have created some similar Census maps.
  3. The single-map 2010/2000 overlay is especially helpful for revealing the increase in diversity over a given area.

For example, you can zoom to Atlanta, GA on the single-map overlay and see the city’s predominantly Black population in 2000 surrounded by suburban Census blocks shaded dark blue, denoting a White population of 90% or more (see images below). As you transition the map from 2000 to 2010, the dark blue in the suburbs fades to a lighter shade (indicating a more mixed population demographically) coupled with more Census blocks shaded green, purple, and orange – each corresponding to communities that are now predominantly (even if only by a few percentage points) Hispanic, Asian, or Black respectively. This pattern is replicated in many of the urban regions featured at the website.

Atlanta & suburbs in 2000

Race/ethnicity change in Atlanta by 2010

Eventually we’ll be moving all this from pre-rendered tiles to vector tiles. CUR’s application architect Dave Burgoon contributed code he developed to TileStache to enable TileStache to produce AMF-based output for use in Flash-based interactive mapping applications. This will give us flexibility in mapping as many Census variables as needed, and also providing complete geographic coverage (hopefully down to the block level) nationwide. That’s the plan, anyway! Stay tuned.

Credits

Funding for much of the Center’s recent work on Census issues has been provided by the Building Resilient Regions Project of the John D. and Catherine T. MacArthur Foundation, the Hagedorn Foundation, as well as support from the CUNY Graduate Center and the City University of New York.

Several people provided feedback and helpful editorial suggestions on earlier versions of the maps and narrative. Though the materials at this site were prepared by the Center for Urban Research, those invdividuals improved our work. We greatly appreciate their contributions.

Mapping the Cityscape exhibit

This summer the Center for Architecture in New York is all about maps.  One of the main exhibits at the Center, “Mapping the Cityscape”, features a dozen or so wall-mounted 8-foot-high maps of Manhattan — different representations and views from 1609 to the present.  Several panel discussions are accompanying the exhibit, including two this week (“Mapping Manhattan” and “Mapping Risk“).  Originally scheduled for the month of July, by popular demand the exhibit will be open through the end of August (Aug. 27th).

The exhibit came out of a panel discussion in May at the Center, organized by Abby Suckle of CultureNOW, with a broad group of participants: Matt Knutzen of the NY Public Library Map Division, John Tauranac of subway mapping fame, Laura Kurgan of Columbia University, and me discussing the OASISnyc mapping site — along with respondents from Google and the Wall Street Journal.  The panel covered lots of ground, from the Library’s invaluable collection of historical maps (now being digitized and geo-referenced) to the evolution of transit mapping in New York to the OASISnyc online mapping site (parks, open space, and much much more) to Columbia’s “Million Dollar Blocks” project to CultureNOW’s mobile maps to the latest from Google and others.  The hall was packed, the audience had lots of questions, and apparently they wanted more — hence the decision by the Center to transform the panel discussion into a summer-long exhibit.

Here are some photos of what you’ll see when you visit:

OASIS maps – each “slice” of the two maps above highlights the different types of mapped data you can display and analyze at the OASIS website.

Million Dollar Blocks – Columbia University’s Spatial Design Lab project that mapped the impact of prison policies on local neighborhoods in New York.

Mannahatta – the Wildlife Conservation Society’s take on what Manhattan likely looked like in 1609; an amazing project, rich with insights, analysis, and visual power.

NY Public Library – historical maps brought to life.

Maps of ecological patterns, historical maps overlain with current geography, transit mapping …

… and land use patterns, cultural icons, and more.

The exhibit opening in early July was packed — Center director Rick Bell & exhibit curator Abby Suckle talk to the crowd about the event:

If you’re interested in how maps — and the very definition of mapping and understanding/visualizing spatial relationships — are changing through the latest interactive technologies,  then this exhibit is for you.  Or if you’re interested in the history of visualizing New York City through maps, then this exhibit is for you.  Either way, please stop by, check out the maps, and attend one of the public programs.  It’ll be a cartographically illuminating experience.

(Exhibit logo from Center for Architecture website.  All photos: Steven Romalewski)

Slippy maps, meet before-and-after jQuery slider (introductions by OpenLayers)

Our team at the Center for Urban Research (at the CUNY Graduate Center) has launched a set of maps showing race/ethnicity patterns from 2000 and 2010 in major cities across the US.  The maps combine several mapping/web technologies that offer a new way of visualizing population change.  This post explains how we did it.

(And by popular demand, we’ve also included a map of Congressman Anthony Weiner’s district in relation to demographic change — you may have heard of him and his Twitter travails recently?)

Race/Ethnicity Change

Briefly, the maps show race/ethnicity change from 2000 to 2010 at the local level throughout major urban regions across the U.S.  So far we include New York City, Los Angeles, Boston, Chicago, Houston, and San Francisco.  (Others are coming soon.)

For our methodology and data analysis (and static maps), we provide that here.  For the mapping and web techniques, see below.

Reactions

So far we’ve received a pretty good response to our maps.  Here are some tweets posted recently:

  • @dancow (web journalist for ProPublica): Cool before/after map from CUNY’s urban research center showing NYC ethnic changes at the block level, from 2000-10.
  • @mericson (deputy graphics editor at NY Times): Nice block-level maps by @SR_spatial & CUNY Urban Research Center showing racial/ethnic change in NYC from 2000 to 2010.
  • @kelsosCorner (former Washington Post cartographer): Digging new 2010 Census plurality maps of NYC.
  • @albertsun (graphics editor at Wall St Journal): Coolest census map I’ve seen yet.
  • @PJoice (HUD employee; tweets are his own): This is the coolest map I have ever seen. Nice work by @SR_spatial and CUNY!
  • @MapLarge: I like how you can use the slider or move the map! Great Visualization!

Technical overview

The map uses the “before and after” technique that media websites have used for images of natural disasters.  We enhanced this technique by integrating it with interactive maps using OpenLayers, the open source mapping framework.  Now the slider works with two sets of overlapping, but perfectly aligned, maps from 2000 and 2010.

As it turns out, we didn’t set out to create an interactive version of these maps. In fact, we originally created static maps, but everyone we showed them to for feedback wanted the ability to zoom in/out and click on the map for more info.  So we developed the OpenLayers version. (And when I say “we”, that mainly means David Burgoon, CUR’s application architect, who I can’t say enough good things about.  I made the maps, and CUR’s Joe Pereira of the CUNY Data Service created the data sets, but Dave brought it all to life.)

OpenLayers enables us to introduce interactivity into the before-and-after images. Maps like these (to our knowledge) have not been available before — where you can move a slider back and forth while also zooming in/out and clicking on individual Census blocks for detailed information. You can also change the transparency of the thematic map layer, and switch between a street view and aerial view basemap.

It involved a good amount of work to integrate the slider technique with OpenLayers and also have two overlapping map instances working in tandem. The two maps need to appear as one, and this involves painstaking effort to ensure that the pixels on your screen are translated accurately to latitude/longitude coordinates in each of the separate but related interactive map instances, and the maps pan together seamlessly as you drag the slider left or right or move the map and it crosses the slider.

Mashup

In order to create the application, we used a mix of software applications, technologies, and techniques, summarized below:

  • We used the statistical software package SPSS to extract the Census block-level data for both years (see our methodology), allocate the 2000 data to 2010 blocks using the Census Bureau’s block equivalency files, and calculate the race/ethnicity plurality for each block.
  • We exported these SPSS files in DBF format and used ESRI’s ArcGIS Desktop to join the DBFs with 2010 TIGER Census block shapefiles.
  • ArcGIS Desktop was also used to create the choropleth maps (based on color schemes from ColorBrewer.org);
  • The map layouts were published as temporary web map services using ESRI’s ArcGIS Server. We used these to create pre-cached tiles (.PNG files) for the 2000 and 2010 maps, corresponding to zoom levels 4 through 10 using the now-standard Google-Microsoft map scales for online web maps. (Our application accesses the choropleth tiles as PNGs directly from the cache created by ArcGIS Server, rather than accessing the ArcGIS web map service in order to assemble the tiles. The latter approach would be too slow and would undermine the transition as you dragged the slider across the map.)
  • The slider technique was adapted from the jQuery plugin by www.catchmyfame.com.
  • OpenLayers provides all the map navigation and serving the maps themselves, modified with customized JavaScript code.
  • The basemap shown beneath the color-shaded map tiles is provided by Microsoft’s Bing map service. The street map and aerial image tiles from Bing are accessed directly via OpenLayers, rather than using the Bing API. This is a key reason we used Bing for these maps; if we used Google Maps as a basemap, we were limited to accessing Google Maps via Google’s API, which would have slowed map drawing times and undermined the slider effect.
  • For geocoding we use the Yahoo! Placefinder API.
  • Some browsers are not able to handle the before/after slider effect smoothly. In particular, Firefox and Safari perform poorly; the slider transition between one map to the other is not smooth. Microsoft’s Internet Explorer is adequate, but Google’s Chrome browser is best.

Data sources/issues

We used block-level data from the Census Bureau’s 100% population counts from the 2000 and 2010 decennial censuses (from Table P2 in the “PL-94-171” files for 2000 and 2010).

The Census Bureau’s block geography changed between 2000 and 2010 — new blocks were created, blocks were merged, and block boundaries were modified in many places. In order to compare population data from 2000 and 2010 using a common set of blocks, we used the Census Bureau’s block relationship file to allocate the 2000 population counts to 2010 geography.

When you’re viewing the map, it is best to use the maps and block-level data to understand trends over a larger area, even over several blocks. Be careful when viewing a specific block on its own. It covers a small area, and the Census Bureau may have made errors.

Credits

Funding for much of the Center’s recent work on Census issues has been provided by the Building Resilient Regions Project of the John D. and Catherine T. MacArthur Foundation, the Hagedorn Foundation, as well as support from the CUNY Graduate Center and the City University of New York.

Several people provided feedback and helpful editorial suggestions on earlier versions of the maps and narrative. Though the materials at this site were prepared by the Center for Urban Research, those invdividuals improved our work. We greatly appreciate their contributions.

MTA subway data in GIS format

UPDATE Oct. 5, 2015: See this post for updates to the GIS files described below, including the new Hudson Yards subway station and 7 line extension.


As promised, I’ve created an updated GIS data set of subway routes in New York based on MTA’s latest GTFS data, which I’ve posted below for anyone to use.  (I’ve also converted the latest MTA GTFS data to GIS format for NYC Transit bus routes, following up on my earlier post this spring.)  Scroll to the end of this post for the links.

I hope my effort provides a template for creating a map layer for apps and/or printed maps that approximates the line symbology on MTA’s map, but improves on this map in several ways — the GIS version is spatially precise, scalable, and may even look better than what Google uses for its transit layer in New York City.  The images below show the map zoomed out and zoomed in, and the post below explains how I did it.

Going forward, hopefully the MTA itself will provide subway route GIS data in the format I’ve described below (or something similar) alongside the GTFS data.  Any feedback or suggestions for improvement will be much appreciated.

Context

After reviewing the GTFS data files in detail, it became clear that GTFS is not necessarily applicable for displaying transit lines on a map and/or analyzing the spatial patterns with GIS.  At least that seems a fair characterization regarding the GTFS version of the city’s subway system.

Even though the GTFS files include a “shapes.txt” file for subways (the spec says this is for “drawing lines on a map to represent a transit organization’s routes“), this is only helpful for basic line representations.  This seems to work fine for bus routes.  But for subways, either the GTFS structure or MTA’s implementation of it poses challenges for creating a map layer of subway lines.

New York’s subway system includes local and express routes that are composed of inbound and outbound trips along the same line, some of which may start or end at different stations.  And there are “skip-stop” trains and “trunk” lines (where multiple routes run on the same set of tracks, such as the E, F, M, and R in Queens from Jackson Heights to Forest Hills).  But the GTFS “shapes” data from MTA only provide a partial representation of this complexity.

To attach route IDs to the “shapes” file, shapes.txt needs to be linked with trips.txt based on shape_id.  But doing so causes the 1, G, and latest version of the M line to drop out, because the trips.txt file does not include any trip entries for these routes.  Also, for some reason, it results in the inclusion of lines that aren’t used anymore (at least for passenger trips, as far as I know).

This is just an early iteration of MTA publishing its GTFS data, so I’m not surprised these limitations exist.  Until these issues are fixed, we have to rely on workarounds.  For example, the MTA has provided a separate shapefile for the 1 and the G (see earlier discussions at the MTA Developer Resources listserv).  According to MTA,

We [MTA] do not have shape data for these lines because of changes in their station configurations have occurred since we lost the staff member who had created the data. We have not had funding to replace him and update the data from 2008. We can provide the data next week with a hand-done solution, and/or better data at some later time, when we are able to acquire the staffing to do so.

This undermines the idea of using GTFS as a “feed” (as its name implies) for automatically displaying subway lines on a map, but hopefully the process will be more seamless as the issues are worked out.

But the and lines are not really missing from the GTFS data.  The GTFS “shapes” file on its own (without filtering it based on the “trips” file) includes line segments for virtually the entire subway system.  It’s just a question of being creative with combining the trip_id and shape_id fields from the trips.txt file to extract the appropriate geometry for the routes in question.  For example, the correct G shape is certainly there; it’s denoted by the “G..N05R” or “G..S05R” shape IDs.  It just so happens that there are no records in the trips.txt file with these shape_id values.  But it’s easy enough to create a new “route” field in the shapes file and populate it with a combination of values from the “routes.txt” file and manual entries for the lines that don’t seem to exist (such as the 1 and G).

Other issues with the shapes.txt file after filtering with the trips.txt file are that the line terminates at 57th Street/7th Avenue and the GTFS data includes shapes that show the running to Jamaica-179th St, the running to New Lots Ave on the 3/4 line, the extending to New Lots, and the running to New Lots.  As far as I can tell, these routes do not exist for the riding public.  Perhaps these are artifacts of older routing schemes, but it makes for an inadequate solution for mapping.  I’m curious how the automated routing and scheduling apps deal with this.

——————————–

UPDATE 7/20/10

After I wrote the preceding paragraph, I did quite a bit more digging into the subway GTFS data for a post at my blog about subway stations.  I realized that my points above about the and lines were wrong.  There are, in fact, stops that these trains make at the stations I mention above.  There aren’t many of them, but they exist. My post about station data explains this more fully.  So for these routes at least, the shapes.txt file is ok.

——————————–

A bigger problem, though, is that the latest version of the line is missing from the shapes data, and can’t be created from the approach described above for the 1 and G.  The images below highlight the challenge – the area in question is circled in blue on both images.  In the old map, the M runs along what is now the J-Z line, and the orange F-V and B-D lines cross the J-M-Z line.  Unfortunately this old routing is what the latest GTFS geometry follows.

But with the latest service changes, the new M line comes in from Brooklyn and then goes north to meet the B-D line. In the latest shapes.txt file, there is no such geometry for the M.  The geometry follows the old J-M-Z line with no obvious shape that follows the M’s new northward jog to meet the B-D line.

Therefore, I created a new line segment for the M, combining segments from the old M and V lines (shape_id values of “M..N89R” and “V..N01R”), along with an arc connecting the two, using ESRI’s ArcGIS editing tools.

Methodology

Once I had the updated set of “shapes” from GTFS, my goal was to somehow convert this data into a GIS version of the MTA’s subway lines in a way that could be replicated (and perhaps integrated back into GTFS format) and also easily symbolized to show separate lines along trunk routes.

As far as I know, to the extent anyone had a GIS dataset of subway routes prior to GTFS (such as this one we had created for the OASIS website by digitizing the MTA’s subway map), the only way to display separate trunk lines was to manually edit the geometry of the GIS line segments along a trunk route by clipping the line and moving it parallel to the trunk line, so it would show up as a distinct line symbol.  Obviously this has problems — the manual work involved is tedious, imprecise, hard to replicate, and it doesn’t scale well — it might work at a certain zoom level, but then zooming in would show the parallel lines farther apart and zooming out would show them merged together — as illustrated by the images below from the NYC Citymap website, going from a wide zoom to a closer zoom:

(The Citymap site is just one example; you can see a similar situation on OASISnyc.net — as you zoom in on the map at this link, you’ll see the and lines become farther apart.)

Divisions and Lines

I remembered that NYC Transit uses “division” and “line” designations that might be helpful in distinguishing the segments. The divisions are a throwback to when the subway system was really three separate systems — the IRT, IND, and BMT. But the line designations are based on more or less current track arrangements (and you can see some of these on the current subway map – see excerpt below).

For example, the movie “The Taking of Pelham One Two Three” refers to the train running on the Pelham Line, leaving the Pelham Bay Park station at 1:23.  Wikipedia has lots of information about the line designations, such as the IND 6th Avenue line or the BMT Nassau Street line.

But how to assign these to the shapes.txt file? The line IDs/names are not included as part of GTFS, and I’ve not seen this information provided anywhere else (publicly anyway).

Station entrance/exit data provides the missing link

Then on July 1 the MTA released a file listing subway entrances and exits with latitude/longitude for each one (the file was updated July 7 to fix some issues in the earlier data). Useful in its own right, the file includes the station name for each entrance/exit along with its division and line. Neat! The entrance/exit points don’t necessarily overlap or intersect the line shapes, so I wouldn’t be able to automatically assign the divisions and lines to the shapes using GIS, but there are only 37 unique lines based on the entrance/exit data so it wouldn’t be that hard or time-consuming to do it manually.

My approach was to create a thematic map of the entrances color-coded by line designation, overlay the GTFS shapes file of subway routes, and then edit the shapes file by splitting the segments where each set of color-coded entrances ended and adding the corresponding line attribute to these new segments.  The image below illustrates the approach.

In other words, instead of a single shape representing the  line, I created six non-overlapping segments to represent the entire 2 train route (along the 7th Ave-Bway, Clark Street, Eastern Parkway, Lenox, Nostrand, and White Plains lines).  I used the ArcGIS “Split Tool” quite a bit, and ended up with a shapefile with 80 unique shapes (including the AirTrain — which is included in GTFS but isn’t managed by MTA so likely doesn’t have an MTA “line” designation).  The attributes from the new file look like this:

This was a manual process based on visual inspection of the line segments, so I’m sure error has crept in.  Also, the way I did it, I allowed for some exceptions.  I didn’t rigorously create new segments, for example, along what appears to be a trunk line in Manhattan where the IND 6th Avenue and 8th Avenue lines meet at the West 4th Street station. And I probably didn’t handle lines travelling over bridges or through tunnels as well as I could have.  And the 5 route along the IRT White Plains line extends from Nereid Avenue to 138th Street/Grand Concourse, but the #5 in the Bronx that just runs during rush hour goes from East 180th Street to Nereid Ave (so on my map the dashed line symbology extends too far south).

Overall, though, I think it works well — it’s pretty good for a first pass.

ArcGIS caused import hassles …

Btw, I should point out that though ArcGIS’s editing tools were great for splitting and re-combining the line segments, ArcGIS misinterpreted important fields when importing the GTFS text files.  Fields that were text (such as “route_id” in the trips.txt file) were imported as numeric, preventing an accurate join.  I needed to use another program (I used SPSS) to save the trips.txt file as a DBF which preserved the text format of the field.  (I had tried using Excel to convert from TXT to CSV and also to XLS, but that also forced the text field to convert to numeric.)

… but ArcGIS provided invaluable cartography tools

Next step was to create the symbology.  I relied on two ArcGIS features to display multiple lines along a trunk route as discrete line symbols: the “cartographic line symbol” feature, and “symbol levels”.  The cartographic line symbol component of ArcGIS’s Symbol Property Editor, among other things, enables you to attach an offset value to the line symbol.  See screen shot below.  The great thing here is that the offset is relative depending on the zoom level of the map — as you zoom in or out the line symbols do not merge together or move further apart, thereby solving the problem parallel copies of line segments.

The Cartographic Line Symbol tool also allows you to create a dashed line symbol, which I used for the Rockaway Park Shuttle and the rush hour extension of the 5 train in the Bronx.

The “symbol levels” feature enabled me to ensure that line segments weren’t inadvertently masked by others along the same geometry.  For example, simply offsetting the and routes from the and  routes running along the IND 8th Avenue line may result in two parallel orange lines, rather than a blue and an orange line.  Setting a priority symbol level ensures that the blue and orange lines will run in parallel.

The resulting trunk line symbology looks good whether you’re zoomed out …

… or zoomed in close:

Soon we’ll have the updated lines and symbology on the OASIS mapping site.

In order to recreate the map symbology, I’ve preserved the color scheme in an ESRI layer (.lyr) file, linked below along with the actual shapefile.  If you’re using a GIS that doesn’t use layer files you’ll need to redo the symbology, but at least you can use the attributes to do so.

(The layer file includes subway route labels that were inspired by an approach provided by ESRI’s New York City office that we first applied for the OASIS site — using the subway route icons a la the MTA subway map. I’ve streamlined it a bit here, and it’s easy to modify further either with ArcGIS or another GIS package. I’ve included a basic MXD file that preserves the labeling.  The MXD uses ESRI’s Maplex labeling engine, but the labels will work with ESRI’s standard labeling engine as well.)

The one missing component to this data is a layer of transfers between subway stations.  I know this has been discussed on the MTA Developers Resource list, but this will have to wait till a more robust data set is available (or I or others have the time to put one together).

Links to the data

Here’s the GIS subway data in shapefile format (zipped):

If you use the data and layer file (which I hope you do), please let me know how it works out.  I’m not including any kind of Creative Commons licensing, but I’d appreciate it if you could reference the “CUNY Mapping Service at the Center for Urban Research” if you use the data and especially the layer symbology in any printed maps or online applications.  Thanks!

Also, here are the post-June 27 service change bus routes in shapefile format (zipped):

top

Better than Google Maps cartography?

I definitely wanted to compare my GIS version of the subway GTFS data with Google Maps, which presumably uses the GTFS data not only for transit directions but also for the basemap itself.   Two things surprised me.  One was that, as of today (July 7) almost two weeks after the MTA’s service changes took effect, Google Maps still shows old subway routes and station information.

The map below, for example, still displays the old  line (see the 23rd St/6th Avenue station) and the discontinued line (see the 23rd St and 28th St Broadway line stations).

Also, the subway lines on Google Maps were choppy and not as smooth as the GTFS-derived GIS lines. The images below compare the two in lower Manhattan.

I don’t think it’s nitpicking to point out the difference. One important aspect of the MTA’s GTFS data from a cartographic perspective is the high-quality route geometry.  It makes it that much more useful not only for good map development, but also for spatial analysis and alignment with other NYC GIS data layers.  Kudos to MTA for providing it.  I’m surprised Google apparently doesn’t use GTFS for their basemap (hopefully they’ll correct me if I’m wrong).

Going forward

For my purposes (and I think I’m far from alone here), I’m more interested in displaying the subway lines in a map layout than developing an application that provides routing and scheduling. Whether or not I use the data for spatial analysis, I’d like to have a subway layer for use in a GIS or any other application that needs the symbology of MTA’s printed map but is more spatially precise than MTA’s map and not as fine-grained as individual trips.

The GTFS format is great for all the web and mobile applications that are being developed.  But for the purposes of local planning work by Community Boards, students, the media, public officials, etc — we want to see the subway lines on a map and analyze them spatially — visualizing and understanding the relationships of nearby land usesdemographics, etc as well as the ability to monitor maintenance and operations trends, determine who represents each line when service changes are being proposed, and more.  So hopefully MTA will see fit to provide subway route data in a systematic way so we can integrate it easily into our maps.

It’s likely that NYC Transit maintains its subway line/route data in a similar structure as I’ve described above in GIS format, either for planning/modeling purposes or for other mapping needs.  Ideally it’s in a format that allows for an automated, rules-driven way of displaying the routes by division/line so changes are handled as seamlessly as possible.  In other words, it would be great if MTA could provide the subway data in a way that doesn’t require the additional staff resources that are involved in coverting the scheduling/routing data to GTFS format.  I’m not expecting anything as simple as “just hitting the export button,” but hopefully something close :).  And since subway routing doesn’t change very often (certainly not as frequently as schedules), this should be much less of a burden on the agency than the work involved in providing the GTFS data.

I look forward to continuing the dialog.

Thoughts about Google’s Styled Maps announcement

At the Google I/O developer event this week, an impressive item was included in the announcement about the latest Javascript Maps API (v3).  Now you can use Javascript to change the style of Google’s base map, in effect creating your own CSS for Google Maps.

Yesterday’s Google Geo Developers Blog highlighted the new feature and capabilities, including a link to a style wizard – see screen shot below – so you can see in real-time what each change looks like (the wizard is kind of like ColorBrewer, without the helpful suggestions for good vs. bad color schemes and symbology).  Note that the Style Wizard won’t work in IE.

Google Maps Style Wizard

This step for Google Maps is big, no doubt, but I’m ambivalent.  On the one hand, it opens up myriad possibilities for using Google’s map API.  Before, some geoweb developers (our team at CUNY included) tended not to use the Maps API because each and every map looked the same.  It got to be boring, and from a cartographic perspective it was basically an acknowledgment that you took the easy way out.  And often the Google map style got in the way of other layers you wanted to add to your map.

As Google’s own blog puts it,

No matter which [Google] Maps API site you are on, every map looks the same. If you want your map to stand out from the crowd, your options are limited to customizing the markers and controls, and if your brand has a particular colour scheme that is reflected on your site, Google Maps may not sit well with it.

EveryBlock’s Paul Smith wrote about these limitations two years ago at his post about “Google Maps Fatigue”.  For our part at CUNY, if we wanted a basemap with our own style that didn’t conflict with other thematic layers and choropleth overlays, either we rolled our own.  Or if we used Google Maps, we used the Flash API so we’d have the ability to desaturate its color (following the New York Times and others) — our Census 2010 Hard to Count mapping site used this technique. 

On the other hand, I’m sure more of us will now use Google Maps as a basemap, but will that further undermine the diversity of locally-crafted and customized online cartography?  With the advent of Google map styles, our team is now looking closely at GMaps for our map applications.  I think we’ll likely only use Google’s basemap for nationwide applications where it’d be too difficult to assemble all the base map layers at that scale.  But … maybe we’ll use it beyond that.

There are still major issues, of course.  Modifying Google’s map styles is one thing, but it’s still the Google basemap with the Google logo.  Using Google’s basemap whether we style it ourselves or not still means we’re stuck with the underlying data — with all its errors, inconsistencies, outdated place names, etc.  Google is trying to fix the errors, and errors are bound to exist, but until the basemap gets to a high level of quality and consistency, in effect you’re endorsing the errors when you use their basemap.  And though we can use the GMap style options to turn on/off place names, for example, this is nowhere near as robust as setting layer-specific filters that display or omit only *certain* place names, or using different road label types for different types of roads, etc.

As I understand it, CloudMade gives you much more control over map content as well as style.  You need to use OpenStreetMap, but perhaps the openness of OSM is more to your liking than Google Maps.  This post at Programmable Web highlights some of the differences between GMaps map styles and CloudMade’s editor features.

Some critics also point out that just making a map look different doesn’t necessarily mean it looks good, or better than the standard Google look-and-feel.  The examples provided via Google’s Geo Developer blog are intriguing …

… but this one via the Javascript Maps v3 documentation is, well, not so much:

Perhaps one cartographic benefit in all this is that it will help the mashup neogeographers of the world realize that there are good and bad map styles — cartography matters.  Web map developers will need to make choices about symbology, it won’t just be on a silver platter from Google.

What do you think?

GIS and Census participation

It’s been too long since my last blog post. Have been quite busy with work, and even though Twitter is a microblogging service, sending a tweet now and then really isn’t an excuse to keep up my actual blog.

One of the projects keeping me (very) busy is our work to help boost participation in the 2010 Census. I thought I’d write about some of our interactive mapping and participation rate analysis along these lines.

In August I described how our team at CUNY’s Center for Urban Research was creating metro-scale maps showing where hard-to-count communities were located so local census advocates could target their outreach. Then in late January we launched our interactive version of those maps at www.CensusHardToCountMaps.org. Originally we designed the site to show hard-to-count areas, but these are only where it was expected there’d be low census participation. Then, on March 23, the Census Bureau started publishing the actual participation rates on a local and national basis. So a week later we updated our site to emphasize the latest participation rates (this link zooms in to Manhattan showing tract-level participation overlaid on a map of hard-to-count tracts).

Though the Census Bureau’s Take 10 map (and a related Google Earth application) display the daily participation rates nationwide, we decided to provide several features that the Census Bureau doesn’t. At our site you can:

  • type in a county and highlight the tracts below a certain participation rate (you can enter whatever rate you want);
  • sort the resulting list so you can see at a glance the highest and lowest performing tracts (this also will be highlighted on the map so you can see how concentrated they are); and
  • compare the 2010 rate map with the 2000 rate map (click the “More…” tab and check the box for “Participation Rate in 2000″).

(Of course, you can also click on any spot on the map to display the latest participation rate for that area — state, county, or tract — depending on how close in or out you’ve zoomed.)

These are the types of data analysis and spatial visualization tools that were requested by census advocates, so they can use the maps to focus on areas that need their help the most.

In order to provide some context for the interactive map, our Center also posted an analysis of the first week’s participation rate. It was a combination of basic statistical analysis and mapping. We examined the correlation between participation rate and hard-to-count scores at the tract level nationwide, and not surprisingly found that rates tended to be lower in hard to count areas. This should help bolster the work of groups who’ve been working in these communities, confirming that they’re focused on areas that need support the most if we want to achieve a 100% count.

We also examined county-level statistics on race/ethnicity using the Census Bureau’s latest population estimates from 2008. (The American Community Survey would provide a richer set of characteristics to examine, but any data from areas with less than 20,000 population are suppressed in the ACS — and this accounts for about 1,300 of the nation’s 3,200 counties.)

The county-level data indicate that race/ethnicity is strongly correlated with census participation (at least in the first week), with participation rates tending to be higher in counties with a greater percentage of whites while the rates tended to be lower in counties with a greater percentage of blacks and Latinos. Because we didn’t have other socio-economic data to evaluate, we weren’t able to disentangle the effects of other characteristics such as low educational attainment, poverty, housing conditions, etc. that may have a stronger correlation while cutting across racial and ethnic categories. An opportunity for further research. As a next step we may also examine county-level unemployment rates and participation rates, as well as evaluating how well the first week’s analysis holds up as time goes on.

Friday (April 2) we added another feature, information about the areas that will receive a second census questionnaire.  (The Funders Census Initiative sent out a news advisory highlighting this service on April 5.)  Now when you click on the www.CensusHardToCountMaps.org map or type in your street address, you’ll see a popup window that (among other things) tells you if households in your tract will be receiving replacement Census forms. We think this will help minimize confusion over people receiving another census form (even if they’ve already mailed their’s in!). This is a “just in case” thing from the Bureau — mailing another form to households in historically low response areas, and mailing another form to households in moderately low response areas who haven’t yet sent their’s in. But the geographic scope of the “blanket” and “target” replacement mailing areas are pretty extensive in most cities (see maps at CUR’s website), so lots of people may be confused. Our mapping site provides a simple way of clearing the air.

We’ve also mapped those second mailing areas. When you visit www.CensusHardToCountMaps.org, select the “More…” tab and zoom in to your area of interest. For example, here’s Boston, MA. Click either or both check boxes in the “April 2010 Replacement Questionnaires” section to map the tracts receiving replacement census forms.

Our hard-to-count mapping site still has its original functionality — such as visualizing the demographic characteristics that will make it difficult to achieve a complete Census count; overlaying ZIP Codes, mail return rates from 2000, recent foreclosure risks by tract; and seeing who’s tweeting about the Census in your area.

But we’re also planning for next steps, thinking of the mapping application as a platform for future Census-related efforts (tracking how successful census advocates were, displaying the 2010 results, enabling the general public to get involved in a meaningful way in the redistricting process). Any ideas? We’d love to hear them.

Mapping “hard to count” areas for 2010 Census

UPDATED 8/21/09: Newsday (Long Island’s daily newspaper) reproduced an island-wide version of one of our maps in their article today (though the map only appeared in the print version of the paper).

People are gearing up across the US for the 2010 Census — not just the Census Bureau, of course, but organizations large and small who are planning myriad outreach efforts to boost participation, especially in typically “undercounted” communities.

It’s important in so many ways for everyone to be counted, but historically not everyone is (and not just because of statistical anomalies or poor street address data).  For various reasons, key constituencies are not fully counted — people of color, renters, recent immigrants, people predominantly speaking languages other than English, etc.  There’s a special effort underway – supported by major foundations, local governments, and spearheaded by advocacy and civil rights groups – to make sure the Census Bureau doesn’t miss these “hard to count” groups.

Our teamat the Center for Urban Research, CUNY Graduate Center is doing its part by mapping the “hard to count” (HTC) population in more than two dozen metropolitan areas nationwide, with the support of the Hagedorn Foundation.  We’ll also be developing an online, interactive mapping application for funders, outreach groups, local officials, the media, and others to easily zoom in on their metro region and create custom maps to help focus their efforts.

Where are the hard-to-count communities?  The Census Bureau analyzed 2000 Census data to:

measure census coverage and to identify reasons people are missed in the census (de la Puente, 1993). The variables include housing indicators (percent renters, multi-units, crowded housing, lack of telephones, vacancy) and person indicators (poverty, not high school graduate, unemployed, complex households, mobility, language isolation).

From this, the Bureau calculated an HTC score for each tract in the nation (on a scale of 0 to 125).  They estimate that any tract with a score of 60 or higher will be at the greatest risk of undercount in 2010.  Our metro maps for the 2010 outreach campaign highlight the concentrations of HTC tracts, as well as the 2000 Census response rate by tract.  Here are two examples, for Chicago and New York City:

Chicago_HTC_2010Census

NYC_HTC_2010Census

Foundations are using these maps to help guide their grantmaking, and local groups are using the maps to target their outreach.  As an example, this month several Long Island-based foundations issued an RFPfor grants to local groups doing Census outreach.  The Hagedorn Foundation asked us to create a special set of Long Island maps to supplement the RFP – you can view the PDFs here and here(see below for a sample – these mapped the HTC scores relative to all tracts in Nassau, so some tracts with scores less than 60 — but still harder to count than others in the county — are shaded as hard to count).

Nassau_HTC_2010Census

Next step for us is to transform these printed maps into a nationwide online mapping website.  The site will enable people to: 

  1. Zoom to their neighborhood, county, city, or state to see the mapped patterns of hard-to-count Census tracts;
  2. Click on individual Census tracts to display detailed information about each one;
  3. Add other layers of Census demographics, 2000 Census response rates, related data from the 2005-07 American Community Survey, and recent foreclosure risks circa 2007-08 so people can see the interrelationships among multiple variables; and
  4. Display state- and local-level resources such as funding opportunities, regional Census contacts, contact info for groups participating in Census outreach, etc.  

We’ve been reviewing other projects that are similar so we don’t duplicate efforts.  These include:

A key question for us is which basemap do we use?  We’ll be serving the tract-level geography and attributes (probably as cached tiles) from our servers using ArcGIS Server on the backend, OpenLayers and ExtJS on the front end — a combination that has served us well (see here and here).  Likely we’ll use Google Maps (and/or their Hybrid or Terrain views) for the underlying street/reference geography.  But perhaps OpenStreetMap would be a better choice?  Or Bing Maps?  Advice from the GeoWeb community would be appreciated.

Timing? We’re shooting for a beta site in late September, seeking feedback from partner groups in the fall, and a production-level site by the end of the year.

Online cartography for richly layered maps

Several recent items have called attention to the growing effort to make really good-looking maps online – see Matt Ball on the coming “cartographic explosion” and Peter Batty’s posts highlighting the great maps from OSM and CloudMade and of course Stamen Design (hardly an inclusive list, but it’s already a long-ish intro sentence).

It’s an exciting time for all that.  We’re hopefully moving well beyond the ubiquitous pushpins of more map mashups than we can count (no dig against Google Maps, but the pushpins can get a bit old — see EveryBlock’s insightful post about “Google Maps fatigue” and FortiusOne’s original — and on-the-mark — blog title “Moving Past Pushpins“). 

From what I can see, however, most of the latest online cartography efforts are focused on road-centric basemaps.  This is great, but there’s a lot more mapped information out there that eventually will be either layered on top of these basemaps or provided online directly.  And we’ll need good cartography to display richly layered online maps effectively.

I wanted to add a note about our humble contribution to that effort.  Our team at the City University of New York (and earlier at the Community Mapping Assistance Project, CMAP) has maintained an online mapping platform for the New York area since early 2001 called OASIS — the Open Accessible Space Information System.  OASIS displays open space resources (broadly defined) to help sustain these resources and visualize the nexus between community greening and broader urban planning issues.  The project is guided by a collaborative partnership of almost 60 greening groups, educators, individuals, businesses, nonprofits, and public agencies.

oasis_mainview

We recently completely revamped the website (see above screen shot).  The old site is at www.oasisnyc.net.  The new one is at http://www.urbanresearchmaps.org/oasis/map.aspx (best viewed in Firefox but tested in IE 6 & 7, Chrome, and Safari).  Our cartographic challenge — as Christy Spielman, a long-time colleague who helped create the original version of OASIS, noted — was to create an interactive map that certainly included transportation features, but in a way that kept them in the background while emphasizing parks, gardens, housing, land use patterns, zoning, schools, and more – plus aerial imagery.

Also, the recent upgrade includes neat new data such as local environmental stewardship “turfs” (in partnership with the USDA Forest Service) and *very* historical imagery and data from the Wildlife Conservation Society’s Mannahatta Project — photorealistic imagery of Manhattan circa 1609 as well as layers representing eco-systems, soils, Native American trails, and the 1609 Manhattan shoreline (more on Mannahatta in a separate post — it’s an amazing project). 

oasis_cityhall1609

So we needed to figure out styles and colors and scale dependency that would work together (we have easily more than 4 dozen layers with distinct map styles), allow room for more layers of data in the future, and potentially be able to integrate data feeds and user generated data on the fly.

I think we did ok, though there’s always room for improvement (feedback is welcome!).  Christy (and I, though to a lesser extent) spent many hours developing color styles and symbology to find the right ones that worked well together.  We used ArcGIS to develop the maps (we’re most familiar with that, and it was quicker than having to learn SLD customization, for example).  We relied heavily on ColorBrewer, an amazing resource for GIS color symbology. 

ESRI’s New York City office — in particular, Patrick Gahagan – also helped us by providing ArcGIS techniques to create the map of subway stations and routes, and providing a DEM mosaic that we used as a backdrop for the new OASIS map.

oasis_subways_and_dem

Cached tiles were key, but deciding which layers to cache and which ones to be dynamic was an involved process.  We wanted to preserve as much flexibility for the map user while of course trying to speed up performance.  We ended up caching most of the transportation data (with the option to turn on/off the entire transporation cache layer) as well as the land use information – lot boundaries, building footprints, and parcel-by-parcel color shading. 

oasis_transportationcachelayer

The parcel and building footprint layers were the most cumbersome as dynamic layers, each of them with literally a million features, so the performance gain by caching them was huge.  But Dave Burgoon (who coded the site from top to bottom – more on his impressive work in a separate post) customized the “land use” section of the legend to enable our users to display either the entire cached land use layer or to display each land use category dynamically (see screen shot below). 

oasis_landuseoptions

This was a good compromise between performance (showing all land use patterns at all scales) and user flexibility (showing one land use category at a time dynamically isn’t too slow, and enables people to see all the commercial property versus residential versus industrial at a time).

The tiles, btw, were generated with ArcGIS Server and integrated into the map directly via OpenLayers.

We also took a page from the O’Reilly book “Designing Web Interfaces” plus Axismaps (who helped design GeoCommons Maker!) plus MapTube – and created a “dynamic transparency” tool that can accomodate each map layer on OASIS. 

oasis_transparencytool

The ExtJS transparency tool allows for a smooth, real-time transition as you slide from fully opaque to 100% transparent.  It makes for a powerful user experience.  The relevant quote from Designing Web Interfaces

Things move smoothly in the real world.  They do not “pop up”.  Transitions smooth out the jarring world of the Web, making changes appear more natural. (p. 233)

… and from Axismaps:

The transparency control lets mapmakers decide what works and what doesn’t.

Hopefully the cartographic result works well.  We’d love to know what you think!