NOTES ON THE STOP-FRISK BUNDLE FROM WNYC & JOHN KEEFE

Here's what's in the full zip file:


FOLDER: nypd_stopfrisk2011_totals_by_block_geo

This collection constitutes an "ESRI Shapefile" that contains the outlines of all NYC census blocks and a field "PCTCNT," which is the number of stops in that block. I calculated this by plotting every stop on a map and then using QGIS software to count the points within each block boundary.

Shapefiles can be read by many free tools, including QGIS (http://qgis.org) and websites such as http://geocommons.com/

The other fields are from the US Census Shapefile on which this shapefile is based -- the 2010 census block file for New York State, which is available here:
http://www.census.gov/cgi-bin/geo/shapefiles2010/main

Descriptions of the census column names can be found here:
http://www.census.gov/geo/www/2010census/tract_rel/tract_rel_layout.html

The "internal point," defined by the latitude and longitude fields INTPTLAT10 and INTPTLON10, are described by the Census bureau: "The Census Bureau calculates an internal point (latitude and longitude coordinates) for each geographic entity. For many geographic entities, the internal point is at or near the geographic center of the entity. For some irregularly shaped entities (such as those shaped like a crescent), the calculated geographic center may be located outside the boundaries of the entity. In such instances, the internal point is identified as a point inside the entity boundaries nearest to the calculated geographic center and, if possible, within a land polygon." (Page 108 of this PDF: https://www.census.gov/prod/cen2010/doc/pl94-171.pdf)

---

FILE: nypd_stopfrisk2011_gunstops_geo

This collection of files contain the locations of stops in which guns were found, according to the NYPD data. I have aggregated multiple stops at the same location as "STOPCNT," and noted the date or dates those stops were made, and the area in which they were made (stairwell, sidewalk, etc.) in "STOPDEETS."

The stops were originally plotted by the NYPD using the New York-Long Island State Plane Coordinate System (XCOORD and YCOORD fields). There were 20 stops that had no coordinates. For 17 of those, I was able to discern the location based on address and/or intersection info elsewhere in the stop record. Three gun-related stops are not plotted because neither I nor the NYPD could determine their location. All points have been converted to EPSG:4326 / WGS 84 ... which is the latitude-longitude projection used by Google maps.

This collection constitutes an "ESRI Shapefile," which can be read by many free tools, including QGIS (http://qgis.org) and websites such as http://geocommons.com/

----

FILE: nypd_stopfrisk2011_gunstops_data.csv

This is the original NYPD data file converted from the .por format and sliced using R for just those stops in which a gun was found. I then saved it as a .csv (comma-separated values) text file.

The column definitions are found in:
nypd_stopfrisk2011_data_file_spec.xlsx

Which I've also provided as .CSV files, for those without Microsoft Excel:
nypd_stopfrisk2011_data_file_spec_labels.csv
nypd_stopfrisk2011_data_file_spec_variables.csv

For those familiar R, to do the conversion I enabled the "foreign" package and did this:

  data<-read.spss("2011.por")
  data<-as.data.frame(data)
  dim(data)
  gunslice<-subset(data, (PISTOL=="Y" | RIFLSHOT =="Y" | MACHGUN=="Y"))
  dim (gunslice)
  write.csv(gunslice,file="nypd_stopfrisk2011_gunstops_data.csv")

----

FILE: nypd_stopfrisk2011_allstops_data.csv

This is the big one. Expanded, it's nearly 500MB. It includes every field for every stop in 2011. It's a direct conversion from the NYPD .por file.

You will be hard-pressed to open this in Excel or similar software, and it exceeds the size limits for Google Fusion Tables. I used the free statistical program R (http://www.r-project.org/); those familiar with Unix and Python might find csvkit useful (http://csvkit.readthedocs.org/en/latest/index.html). And programmers can definitely call it in line-by-line.

The column definitions are found in:
nypd_stopfrisk2011_data_file_spec.xlsx

Which I've also provided as .CSV files, for those without Microsoft Excel:
nypd_stopfrisk2011_data_file_spec_labels.csv
nypd_stopfrisk2011_data_file_spec_variables.csv

For those familiar R, to do the conversion I enabled the "foreign" package and did this:

  data<-read.spss("2011.por")
  data<-as.data.frame(data)
  dim(data)
  write.csv(data,file="nypd_stopfrisk2011_allstops_data.csv")

----

FILE: nypd_stopfrisk_StatisticalNotes.txt

This is the original NYPD statistical notes file from: 
http://www.nyc.gov/html/nypd/html/analysis_and_planning/stop_question_and_frisk_report.shtml

----

FILE: nypd_stopfrisk2011_data_file_spec.xlsx

This is the original data specifications file from:
http://www.nyc.gov/html/nypd/html/analysis_and_planning/stop_question_and_frisk_report.shtml

I've also provided them as .CSV files, for those without Microsoft Excel:
nypd_stopfrisk2011_data_file_spec_labels.csv
nypd_stopfrisk2011_data_file_spec_variables.csv


John Keefe
Senior Editor, Data News & Journalism Technology
WNYC, New York Public Radio
jkeefe@wnyc.org
@jkeefe on Twitter