Street Flooding#
Source: 311 Service Requests from 2010 to Present#
About#
Key |
Value |
|---|---|
URL |
https://data.cityofnewyork.us/Social-Services/311-Service-Requests-from-2010-to-Present/erm2-nwe9 |
Description |
All 311 Service Requests from 2010 to present. |
Updated |
2023-02-15 |
Update Frequency |
Daily |
Views |
440K+ |
Data Provided by |
311, DoITT |
Category |
|
API Docs |
https://dev.socrata.com/foundry/data.cityofnewyork.us/erm2-nwe9 |
API Endpoints |
|
Data Dictionary |
|
|
Sewer |
|
Street Flooding (SJ) ☔ |
Import Libraries#
Built-in Libraries#
# import json
import os
# from datetime import datetime as dt
# from pathlib import Path
External Libraries#
# import pyproj
import geopandas as gpd
# import pandas as pd
# import geojson as gj
Define Variables#
Default limit = 1000
%%script echo skip
NYC_OPEN_DATA_311_API_JSON = 'https://data.cityofnewyork.us/resource/erm2-nwe9.json?descriptor=Street%20Flooding%20(SJ)'
NYC_OPEN_DATA_311_API_GEOJSON = 'https://data.cityofnewyork.us/resource/erm2-nwe9.geojson?descriptor=Street%20Flooding%20(SJ)'
NYC_OPEN_DATA_311_API_CSV = 'https://data.cityofnewyork.us/resource/erm2-nwe9.csv?descriptor=Street%20Flooding%20(SJ)'
skip
Download 311 Service Complaints for Street Flooding (SJ)#
Save .geojson data locally#
!python obtainflooddata.py
Downloading Latest Street Flooding Dataset...
Street flooding dataset has already been download for 2023-04-03.
Download Complete
Define prefix for output variable#
%%script echo skip
output_prefix = 'data/street-flooding/street_flood-complaints.'
skip
Save .json data locally#
%%script echo skip
street_flooding_jdf = pd.read_json(NYC_OPEN_DATA_311_API_JSON)
street_flooding_jdf.to_json(output_prefix + 'json')
skip
Save .csv data locally#
%%script echo skip
street_flooding_cdf = pd.read_csv(NYC_OPEN_DATA_311_API_CSV)
street_flooding_cdf.to_csv(output_prefix + 'csv')
skip
View Street Flooding Metadata#
nyc_street_flooding_geojson = 'data/street-flooding/street-flood-complaints_rows-all.geojson'
street_flooding_gdf = gpd.read_file(nyc_street_flooding_geojson)
street_flooding_gdf.info()
<class 'geopandas.geodataframe.GeoDataFrame'>
RangeIndex: 35225 entries, 0 to 35224
Data columns (total 45 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 location_state 34213 non-null object
1 facility_type 23296 non-null object
2 intersection_street_2 11344 non-null object
3 city 34360 non-null object
4 location_zip 34213 non-null object
5 park_borough 35221 non-null object
6 latitude 34213 non-null object
7 road_ramp 0 non-null float64
8 created_date 35225 non-null datetime64[ns]
9 agency 35225 non-null object
10 park_facility_name 35225 non-null object
11 location_address 34213 non-null object
12 agency_name 35225 non-null object
13 descriptor 35225 non-null object
14 bbl 21819 non-null object
15 location_city 34213 non-null object
16 open_data_channel_type 35225 non-null object
17 cross_street_2 30199 non-null object
18 bridge_highway_direction 0 non-null float64
19 longitude 34213 non-null object
20 bridge_highway_segment 0 non-null float64
21 street_name 23933 non-null object
22 incident_address 23933 non-null object
23 address_type 35219 non-null object
24 incident_zip 34356 non-null object
25 unique_key 35225 non-null object
26 complaint_type 35225 non-null object
27 y_coordinate_state_plane 34213 non-null object
28 status 35225 non-null object
29 bridge_highway_name 0 non-null float64
30 location_type 0 non-null float64
31 due_date 1 non-null datetime64[ns]
32 taxi_company_borough 0 non-null float64
33 taxi_pick_up_location 0 non-null float64
34 x_coordinate_state_plane 34213 non-null object
35 resolution_description 35188 non-null object
36 community_board 35221 non-null object
37 resolution_action_updated_date 35197 non-null datetime64[ns]
38 intersection_street_1 11344 non-null object
39 closed_date 35189 non-null datetime64[ns]
40 vehicle_type 0 non-null float64
41 cross_street_1 30209 non-null object
42 borough 35221 non-null object
43 landmark 0 non-null float64
44 geometry 34213 non-null geometry
dtypes: datetime64[ns](4), float64(9), geometry(1), object(31)
memory usage: 12.1+ MB
Coordinate Reference System (CRS)#
street_flooding_gdf.crs
<Geographic 2D CRS: EPSG:4326>
Name: WGS 84
Axis Info [ellipsoidal]:
- Lat[north]: Geodetic latitude (degree)
- Lon[east]: Geodetic longitude (degree)
Area of Use:
- undefined
Datum: World Geodetic System 1984
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich
Preview Data#
street_flooding_gdf[['created_date', 'borough', 'bbl', 'geometry']].head(10)
| created_date | borough | bbl | geometry | |
|---|---|---|---|---|
| 0 | 2010-01-02 08:26:00 | BROOKLYN | 3089000064 | POINT (-73.92178 40.58778) |
| 1 | 2010-01-02 12:00:00 | STATEN ISLAND | NaN | POINT (-74.14329 40.63866) |
| 2 | 2010-01-02 17:45:00 | QUEENS | 4120050012 | POINT (-73.79530 40.68140) |
| 3 | 2010-01-04 16:47:00 | QUEENS | 4106210008 | POINT (-73.73843 40.72006) |
| 4 | 2010-01-05 10:37:00 | BROOKLYN | 3086550021 | POINT (-73.90969 40.61250) |
| 5 | 2010-01-06 19:26:00 | BROOKLYN | 3029270015 | POINT (-73.93297 40.71584) |
| 6 | 2010-01-06 08:24:00 | QUEENS | 4119960122 | POINT (-73.80255 40.67925) |
| 7 | 2010-01-06 09:17:00 | STATEN ISLAND | 5040740044 | POINT (-74.10646 40.55866) |
| 8 | 2010-01-06 06:00:00 | QUEENS | 4030030044 | POINT (-73.87694 40.71804) |
| 9 | 2010-01-07 10:16:00 | STATEN ISLAND | 5014850078 | POINT (-74.14943 40.61979) |
street_flooding_gdf[['created_date', 'borough', 'bbl', 'geometry']].tail(10)
| created_date | borough | bbl | geometry | |
|---|---|---|---|---|
| 35215 | 2023-03-31 14:50:00 | QUEENS | 4035540010 | POINT (-73.90314 40.69508) |
| 35216 | 2023-03-31 15:56:00 | BROOKLYN | NaN | POINT (-73.97547 40.67447) |
| 35217 | 2023-04-01 17:05:00 | BROOKLYN | 3050530018 | POINT (-73.96291 40.65447) |
| 35218 | 2023-04-01 19:56:00 | STATEN ISLAND | 5035730072 | POINT (-74.09985 40.58014) |
| 35219 | 2023-04-01 12:35:00 | QUEENS | 4062770002 | POINT (-73.77331 40.76058) |
| 35220 | 2023-04-01 21:32:00 | QUEENS | 4010500002 | POINT (-73.89320 40.76207) |
| 35221 | 2023-04-01 23:17:00 | BROOKLYN | 3083070080 | POINT (-73.88413 40.63804) |
| 35222 | 2023-04-01 16:14:00 | BROOKLYN | 3077180057 | POINT (-73.93635 40.61422) |
| 35223 | 2023-04-01 14:32:00 | BRONX | NaN | POINT (-73.90305 40.84063) |
| 35224 | 2023-04-01 14:17:00 | BRONX | NaN | POINT (-73.88099 40.87091) |
References#
Python Standard Library#
Format#
How to pad zeros to a String in Python | Python Engineer
Contributer: @Patrick Loeber
Date and Time#
datetime — Basic date and time types | Python > Documentation
File & Path I/O#
pathlib — Object-oriented filesystem paths | Python > Documentation
GeoPandas#
User Guide#
Reading and Writing Files | GoePandas - Documentation - User Guide
API Reference#
Jupyter Notebook#
Magic (%%) Commands#
How to (intermittently) skip certain cells when running IPython notebook? | stackoverflow
Contributor: @Mark
pyproj#
Fix#
Un-install pyproj
conda remove --force pyproj
Re-install pyproj via pip instead of conda
pip install pyproj