Day 30 - Application Programming Interfaces (APIs)

Fall 2022

Dr. Jared Joseph

November 16, 2022

Overview

Timeline

  • What is an API
  • Using APIs
  • APIs in R

Goal

Create our own custom API call in R

Coming Soon

What is an API

Application Programming Interfaces (APIs)

APIs are a way for computers to talk with each other.


APIs are used behind the scenes for a ton of services:

  • Weather data
  • social media embeds
  • location services
  • internet searches
  • (Nearly) any app with an online component

APIs vs Scraping

Scraping

Pros:

  • Very versatile
  • Can attempt to get pretty much anything on the internet

Cons:

  • Often explicitly forbidden
  • Often bespoke
  • Very fragile
  • Requires understanding web design

APIs

Pros:

  • Easier access to data
  • Raw data
  • Faster
  • More robust

Cons:

  • Not always available
  • Restrictions on data access
  • Need to register for a key
  • Often paid services

Why Would Corps Offer This?

APIs are generally seen as beneficial for everyone involved.

You

  • Get access to private data
  • Raw data
  • Stable
  • Support

They

  • Set quotas
  • Control what data is available
  • Disable access if needed
  • Track usage

An Analogy

Web Scraping

APIs

Using APIs

Use Case - NYC 311

Requests for public service in NYC

  • Updated daily
  • Created October 2011
  • Each row is a 311 request
  • 31,500,000 rows
  • Too big to open normally

Not worth downloading everything every day for new data.


With APIs, we can subset that data before we load it into R.

https://data.cityofnewyork.us/Social-Services/311-Service-Requests-from-2010-to-Present/erm2-nwe9

Speaking API

Example API Call:

https://data.cityofnewyork.us/resource/erm2-nwe9.json?complaint_type=Illegal Parking&borough=BRONX

The most common form of an API requests look like normal web URLs.


For example, to get data from the 311 dataset, an API request would look like:

It is made up of:

Endpoint
https://data.cityofnewyork.us
Dataset we Want
resource/erm2-nwe9
Type of Data
.json
Our Query (Search/Subset)
?complaint_type=Illegal Parking&borough=BRONX

What are the Choices?

Mix and Match!

https://data.cityofnewyork.us/resource/erm2-nwe9.json?complaint_type=XXXX&borough=XXXX

Complaint types

  • Noise - Residential
  • HEAT/HOT WATER
  • Illegal Parking
  • Street Condition
  • Street Light Condition

Boroughs

  • BROOKLYN
  • QUEENS
  • MANHATTAN
  • BRONX
  • STATEN ISLAND

APIs and JSON

Most APIs return JSON data.

JSON is a nested series of Keys and Values.

Similar to:

  • A long dataframe
  • Dictionaries in Python
  • Lists in R
  • HTML (kinda)
  • XML

Sometimes CSV is an option too.

JSON Example:

{
  "student": [ 
     { 
        "id":"01", 
        "name": "Paul", 
        "lastname": "Atreides" 
     }, 
     { 
        "id":"02", 
        "name": "Feyd-Rautha", 
        "lastname": "Harkonnen" 
     } 
  ]   
}

APIs in R

Specific Services

Many large APIs have R packages specific to them.

General API Tools

httr
Get the data from the site.
jsonlite
Work with json data in R. 
```{r}
library(httr)
library(jsonlite)

# Get the data from the API
# Pay attention to the spaces!
# Replaced with %20 (apis don't like spaces)
api_return = GET("https://data.cityofnewyork.us/resource/erm2-nwe9.json?complaint_type=Illegal%20Parking&borough=BRONX")

api_return
```
Response [https://data.cityofnewyork.us/resource/erm2-nwe9.json?complaint_type=Illegal%20Parking&borough=BRONX]
  Date: 2022-11-17 01:08
  Status: 200
  Content-Type: application/json;charset=utf-8
  Size: 1.25 MB
[{"unique_key":"55985882","created_date":"2022-11-15T00:45:39.000","agency":"...
,{"unique_key":"55984459","created_date":"2022-11-15T00:17:49.000","agency":"...
,{"unique_key":"55981806","created_date":"2022-11-14T23:52:28.000","agency":"...
,{"unique_key":"55981487","created_date":"2022-11-14T23:50:06.000","agency":"...
,{"unique_key":"55981480","created_date":"2022-11-14T23:39:13.000","agency":"...
,{"unique_key":"55980398","created_date":"2022-11-14T23:34:08.000","closed_da...
,{"unique_key":"55983006","created_date":"2022-11-14T23:30:41.000","agency":"...
,{"unique_key":"55984466","created_date":"2022-11-14T23:20:03.000","agency":"...
,{"unique_key":"55984467","created_date":"2022-11-14T22:54:06.000","agency":"...
,{"unique_key":"55987417","created_date":"2022-11-14T22:50:47.000","agency":"...
...

From JSON to DF

```{r}
# Convert raw data to JSON
api_strings = rawToChar(api_return$content)

# convert JSON to df
api_df = fromJSON(api_strings)
```
# show results
head(api_df)
unique_key created_date agency agency_name complaint_type descriptor location_type incident_zip incident_address street_name cross_street_1 cross_street_2 intersection_street_1 intersection_street_2 address_type city landmark status community_board bbl borough x_coordinate_state_plane y_coordinate_state_plane open_data_channel_type park_facility_name park_borough latitude longitude location resolution_description resolution_action_updated_date closed_date
55985882 2022-11-15T00:45:39.000 NYPD New York City Police Department Illegal Parking Commercial Overnight Parking Street/Sidewalk 10469 2420 LURTING AVENUE LURTING AVENUE WARING AVENUE MACE AVENUE WARING AVENUE MACE AVENUE ADDRESS BRONX LURTING AVENUE In Progress 11 BRONX 2044490012 BRONX 1023832 253250 ONLINE Unspecified BRONX 40.86169872990346 -73.85690227359987 40.86169872990346 NA NA NA
55984459 2022-11-15T00:17:49.000 NYPD New York City Police Department Illegal Parking Blocked Hydrant Street/Sidewalk 10461 1525 LURTING AVENUE LURTING AVENUE EAST TREMONT AVENUE POPLAR STREET EAST TREMONT AVENUE POPLAR STREET ADDRESS BRONX LURTING AVENUE In Progress 11 BRONX 2040680021 BRONX 1025715 246623 ONLINE Unspecified BRONX 40.843500966964086 -73.85013581133983 40.843500966964086 NA NA NA
55981806 2022-11-14T23:52:28.000 NYPD New York City Police Department Illegal Parking Blocked Crosswalk Street/Sidewalk 10461 2955 CODDINGTON AVENUE CODDINGTON AVENUE GILLESPIE AVENUE JOSEPH ODDO WAY GILLESPIE AVENUE JOSEPH ODDO WAY ADDRESS BRONX CODDINGTON AVENUE In Progress 10 BRONX 2053920034 BRONX 1031195 245472 MOBILE Unspecified BRONX 40.8403143810083 -73.83033791607889 40.8403143810083 NA NA NA
55981487 2022-11-14T23:50:06.000 NYPD New York City Police Department Illegal Parking Blocked Hydrant Street/Sidewalk 10468 2539B GRAND AVENUE GRAND AVENUE PASTOR ROBERT LEWIS FOLEY SR WAY WEST 192 STREET PASTOR ROBERT LEWIS FOLEY SR WAY WEST 192 STREET ADDRESS BRONX GRAND AVENUE In Progress 07 BRONX 2032140142 BRONX 1011539 254569 PHONE Unspecified BRONX 40.86536554285058 -73.90133878369453 40.86536554285058 Your complaint has been received by the Police Department and additional information will be available later. 2022-11-15T01:46:37.000 NA
55981480 2022-11-14T23:39:13.000 NYPD New York City Police Department Illegal Parking Blocked Hydrant Street/Sidewalk 10452 125 MARCY PLACE MARCY PLACE WALTON AVENUE GRAND CONCOURSE WALTON AVENUE GRAND CONCOURSE ADDRESS BRONX MARCY PLACE In Progress 04 BRONX 2028410083 BRONX 1007484 244596 MOBILE Unspecified BRONX 40.83800423364504 -73.91603386694314 40.83800423364504 NA NA NA
55980398 2022-11-14T23:34:08.000 NYPD New York City Police Department Illegal Parking Blocked Hydrant Street/Sidewalk 10472 1329 COMMONWEALTH AVENUE COMMONWEALTH AVENUE EAST 172 STREET EAST 174 STREET EAST 172 STREET EAST 174 STREET ADDRESS BRONX COMMONWEALTH AVENUE Closed 09 BRONX 2038740063 BRONX 1020596 242945 ONLINE Unspecified BRONX 40.83342849023004 -73.86865697421297 40.83342849023004 The Police Department responded to the complaint and determined that police action was not necessary. 2022-11-15T00:37:51.000 2022-11-15T00:37:44.000

Code-Along

For Next Time

Topic

  • Lab 8
  • Quiz 3 Open

To-Do

  • Finish Worksheet
  • Work on Project 2

SDS 192-03: Intro to Data Science