15. Retrieving Data from APIs¶
What is an API?
APIs are mechanisms that enable two software components to communicate with each other using a set of definitions and protocols. Many organizations now make their data available via API. Web-based can access this data and use them visualization or analysis.
There are many different types of web APIs. One of the most common types is a REST API. This type of API uses URL arguments to specify what information should be returned via the API.
API Endpoints
When working with APIs, you will frequently hear the term “endpoint”. The API endpoint refers to the dataset that is being retrieved by the API’s URL. These data may be in the form of census data, geospatial base maps, elevation data or any type of data that has been made available through the API. As an example, let’s look at the Colorado Information Marketplace which is a data warehouse that provides access to a wide range of Colorado-specific open datasets available via a REST API called the Socrata Open Data API (SODA) - https://data.colorado.gov/browse
There are lots of API endpoints or data sets available through this API. Click on this link to see the response that you get from using a browswer to make a basic request with no URL parameters specified: https://data.colorado.gov/resource/tv8u-hswn.json. Notice that the response in this case is returned in JSON format.
Retrieving API Data Using Python
Instead of using the browser to view the json file at the API endpoint, we can use Python to make a call to the API, retreive the data, and view the data. The script below illustrates how this works:
import requests
import json
colorado_data = requests.get("https://data.colorado.gov/resource/tv8u-hswn.json")
print (colorado_data.json())
The output from the above script is a json object. Although this is a human readable format, we can use the pandas library to format the json and display it in a more user-friendly way. The code sample below shows how this can be done.
Viewing Json with Pandas
import requests
import json
import pandas as pd
url = "https://data.colorado.gov/resource/tv8u-hswn.json"
colorado_data = pd.read_json(url)
colorado_data.head()
Geocoding Using the OpenStreetMap Nominatim API
The Python script below uses the OpenStreetMap Nominatim API to geocode an address, retrieve a JSON response, parse the JSON response, and print out the latitude and longitude coordinates.
import requests
import json
# Define the address to geocode
address = input("Enter the address of interest ")
#address = "1600 Pennsylvania Avenue NW, Washington, DC 20500, USA"
# Build the URL for the Nominatim API request
url = f"https://nominatim.openstreetmap.org/search?q={address}&format=json"
# Make a GET request to the Nominatim API
response = requests.get(url)
# Parse the JSON response
data = json.loads(response.content)
# Extract the latitude and longitude coordinates from the response
latitude = data[0]['lat']
longitude = data[0]['lon']
# Print the latitude and longitude coordinates
print(f"Latitude: {latitude}")
print(f"Longitude: {longitude}")
Downloading Json and Mapping the Data Points
As GIS researchers, we are also interested in mapping latlong coordinates that may be contained in API endpoints. The script below takes a json file, parses it and extracts the latitude/longitde data. Afterwards, the folium library is used to map the data. The data comes from this world population website
import json
import requests
import folium
# Load the JSON file
with open("/Users/.../cities.json", 'r') as f:
data = json.load(f)
# Parse the JSON data to extract city names
cities = [row['name'] for row in data]
# Use OpenStreetMap's Nominatim to obtain latitude and longitude for each city
locations = []
for city in cities:
# Construct the API request URL
url = f"https://nominatim.openstreetmap.org/search?format=json&q={city}"
# Make a GET request to the API endpoint
response = requests.get(url)
# Parse the response content as JSON
data = json.loads(response.content)
# Extract the latitude and longitude from the JSON data
latitude = data[0]['lat']
longitude = data[0]['lon']
# Append the latitude and longitude to the locations list
locations.append([latitude, longitude])
print (locations)
# Create a map with a marker for each city's latitude and longitude
map = folium.Map(location=[locations[0][0], locations[0][1]], zoom_start=12)
for location in locations:
folium.Marker(location).add_to(map)
# Display the map
map
import requests
import json
import folium
# Make a GET request to the API endpoint
response = requests.get("https://example.com/api")
# Parse the response content as JSON
data = json.loads(response.content)
# Extract the latitude and longitude from the JSON data
latitude = data['latitude']
longitude = data['longitude']
# Create a map using the extracted latitude and longitude
map = folium.Map(location=[latitude, longitude], zoom_start=12)
# Add a marker to the map at the extracted latitude and longitude
folium.Marker([latitude, longitude]).add_to(map)
# Display the map
map
Mapping with Python & Folium - Creating Maps from Raw CSV Data
This video tutorial shows how to create Folium maps from raw CSV data in Python. The video covers loading the data into Python, transforming the data, extracting latitude/longitude, and creating interactive maps based on the data.
Retrieving Data from the Twitter API and Mapping the Coordinates of Tweets
Here is the author's code: .. code-block:: python from TwitterAPI import TwitterAPI import matplotlib.pyplot as plt import descartes import geopandas as gpd import json consumer_key= "" consumer_secret= "" access_token_key= "" access_token_secret= "" api = TwitterAPI(consumer_key, consumer_secret, access_token_key, access_token_secret) response = api.request("statuses/filter", {"track": ["harry", "and", "meghan"]}) tweets = response.get_iterator() coordinates = [] count = 0 while count < 100: tweet = next(tweets) if "place" in tweet and tweet["place"] != None: location = tweet["place"]["bounding_box"]["coordinates"][0][0] coordinates.append(location) count += 1 world_map = gpd.read_file("C:/Users/Brian/Desktop/TM_WORLD_BORDERS_SIMPL-0.3/TM_WORLD_BORDERS_SIMPL-0.3.shp") fig, ax = plt.subplots(figsize = (15, 15)) world_map.plot(ax=ax) for x, y in coordinates: plt.scatter(x, y, marker="o", c="red") plt.savefig("map.png")Readings
Getting longitude-latitude coordinates for a (long) list of cities using Python and a free API. https://amaral.northwestern.edu/blog/getting-long-lat-list-cities
API’s and Natural Language Processing in Open Source Python https://www.earthdatascience.org/courses/use-data-open-source-python/intro-to-apis/
JSON Parser- https://jsonformatter.org/json-parser