Skip to content

Data Collection: APIs

65 min Real data Needs: Web Scraping

An API is a service that agrees to give you data in a format you can rely on. After the last page, this will feel like a holiday.

Why this matters

Scraping is what you do when nobody provided a way in. An API is that way in. The data arrives as data rather than as layout, the fields have names, and the thing does not break because somebody changed a CSS class.

Most of your working life will involve calling APIs: a model provider, a payment system, an internal service at your own company. The three things this page covers, pagination, rate limits and messy fields, are the three things that turn a five line example into real code.

What an API gives you

Someone published an interface and promised it will behave a certain way. You send a request to an address, you get back JSON, and the shape of that JSON is documented rather than guessed.

Read the documentation first. Ten minutes there saves an hour of experiments, and it will tell you the things you cannot discover by poking: how pagination works, what the limits are, whether you need a key.

The parts of a request and a response A URL is made of a scheme, a host, a path and a query string. The request carries a method and headers. The response carries a status code, headers and a JSON body. https:// scheme swapi-api.hbtn.io host /api/starships/ path ?page=2 query the request GET, the only verb you need here params, your query string headers, who you are and what format you want back the response status code, did it work headers, including your rate limit body, the JSON you came for Beginners read only the body. The status code and the headers are where the useful information about failure lives.

A URL carries most of what you are asking for. The scheme and host say where, the path says what kind of thing, and the query string narrows it down. In requests you rarely build that string yourself.

import requests

response = requests.get(
    "https://swapi-api.hbtn.io/api/starships/",
    params={"page": 2},
    timeout=10,
)
print(response.url)
print(response.status_code)

Passing params as a dictionary rather than gluing the string together handles the encoding for you, which matters the moment a value contains a space or an accented character.

Use the base URL the task gives you

The original Star Wars API has moved more than once. Use whichever address the project specifies, and define it once at the top of your file as a constant so that changing it is a one line edit.

Reading the response

Two parts matter and beginners read only one of them.

What the four families of status code mean Two hundreds mean success. Three hundreds are redirects and are followed automatically. Four hundreds mean your request was wrong. Five hundreds mean the server failed and you should retry later. 2xx it worked use the body 200 OK 3xx moved requests follows it for you 301, 302 4xx your mistake fix the request 404, 401, 403, 429 5xx their problem wait and retry 500, 503 response.raise_for_status() turns everything in the last two columns into an exception. Sometimes that is what you want. Sometimes you need to read the code yourself and act on it. 

if response.status_code == 200:
    data = response.json()

.json() parses the body into Python dictionaries and lists. There is no need for the json module unless you are handed a string rather than a response.

You have two ways of dealing with failure and they suit different situations.

response.raise_for_status()

That turns any 4xx or 5xx into an exception, which is right when you want the script to stop. But the GitHub task needs the opposite: a 404 should print Not found and a 403 should tell you when the limit resets. When the status code is information you need to act on, read it yourself.

if response.status_code == 404:
    print("Not found")
elif response.status_code == 403:
    print("rate limited")
else:
    data = response.json()

Working with JSON

JSON becomes dictionaries, lists, strings, numbers, booleans and None. You already know how to walk that, from the Python warm up.

data = response.json()
print(data["count"])
for ship in data["results"]:
    print(ship["name"], ship["passengers"])

The habit that saves you: look at one record before writing any logic.

from pprint import pprint

pprint(data["results"][0])

And use .get() when a field may be absent, because a missing key raises while a missing value does not.

homeworld = species.get("homeworld")

Some SWAPI species have null for homeworld. species["homeworld"] gives you None there, and calling requests.get(None) fails in a way that takes a while to trace back to its cause.

Pagination

Almost no API gives you everything at once. It hands you a page and tells you where the next one is.

How a paginated JSON response is shaped Each page is an envelope holding a count, a link to the next page, and an array of results. You follow the next link until it is null. page 1 "count": 82 "next": ".../?page=2" "results": [ ... ] 10 of 82 collected page 2 "count": 82 "next": ".../?page=3" "results": [ ... ] 20 of 82 collected page 9 "count": 82 "next": null "results": [ ... ] 82 of 82 collected dots stand for the pages in between The API tells you where the next page is. Read that field rather than counting pages yourself, because the number of pages changes whenever the data does. stop when next is null 

def fetch_all(url):
    """Follow the next links and return every result."""
    results = []
    while url:
        payload = requests.get(url, timeout=10).json()
        results.extend(payload["results"])
        url = payload.get("next")
    return results

Four lines, and it is the whole of pagination for this style of API. The loop ends by itself because next is null on the last page, and while None is false.

Note what is not there: no page counting, no guessing how many there are. If the data grows tomorrow, this code still works.

Not every API paginates the same way

This one gives you a full URL for the next page. Others give you a page number you have to build the URL from, or a cursor token, or a Link header. The documentation tells you which. The concept is identical: ask, collect, ask again, stop when the API says there is no more.

The data is messier than you expect

This is the part of the objectives that sounds innocuous and is not. Here is what SWAPI actually returns for a starship's passenger capacity:

"843"
"n/a"
"unknown"
"30-165"
"1,000"

Every one of those is a string. Two of them are not numbers at all, one is a range, and one has a thousands separator that will break int().

So the first task is not really about pagination. It is about deciding what to do with each of those cases before you can compare anything to passengerCount.

def parse_passengers(raw):
    """Turn a passengers field into a number, or None if it is not one."""
    if raw is None:
        return None
    cleaned = raw.replace(",", "").strip().lower()
    if cleaned in ("n/a", "unknown", ""):
        return None
    if "-" in cleaned:
        cleaned = cleaned.split("-")[0]
    return int(cleaned) if cleaned.isdigit() else None

Returning None rather than zero is deliberate. A ship whose capacity is unknown is not a ship that carries nobody, and quietly turning one into the other is how wrong answers get produced confidently.

ML connection

You have now met this three times: missing values in pandas, missing data in Fundamentals, and unknown fields here. The pattern is always the same. An absent value is not a zero, and deciding what it means is a judgment you should make on purpose and write down.

Then the actual task becomes short.

def availableShips(passengerCount):
    """Return the names of ships that can carry at least passengerCount."""
    ships = []
    for ship in fetch_all("https://swapi-api.hbtn.io/api/starships/"):
        capacity = parse_passengers(ship.get("passengers"))
        if capacity is not None and capacity >= passengerCount:
            ships.append(ship["name"])
    return ships

The second task is the same shape with a different filter. A species counts as sentient if the word appears in either classification or designation, so check both, and lowercase them before comparing because the data is not consistent about case.

One request per item adds up

The sentient planets task has a trap in it. Each species carries its homeworld as a URL rather than a name, so you have to fetch it. Do that naively and you send one request per species, including for planets you already looked up.

Requesting the same thing over and over Without a cache, every species triggers its own request for a home planet, including planets already fetched. With a cache, each planet is fetched once and reused. without a cache with a cache species 1 Tatooine species 2 Naboo species 3 Tatooine species 4 Kashyyyk species 5 Naboo species 6 Tatooine species 1 species 2 species 3 species 4 species 5 species 6 Tatooine Naboo Kashyyyk 6 requests 3 requests, 3 cache hits Across 37 species the saving is far larger, and it is the difference between finishing and hitting a rate limit. 

def planet_name(url, cache):
    """Fetch a planet name, remembering ones already seen."""
    if url is None:
        return None
    if url not in cache:
        cache[url] = requests.get(url, timeout=10).json()["name"]
    return cache[url]

A dictionary is all the cache you need. Pass the same one through your loop and every repeated homeworld becomes free.

This matters more than the time saved. Every avoided request is one you have not spent from your rate limit, and rate limits are the subject of the next section.

Rate limits

An API decides how many requests you may send in a period. Exceed it and you get refused, usually with 403 or 429.

A rate limit budget over time Each request lowers the remaining allowance. When it reaches zero the API refuses with a 403 until the reset time, when the allowance is restored. X-RateLimit-Remaining 60 0 403 every request refused X-RateLimit-Reset allowance restored time The reset time arrives as a Unix timestamp in seconds. Subtract the current time to get how long you must wait. Read Remaining as you go, and slow down before you hit zero rather than after.

GitHub tells you exactly where you stand, in the response headers.

print(response.headers.get("X-RateLimit-Limit"))
print(response.headers.get("X-RateLimit-Remaining"))
print(response.headers.get("X-RateLimit-Reset"))

Reset is a Unix timestamp, meaning seconds since the start of 1970. To turn it into something a person can read, subtract the current time.

import time

def minutes_until_reset(response):
    """How many minutes until the rate limit is restored."""
    reset_at = int(response.headers.get("X-RateLimit-Reset", 0))
    return round((reset_at - time.time()) / 60)

That is the third task. When the status is 403, print Reset in X min using this number.

#!/usr/bin/python3
"""Print the location of a GitHub user."""
import sys
import time
import requests

if __name__ == '__main__':
    response = requests.get(sys.argv[1])
    if response.status_code == 404:
        print("Not found")
    elif response.status_code == 403:
        reset_at = int(response.headers.get("X-RateLimit-Reset", 0))
        print("Reset in {} min".format(round((reset_at - time.time()) / 60)))
    else:
        print(response.json().get("location"))

The if __name__ == '__main__': guard is required by the task and you met it in the Python warm up. It means the file can be imported without running the script, which is what the checker relies on.

Not hitting the limit in the first place

Three habits, in order of how much they help.

Cache, as above. The cheapest request is one you do not send.

Slow down on purpose. A short sleep between requests keeps you well inside almost any limit.

Back off when refused. Do not retry immediately, and do not retry at a fixed interval. Double the wait each time.

def get_with_backoff(url, attempts=5):
    """Retry with a doubling delay when the server pushes back."""
    delay = 1
    for _ in range(attempts):
        response = requests.get(url, timeout=10)
        if response.status_code not in (429, 500, 502, 503):
            return response
        time.sleep(delay)
        delay *= 2
    return response

Waiting 1, 2, 4, 8 seconds gives a struggling server room to recover. Retrying every half second gives it more of what caused the problem.

Keys and secrets

Many APIs want a key. GitHub works without one at 60 requests an hour, and with one at 5000, which is the difference between finishing the task and waiting.

A key goes in a header:

headers = {"Authorization": "Bearer " + token}
response = requests.get(url, headers=headers, timeout=10)

Where the token does not go: in your code, in your notebook, or anywhere inside your repository. Put it in a file called .env, and put .env in .gitignore.

echo "GITHUB_TOKEN=ghp_yourtokenhere" > .env
echo ".env" >> .gitignore

import os

token = os.environ.get("GITHUB_TOKEN")

A committed key is a public key

This is the same lesson as the Git tutorial, where we did not put a token in a clone URL. Deleting a secret in a later commit does not remove it: it is still in the history, and bots scan public repositories for exactly this within minutes. If it happens, revoke the key immediately rather than trying to rewrite history.

From API to DataFrame

Once you have the records, pandas takes over.

import pandas as pd

ships = fetch_all("https://swapi-api.hbtn.io/api/starships/")
df = pd.DataFrame(ships)
print(df[["name", "passengers", "cost_in_credits"]].head())

When the JSON is nested, json_normalize flattens it into columns.

df = pd.json_normalize(records, sep="_")

Save the raw response before you transform anything.

import json

with open("starships_raw.json", "w") as f:
    json.dump(ships, f)

That file means the next twenty times you change your parsing logic you do not touch the network. It also means you can prove later what the API actually said, which matters when your results and somebody else's disagree.

When it goes wrong

JSONDecodeError

The body was not JSON. Usually an error page or a redirect to a login. Print response.status_code and the first 200 characters of response.text and you will see what it really sent.

KeyError: 'results'

The response is not shaped the way you assumed, often because the request failed and you parsed the error body. Check the status code before reading fields.

TypeError: '>=' not supported between instances of 'str' and 'int'

You compared a field straight from the JSON to a number. Almost every numeric-looking field in SWAPI is a string. Parse it first.

403 Forbidden on GitHub

Either you are rate limited, in which case the reset header tells you when, or the endpoint needs authentication. X-RateLimit-Remaining at zero distinguishes the two.

My loop never ends

You are re-requesting the same page. Make sure you assign the next URL from the payload rather than incrementing something yourself, and print the URL each time round while you are debugging.

It works for me and fails for the checker

Often a trailing slash, an assumed page count, or a hard-coded number of results. Follow the links the API gives you and do not assume the data is the same size it was when you wrote the code.

Check yourself

  1. Request one page of starships and print the status code, the number of results, and the value of next.
  2. Write fetch_all and confirm the total matches the count field on page one.
  3. List every distinct value that appears in the passengers field across all ships. Decide what each one should become, and say why.
  4. Implement availableShips(passengerCount) and check that an impossible number returns an empty list rather than raising.
  5. Implement sentientPlanets() with a cache. Count how many requests you sent with it and without it.
  6. Call the GitHub API for a user that does not exist, and for one that does. Print X-RateLimit-Remaining each time and watch it fall.
  7. Save a raw response to disk and rebuild your DataFrame from the file with no network connection.

Quick reference

Task Code
GET with parameters requests.get(url, params={"page": 2}, timeout=10)
Status code response.status_code
Raise on failure response.raise_for_status()
Parse the body response.json()
A header response.headers.get("X-RateLimit-Reset")
Send a token headers={"Authorization": "Bearer " + token}
Safe field access record.get("homeworld")
Next page url = payload.get("next")
Timestamp to minutes round((reset_at - time.time()) / 60)
Read a secret os.environ.get("GITHUB_TOKEN")
Records to a table pd.DataFrame(records)
Flatten nested JSON pd.json_normalize(records, sep="_")

Questions to sit with

  1. An API publishes a rate limit and a documented format. A scraper has neither. What does that difference say about what the owner expects from you?
  2. Fields like "n/a" and "unknown" and "30-165" all end up as None in your parser. What have you lost by treating them the same, and when would that matter?
  3. Caching means your results reflect the data as it was when you first fetched it. When is that a feature, and when is it a bug?
  4. Your script works because the API returns next links. What happens to everything you built if the provider changes that, and what would reduce the damage?

Next

You can collect data from pages and from services. Next, what to do in the first hour with a dataset you have never seen.

Data Preparation and Visualization

Worth your time: the requests documentation is one of the best written in Python, and GitHub's rate limit page explains the headers used in the third task.