Calling REST APIs and Extracting JSON Fields

Hello! In the previous lesson, you summarized request logs with pandas and joined those summaries to another table. Those logs were already available as rows. In this lesson, you’ll retrieve data from a web service instead: make a REST API call, check its response, and extract the fields you need from JSON.

Last time I pointed ahead to SQL; before that, we need this remaining Python workflow. Allow about 35–40 minutes to watch, read, and run the examples.

What comes back from an API call?

A REST API exposes addresses called endpoints. Your program sends an HTTP request to an endpoint, and the server returns a response. Here we’ll use GET, the method for retrieving data. A response has a status code, which reports what happened, and may have a body, which contains the requested data.

Watch two portions of Tech With Tim’s Master Python Requests In 15 Minutes. Call Any API. The first establishes what belongs to a request versus a response; the second makes a Python request and reads a field from the returned JSON.

The video uses a public demonstration API at https://jsonplaceholder.typicode.com/posts/1. The final /1 identifies the post being requested. You do not need credentials for this example, though real services may require them.

Make the call, then decode the body

Use the isolated Python environment you set up earlier. If requests is not installed in it, run python -m pip install requests; if this is a project you intend to keep, add the dependency to your pinned project dependencies as you learned earlier.

Create api_demo.py:

import requests

url = "https://jsonplaceholder.typicode.com/posts/1"

response = requests.get(url, timeout=10)
response.raise_for_status()

post = response.json()
print(post["id"])
print(post["title"])

requests.get() returns a Response object, not the post dictionary itself. raise_for_status() stops the script with an HTTP error if the server reports a client or server failure, such as 404 or 500. Only then does json() decode the response body into Python data. The timeout prevents the request from waiting indefinitely if the service does not respond.

Read the Requests Quickstart documentation for the two checks behind this sequence: decoding JSON and checking HTTP status. You can concentrate on the indicated subsections rather than the rest of the page.

JSON is a data format; response.json() converts it to familiar Python structures. A JSON object becomes a dictionary, and a JSON array becomes a list. For this endpoint, the response is expected to be an object with keys such as userId, id, title, and body. Thus post["title"] selects one field; it does not make another network request.

There are two separate ways this sequence can fail. A server can return an error status with valid JSON describing the error, so successful decoding is not enough. Conversely, an HTTP request can have a successful status but a body that is empty or not valid JSON; then response.json() raises a decoding error. Checking status and decoding the body answer different questions.

Extract fields from nested JSON

Not every response is as flat as a post. The following image shows the structure of an example order document—not a response from the posts endpoint. Boxes labeled object contain named fields; line_items is an array whose entries are objects.

Suppose an API returned an order with that structure. After decoding, a small example could look like this:

order = {
    "orderId": 731,
    "customer": {"custId": "C42", "name": "Avery"},
    "line_items": [
        {"sku": "PEN-01", "qty": 2, "unitPrice": 1.5},
        {"sku": "PAD-02", "qty": 1, "unitPrice": 4.0},
    ],
}

Use a string key to enter a dictionary and a numbered index to select an item from a list:

print(order["customer"]["custId"])       # C42
print(order["line_items"][0]["sku"])      # PEN-01

skus = [item["sku"] for item in order["line_items"]]
print(skus)                               # ['PEN-01', 'PAD-02']

The [0] selects the first line item, not a field literally named "0". If you need every item, iterate through the list rather than assuming there is only one. Also, [0] fails for an empty list.

Square-bracket access is appropriate when a field is required: a missing key raises KeyError, making a mismatch with the expected response visible. For a genuinely optional field, order["customer"].get("contact") returns None when it is absent. Do not substitute .get() for every required field; that can conceal a changed or malformed response.

Put the workflow in a reusable function

Your earlier pandas scripts worked with named columns such as request_id and model. Here you can similarly choose a small set of names for the data you extract, instead of passing an entire API response through the rest of a program.

Replace the first version of api_demo.py with:

import requests


def fetch_post_summary(post_id):
    url = f"https://jsonplaceholder.typicode.com/posts/{post_id}"

    response = requests.get(url, timeout=10)
    response.raise_for_status()
    post = response.json()

    if not isinstance(post, dict):
        raise ValueError("Expected a JSON object for one post")

    return {
        "post_id": post["id"],
        "author_id": post["userId"],
        "title": post["title"],
    }


try:
    print(fetch_post_summary(1))
except (requests.exceptions.RequestException, ValueError, KeyError) as exc:
    print(f"Could not load post: {exc}")

Read the function in order: it identifies the endpoint, retrieves the response, checks HTTP status, decodes JSON, checks that the result has the expected overall shape, and selects three fields. author_id is just a clearer name for the API’s userId; it is an identifier, not an author’s name.

The exception handler keeps a failed live request from producing an unexplained crash in this small script. RequestException covers request-related failures, including HTTP errors and timeouts, as well as Requests’ JSON decoding error. ValueError covers the explicit shape check, and KeyError identifies a missing required field. In a larger application, you would decide whether to report, retry, or pass each failure to the caller; the important point here is not to treat a failed call as valid post data.

Takeaways

A REST GET request retrieves a response; it does not directly return a Python dictionary. Check the HTTP result with raise_for_status(), decode a JSON body with response.json(), and then use dictionary keys and list indices according to the response’s structure. Select the fields your program needs, and keep missing fields or failed requests visible rather than silently accepting them.

Next, you’ll use pytest to test Python functions—including extraction logic—without depending on a live API being available for every test.