Making API Requests with `requests`

What is an HTTP Request?

At its core, an HTTP request is a message a client (like your Python script) sends to a server to ask for information or to perform an action. You're already familiar with this from web browsing: typing a URL and hitting Enter sends an HTTP GET request to retrieve a webpage.

When we build agents, we use these requests programmatically to talk to APIs (Application Programming Interfaces). An API is a set of rules and endpoints that a server exposes for other programs to interact with.

For our purposes, the two most important HTTP methods are:

• GET: Used to retrieve data from a server.
• POST: Used to send data to a server to create or update a resource.

Python has built-in ways to handle this, but the third-party requests library has become the standard due to its simplicity and power.

Making Your First Request: GET

Let's start by fetching some data. The GET request is the simplest type, used purely to retrieve information. We'll use the requests library to do this.

The following reading from Real Python is an excellent guide. We'll walk through it step-by-step.

Once you have requests installed in your virtual environment, you can make a GET request with a single line of code. Let's try hitting the base endpoint of the GitHub API:

import requests

# The URL of the API endpoint we want to access
url = "https://api.github.com"

# Make the GET request
response = requests.get(url)

# The 'response' object contains all the information from the server
print(response)

If you run this, you'll see something like <Response [200]>. This tells you that you received a Response object and the request was successful (indicated by the 200 status code). Now, let's see what's inside that object.

Inspecting the Server's Response

The Response object is a treasure trove of information. The three most critical pieces are the status code, the headers, and the body (content).

1. Status Code: A number indicating the outcome of the request. 200 means success, 404 means not found, 500 means server error, etc.
2. Headers: Metadata about the response, like the content type (application/json, text/html).
3. Body: The actual data returned by the server. This could be HTML, JSON, an image, etc.

This next reading section shows you how to access all this information.

Let's expand our previous example to properly inspect the response. The .json() method is particularly powerful because it takes the raw JSON text from the response body and directly converts it into a Python dictionary for you.

import requests
import json # Import for pretty-printing

url = "https://jsonplaceholder.typicode.com/todos/1"

try:
    response = requests.get(url)

    # Check if the request was successful (status code 200-399)
    response.raise_for_status() # This will raise an exception for bad status codes

    print(f"Status Code: {response.status_code}")
    
    # Get the response body as a Python dictionary
    data = response.json()
    
    print("\n--- Response JSON ---")
    # Use the json library to pretty-print the dictionary
    print(json.dumps(data, indent=2))
    
    # Now you can work with the data as a normal Python dict
    print(f"\nTask Title: {data['title']}")
    print(f"Is completed: {data['completed']}")

except requests.exceptions.HTTPError as err:
    print(f"An HTTP error occurred: {err}")
except requests.exceptions.RequestException as err:
    print(f"An error occurred: {err}")

In this example, we use jsonplaceholder, a free fake API perfect for testing. Notice how response.json() gives us a dictionary we can immediately start working with, just like we practiced in the previous lessons.

Sending Data to the Server: POST

While GET is for reading data, POST is for writing it. This is what your agent will use to submit forms, create resources, or send instructions to an LLM.

When you send a POST request, you include a "payload" in the body of the request. Thanks to our last lesson, you know exactly how to prepare this payload: as a Python dictionary that requests can automatically convert to JSON.

When you pass a dictionary to the json parameter of requests.post(), the library does two things for you:

1. Serializes your dictionary into a JSON string.
2. Sets the Content-Type header to application/json, telling the server what kind of data you're sending.

Let's try creating a new "todo" item.

import requests
import json

url = "https://jsonplaceholder.typicode.com/posts"

# This is our payload, created as a Python dictionary.
# The agent could generate this dictionary based on its goals.
new_post_data = {
    "title": "My Agent's First Post",
    "body": "This is the content of the post.",
    "userId": 101 # A made-up user ID for this example
}

try:
    # We use requests.post() and pass our dictionary to the 'json' parameter
    response = requests.post(url, json=new_post_data)
    
    response.raise_for_status()
    
    # The server often responds with the created object, including its new ID
    print(f"Status Code: {response.status_code}") # Should be 201 Created
    
    returned_data = response.json()
    
    print("\n--- Server Response ---")
    print(json.dumps(returned_data, indent=2))
    
    # Notice the server assigned an 'id'
    print(f"\nNew post created with ID: {returned_data['id']}")

except requests.exceptions.RequestException as err:
    print(f"An error occurred: {err}")

A 201 Created status code is the standard successful response for a POST request that results in a new resource being created.

Customizing Requests: Parameters and Headers

Sometimes you need to provide more information than just a URL and a body.

• Query Parameters: For GET requests, you can filter or specify data using parameters in the URL (e.g., ?sort=stars&order=desc). The requests library lets you provide these as a dictionary to the params argument.
• Headers: You can provide custom HTTP headers to do things like specify the data format you accept or, crucially, provide authentication credentials.

Here is a conceptual example of how you will use a header for authentication when we get to the LLM module. You don't need to run this, but see how the API key is passed in the headers.

# Conceptual example for a future lesson
api_key = "YOUR_SECRET_API_KEY"
url = "https://api.openai.com/v1/chat/completions"

headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json"
}

payload = {
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Hello!"}]
}

# The headers dictionary is passed to the request
# response = requests.post(url, headers=headers, json=payload)

import requests
import json

# --- Part 1: GET posts for a user ---
posts_url = "https://jsonplaceholder.typicode.com/posts"
params = {"userId": 5}

try:
    print("Fetching posts for userId=5...")
    response_get = requests.get(posts_url, params=params)
    response_get.raise_for_status() # Check for errors

    posts_data = response_get.json()
    
    # --- Part 3: Print the title of the first post ---
    if posts_data:
        first_post = posts_data[0]
        first_post_id = first_post['id']
        print(f"Title of first post: '{first_post['title']}'")
        print(f"Post ID is: {first_post_id}")
        
        # --- Part 4 & 5: POST a new comment ---
        comments_url = "https://jsonplaceholder.typicode.com/comments"
        new_comment_payload = {
            "postId": first_post_id,
            "name": "Test User",
            "email": "test@example.com",
            "body": "This is a test comment from my script."
        }
        
        print("\nPosting a new comment...")
        response_post = requests.post(comments_url, json=new_comment_payload)
        
        # --- Part 6: Check result and print new ID ---
        if response_post.status_code == 201:
            print("Comment created successfully!")
            new_comment = response_post.json()
            print(f"New comment ID from server: {new_comment['id']}")
            print("--- Comment details ---")
            print(json.dumps(new_comment, indent=2))
        else:
            print(f"Failed to create comment. Status: {response_post.status_code}")

    else:
        print("No posts found for that user.")

except requests.exceptions.RequestException as err:
    print(f"An error occurred: {err}")

Conclusion

You've just learned one of the most essential skills in modern programming: communicating with APIs over the web. This is the foundation upon which your agent will perceive and act upon the digital world.

Here are the key takeaways from this lesson:

• requests is the standard library for making HTTP requests in Python due to its simplicity.
• requests.get(url, params=...) is used to retrieve data, with params allowing you to filter results.
• requests.post(url, json=...) is used to send data to a server to create or modify a resource. Passing a dictionary to the json parameter is the easiest way to send JSON payloads.
• The Response object contains the server's full reply. You can check the outcome with .status_code and get the data with .json().
• Headers are used to send metadata, including critical authentication tokens which we will use extensively later.

In this lesson, we optimistically assumed our requests would work. However, in the real world, networks fail, servers have issues, and APIs change. Our agent needs to be able to handle this. In the very next lesson, we will implement robust error handling for network operations using try-except blocks, making our agent far more resilient.