Checking Web Services with Curl

1. What is curl? Basic Requests and Headers

curl stands for "Client URL". It's a powerful command-line tool that can transfer data to or from a server using dozens of protocols. We'll focus on its most common use: making HTTP and HTTPS requests, just like a web browser does, but with surgical precision.

At its simplest, curl fetches the content of a URL and displays it in your terminal.

Open your terminal and try it:

curl example.com

You'll see the HTML source code of the example.com homepage. This simple command already confirms that the web server at example.com is available and serving content.

But a web response is more than just the body (the HTML). It also includes headers, which contain crucial metadata. Let's watch a short video that introduces the most fundamental curl commands for fetching content and inspecting headers.

As the video demonstrated, you have several options for inspecting responses:

• curl <url>: Fetches only the response body (e.g., HTML, JSON).
• curl -I <url> (--head): Fetches only the response headers. This makes a HEAD request, which is a lightweight way to check a server's status and metadata without downloading the full content.
• curl -i <url> (--include): Fetches both the headers and the body. The headers are printed first, followed by a blank line, and then the body.
• curl -L <url> (--location): Follows redirects. If you request http://google.com, the server will respond with a redirect (like 301 Moved Permanently) to https://www.google.com. The -L flag tells curl to automatically make a new request to the new location.

The very first line of the headers is the most important for checking availability: the HTTP status line.
HTTP/2 200
This tells you the protocol version (HTTP/2) and the status code (200). A 200 OK status means your request was successful! Other common codes include 301 (redirect), 404 Not Found, and 500 Internal Server Error.

2. Deeper Inspection: Verbose Mode and SSL

To effectively debug a web service, you need to see the entire conversation: the request that curl sends and the response the server gives back.

Verbose Mode (-v)

The -v (--verbose) option is your best friend for troubleshooting. It prints a detailed log of the connection, including the request headers curl sends (prefixed with >) and the response headers the server returns (prefixed with <).

Let's see what the creator of curl has to say about using verbose mode for debugging.

Handling SSL/TLS Certificates (-k)

When you're developing locally, you might set up a web server (like Nginx, which we'll cover later) with a "self-signed" SSL certificate. curl, like your browser, will refuse to connect to a server with a certificate it doesn't trust. This is a security feature, but it gets in the way during testing.

The -k (--insecure) flag tells curl to bypass certificate validation.

3. curl for the Full Stack Developer: Interacting with APIs

So far, we've been acting like a browser, fetching web pages. But as a developer, you'll more often use curl to act like a client application, talking to a REST API. This usually involves sending data using different HTTP methods, like POST.

• -X <METHOD>: Specifies the HTTP method (e.g., -X POST, -X PUT, -X DELETE). If omitted, curl defaults to GET.
• -d '{"key":"value"}' (--data): Sends data in the request body. This is used for POST and PUT requests.
• -H "Header-Name: Header-Value" (--header): Adds a custom header to your request. For modern APIs, Content-Type: application/json is one of the most common headers you'll need to send.

Let's look at an example of curl sending data to a test API.

To put these pieces together for API development, let's consult a reference guide.

4. Do-It-Yourself: Hands-On with curl

Time to put theory into practice. These exercises use httpbin.org, a fantastic free service designed for testing HTTP clients.

Task 1: Inspect a Website

1. Get headers only: Check the response headers from the Fedora Project's website. What is the server? What is the content-type?

   curl -I https://fedoraproject.org
   

2. Follow redirects: Use curl to see the redirect chain from the insecure http:// version of google.com.

   curl -I -L http://google.com
   

   The -I and -L flags can be combined. You should see at least two sets of headers: one for the initial 301 redirect and one for the final 200 OK.

Task 2: Interact with a Test API

1. Make a GET request: httpbin.org/get is an endpoint that inspects your incoming request and echoes it back to you as a JSON response. The -s flag silences the progress meter for a cleaner output.

   curl -s https://httpbin.org/get
   

   If you have jq installed, you can pipe the output to it for pretty-printing: curl -s https://httpbin.org/get | jq

2. Make a POST request: Now, send some JSON data to the post endpoint.

   curl -X POST \
     -H "Content-Type: application/json" \
     -d '{"user":"test-user", "status":"learning-curl"}' \
     https://httpbin.org/post
   

   Examine the JSON response. Look for the json field. Does it contain the data you sent? This is exactly how you would test if your Django API is correctly receiving data.

Task 3: Troubleshoot a Secure Connection

1. Attempt a connection: Try to connect to a site that uses a self-signed certificate.

   curl https://self-signed.badssl.com/
   

   You should get an error message about the SSL certificate.

2. Investigate and bypass: Now, use verbose mode (-v) to see the details of the failed SSL handshake, and use the insecure flag (-k) to bypass the error and fetch the page content.

   curl -vk https://self-signed.badssl.com/
   

   This combination is invaluable for debugging local HTTPS setups.

Conclusion

In this lesson, you've gained a powerful tool for your developer toolkit. You've moved beyond simply checking if a server is online and are now able to have a detailed conversation with the web services running on it.

Key Takeaways:

• curl is a versatile command-line tool for making HTTP requests to test web service availability and functionality.
• You can inspect headers with -I (headers only) or -i (headers + body), and you should always check the HTTP status code first.
• Always use -L to follow redirects when testing public websites.
• -v (verbose) is essential for debugging, showing you the exact request sent and response received.
• For API development, you'll frequently use -X POST, -H "Content-Type: application/json", and -d to send data.
• -k allows you to work with services using self-signed certificates in trusted development environments.

You can now test if a web server is up and responding correctly. But what if the server is fine, but your system can't even find its IP address? In our next lesson, we'll dive into dig, a tool specifically designed to query the Domain Name System (DNS) and troubleshoot name resolution issues.