API Gateway to Event-Driven Post-Processing: End-to-End Testing

1. The Anatomy of an End-to-End Flow

Before we run the test, let's visualize the journey of a request through our system. It's a hybrid flow, combining two distinct communication patterns that we've discussed throughout this course: request-response and event-driven.

Phase 1: The Synchronous Request

This is the immediate, blocking part of the interaction. The client needs a quick confirmation that their order has been accepted.

The sequence is:

1. A client sends an HTTP POST request to the /orders endpoint on our FastAPI API Gateway.
2. The gateway validates the request and makes a synchronous, unary gRPC call to the Order Service.
3. The Order Service performs its initial logic (e.g., creates an order entry with a "PENDING" status) and immediately returns a success response to the API Gateway.
4. The gateway translates this back into an HTTP 201 Created response for the client.

From the client's perspective, the interaction is now complete. But for our system, the work is just beginning.

Phase 2: Asynchronous Post-Processing

After sending the synchronous response, the Order Service publishes an OrderCreated event to a Redis Stream. This decouples the initial order creation from all the subsequent processing, making the system more resilient and scalable.

The complete flow, including both phases, looks like this:

Client ---HTTP POST /orders---> [API Gateway]
   |                                  |
   |                            gRPC Call
   |                                  |
   '---<--- HTTP 201 Created --- [Order Service] --publishes 'OrderCreated' event--> [Redis Stream]
                                                                                     |
                                                                                 (async)
                                                                                     |
                                           +-----------------------------------------+
                                           |                                         |
                                     [Inventory Svc]                           [Payment Svc]
                                     (consumes event,                           (consumes event,
                                      reserves stock)                              processes payment)

Now, let's trigger this entire chain reaction and watch it happen live.

2. Hands-On: A Live End-to-End Test

In this exercise, we will act as a client, send a request to create an order, and then become detectives, following the trail of that order through the system's logs, message broker, and database.

Step 1: Start the System

Open a terminal in the root of your microservices-platform project and start all the services in detached mode. The --build flag ensures we're using the latest version of your code.

docker-compose up --build -d

Confirm that all services are running and healthy.

docker-compose ps

You should see all your containers (api-gateway, user-service, order-service, etc.) in the Up state.

Step 2: Tail the Logs

To see the story unfold, we need to watch the logs of the key actors. Open four separate terminal windows. In each one, run one of the following commands:

Terminal 1 (API Gateway):

docker-compose logs -f api-gateway

Terminal 2 (Order Service):

docker-compose logs -f order-service

Terminal 3 (Inventory Service):

docker-compose logs -f inventory-service

Terminal 4 (Payment Service):

docker-compose logs -f payment-service

Arrange these windows so you can see them all at once.

Step 3: Trigger the Flow

Now, let's act as a client. In a fifth terminal, create a simple Python script named test_client.py to send the HTTP request to our running API Gateway.

# test_client.py
import requests

GATEWAY_URL = "http://localhost:8000/orders"




# This payload assumes your Order Service expects these fields.
# Adjust if your implementation is different.
order_data = {
    "user_id": "user-from-e2e-test",
    "items": [
        {"product_id": "laptop-x1", "quantity": 1},
        {"product_id": "monitor-z1", "quantity": 2}
    ]
}

try:
    response = requests.post(GATEWAY_URL, json=order_data)
    response.raise_for_status()  # Raises an HTTPError for bad responses (4xx or 5xx)

    print(f"✅ Success! Order created.")
    print(f"   Status Code: {response.status_code}")
    print(f"   Response: {response.json()}")

except requests.exceptions.RequestException as e:
    print(f"❌ Failed to create order: {e}")

Run the script:

python test_client.py

If everything in the synchronous path works, you will see a ✅ Success! message.

Step 4: Observe and Verify

This is the exciting part. As soon as you run the script, watch your four log terminals. You should see a cascade of activity:

1. API Gateway Log: You'll see an incoming POST /orders request, followed by a log message indicating it's making a gRPC call to the Order Service.
2. Order Service Log: It will log the receipt of the gRPC call and, crucially, a message like "Publishing OrderCreated event...".
3. Inventory Service Log: A moment later, it will spring to life, logging "Consumed OrderCreated event..." and a message about reserving stock.
4. Payment Service Log: Similarly, it will log that it has consumed the OrderCreated event and is initiating payment processing.

The trail doesn't end with the logs. Let's verify the state of the system.

Verify the Event in Redis:
Check if the OrderCreated event was actually written to the stream.

# Check the number of events in the stream
docker-compose exec redis redis-cli XLEN order_events




# Read the latest event from the stream
docker-compose exec redis redis-cli XREVRANGE order_events + - COUNT 1

Verify the Payment Record in PostgreSQL:
Connect to the database and see if the Payment Service created a record.

# Connect to the psql client
docker-compose exec postgres psql -U your_db_user -d payments_db




# Once inside the psql prompt, run a query:
SELECT * FROM payments ORDER BY created_at DESC LIMIT 1;




# Type \q to exit psql

You should see a new row in the payments table corresponding to the order you just created.

If you've successfully followed this trail from HTTP request to database record, congratulations! You have just performed a full end-to-end test of your microservices application.

3. Exploring a Reference Architecture: OrderShop v2

To solidify these concepts, let's explore a similar, publicly available reference application from Redis. This will show you the same patterns we're using in a slightly different implementation, which is excellent for broadening your understanding.

4. From Manual to Automated Testing

Our manual test was invaluable for understanding the flow, but it's not scalable for a real project. In a production environment, you would automate this process.

An automated end-to-end test would typically be a script (e.g., using pytest) that:

1. Programmatically starts the entire environment using Docker Compose.
2. Sends the HTTP request, just like our test_client.py script.
3. Waits and polls the system to verify the expected side effects (e.g., checks the database for a new record, queries another service's API for a status update).
4. Asserts that the final state of the system is correct.
5. Tears down the environment.

This automates the entire "observe and verify" process we just did by hand.

Conclusion

In this lesson, you validated the architecture you have painstakingly built over this entire module. You saw firsthand how a single client request can trigger a complex but predictable workflow that spans multiple services, combining both synchronous and asynchronous communication.

Key Takeaways:

• An end-to-end flow in microservices often has a fast, synchronous phase (for user feedback) and a slower, robust, asynchronous phase (for backend processing).
• Testing this flow involves stimulating the system at its entry point (the API Gateway) and verifying the effects across all involved components: logs, message queues, and databases.
• Orchestration tools like Docker Compose are fundamental for creating a realistic, isolated, and repeatable testing environment for your entire microservices system.

Preview of the Next Lesson

Our system now works perfectly... in a perfect world. But distributed systems live in an imperfect world where networks fail and services crash. In the final lesson of this module, "Demonstrate system resilience by simulating a service failure," we will move from testing for success to testing for failure. We will intentionally shut down a critical service mid-flow and observe the consequences, setting the stage for building more resilient, fault-tolerant systems.