Simulating Service Failures for Resilience

1. Why Break Things on Purpose? The Principles of Chaos Engineering

In a monolithic application, failure is often an all-or-nothing event. In a distributed system, with dozens of services communicating over an unreliable network, partial failure is the normal state of operation. A single service can fail due to a bug, a hardware issue, or a network partition. Understanding how these small failures impact the broader system is one of the most critical aspects of microservice architecture.

To get into the right mindset, let's hear from the pioneers at Netflix, who popularized this discipline.

The practice of intentionally testing for failure is called Chaos Engineering. It's not about causing random destruction; it's a scientific method for uncovering weaknesses.

Today's experiment will follow these steps:

1. Define Steady State: Establish a baseline of normal, successful system behavior.
2. Formulate a Hypothesis: Make an educated guess about what will happen when we introduce a specific failure.
3. Inject Failure: Introduce a controlled failure into the system.
4. Observe and Verify: Measure the impact of the failure and compare it to your hypothesis.
5. Learn and Improve: Use the findings to make the system more resilient.

2. Experiment: Simulating a Payment Service Outage

For our experiment, we will simulate an outage of the Payment Service. This is an interesting choice because it's part of the asynchronous workflow, not the initial, user-facing request.

Step 1: Define Steady State & Formulate Hypothesis

• Steady State: Our steady state is the successful end-to-end flow we verified in the last lesson. A client sends a request, gets a quick 201 Created response, and behind the scenes, the order event is processed by both the Inventory and Payment services, resulting in a new record in the payments database.

• Hypothesis: What do we predict will happen when the payment-service is down?

  • Synchronous Flow: The initial POST /orders request to the API Gateway should still succeed. The Order Service doesn't talk to the Payment Service directly; it only publishes an event to Redis. The client should be unaware of any problem.
  • Asynchronous Flow: The OrderCreated event will be published to the order_events stream in Redis. The Inventory Service should consume it and work normally. The Payment Service, being offline, will not consume the event.
  • System State: After the experiment, we expect to see the event in Redis, but no corresponding entry in the payments database table.
  • Recovery: When we bring the payment-service back online, our consumer group logic should allow it to read the "pending" message from the stream and process the payment, achieving eventual consistency.

Let's test this hypothesis.

Step 2: Prepare and Run the Experiment

First, ensure your environment is clean and running.

1. Start the System: In the root of your project, run:

   docker-compose up --build -d
   

2. Monitor the Logs: Just like last time, open separate terminal windows to tail the logs of our key services. This is our observation deck.

   
   
   
   # Terminal 1
   docker-compose logs -f api-gateway
   
   
   
   
   # Terminal 2
   docker-compose logs -f order-service
   
   
   
   
   # Terminal 3
   docker-compose logs -f inventory-service
   
   
   
   
   # Terminal 4 (This one will be quiet for a while)
   docker-compose logs -f payment-service
   

3. Inject the Failure: Now, let's take down the Payment Service. In a new terminal, run:

   docker-compose stop payment-service
   

   Verify that it has stopped by running docker-compose ps. You should see its state as Exited.

4. Trigger the Flow: With the Payment Service down, run the test_client.py script from the previous lesson to create an order.

   python test_client.py
   

Step 3: Observe and Verify the Impact

Carefully examine your logs and the state of the system.

1. Client & Synchronous Logs: Your test_client.py script should report ✅ Success!. The logs for api-gateway and order-service should show the normal, successful synchronous flow. This confirms the first part of our hypothesis: the user-facing interaction is unaffected.

2. Asynchronous Logs:

   • The inventory-service log should show that it consumed the OrderCreated event.
   • The payment-service log should remain silent. It's offline and cannot process the event.

3. System State Verification:

   • Check Redis: Confirm the message is waiting in the stream. The order_events stream should have one message, and the consumer group should have one pending message for the payment-consumer.

     
     
     
     # Check for pending messages in the group
     docker-compose exec redis redis-cli XPENDING order_events payment-group
     

     This command will show you that there is a message that has been delivered but not yet acknowledged.
   • Check the Database: Connect to the payments database and verify that no new payment record was created.

     docker-compose exec postgres psql -U your_db_user -d payments_db -c "SELECT * FROM payments ORDER BY created_at DESC LIMIT 5;"
     

     You should not see a record for the order you just placed.

Our observations match our hypothesis perfectly. The system has gracefully degraded.

Step 4: Simulate Recovery

Now, let's see if the system can heal itself. Bring the Payment Service back online.

docker-compose start payment-service

Immediately switch to the payment-service log terminal. You should see it spring to life! It will connect to Redis, find the pending message in its consumer group, and process the payment.

Finally, verify the database one last time:

docker-compose exec postgres psql -U your_db_user -d payments_db -c "SELECT * FROM payments ORDER BY created_at DESC LIMIT 5;"

The payment record should now be present. We have achieved eventual consistency.

3. Analysis: What Did We Learn?

This simple experiment reveals profound characteristics of our architecture:

• Decoupling works: By using a message broker (Redis Streams), the order-service (the producer) is completely decoupled from the payment-service (the consumer). The producer doesn't know or care if the consumer is online. This is the foundation of resilience in event-driven systems.
• Graceful Degradation: The system didn't crash. The most critical part of the user experience—placing the order—remained fully functional. The backend processing was delayed, not lost.
• Identifying Criticality: We've implicitly identified the payment-service as a non-critical component for the initial synchronous request. What would have happened if we stopped the order-service instead? The entire flow would have failed immediately. This makes the order-service part of the critical path.

To see a practical implementation of injecting a different kind of failure—latency—and observing its effects, this article on ToxiProxy is an excellent read.

Conclusion: A Foundation for Resilient Systems

Congratulations on completing not just this lesson, but the entire course! Today, you moved beyond testing for success and deliberately tested for failure. You demonstrated that our event-driven architecture provides tangible resilience, allowing parts of the system to fail without causing a total outage. This is a fundamental skill for any engineer building distributed systems.

Key Takeaways from this Lesson:

• Failure in distributed systems is inevitable; we must design and test for it.
• Chaos Engineering is the practice of injecting controlled failures to verify system resilience.
• Asynchronous, event-driven communication is a powerful pattern for decoupling services and enabling graceful degradation.
• Understanding the "critical path" of a request helps prioritize which services must have the highest availability.

This project has given you a solid, practical foundation. You've journeyed from the high-level concepts of microservices to the low-level details of writing async gRPC code, containerizing services, orchestrating them with Docker Compose, and testing them for both success and failure. You now possess the core vocabulary, patterns, and hands-on skills to confidently design, build, and reason about modern, high-performance distributed systems in Python.