Kubernetes Rolling Deployments for Zero-Downtime Updates

1. The Rolling Update Strategy

When you change a Deployment's Pod template (for example, by updating the container image to a new version), Kubernetes needs a strategy to replace the old Pods with new ones. By default, it uses the RollingUpdate strategy.

This strategy incrementally replaces old Pods with new ones, ensuring that the application remains available throughout the update process.

The speed and safety of this process are controlled by two key parameters in your deployment.yaml under spec.strategy.rollingUpdate:

• maxUnavailable: The maximum number of Pods that can be unavailable during the update. This can be an absolute number (e.g., 1) or a percentage (e.g., 25%).
• maxSurge: The maximum number of new Pods that can be created above the desired replicas count. This can also be an absolute number or a percentage.

A common and very safe strategy for production systems is to ensure you never have fewer pods than your desired replica count. This is often called a "create-before-destroy" approach.

2. The Hidden Dangers of a Default Rollout

You might assume that the default rolling update is sufficient for zero downtime. However, it often isn't. Kubernetes, by default, doesn't fully understand your application's internal state. This can lead to two common problems during a rollout:

1. Premature Traffic Routing: Kubernetes may start a new Pod and immediately have the Service route traffic to it. If your application takes a few seconds to initialize (e.g., load configuration, establish database connections), it might receive requests before it's ready, resulting in errors.
2. Abrupt Connection Termination: Kubernetes might terminate an old Pod while it's in the middle of processing a long-running request. This will cause errors like connection reset by peer for the client.

Let's see a practical demonstration of this problem.

3. The Three Pillars of a Zero-Downtime Deployment

To solve these problems and achieve true zero-downtime, we need to configure our deployment to work with our application. This relies on three pillars: graceful startup, graceful shutdown, and a robust update strategy.

The video "Zero Downtime Deployments in Kubernetes Explained" by That DevOps Guy provides an excellent, in-depth walkthrough of all the pieces we need. We will use it as our main guide for this section.

Pillar 1: Graceful Startup with Readiness Probes

To prevent Kubernetes from sending traffic to a Pod that isn't ready, we use a Readiness Probe. This is a check that Kubernetes periodically performs to ask your application, "Are you ready to serve traffic?" The Service will only route requests to a Pod after its readiness probe succeeds.

For our FastAPI application, the root path / returns a successful HTTP 200 response, making it a simple but effective readiness check. We add the readinessProbe to the container spec in our deployment.yaml.

Watch how readiness probes are implemented and how they solve the premature startup problem.

Pillar 2: Graceful Shutdown with terminationGracePeriodSeconds

When Kubernetes decides to terminate a Pod, it first sends a SIGTERM signal to the main process in the container. It then waits for a certain amount of time before forcefully killing the process with SIGKILL. This waiting period is the termination grace period.

If your application has in-flight requests that take longer than the default grace period (30 seconds), they will be cut off. You must configure terminationGracePeriodSeconds in your Pod spec to a value that is safely longer than your longest expected request time.

Pillar 3: The preStop Hook for Network Race Conditions

There is one final, subtle race condition. When a pod is terminated, there is a slight delay between the pod being removed from the Service's endpoints and the actual termination signal (SIGTERM) being sent. If your application immediately stops accepting connections upon receiving SIGTERM, it might reject a request that was already in flight from the load balancer.

The solution is to use a preStop lifecycle hook. This hook executes a command after the Pod is marked for termination but before SIGTERM is sent. By adding a small sleep, we give the networking layer enough time to stop sending new traffic to the pod before we begin the application's shutdown process.

4. Hands-On: A Production-Ready Rolling Update

Let's apply these three pillars to the deployment.yaml from our previous lesson.

Update your deployment.yaml to match the following. The changes are highlighted with comments.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fastapi-app
spec:
  replicas: 3 # Increased replicas for more resilience



  # --- Pillar 3: A robust update strategy ---
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: fastapi
  template:
    metadata:
      labels:
        app: fastapi
    spec:



      # --- Pillar 2: Graceful shutdown period ---
      terminationGracePeriodSeconds: 60
      containers:
      - name: api
        image: public.ecr.aws/l6j4t5e6/fastapi-k8s-demo:latest
        ports:
        - containerPort: 8000



        # --- Pillar 1: Graceful startup probe ---
        readinessProbe:
          httpGet:
            path: / # Our app's root path responds with HTTP 200
            port: 8000
          initialDelaySeconds: 5 # Wait 5s before first probe
          periodSeconds: 5       # Probe every 5s
          failureThreshold: 3    # Mark as not ready after 3 failures



        # --- Pillar 3 (cont'd): preStop hook for network race condition ---
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 10"]

Now, let's trigger and monitor the update.

1. Apply the new configuration:
   Save your updated deployment.yaml and apply it:

   kubectl apply -f deployment.yaml
   

2. Monitor the rollout status:
   Use kubectl rollout status to watch the update in real time. The -w flag will watch for changes.

   kubectl rollout status deployment/fastapi-app -w
   

   You will see Kubernetes reporting the progress as it terminates old pods and brings up new ones according to your strategy.

3. Check the history:
   Once the rollout is complete, you can view the history of your deployments.

   kubectl rollout history deployment/fastapi-app
   

4. (Optional) Roll back the deployment:
   If you ever deploy a bad version, rolling back is simple. This command will revert to the previous revision.

   kubectl rollout undo deployment/fastapi-app
   

Conclusion

In this lesson, you elevated your Kubernetes deployment skills from basic to production-ready. You learned that achieving zero downtime requires a deliberate partnership between the platform's configuration and the application's behavior.

Key Takeaways:

• A Rolling Update is Kubernetes' default strategy for replacing Pods, controlled by maxSurge and maxUnavailable. A setting of maxSurge: 1 and maxUnavailable: 0 is a very safe starting point.
• Readiness Probes are essential for graceful startup. They tell Kubernetes when a new Pod is truly ready to handle traffic, preventing errors from premature routing.
• Graceful shutdown requires setting terminationGracePeriodSeconds to a value that allows in-flight requests to complete.
• A preStop hook with a small sleep is a best practice to mitigate network race conditions during pod termination.

Preview of the Next Lesson

Our application is now deployed and updatable without downtime. However, when things go wrong, how do we find out what happened? Plain text logs are hard to search and analyze. In the next lesson, we will focus on observability by learning how to configure structured JSON logging in a Python microservice, making our logs machine-readable and far more powerful for debugging in a distributed environment.