Kubernetes API Server & etcd: Request Handling and State Management

The Cluster's Gateway and Brain

At the heart of the control plane, the API server and etcd work in a tight, exclusive partnership.

• The API Server is the front door for the entire Kubernetes control plane. Every interaction—from you using kubectl, to controllers watching for changes, to kubelets reporting status—goes through the API server.
• etcd is a distributed key-value store that serves as Kubernetes' single source of truth. It stores the configuration and state of every object in the cluster.

Crucially, the API server is the only component that is allowed to talk directly to etcd. This centralizes access, validation, and security, forming a robust foundation for the entire system.

Let's start with a video that recaps the roles of all control plane components, paying special attention to how the API server and etcd are positioned.

etcd: The Single Source of Truth

As the video explained, etcd is the database for your Kubernetes cluster. Your experience with distributed systems will help you appreciate its design.

• What it is: etcd is a consistent and highly-available key-value store. It's "distributed" because it typically runs on multiple master nodes and uses the Raft consensus algorithm to ensure that all nodes agree on the cluster's state. This provides fault tolerance; the cluster can lose a master node without losing its state.
• What it stores: It stores the complete state of the cluster, both the desired state (what you want to be running, e.g., "I want 3 replicas of my Nginx app") and the actual state (what is currently running, e.g., "2 replicas are running, 1 is pending"). This data is organized in a key-space, with paths like /kubernetes.io/pods/default/my-pod.
• The Watch Mechanism: This is etcd's most powerful feature in the context of Kubernetes. Clients (specifically, the API server) can "watch" for changes on specific keys or directories. When a key is updated, etcd notifies the watcher. This event-driven model is what allows the entire Kubernetes system to be so responsive and loosely coupled.

To solidify your understanding of etcd's role and this critical watch feature, the following reading provides a clear explanation.

A Request's Journey Through the API Server

Now that we know where the state is stored, let's trace the path of a request to understand how the API server processes it before it ever reaches etcd. When you run a command like kubectl apply -f deployment.yaml, your request goes on a multi-stage journey inside the API server.

The following video provides an excellent and concise walkthrough of this entire process.

Let's break down those stages:

1. HTTP Handling: The journey begins here. kubectl converts your YAML manifest into a JSON payload and sends it as an HTTP request to the API server's endpoint.

2. Authentication: The first security gate. The API server needs to know who is making the request. It checks for credentials (like a client certificate or a bearer token) and verifies the identity of the user, group, or ServiceAccount. If authentication fails, the request is rejected immediately.

3. Authorization: The second security gate. Once the user is authenticated, the API server determines if they have permission to perform the requested action. For example, "Is this user allowed to create deployments in the production namespace?" This is typically handled by the Role-Based Access Control (RBAC) subsystem.

4. Admission Control: This is a powerful and extensible gatekeeping stage. After a request is authenticated and authorized, it's intercepted by one or more Admission Controllers. These are plugins that can examine, modify, or even reject the request before it's persisted. There are two types:

   • Mutating Admission Controllers: These can change the object. A common use case you might recognize from your work with Envoy is injecting a sidecar proxy container into every Pod created in a namespace. Another example is setting a default storageClass for a PersistentVolumeClaim if one isn't specified.
   • Validating Admission Controllers: These can reject the object based on custom policies, but cannot change it. For example, a validating controller could enforce a policy that all container images must come from a trusted corporate registry, or that all Pods must have resource limits defined.

5. Schema Validation & Persistence: After passing through all admission controllers, the object is validated against the internal schema for that resource type. If it's valid, the API server serializes it (often using an efficient format like Protobuf) and writes it to etcd.

Only after the object is successfully written to etcd is the request considered complete. At this point, etcd sends a notification back to the API server, which in turn notifies other components (like the scheduler or controller manager) that are watching for these changes, triggering the next set of actions.

Conclusion

In this lesson, you've taken a close look at the communication hub and memory of the Kubernetes cluster. You now have a solid mental model of how state is managed and how user intent is processed.

Key Takeaways:

• etcd is the cluster's consistent and highly-available key-value store. It is the single source of truth for all cluster data.
• The API server is the central management endpoint. It is the only component that communicates directly with etcd, acting as a gateway for all other components.
• Requests to the API server undergo a strict, multi-stage pipeline: Authentication (who are you?), Authorization (can you do this?), and Admission Control (does this request meet our policies?).
• Admission Controllers (both Mutating and Validating) are powerful plugins for enforcing custom policies and automating modifications, a key area of interest for Platform Engineers.

In our next lesson, we will build directly on this foundation. Now that you understand how the API server stores state in etcd, we'll explore the philosophy behind that state by discussing declarative management. You'll learn why telling Kubernetes "what you want" is far more powerful than telling it "what to do."