Joining Worker Nodes with Kubeadm

1. The kubeadm join Process: Establishing Trust

Adding a new node to a cluster isn't as simple as just connecting it to the network. A crucial security challenge must be solved: bidirectional trust.

• The new node must trust the control plane. How does the worker node know it's connecting to the legitimate API server and not a malicious imposter?
• The control plane must trust the new node. How does the API server know the joining node is authorized and not an intruder trying to gain access to the cluster?

kubeadm solves this "chicken-and-egg" problem through a two-part process: Discovery and TLS Bootstrap.

To understand these concepts in detail, let's turn to a well-written article that breaks down the theory.

To summarize and reinforce the key points from the article:

1. Discovery (Node -> Control Plane Trust): The worker node verifies the control plane's identity. The default and most secure method is CA Pinning.

   • The kubeadm join command includes a hash of the cluster's root Certificate Authority (CA) public key (--discovery-token-ca-cert-hash).
   • The joining node fetches the CA certificate from the control plane and calculates its hash.
   • If the calculated hash matches the one provided in the join command, the node trusts the control plane. This prevents man-in-the-middle attacks.

2. TLS Bootstrap (Control Plane -> Node Trust): The control plane verifies the worker node's identity using a temporary bootstrap token.

   • The join command includes a short-lived token (--token).
   • The new node uses this token to make its first authenticated request to the API server.
   • Once temporarily authenticated, the node's kubelet generates a new private key and submits a Certificate Signing Request (CSR) to the API server.
   • The control plane's controller manager automatically approves this CSR.
   • The kubelet receives a unique, long-term client certificate, which it stores in /etc/kubernetes/kubelet.conf and uses for all future secure communication. The bootstrap token is no longer needed for this node.

For a formal overview of the kubeadm join command, its workflow, and different discovery methods, the official Kubernetes documentation is the definitive source.

2. Hands-On: Adding a Worker Node

Now, let's put the theory into practice and add a worker node to your cluster.

Prerequisites on the Worker Node

First, ensure your worker node is prepared just as the control-plane node was. This means:

• Swap is disabled.
• Kernel modules (br_netfilter, overlay) are loaded and sysctl parameters for networking are set.
• containerd, kubelet, and kubeadm are installed.

These steps were covered in the lesson prior to initializing the control plane.

Obtaining the Join Command

In the last lesson, the output of kubeadm init provided the exact kubeadm join command needed. If you saved it, you're ready to go.

If you lost the command or the token has expired (they are valid for 24 hours by default), you can easily generate a new one. Run this command on your control-plane node:

sudo kubeadm token create --print-join-command

This will output a new, valid kubeadm join command.

Executing the Join

Now, log in to your worker node and execute the kubeadm join command as the root user or with sudo.

Let's watch a demonstration of this process.

The command you run will look like this:

# Run this on your WORKER node
sudo kubeadm join <control_plane_ip>:6443 --token <token> \
    --discovery-token-ca-cert-hash sha256:<hash>

Upon successful execution, you will see output confirming the node has joined the cluster.

Verifying the Cluster State

Switch back to your control-plane node. Check the status of your nodes:

kubectl get nodes

You should now see your new worker node in the list. Initially, its status might be NotReady, but after a minute or two, it should transition to Ready.

NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   25m   v1.30.x
workernode1    Ready    <none>          90s   v1.30.x

Why does the node become Ready?
This happens automatically because of the CNI plugin (Calico) you installed in the previous lesson. The Calico components run as a DaemonSet, which ensures that its networking agent (calico-node) is automatically deployed to any new node that joins the cluster. Once the CNI agent is running, the kubelet on the worker node reports its network as ready, and the node's status changes to Ready.

You can verify that the CNI and kube-proxy pods are running on your new worker node:

# Run this on the control-plane node, replacing <worker-node-name>
kubectl get pods --all-namespaces -o wide | grep <worker-node-name>

The output should show pods for kube-proxy and calico-node running on the worker node's IP address.

3. Troubleshooting Common Join Issues

In a perfect world, kubeadm join always works. In reality, you may encounter issues. Knowing how to troubleshoot is a key skill for a Kubernetes administrator.

The most common failure is the command hanging indefinitely during preflight checks.

Here is a summary of the diagnostic flow for a hanging kubeadm join:

1. Test Connectivity: From the worker node, try to reach the API server's health endpoint.

   curl -k https://<control_plane_ip>:6443/healthz
   

   If you get ok, connectivity is fine. If the command hangs or fails to connect, you have a network problem.

2. Check Firewalls: If the curl command fails, the most likely culprit is a firewall on the control-plane node blocking incoming traffic on port 6443.

   • On CentOS/RHEL/Rocky: sudo systemctl status firewalld
   • On Ubuntu/Debian: sudo ufw status
   • If a firewall is active, you must configure it to allow TCP traffic on port 6443.

3. Check API Server Binding: On the control plane, ensure the API server is listening on its public IP, not just localhost.

   ss -tlnp | grep 6443
   

   You should see it listening on 0.0.0.0:6443 or <control_plane_ip>:6443, not 127.0.0.1:6443.

Conclusion

In this lesson, you successfully expanded your Kubernetes cluster from a single node to a multi-node environment capable of distributing workloads.

Here are your key takeaways:

• The kubeadm join command securely adds worker nodes to an existing cluster.
• The process relies on a bidirectional trust model: Discovery (via CA hash) for the node to trust the control plane, and TLS Bootstrap (via bootstrap token) for the control plane to trust the node.
• You can generate a new join token and command at any time from the control plane using kubeadm token create --print-join-command.
• A successful join can be verified with kubectl get nodes on the control plane. The automatic deployment of CNI pods via a DaemonSet is what allows new nodes to become Ready.

You now have a proper, multi-node Kubernetes cluster. The foundation is built. In our next lesson, we will shift focus from building the cluster to using it. You'll learn how to use kubectl to inspect the state of your cluster and its resources, starting with the get and describe commands.