Recommended resources
The freely available resources this lesson will direct you to.
Hello! Last lesson turned a workload trace into estimates for compute, memory, network, and storage. Those estimates also reveal a useful architectural distinction: the system that decides where a training job should run has different responsibilities from the job that uses accelerators, reads data, and writes checkpoints.
In this lesson, you’ll separate those responsibilities into a control plane and a data plane. We’ll apply the distinction to both training and inference, then use Kubernetes observations to make it concrete. The goal is an architectural boundary you can use when designing, operating, or troubleshooting an AI platform—not a rule that every component must fit neatly into one box.
Decisions about work versus execution of work
A control plane accepts desired state and works to make it real. In an AI platform, it may validate a training-job specification, select resources, coordinate a pipeline, deploy a model version, or change the number of serving replicas. Its inputs include configuration, policy, metadata, and measurements of current state. Its outputs are decisions and configuration changes.
A data plane performs the workload. It reads training examples and model artifacts, runs accelerator kernels, synchronizes distributed workers, writes checkpoints, or handles prediction requests. It carries the bytes and computation for which the platform exists.
This is a responsibility boundary, not necessarily a boundary between machines. A node agent may receive instructions from the control plane while starting data-plane containers on the same node. Likewise, a serving gateway may apply a routing policy created elsewhere while handling every live request itself.
Watch the managed-service example in Data Plane vs. Control Plane from IBM Technology. The database is not an AI workload, but its distinction between an administrative request and the work that request initiates transfers directly to training jobs and model deployments.
Watch Data Plane vs. Control Plane from IBM Technology for a compact account of the boundary and its costs.
In the service example, distinguish the API request to initiate a backup from the background work that produces it. Then watch the trade-offs: separating responsibilities also adds interfaces and infrastructure to maintain.
The transferable question is: Is this component deciding or configuring what should run, or is it carrying out the workload or serving its requests? That question is more reliable than classifying components by their names. A “scheduler” usually makes a control decision; a “gateway” usually handles data-plane traffic, even when it enforces a centrally configured policy.
A responsibility map for an AI platform
Consider a platform that trains models and publishes them for online use. Its principal responsibilities divide as follows:
| Concern | Control-plane responsibility | Data-plane responsibility |
|---|---|---|
| Training | Validate the job; coordinate dependencies; select placement and retry policy | Read examples; compute gradients; exchange collective traffic; write checkpoints |
| Model delivery | Record approvals and desired model version; configure rollout and replicas | Fetch model bytes; load weights; execute predictions |
| Traffic | Define routes, service tiers, and rate-limit policy | Match requests to routes; enforce limits; forward and serve requests |
| Capacity | Interpret demand signals; request more or fewer replicas | Report measurements; process requests on the available replicas |
| Storage | Track artifact identity, lineage, and access policy | Transfer dataset, checkpoint, and model bytes |
Two distinctions in that table prevent common design mistakes.
First, metadata is not the artifact it describes. A registry entry can say that model version 12 is approved for production; the model weights may reside in object storage. Updating the entry is a control operation. Downloading several gigabytes of weights to a serving replica is a data transfer, even if the download was triggered by that update.
Second, setting policy is not the same as enforcing it. A platform API may configure a tenant’s request limit. The gateway must apply that limit on the live request path without asking the management API for a fresh decision on every token or prediction. Some enforcement needs shared state or a dependency, so independence must be designed and verified rather than assumed.
Kubernetes offers a familiar instance of this pattern. Its API, reconciliation controllers, and scheduler manage desired and observed state; containers on worker nodes perform AI work. The boundary of an AI platform’s control plane can be wider than the Kubernetes control plane: a pipeline coordinator or model-release service may also make management decisions. Avoid equating “control plane” solely with a particular Kubernetes node or set of pods.
Follow a serving request, then follow a deployment change
KServe provides a concrete implementation to examine without making its components mandatory for a vendor-neutral architecture. Read its short control-plane introduction first. The point is the lifecycle responsibility of the controller, not the choice between its deployment modes.
Read KServe’s Control Plane page to see how an inference-service controller is positioned relative to serving execution.
At the top of the page, read the introductory paragraph beginning “The KServe control plane is responsible” through lifecycle management. In “Overview,” read the paragraph beginning “The control plane operates independently” through plane separation. Stop before the deployment-mode details.
Now look at the execution side. The same documentation places request-handling components—including a gateway and model-serving containers—on the data-plane path. Notice that its generative-inference description also shows controllers alongside that path. A diagram can depict both planes without turning every pictured component into a data-plane component.
Read KServe’s Data Plane page to identify what handles live inference, and to compare generative and predictive serving.
Read the opening description and “Overview,” from execution role. In “Generative Inference Architecture” → “Key Components,” read from “Envoy AI Gateway” through “Infrastructure Components,” beginning at the component map; distinguish the explicitly labeled “Control Plane Components” from the gateway and inference deployments. Finally, in “Predictive Inference Architecture” → “Data Plane Components,” read “Predictor,” “Transformer (Optional),” and “Gateway,” beginning with the predictor. Focus on which components touch an actual request.
Here is the generative-serving architecture described in the reading. Treat it as a labeled implementation example, not as a required stack for an AI platform.
Suppose an operator changes the desired model version and replica count. That is a management operation. Controllers reconcile the change, arrange workloads, and configure endpoints. Serving processes then load the selected weights and handle requests. Conversely, when a user sends a generation request, it should traverse the gateway and serving processes—not the deployment controller. The gateway’s live routing and rate-limit enforcement are data-plane actions, although their policies may have been supplied by the control plane.
The same separation applies to ordinary predictive inference. A controller establishes which predictor should exist; the predictor executes the model. An optional transformer performs preprocessing or postprocessing on the request path, so it is data-plane work even though someone configured it through an administrative API.
Follow a training job across the boundary
Inference is only half of this course’s platform. Reuse the representative training job from the last lesson: four accelerators, a prepared tokenized dataset, and periodic durable checkpoints.
A user submits a job specification stating the image, resource needs, data snapshot, output location, and retry constraints. Platform policy checks whether that request is permitted. A scheduler finds suitable capacity; orchestration records job state and may coordinate dependent pipeline stages. These are control-plane decisions. None of them performs the estimated FLOPs of training.
Once workers start, they read tokens, retain training state in accelerator memory, compute gradients, exchange synchronization traffic, and write checkpoints. Those are data-plane operations. This distinction changes how you size the system: the collective-communication estimate from last lesson applies to the worker interconnect, not to the management API. Checkpoint bandwidth applies to the workers’ path to storage, not to the metadata service that records a checkpoint’s identity.
There are interfaces between the planes throughout a run. Workers report status and metrics; a controller may decide to retry a failed worker or start a replacement. Status reporting does not turn the training process into a controller, and retry decisions do not turn the controller into a training worker.
An artifact store illustrates why “plane” describes an operation more accurately than a product label. The same storage service might hold model bytes read by workers, while a separate metadata interface tracks versions and approval state. Identify the API call and bytes on each path rather than assigning the entire storage product to one plane.
Make the boundary observable in Kubernetes
Here is a read-only operational walkthrough you can use on a Kubernetes cluster that already hosts a training or inference workload. It makes no deployment changes. Substitute a namespace and, where available, a serving deployment and Service; if you do not have such a cluster, use the listed objects as a specification for what you would inspect.
# Management objects and the workloads they request
kubectl -n <namespace> get deployments,jobs,services
# Current execution instances and their placement
kubectl -n <namespace> get pods -o wide
# Desired versus ready serving replicas
kubectl -n <namespace> describe deployment <serving-deployment>
# Backends currently published for a serving Service
kubectl -n <namespace> get endpointslices \
-l kubernetes.io/service-name=<serving-service> -o wide
Read the results in pairs. A Deployment or Job specification expresses desired work; the pods show the instances created to carry it out. Deployment conditions and events help explain whether reconciliation reached the intended state. EndpointSlices show which backends a Service can direct traffic toward; they do not prove those backends produce correct model outputs. The kubectl queries themselves contact the Kubernetes API, so they inspect control-plane records of execution rather than directly measuring request latency or accelerator utilization.
For an offline version, a compact architecture record can capture the same relationships:
Desired state: model version, replica target, job resource request
Decision owner: release controller, autoscaler, scheduler
Execution: serving pods or training workers
Traffic/bytes: inference requests, training data, collectives, artifacts
Feedback: readiness, queue depth, job status, workload metrics
Fill that record for each workload in an actual platform design. It is especially useful when a proposed component seems ambiguous: document the decision it makes, the traffic it handles, and the dependency it has on the other plane. A gateway, for example, may both receive routing configuration and process high-volume requests. Those are distinct responsibilities even if implemented by closely related software.
The next operational step is to trace a single change and a single request separately. For a model rollout, inspect desired version, deployment state, readiness, and published backends. For a prediction, inspect gateway and server request telemetry. Mixing those traces can lead to a misleading diagnosis—for example, declaring a rollout healthy because the controller reports completion when new replicas cannot load their model weights.
Design for different failures and scaling pressures
Separation earns its keep when the planes need different treatment. Management APIs and controllers need sound authorization, consistent desired state, reconciliation, and auditable changes. Inference gateways and workers need capacity for live traffic and predictable latency. Training workers need accelerator and data-path throughput. You can scale and isolate these concerns independently—but doing so introduces interfaces and operational overhead.
The distinction also gives you a sharper failure analysis:
- Management API or controller unavailable: already-running serving replicas may continue serving with their existing configuration. New deployments, scaling decisions, or recovery requiring reconciliation may stop. Continued service depends on the request path not making synchronous calls to the failed component.
- Serving gateway or model replicas unavailable: the control plane can still record the correct desired state, but requests fail or queue until usable data-plane capacity returns.
- Artifact storage unavailable: existing warm replicas may continue using loaded weights, while cold starts or replacements fail to load. A running training job may fail when it next needs input or writes a checkpoint.
- Metrics unavailable: current workers may keep running, but an autoscaler or operator can make poor decisions—or none—without reliable feedback.
These are conditional behaviors, not guarantees. In particular, “independent data plane” does not mean “independent of all shared dependencies.” Authentication, routing state, caches, and artifact access can reintroduce a management dependency onto the live request path. A robust design states which operations must survive a control-plane outage, then tests that claim.
Finally, physical isolation is a choice rather than the definition of the planes. Separate networks, permissions, or node pools may protect management services from workload traffic and limit what compromised workers can reach. They also add deployment and maintenance costs. Start with a clear logical boundary and explicit interfaces; choose physical separation in proportion to traffic, security, and reliability requirements.
Conclusion
A useful control-plane/data-plane split identifies who declares and reconciles desired work and who moves bytes and performs that work. For AI infrastructure, that means distinguishing scheduling from accelerator execution, model approval from model loading, and routing policy from live request handling. Components at the boundary should be described by their individual decisions and request-path duties rather than forced into a label.
That separation gives you better capacity estimates, fault diagnosis, and security boundaries. It also prepares the next lesson: a unit-cost model should charge substantial compute, transfer, and storage consumption to the workloads that incur them while still accounting for the shared control services needed to operate the platform.