Infrastructure Requirements for AI Workloads

Hello! We’ll begin by looking at AI infrastructure from the workload outward. Before choosing accelerators, storage, or a deployment pattern, you need to know what the system is doing and when its result must be available. This first module develops that reasoning, then moves toward measurable requirements, architecture, capacity, and cost.

In this lesson, you’ll classify training, batch inference, online prediction, and generative AI by the resources they consume and the operational constraints they impose. The important distinction is not simply “GPU or no GPU.” It is whether the work updates a model, waits for a schedule or a request, and produces a fixed-size result or a potentially long sequence of outputs.

---

Start with the workload’s clock

Consider a model that recommends items. You might train it overnight, calculate recommendations for every account every morning, or score candidates when someone opens an application. Those are different infrastructure workloads even if they use the same model.

Two questions provide a useful first classification:

1. Does the computation update model parameters? If yes, it is training. If it only uses an existing model to produce results, it is inference.
2. What triggers inference, and how long can the result wait? A scheduled job can optimize for completion by a deadline. A request-time prediction must fit within a user-facing latency budget.

A third question matters increasingly: Does one request produce a bounded result or generate a variable-length sequence? A fraud score might be one number. A text-generation request may occupy serving resources through many successive output tokens.

The following short reading supplies a vendor-neutral way to apply these questions, even though it appears in a cloud provider’s architecture guide.

Latency tolerance is an infrastructure property. If a job can finish an hour later without consequence, you have room to queue it, retry it, or use otherwise idle capacity. If a prediction is needed before an application can respond, that flexibility largely disappears. We will turn such statements into precise objectives in the next lesson; for now, use them to identify the workload class.

---

Training: optimize sustained progress, not individual response time

Training repeatedly reads examples, performs a forward pass, measures an error, calculates gradients in a backward pass, and updates parameters. Its infrastructure must support this cycle for long enough to produce a usable model. A failed request-time prediction may be retried in milliseconds; a failed training job might lose hours of computation unless it can resume from a checkpoint.

That extra state explains why training a model commonly requires more accelerator memory than serving the same model. In addition to parameters, a training device may hold activations, gradients, and optimizer state. Larger batches can increase activation memory. Meanwhile, slow delivery of examples from storage can leave expensive compute idle.

Training does not inherently require a cluster or even a GPU. Small models may train effectively on CPUs or one accelerator. At larger scales, however, the model or target completion time may require multiple devices. Workers then need to exchange information, making interconnect bandwidth and placement relevant alongside compute. Checkpoint storage must also absorb periodic writes and support recovery. These requirements favor planned capacity and monitoring of useful training progress, rather than measuring success by HTTP request latency.

To see why network topology enters the picture, watch the cluster-hierarchy portion of this lecture. Treat the named hardware as an example, not a prescribed platform.

The operational picture is therefore a long-running job with sizeable memory needs, sustained data input, possible worker-to-worker communication, and a recovery plan. How much of each it needs depends on the model and training configuration; “training” alone does not imply one machine size.

---

Batch inference and online prediction: the same model, different deadlines

Batch inference runs an existing model over a collection of inputs. A scheduled process might score all accounts each night and write results to a table or object store. Its primary question is often: Can the full dataset be processed before the results are needed? That encourages large input batches, sequential reads, high aggregate throughput, and efficient accelerator utilization where accelerators help.

Batch work can often be divided into independently retryable partitions. If capacity briefly disappears, a well-designed job can resume unfinished partitions—provided its deadline still has room. Storage bandwidth, output-write capacity, and orchestration matter as much as model execution. Batch inference need not use an accelerator: the model, dataset size, and deadline determine that choice.

Online prediction instead runs inference in response to a request. Its defining constraint is the end-to-end response path. Model execution shares the latency budget with admission, preprocessing, feature retrieval, and response handling. Capacity must be available when requests arrive, including during bursts; merely achieving good average throughput is insufficient if a queue makes some responses too slow. This is why replica readiness, autoscaling behavior, and tail latency matter.

There is a subtle architectural boundary here: retrieving a prediction that was computed in batch is not the same as performing online inference. The retrieval endpoint may be online and fast, while the prediction itself remains as old as the last batch run. Conversely, online inference can use features computed earlier in a batch. It does not automatically require a streaming feature pipeline.

Chip Huyen’s discussion makes these distinctions concrete:

Online serving replicas can be replaceable while their dependencies hold state: for example, an online feature store may retain recently computed values. Likewise, adding real-time features is a separate decision from making predictions online. It can improve freshness, but adds ingestion, processing, and storage work that must also remain healthy.

---

Generative AI: classification needs an additional dimension

“Generative AI” does not sit beside training, batch inference, and online prediction as a mutually exclusive category. You can train a generative model, use it to generate thousands of summaries offline, or serve an interactive conversation. Classify its execution mode first, then account for what generation changes within that mode.

For autoregressive text generation, a request usually has two different phases. During prefill, the model processes the input prompt. During decode, it produces output tokens successively. The amount of work therefore varies with prompt length and output length. Serving systems must retain model weights and, for active requests, a KV cache that stores information needed to continue generation. Longer contexts and more simultaneous requests increase that cache’s memory demand.

These characteristics make accelerator memory capacity and bandwidth especially important, even when raw compute capability looks ample. They also change what “fast” means: an interactive service cares both about how soon the first token appears and how steadily subsequent tokens arrive. Offline generation can batch work more aggressively and prioritize total tokens produced before its deadline. If a model must be split across devices, communication becomes another constraint, but that is not a requirement for every generative deployment.

This brief reading offers a useful checklist for identifying constraints without assuming that a hardware specification predicts real-world performance.

The key caution is that a workload label suggests likely pressures, not a guaranteed bottleneck. A small online model might be limited by feature retrieval; a large batch job might be limited by storage reads; a generation service might run out of memory before it exhausts compute. Measure the suspected constraint on the intended stack.

---

Put the classification to work

Suppose one system produces a daily list of recommended items and also offers an interactive assistant that explains an item on request. Several workloads coexist:

Work	Classification	First infrastructure concern
Update the recommendation model using historical interactions	Training	Sustained data supply, training state in memory, and recoverable progress
Calculate tomorrow’s recommendations for all accounts	Batch inference	Dataset-wide throughput and completion before publication
Score a new candidate while someone browses	Online prediction	Available serving capacity and end-to-end request latency
Generate an explanation in the interactive assistant	Online generative inference	Model and KV-cache memory, first-token delay, and ongoing token delivery

If the explanation were generated for every item overnight instead, it would be batch generative inference. If a fast endpoint merely fetched those saved explanations, it would not perform request-time generation. These distinctions prevent a common planning error: sizing all four paths as though they were interchangeable “model-serving” deployments.

A useful infrastructure classification records four things before selecting machines: what triggers the work, when its result is due, what state must fit or persist, and which resource is most likely to constrain progress. It should also record uncertainty. Where demand, model size, or input length is unknown, benchmark or measure rather than treating an initial estimate as a capacity guarantee.

---

Conclusion

Training favors sustained, recoverable progress and may need substantial memory, data throughput, and inter-worker communication. Batch inference favors efficient processing of a dataset by a deadline. Online prediction trades scheduling flexibility for immediate capacity and controlled request latency. Generative inference adds variable-length execution and, for many models, significant model-weight and KV-cache memory pressure. Generation can still be either batch or online.

In the next lesson, you’ll turn these qualitative differences into measurable availability, latency, throughput, freshness, and quality objectives—the requirements an architecture can actually be tested against.