Back-of-the-Envelope System Estimation

The Art of Approximation

In a system design interview, and in real-world planning, you are not expected to produce perfectly accurate figures. The goal is to be directionally correct and to demonstrate a structured thought process. It's about knowing whether you need 10 servers or 100, a terabyte of storage or a petabyte. This ability to reason about scale is what separates a speculative design from a grounded one.

Let's begin by understanding the mindset and importance of this skill.

To reinforce this, the following short video explains how these quick calculations help validate design choices instantly.

A Framework for Estimation

To avoid getting lost in the numbers, it's essential to follow a structured approach. This not only keeps your thoughts organized under pressure but also signals to an interviewer that you have a repeatable, mature process.

A robust framework for BOE calculations involves five steps:

1. Clarify Scope: Explicitly state what you are estimating (e.g., peak write QPS, 5-year storage growth).
2. State Assumptions: Announce your assumptions about user numbers, behavior, data sizes, etc. This makes your reasoning transparent and invites collaboration.
3. Decompose the Problem: Break the large estimation task into smaller, manageable parts (e.g., traffic, storage, bandwidth).
4. Estimate Components: Calculate each part using simple math and aggressive rounding.
5. Synthesize and Sanity-Check: Combine the estimates and evaluate if the final number seems reasonable. Don't be afraid to self-correct.

The following resource details this framework and provides the fundamental units you'll need.

Now, let's see this framework applied in a practical example.

Estimation in Practice: A Deeper Dive into LLM Inference

While the general framework applies broadly, your background in generative AI means you'll encounter more specific and complex estimation challenges. Let's apply these principles to a modern use case: sizing the infrastructure for Large Language Model (LLM) inference.

Here, the key resources to manage are not just server counts, but specifically GPU memory, which is often the primary bottleneck. The main components consuming GPU memory during LLM inference are:

1. Model Weights: The parameters of the model itself. A simple estimate is Number of parameters * bytes per parameter. For FP16/BF16 precision, this is 2 bytes per parameter.
2. KV Cache: A stateful cache that stores the keys (K) and values (V) for all preceding tokens in a sequence. This is what allows the model to generate subsequent tokens without reprocessing the entire input, making generation fast. Its size is dynamic and depends on the number of concurrent users and the length of their sequences.
3. Activation Memory: Intermediate results stored during the forward pass of the model.

The image below shows a tool that automates these calculations, highlighting the key input parameters and the resulting memory breakdown.

Let's break down how to calculate these numbers yourself. The following article provides a detailed guide.

With the memory footprint established, we can then estimate the system's capacity (how many users it can serve) and latency (how fast it responds).

• Capacity: The total available GPU memory (after loading the model weights) divided by the memory required per user determines the maximum number of concurrent requests.
• Latency: The total response time is the sum of two phases:
  • Prefill Time (Time to First Token - TTFT): The time to process the initial prompt. This is typically a compute-bound operation.
  • Generation Time (Time Per Output Token - TPOT): The time to generate each subsequent token. This is a memory-bandwidth-bound operation.

Let's continue with the same resource to see how these are calculated.

These LLM-specific calculations are a powerful extension of the general BOE framework. They demonstrate a deep understanding of the underlying workload, which is exactly what top-tier companies look for when hiring for specialized ML system design roles.

Conclusion

This lesson concludes our foundational module by equipping you with a quantitative lens to view system design. You've learned that back-of-the-envelope estimation is not about chasing precision but about establishing scale and making informed architectural trade-offs.

Here are the key takeaways:

• Purpose of BOE: To quickly create reasonable, order-of-magnitude estimates that ground your design in reality and guide architectural choices like scaling, caching, and data partitioning.
• A Structured Framework: A five-step process—Clarify, Assume, Decompose, Estimate, Sanity-Check—ensures a clear and defensible analysis, especially in an interview setting.
• Core Estimation Areas: We focused on the three pillars of system resources: Compute (QPS from user traffic), Storage (data size considering growth and retention), and Network (bandwidth from read/write patterns).
• Specialized ML Estimation: For complex workloads like LLM inference, we extended the framework to estimate GPU memory, system capacity (concurrent users), and latency (TTFT and TPOT), which are the critical constraints.

With this, you have completed the first module, "Foundations of Distributed Systems for ML." You now possess a solid understanding of the core principles and technologies that underpin scalable ML systems.

In our next lesson, we will begin Module 2: Data Management and Governance in ML. We'll move up the stack to the lifeblood of any ML system—the data itself—and start by learning how to design robust data ingestion pipelines for both batch and streaming sources.