AI System Non-Functional Requirements

The Twin Challenges: Inference Latency and Cost

While model training gets much of the public attention, it is a one-time, offline cost. For a production system, the ongoing, real-time cost of inference—the process of using a trained model to make predictions—is the dominant factor. This is a critical insight for an architect, as it directly impacts both business viability and user experience.

The graph below illustrates this economic reality. While the initial training investment can be substantial, the cumulative cost of inference often surpasses it in a matter of months.

This makes two NFRs paramount: inference latency and cost. They are deeply intertwined and often present a direct trade-off. Let's explore why they are so critical and how they are measured.

As the article emphasizes, optimizing for one metric often negatively impacts another. For example, aggressive batching of requests can increase throughput (requests per second) and lower per-request cost, but at the expense of higher latency for individual users. For an interactive application like a chatbot, low latency is critical for a good user experience. For an offline data processing job, high throughput and low cost are the priorities.

To go deeper into the mechanics, the following video from the CTO of Mistral AI provides a quantitative look at what drives these metrics at the hardware level.

The key architectural insight here is that you are always operating on a curve defined by your model, your code, and your hardware. Your job as an architect is to choose a point on that curve (a specific trade-off) that meets your application's NFRs and then find ways to shift the entire curve in your favor through optimization.

Common strategies for managing latency and cost include:

• Model Selection: Choosing a smaller, more specialized model is often the most effective optimization.
• Hardware Choice: Selecting the right GPU (or other accelerator) for the job.
• Quantization: Reducing the numerical precision of the model's weights to decrease its memory footprint and speed up computation.
• Caching: Storing the results of previous LLM calls to serve identical or similar future requests.

Let's briefly look at caching, a classic system design pattern applied to this new domain.

Semantic caching, which uses vector embeddings to find semantically similar queries, is a powerful example of how AI-specific techniques can enhance traditional architectural patterns.

Model Versioning: Managing Change and Depreciation

In traditional software, you control your dependencies. You decide when to upgrade a library, and you can test against the new version before deploying. With AI models, especially those consumed via third-party APIs, this control is often lost. A provider might update or deprecate a model with little notice, breaking your application in subtle ways. This introduces model versioning and lifecycle management as a critical NFR.

The video makes a crucial point: when a model provider forces an upgrade, it's not just an API change. The new model might behave differently, require different prompt structures, and produce different outputs for the same input, leading to a cascade of failures.

This problem isn't limited to the model itself. In an LLM-powered system, the prompt template is a piece of application logic. A small tweak to a prompt can cause significant behavioral changes. This means prompts, like code, must be versioned, tested, and managed systematically.

An architectural solution to this NFR often involves creating a "model-as-a-service" layer within your own infrastructure, even if you are just proxying calls to an external provider. This layer gives you a single point of control to manage model versions, route traffic to different models for A/B testing, and implement fallbacks, thus insulating the rest of your system from upstream volatility.

Data Privacy and Security: The Foundation of Trust

Perhaps the most significant NFR introduced by AI, especially for enterprise applications, is data privacy. When you use a public AI API, you are sending your data—and potentially your customers' data—to a third party. For organizations in regulated industries like finance or healthcare, or those handling any form of personally identifiable information (PII), this is often a non-starter.

The primary architectural decision driven by the data privacy NFR is the choice between using a public, proprietary model API versus self-hosting an open-source model.

• Proprietary APIs (e.g., OpenAI, Anthropic):
  • Pros: Easy to use, no infrastructure management, access to state-of-the-art models.
  • Cons: Data is sent to a third party, creating privacy and compliance risks. Costs can be high and unpredictable at scale. You are subject to the provider's model lifecycle.
• Self-Hosted Open-Source Models (e.g., Llama, Mistral):
  • Pros: Full control over data (it never leaves your infrastructure), predictable costs (after initial hardware investment), control over model versions.
  • Cons: Requires significant in-house MLOps and infrastructure expertise, high upfront cost for GPU hardware, and you are responsible for scaling and maintenance.

This "build vs. buy" decision is fundamental. The need for data sovereignty is a powerful forcing function that pushes many organizations towards the self-hosting path, despite its complexity.

Tying it all Together: An Architectural View

These NFRs are not abstract concerns; they manifest as concrete components and design choices in an LLM serving architecture. The diagram below shows a typical architecture for serving an LLM within a Kubernetes cluster.

Let's map the NFRs we've discussed to this diagram:

• Latency & Cost: The entire right side of the diagram is dedicated to managing this. The GPU provides the raw compute power. The Inference Engine (e.g., vLLM, TensorRT-LLM) uses techniques like paged attention and continuous batching to optimize performance. The KV Cache stored in VRAM is a specialized cache that dramatically speeds up the autoregressive decoding loop, reducing token-to-token latency. The choice of model, its quantization level, and the GPU type are all decisions made to meet latency/cost targets.
• Model Versioning: The Model Weights (Storage) component represents a model repository (like a Hugging Face hub or an internal artifact store). A robust architecture requires this repository to support versioning, allowing the Inference Server to load specific, tested versions of a model. The Inference Server itself might contain logic to route requests to different model versions for A/B testing.
• Data Privacy: The blue box labeled "LLM Serving Infrastructure" represents a boundary. By deploying this entire stack on your own infrastructure (on-premise or in a private cloud), you ensure that all data, from the initial prompt request to the model weights, remains within your control. The Inference Server can also be a place to implement PII scrubbing or other data masking logic before the data ever reaches the Inference Engine.

Conclusion

Integrating AI is not as simple as making an API call. It forces architects to confront a new set of demanding non-functional requirements that have profound implications for system design, cost, and user trust.

Key Takeaways:

• Cost and Latency are Paramount: Ongoing inference costs often dwarf initial training costs, making cost a first-class architectural metric. Latency is critical for user experience, and there is a constant trade-off between latency, throughput, and cost that you must manage.
• Versioning is a Reliability Concern: Models and prompts are dynamic dependencies. Without rigorous versioning and management, your system is vulnerable to upstream changes that can cause unpredictable failures. Treat prompts as code and models as versioned artifacts.
• Data Privacy Drives Infrastructure Choices: The need for data privacy and sovereignty is a primary driver for self-hosting models. The "build vs. buy" decision between a third-party API and a self-hosted stack is one of the most fundamental architectural choices you will make.
• NFRs Shape the Architecture: These requirements directly translate into concrete components and patterns, such as caching layers, model repositories, and optimized inference engines running on specialized hardware.

Preview of the Next Lesson:

Now that we understand the key constraints (the "what" and "why"), our next lesson will focus on the solutions. We will describe common architectural patterns for integrating AI services into existing systems, such as the sidecar, gateway, and embedded model patterns. We will see how these patterns provide different ways to address the NFRs we've discussed today.