Cost-Aware Request Routing for LLMs

Hello! Welcome back to your course on preparing for a software engineering role at Anthropic.

In our last lesson, we explored several technical strategies for reducing token costs, such as prompt engineering and caching. A key takeaway was the idea of using the "cheapest capable model" for a given task. But how do you automate that decision-making process at scale?

Today, we will answer that question. The learning outcome for this lesson is to design a cost-aware request routing system that balances model quality, latency, and token cost. This is a classic system design problem that moves beyond simple API calls and into the realm of building intelligent, efficient infrastructure—something that directly aligns with your experience building the Rudi inference scheduler at AWS. Instead of routing based on priority or capacity, we'll be routing based on the content of the request itself to optimize for a multi-dimensional objective.

1. The Core Problem: The LLM Router

At its heart, a cost-aware routing system—often called an LLM Router—acts like an intelligent switchboard. Imagine you have access to a fleet of models, each with a different profile:

• High-Capability Model (e.g., Claude 3 Opus): Highest quality, but also highest cost and latency.
• Balanced Model (e.g., Claude 3.5 Sonnet): Good quality, moderate cost and latency.
• High-Speed Model (e.g., Claude 3 Haiku): Lower quality for complex reasoning, but very low cost and latency.

The router's job is to inspect each incoming user request and dynamically send it to the most appropriate model, creating the best possible user experience for the lowest possible cost.

2. The Fundamental Trilemma: Quality, Latency, and Cost

Designing a router is an exercise in managing trade-offs. As the OpenAI DevDay talk emphasizes, there is no single "playbook." The art is in balancing three competing factors for your specific application.

As the video highlights, the three pillars of our routing decision are:

1. Quality: The correctness, relevance, and depth of the model's response. This is paramount but hard to quantify.
2. Latency: The perceived speed of the application. This is a combination of network latency, prompt processing time (TTFT), and token generation time (TBT).
3. Cost: The direct expense of the API call, determined by the model and the number of input/output tokens.

A router's logic is essentially an algorithm for navigating this trilemma. Sending everything to Opus maximizes quality but results in high cost and latency. Sending everything to Haiku minimizes cost and latency but may fail on complex user queries. The goal of the router is to find the optimal point on this spectrum for every single request.

3. Architectures for Dynamic Routing

So, how do we build the "brain" of the router? The most common approach is dynamic routing, where the routing decision is made at runtime based on the prompt's content.

Let's explore the primary architectural patterns for this. The following AWS blog post provides an excellent, detailed overview that will feel familiar given your background.

As the article explains, the two dominant patterns for the routing logic are:

A. LLM-Assisted Routing (Classifier-Based)

In this pattern, you use another LLM—typically a small, fast, and cheap one—as a classifier.

1. The user's prompt arrives at the router.
2. The router sends the prompt to a "classifier LLM" (e.g., Haiku) with a special instruction like: "Classify this query as 'simple' or 'complex'."
3. The classifier LLM returns its classification.
4. The router uses this classification to send the original prompt to the appropriate model (e.g., 'simple' -> Sonnet, 'complex' -> Opus).

Trade-offs:

• Pros: Can capture sophisticated and nuanced distinctions between prompts. The logic is expressed in natural language, making it easy to iterate on.
• Cons: Introduces extra latency and cost for the classification step. The performance depends on how well you can prompt the classifier LLM.

B. Semantic Routing (Embedding-Based)

This pattern uses vector embeddings to route based on semantic similarity, a concept we touched on for caching in the last lesson.

The flow is as follows:

1. Offline Step: You define categories (e.g., "Math Problem," "History Question") and create embeddings for several example prompts in each category. These are stored in a vector database.
2. Runtime Step:
   • The user's prompt arrives.
   • The router generates an embedding for the user's prompt.
   • It performs a similarity search in the vector database to find the closest matching category.
   • It routes the prompt to the model assigned to that category.

Trade-offs:

• Pros: Typically much faster and cheaper than an LLM-assisted approach because embedding generation and vector search are highly optimized operations.
• Cons: Can be less precise than a classifier LLM for nuanced tasks. Its effectiveness depends entirely on the quality and coverage of your example prompts in the vector database. It also adds the operational complexity of managing a vector store.

The following video provides a concise, high-level overview of these router types and a few others, giving you a broader perspective.

4. A Practical Guide: Building a Causal LLM Router

Let's get practical. How would you actually build an LLM-assisted router? One of the most powerful techniques is to fine-tune a causal LLM to act as a specialized classifier. The following guide walks through this process end-to-end.

This "LLM-as-a-Judge" approach for data generation is a powerful pattern. It automates the creation of a high-quality dataset needed to train the router, transforming the routing problem into a standard supervised learning task.

5. Analyzing Trade-offs and Making a Decision

Your experience at AWS has surely taught you that there's no one-size-fits-all solution. Choosing a routing strategy requires a careful cost-benefit analysis.

Let's return to the AWS blog post, which has an excellent section on cost analysis.

Key Decision Criteria:

• Cost & Latency Overhead: Semantic routing is generally cheaper and faster. The AWS example shows it costs about half as much to implement as LLM-assisted routing ($107.9 vs $188.9) and is significantly faster at classification (~0.1s vs ~0.5s).
• Accuracy & Nuance: LLM-assisted routing can handle more complex, nuanced logic that is difficult to capture with a few example prompts for semantic search.
• Operational Complexity: Do you want to manage a vector database (semantic routing) or a fine-tuning pipeline (LLM-assisted routing)?
• Build vs. Buy: Is a managed solution like Amazon Bedrock Intelligent Prompt Routing sufficient? Or do you need the full control and flexibility of a custom-built router to use models from different providers or implement highly specific logic? Given your background, designing and building a custom solution is well within your skillset.

Conclusion

Today, we designed a cost-aware request routing system, a critical piece of infrastructure for any production-scale LLM application. We saw that this is not just a simple load balancer but a sophisticated system that must intelligently balance the trade-offs between response quality, latency, and cost.

Key Takeaways:

• An LLM Router dynamically selects the most appropriate model for a given request to optimize for quality, latency, and cost.
• The two primary architectures for dynamic routing are LLM-assisted (classifier-based) and semantic (embedding-based) routing.
• LLM-assisted routing offers higher precision for nuanced tasks but incurs greater cost and latency for the classification step.
• Semantic routing is faster and cheaper but depends heavily on the quality of reference embeddings and may be less precise.
• Building a router often involves using an LLM-as-a-Judge pattern to generate labeled data for fine-tuning a classifier model.
• The final choice of architecture depends on a thorough trade-off analysis and a classic build-vs-buy decision.

Next Lesson Preview

For the past several lessons, we've focused on the systems and economics surrounding LLMs—billing, cost reduction, and now routing. We've treated the models themselves as black boxes. In our next module, we will start to open that box. We'll begin our deep dive into the technology that powers these models, starting with the fundamental building blocks. In the next lesson, we will explore the question: how do you represent the meaning of words and their position in a sequence so that a machine can understand them? We'll cover word embeddings and positional encodings, the first steps on our journey into the Transformer architecture.