Introduction
In our previous lesson, we established the key non-functional requirements (NFRs) that make integrating AI a unique architectural challenge: inference latency, cost, model versioning, and data privacy. Understanding these constraints is crucial because they act as the forcing functions that shape our design choices.
This lesson builds directly on that foundation by exploring the solutions. We will move from the "why" of the problem to the "how" of the solution by describing the common architectural patterns used to integrate AI services into existing systems. Specifically, you will learn to describe the sidecar, gateway, and embedded model patterns. For each pattern, we will analyze how it helps manage the NFRs we've discussed, its trade-offs, and the scenarios where it is most applicable. This will provide you with a practical toolkit for making sound architectural decisions when bringing AI capabilities into your software.
The Central Hub: The LLM Gateway Pattern
For organizations moving beyond trivial AI usage, the most common and powerful integration pattern is the LLM Gateway. You're already familiar with the concept of an API Gateway in microservices architecture, which acts as a single entry point for all clients. An LLM Gateway applies the same principle but is specialized for the unique traffic patterns and requirements of Large Language Models.
Instead of each service in your system directly calling OpenAI, Anthropic, or a self-hosted model, all requests are routed through this central gateway. This single point of contact provides enormous leverage for managing the NFRs we identified.
To understand the core functions and architectural variants of an LLM Gateway, the following article provides an excellent, architect-focused overview.
LLM Gateway Architecture and Multi-Model Routing | Collin Wilkins
This article by Collin Wilkins clearly articulates the business case for an LLM Gateway and breaks down its deployment patterns. It's a pragmatic guide that resonates with the challenges faced by engineering teams at scale.
Please read the following sections: Start with the introduction to understand what the gateway does and why reliability and cost visibility are its primary benefits. Next, read the section on when to add it, which provides clear heuristics based on service count and monthly spend. Finally, focus on the core of the article, "Three Architecture Patterns," which details the Sidecar Proxy, Centralized Gateway, and Edge Routing deployment models. Pay close attention to the trade-offs listed in the table.
The article makes a critical distinction between a centralized gateway service and using a sidecar as a deployment model for that gateway logic. Let's formalize the sidecar pattern itself, as it's a powerful pattern for decoupling concerns in distributed systems, including AI.
The Decoupled Helper: The Sidecar Pattern
The Sidecar pattern involves deploying a helper container alongside your main application container in the same pod (in a Kubernetes context) or on the same virtual machine. The two containers share a network namespace and storage, allowing them to communicate efficiently over localhost. The main application focuses on its core business logic, while the sidecar handles cross-cutting operational concerns.
In the context of AI integration, a sidecar is an excellent way to encapsulate all the complexity of interacting with an LLM:
- API Interaction: The sidecar can manage API key authentication, request formatting, and response parsing for one or more LLM providers. The main application simply makes a local HTTP or gRPC call to the sidecar.
- Resilience: The sidecar can implement retry logic, timeouts, and even fallback logic between different models or providers, completely shielding the main application from this complexity.
- Observability: The sidecar can be responsible for logging all prompts, responses, token counts, and latencies, providing consistent observability without cluttering the application code.
- Caching: A sidecar is a natural place to implement a cache for LLM responses, reducing both latency and cost for repeated requests.
By using a sidecar, you keep your primary service's domain logic clean and independent of the specific AI implementation details. If you decide to switch from OpenAI to a self-hosted Llama model, you only need to update the sidecar, not every service that uses AI. This aligns perfectly with the principles of modular, maintainable microservice architectures.
The High-Performance Option: The Embedded Model Pattern
While gateways and sidecars are excellent for interacting with external or remote model APIs, some use cases demand the lowest possible latency and absolute data privacy. This is where the Embedded Model pattern comes in. In this pattern, the AI model runs directly within the application's process or as a separate process on the same machine. There is no network call to an external service.
This approach is particularly suitable for:
- Low-latency requirements: Features like real-time type-ahead suggestions or semantic search in a dropdown, where a 500ms+ network round trip is unacceptable.
- Data privacy and security: When handling highly sensitive data that cannot leave the application's infrastructure boundary under any circumstances.
- Offline functionality: Applications that need to function without a persistent internet connection.
- Specialized tasks: When a smaller, fine-tuned model can outperform a large, general-purpose one on a specific task like classification or sentiment analysis.
Of course, this pattern comes with the significant operational overhead of managing the model's lifecycle and the underlying compute resources.
The following guide provides a clear comparison of these patterns and their ideal use cases.
AI Implementation Strategy: A Practical Guide for Product Teams ...
This guide frames the architectural patterns as strategic choices in an AI implementation plan. It offers a practical perspective on when and why to choose one over the other.
Read the section "Phase 3: Architecture decisions," which describes three patterns: Pattern 1: API-mediated inference (which corresponds to our Gateway/Sidecar discussion). Pattern 2: Async pipeline with pre-computation (a valuable related pattern for non-real-time needs). Pattern 3: Embedded model with edge inference. Pay close attention to the "Watch out for" points for each pattern. After that, read the following subsection, "Regardless of pattern," as its advice on building in a fallback path, evaluation harness, and feedback mechanism is critical for any robust AI integration.
To see a concrete example of an embedded model in action, the following video demonstrates running a small embedding model directly in the browser using JavaScript. This is the "embedded" pattern taken to its logical extreme—running on the client device itself.
Add Useful AI to Your Web App (Not Just Chatbots) • Steve Sanderson • GOTO 2024
In this segment from his GOTO 2024 talk, Steve Sanderson demonstrates how to implement semantic search using a small model that runs entirely on the client-side.
Watch the section on "Semantic Lookup." Notice how he uses a small, specialized sentence-similarity model (around 20 MB) to power a UI feature without needing a call to a large, expensive backend LLM. This is a perfect illustration of the embedded pattern for a specific, low-latency use case.
A Framework for Choosing the Right Pattern
As an architect, your role is not just to know the patterns but to decide which one to apply based on a set of trade-offs. The choice between a Gateway, Sidecar, and Embedded model depends on your specific system requirements, organizational context, and the NFRs you are prioritizing.
Here is a decision framework to guide your thinking:
| Pattern | Best For | Primary Benefit | Main Drawback |
|---|---|---|---|
| Centralized Gateway | Organizations with many services using LLMs; need for centralized cost control, observability, and governance. | Control & Visibility. Single point to manage cost, security, and routing. | Latency. Adds a network hop to every request. Potential single point of failure. |
| Sidecar | Decoupling AI concerns from a single service; teams wanting autonomy while following a standard pattern. | Decoupling & Encapsulation. Clean separation of business logic from AI logic. | Operational Complexity. Higher resource consumption per service; requires robust pod/container management. |
| Embedded Model | Latency-critical features (sub-100ms); absolute data privacy requirements; offline-first applications. | Performance & Privacy. No network latency; data never leaves the host. | High Operational Overhead. Requires MLOps expertise to manage models and infrastructure. |
Your decision will likely be a hybrid. A common strategy is to use a Centralized Gateway for most interactions with large, powerful models (e.g., for complex generation tasks) while using Embedded Models within specific services for specialized, low-latency tasks like semantic search or classification. The Sidecar pattern can be the implementation vehicle for either approach at the service level.
Conclusion
In this lesson, we have moved from the problems of AI integration to the architectural patterns that provide solutions. You learned to describe and differentiate between the Gateway, Sidecar, and Embedded model patterns, analyzing how each addresses specific non-functional requirements.
Key Takeaways:
- The LLM Gateway is a specialized API gateway that provides centralized control over cost, reliability, and security for LLM usage at scale. It's the default choice for organizations with multiple services consuming AI.
- The Sidecar Pattern is an excellent way to decouple AI-related concerns (like API calls, retries, and caching) from your application's core business logic, promoting modularity and maintainability.
- The Embedded Model Pattern offers the ultimate in low latency and data privacy by running the model directly on the application host, but at the cost of significant operational overhead.
- The choice of pattern is a strategic architectural decision based on a trade-off analysis of latency, cost, privacy, and operational complexity.
Preview of the Next Lesson:
Now that we have established the fundamental patterns for integrating a single AI component into a system, we can move to a higher level of complexity. In the next lesson, we will begin to explore AI Agentic Architectures, where systems are composed of multiple, often autonomous, AI agents that communicate and collaborate to achieve a goal. We will distinguish these agents from simpler AI integrations and describe their core components.