Designing a Multi-Agent Customer Support Platform

Welcome back! In our last session, we meticulously designed and diagrammed a production-grade, scalable RAG architecture. You learned how to separate data ingestion from real-time inference, and how to incorporate essential components like caching, re-ranking, and load balancing. The key lesson was to think like a senior engineer: design for scale, availability, and resilience, not just for a simple prototype.

Today, we're taking that knowledge and applying it in a high-stakes, realistic scenario. The learning outcome for this lesson is to perform a mock system design for a multi-agent customer support platform. This is one of the most common and challenging system design questions for AI Engineer roles. It requires you to synthesize everything we've covered—from RAG and tool-calling to scalability and error handling—into a single, coherent system. We will structure this lesson as if you are in the interview, building your answer step-by-step.

The Problem Statement

Imagine an interviewer presents you with the following prompt:

"Design an AI agent system to handle customer support for a large e-commerce platform. The system must handle 50,000 tickets per day, with potential 5x spikes during outages. It needs to access customer order data, issue refunds, and consult a knowledge base. Keep the average cost below $0.08 per ticket and ensure all decisions are auditable for compliance."

A junior engineer might immediately start drawing boxes for "LLM" and "Vector DB." A senior engineer starts by asking questions. Let's learn to think like a senior engineer.

The AGENT Framework: Structuring Your Answer

In a high-pressure interview, a structured approach is your best friend. It prevents you from rambling and shows the interviewer you have a methodical way of thinking. We'll use a framework to guide our design.

Now, let's apply this framework to the problem.

Step A: Assess the Problem (Ask Clarifying Questions)

Before we draw a single line, we need to treat the constraints not as background noise, but as the primary drivers of our architecture. A great answer begins with probing these constraints.

The following video demonstrates the critical difference between a weak answer that describes components and a strong answer that starts by clarifying constraints.

Inspired by the video, here are the questions we would ask the interviewer, and the hypothetical answers we'll use to guide our design:

1. What is the target automation rate?

   • Hypothetical Answer: "We're aiming for 80% of tickets to be resolved fully autonomously. The remaining 20% should be escalated to humans with a complete summary of the AI's interaction." This tells us where to set our human escalation boundary.

2. What level of auditability is required? Operational or Compliance?

   • Hypothetical Answer: "Full compliance. A regulator must be able to review a decision six months later and see exactly which policy version and data justified it." This means simple logging is not enough; we need a more robust solution.

3. Does the $0.08 cost envelope include only inference, or is it all-in (infra, tooling)?

   • Hypothetical Answer: "That's the all-in, fully-loaded cost." This forces us to be extremely deliberate about our choice of models and architecture to manage costs.

4. Is there a dollar threshold for autonomous refunds?

   • Hypothetical Answer: "Yes, the system can autonomously process refunds up to $50. Anything higher requires human approval." This defines a critical safety boundary for our agent.

With these answers, we can now move on to designing the architecture.

Step G & E: Ground the Architecture and Enumerate Components

Our first major decision is whether to use a single, monolithic agent or a multi-agent system. Given the diverse tasks (billing, order tracking, policy questions) and the need for specialized tools, a multi-agent approach is superior. It allows for modularity, easier maintenance, and the use of specialized, fine-tuned prompts for each agent.

This aligns with a core principle for designing multi-agent systems: use them when you have clearly separable roles.

Our architecture will be a Supervisor-Worker pattern. A central Supervisor agent routes tasks to specialized Worker agents.

Let's break down the key components shown in this diagram:

1. User & Ingestion Layer: The user query comes in via channels like chat, email, or a web form. The very first step is an ingestion layer that scrubs all PII (Personally Identifiable Information), replacing it with tokens, and stores the mapping in a secure vault. This is a critical security and compliance step.

2. Supervisor Agent (The Router): This is the brain of the operation. Its only job is to analyze the PII-scrubbed user query and route it to the correct specialist agent.

   • Cost Optimization: This agent must be fast and cheap. It would use a small, efficient classification model (e.g., Claude Haiku, GPT-3.5-Turbo, or a fine-tuned open-source model). This is our first and most important cost-control mechanism. We spend fractions of a cent here to avoid unnecessarily invoking expensive models.

3. Specialized Agents (The Workers): These are the experts. Each one has a focused prompt and is given access to a limited set of tools.

   • Policy Agent: Answers "how-to" questions. It uses a RAG pipeline (like the one we designed last lesson) over the company's knowledge base and policy documents.
   • Billing Agent: Handles questions about charges and invoices. It has a tool to get_billing_info.
   • Order Agent: Tracks orders and manages returns. It has tools like get_policy_detail (to check return eligibility) and initiate_refund.
   • Human Escalation Agent: This agent's job is not to solve the problem, but to package the entire conversation history, including the agent's internal "thoughts" and attempted actions, into a concise summary for a human support officer.

4. Tools & Data: These are the functions and databases the agents interact with. Critically, agents don't have direct database access. They use well-defined API-based tools like get_payment_history(order_id). This provides a layer of abstraction and security.

Step N: Navigate Failure Modes and Production Realities

This is where you demonstrate senior-level thinking. A system diagram is just a set of happy paths. The real challenge is designing for when things go wrong.

The Manifold AI Learning video provides an exceptional walkthrough of how to build an architecture driven by constraints and failure modes.

Let's integrate these critical concepts into our design:

• Cost Control (Model Tiering): As mentioned, the Supervisor uses a cheap model. The more powerful, expensive reasoning models (e.g., GPT-4, Claude Opus) are only activated for the specialist agents when a query is complex enough to require policy validation or multi-step tool use. This tiered approach is fundamental to staying within our $0.08/ticket budget.

• Auditability (Compliance-Grade Logging): We won't just log messages. We will have a dedicated Audit Stream (e.g., writing to a Kafka topic). Every time an agent makes a decision or calls a tool, it emits a structured event: {"agent": "OrderAgent", "tool_called": "get_policy_detail", "params": {"order_id": "XYZ"}, "tool_output": {"eligible": true}, "decision": "Proceeding with refund"}. Crucially, for compliance, when referencing a policy, we snapshot the relevant text snippet into the audit log, not just a link to the document. This handles the "audit drift" problem.

• Safety (Guardrails): We implement synchronous checks, or "guardrails," before executing sensitive actions. For example, before the initiate_refund tool is executed, a guardrail function intercepts the call. if refund_amount > 50: raise Exception("Refund amount exceeds autonomous limit. Escalate to human."). This is a hard-coded safety rule, not a suggestion to the LLM.

• Handling Spikes (Classifier Fallback): During a 5x spike, it's likely many tickets are about the same incident. Our ingestion layer will have a simple semantic similarity check. If it detects a high volume of very similar incoming tickets, it can bypass the expensive agent system and trigger an "incident mode," providing a static response ("We are aware of a platform issue and are working on it...") and saving the cost of processing 200,000 redundant tickets.

• Tool Failures (Clean Handoff): If a tool (e.g., the Order DB API) times out mid-execution, a simple retry might not be enough. Our design specifies that for in-flight customer issues, a tool failure triggers an immediate, clean escalation to a human. The system's state management is optimized for a clean handoff, not just for resuming, ensuring the customer isn't stuck in a failing loop.

Step T: Talk Through Trade-offs

Finally, we acknowledge the compromises in our design.

• Multi-Agent vs. Single Agent: We chose a multi-agent system for its specialization and modularity. The trade-off is increased architectural complexity and potential for slightly higher latency due to the handoffs between agents. However, we mitigate this by using a very fast router and running specialist agents in parallel where possible.

• RAG vs. Tool Use: An interviewer might ask, "Why not just use RAG for everything?" Our answer: "RAG is excellent for retrieving information from a static knowledge base, and we use it for our Policy Agent. However, it cannot perform actions or access real-time, transactional data. For that, we need structured tool-calling APIs to interact with live systems like the order database and payment gateway. Our system uses both, applying the right technique for the right job."

• Automation vs. Safety: We drew a clear line at an $50 refund limit. We could have tried to build a more complex reasoning agent to handle larger refunds, but we traded off a higher automation rate for absolute safety and predictability, which is a key requirement in a production system involving money.

Conclusion

In this mock design session, we've constructed a robust, production-ready architecture for a multi-agent customer support platform. We went far beyond just listing components and instead focused on designing from the constraints—cost, scale, auditability, and safety.

Key Takeaways:

• Use a Framework: The AGENT framework (Assess, Ground, Enumerate, Navigate, Talk through) provides a clear structure for any system design answer.
• Clarify First, Design Second: Senior engineers design from constraints. Questions about automation targets, auditability, and cost are the most powerful tools to start with.
• Design for Failure: A production system isn't about the happy path. It's about how the system behaves during tool timeouts, traffic spikes, and when faced with malicious inputs. Explicitly discussing failure modes is crucial.
• The Supervisor-Worker Pattern: This is a powerful and common pattern for multi-agent systems, where a cheap, fast router (Supervisor) delegates tasks to specialized, powerful experts (Workers).
• Cost is an Architectural Driver: Decisions like model tiering and incident-mode fallbacks are not afterthoughts; they are core architectural components driven by the cost envelope.

In our next lesson, we will zoom in on one of the most important trade-off discussions in AI system design: evaluating and articulating the trade-offs between RAG, fine-tuning, and sophisticated prompt engineering for different interview scenarios.