Multi-modal Routing with Differentiated Request Handling

Recommended resources

The freely available resources this lesson will direct you to.

Welcome back! In our previous lesson, we set up a foundational LLM router using LiteLLM, directing requests to different models based on simple tags in the request payload. This gave us a powerful way to use the right text model for the right text-based task.

Today, we'll build on that foundation to tackle a more complex and increasingly common scenario: multimodal applications. Many modern AI systems need to understand not just text, but images as well. This lesson focuses on extending our router to differentiate between requests that are purely text-based and those that contain images, routing them to the appropriate specialized models. This capability is essential for building sophisticated, multimodal AI services that can see and read.

By the end of this lesson, you will be able to design a routing system that intelligently handles both text and image inputs, a key component in the architecture of any advanced AI platform.

1. The Challenge: Inconsistent Multimodal APIs

Before we dive into building, let's understand the core problem our router needs to solve. When you move beyond text-only models, you'll find that different AI providers have designed their APIs for multimodal inputs in different, often incompatible, ways. A request format that works for OpenAI's Vision API might fail with Anthropic's or Google's.

An AI Gateway or router must act as a normalization layer, providing a single, consistent API to your client applications while handling the diverse formats of the underlying models.

The following video from Mehul Mohan clearly illustrates this issue by comparing the API structures for image inputs from different providers.

How to build your own AI gateway (and do a generational ragebait on twitter)

In "How to build your own AI gateway," Mehul Mohan highlights the inconsistencies between different LLM providers when handling multimodal data.

Watch this segment to see a concrete example of the problem we're solving today. API variations: Pay close attention to how Anthropic Claude and OpenAI's GPT-4 Vision expect image data to be formatted. Notice the differences in the JSON structure, such as the keys used (image vs. image_url). This divergence is precisely why a router with a normalization layer is critical.

As you can see, forcing your application developers to write custom logic for each provider is inefficient and brittle. A well-designed router abstracts this complexity away, which is a core principle of good platform engineering.

2. The Solution: Detecting Modality from Request Content

To route requests based on modality, our router must first inspect the incoming request and detect whether it contains images. A widely adopted standard for this, pioneered by OpenAI, is to structure the content field of a message as an array of objects, where each object represents a different part of the input.

Let's examine a practical implementation of this pattern. The following resource is a detailed summary of a code change that adds multimodal capabilities to a FastAPI service. It provides a perfect, real-world example of the detection and routing logic.

Add multimodal AI capabilities with image-text-to-text pipeline

This document, sourced from a Hugging Face Space commit, breaks down how a FastAPI backend was upgraded to handle both text and image inputs.

Please review the following sections to understand the end-to-end implementation: Core Integration & Message Format: Start with the sections "Core Integration" and "Message Format Support". Notice how the request payload can now contain a list of content parts, such as {"type": "image", ...} and {"type": "text", ...}. This structure is the key to our detection logic. Usage Examples: Look at the curl commands under "Usage Examples". Compare the JSON body for the "Image-Only," "Multimodal," and "Text-Only" requests. This shows you exactly how a client communicates different modalities to the API. Intelligent Content Detection: Read the "Intelligent Content Detection" section, which explicitly states that the system automatically detects the request type and routes it accordingly. Pydantic Models: In the code diffs, find the Pydantic models TextContent, ImageContent, and ChatMessage. The content field in ChatMessage uses Union[str, List[...]] to accept either a simple string (for text-only) or a list of content blocks (for multimodal). This is a common and robust pattern for defining such an API in Python. Detection Function: Locate the has_images function. This simple but crucial function iterates through the message content, checking for any part where item.type == "image". This is the detection logic in action. Routing Logic: Finally, look at the create_chat_completion endpoint. The if is_multimodal: block is the router's decision point. If has_images returns true, the request is dispatched to generate_multimodal_response; otherwise, it falls through to the existing text-only logic.

This walkthrough demonstrates a complete, self-contained implementation of modality-based routing at the application level. Given your experience with Python and building services, this pattern should be very familiar.

3. The Infrastructure View: Declarative Multimodal Routing

Implementing this logic in your application code, as shown in the FastAPI example, is a perfectly valid approach. However, for a large-scale system, handling this at the infrastructure level—within the API gateway itself—is often preferable. This aligns with the SRE principle of separating infrastructure concerns from application logic.

Let's see how a production-grade AI gateway like Kong handles this declaratively.

This diagram illustrates how a "Task Router Agent" directs a user's query to the appropriate tool based on its modality, such as an image-to-text tool or a text generation tool. Our goal is to build the infrastructure that enables this routing.

Modern API gateways are being extended with specific features for GenAI workloads. The following article from Kong's engineering blog discusses new capabilities for routing different GenAI categories.

Build a Multi-LLM AI Agent with Kong AI Gateway & LangGraph

This section of the Kong engineering blog announces new features in their AI Gateway for handling diverse GenAI models beyond just LLMs.

Focus on the section "Kong AI Gateway 3.11 and Support for New GenAI Models." Pay close attention to the new configuration parameters introduced: genai-category: This parameter allows you to classify the backend service. You can see examples like image/generation, text/generation, and audio/speech. route_type: This parameter specifies the nature of the API endpoint itself, with values like llm/v1/responses for text and image/v1/images/generations for images. By using these declarative parameters in your gateway configuration, you can build sophisticated routing rules for different modalities without writing any custom application code. The gateway handles the inspection and routing based on the route the client calls.

This infrastructure-centric approach allows you to manage multimodal routing through configuration files (e.g., YAML), which fits perfectly into GitOps and IaC workflows you're already proficient with.

Your Turn: Design the Routing Logic

Let's solidify your understanding. Based on the patterns you've seen in the FastAPI and Kong examples, take about 5-10 minutes to write down the pseudocode for a function that would sit inside a custom AI gateway.

Task:
Write a function get_target_service(request) that inspects an incoming HTTP request and decides which backend service to route it to.

The gateway exposes a single endpoint: /v1/chat/completions.
If the request body follows the multimodal format and contains an image content part, the function should return the string "vision-service.internal".
If the request body is text-only (either a simple string or a content array with only text parts), it should return "text-service.internal".
Assume you have a helper function has_images(payload) that returns True or False.

Solution

Here is one possible pseudocode implementation:




# Assume helper functions are available:
# get_request_body(request) -> returns parsed JSON payload
# has_images(payload) -> returns True if image content is present

function get_target_service(request):



  # Check if the request path matches our target endpoint
  if request.path != "/v1/chat/completions":
    return "default_service.internal" # or handle as an error




  # Get the JSON body of the request
  payload = get_request_body(request)




  # Use our helper to detect modality
  if has_images(payload):



    # Route to the service that handles vision models
    log("Routing to vision service.")
    return "vision-service.internal"
  else:



    # Route to the standard text generation service
    log("Routing to text service.")
    return "text-service.internal"

This simple logic is the heart of a multimodal router.

Conclusion

In this lesson, we moved beyond simple text-based routing to address the complexities of multimodal AI systems. You learned how to design a router that can intelligently differentiate between text and image inputs, a critical capability for modern AI platforms.

Key Takeaways:

The Problem: Different AI providers use inconsistent API formats for multimodal inputs, necessitating a normalization layer in your gateway.
The Detection Mechanism: A common standard is to inspect the request payload for a content array containing objects with a "type": "image" field.
Implementation Patterns: This routing logic can be implemented either within the application's code (e.g., using an if/else block in a FastAPI endpoint) or declaratively at the infrastructure level (e.g., through configuration in an API gateway like Kong).
Infrastructure Approach: Using declarative configuration in an AI gateway aligns with SRE and IaC best practices, separating routing concerns from application logic.

In our next lesson, we will make our router even more intelligent. We will explore how to implement policy-based routing, where decisions are made not just on content type, but on dynamic factors like cost, latency, and required model capability.