From Pixels to Prompts: Vision Encoders Explained

Introduction

Welcome to the first lesson in our module on Multimodal Models and Specialist LLMs. In the previous module, we concluded our deep dive into the DeepSeek-R1 training pipeline by examining the final alignment stage. We saw how a hybrid reward system, combining rule-based checks for reasoning and preference-based models for general helpfulness and safety, transforms a powerful reasoning engine into a polished, user-ready AI assistant.

Now, we shift our focus from purely text-based models to the exciting world of multimodality. State-of-the-art models like GPT-4o and the generative planners central to this course, such as DAP and Plan-R1, have the ability to "see." They can process visual information and reason about it, a critical capability for autonomous driving and many other applications.

This lesson addresses the fundamental mechanism that makes this possible. Our learning outcome is to explain how vision encoders produce visual tokens that can be processed by a language model. We will deconstruct the process of converting a grid of pixels into a sequence of vector embeddings that a Transformer can understand, laying the groundwork for understanding how an LLM perceives the world.

The Challenge: Why Can't Language Models See?

At their core, Large Language Models (LLMs) are sequence processors. They are designed to operate on a sequence of tokens, where each token is represented by a high-dimensional vector (an embedding). A text sentence is naturally a sequence of words or sub-words, but an image is a continuous grid of pixels.

Let's train Vision Language Models (VLM) from scratch using just Text-Only LLMs!

To start, let's watch a brief segment that clearly articulates this core problem.

Watch the first 33 seconds of this video from the Neural Breakdown channel. It perfectly frames the challenge: you can't just pass raw image pixels to a text-only LLM.

To bridge this modality gap, we need a component that can translate an image into the "language" of the Transformer: a sequence of vector embeddings. This component is the vision encoder.

The Vision Encoder Pipeline: From Pixels to Tokens

The dominant architecture for modern vision encoders is the Vision Transformer (ViT). The key insight of ViT is to restructure the image into a sequence of smaller pieces that a standard Transformer can process. This involves a few key steps.

Let's look at the overall architecture first.

Vision Transformer (ViT) Architecture for Visual Tokenization — This diagram shows the standard Vision Transformer pipeline. An image is broken into patches, which are then converted into a sequence of embeddings (visual tokens) and processed by a Transformer Encoder. This is the fundamental process for creating visual tokens.

We will now break down this process step-by-step.

1. Image Patching

The first step is to divide the input image into a grid of non-overlapping patches. For example, a 224x224 pixel image might be divided into a 16x16 grid of patches, where each patch is 14x14 pixels. This simple but powerful step converts the 2D image grid into a 1D sequence of patches.

PATCH EMBEDDING | Vision Transformers explained

This video from ExplainingAI provides a clear, animated walkthrough of the patch embedding process.

Watch from 01:11 to 01:54 to see how an image is divided into a sequence of patches.

2. Flattening and Linear Projection

Each 2D patch is then "flattened" into a single, long 1D vector of pixel values. For a 14x14 patch with 3 color channels (RGB), this results in a vector of size 14 * 14 * 3 = 588.

However, this raw pixel vector is not yet in a format the Transformer can use. The Transformer layers expect inputs of a consistent, fixed dimension (e.g., 768 or 4096). Therefore, each flattened patch vector is passed through a standard linear layer (a fully-connected layer) that projects it from the raw pixel dimension (e.g., 588) to the required Transformer embedding dimension, $D$ .

The output of this projection is a sequence of vectors, each of dimension $D$ . These are the visual tokens.

PATCH EMBEDDING | Vision Transformers explained

Continuing with the ExplainingAI video, let's see how these patches are converted into vector embeddings.

Watch from 01:54 to 02:55. Focus on how each patch is flattened and then projected into the target Transformer dimension, D.

3. Contextualization with Self-Attention

At this point, we have a sequence of independent patch embeddings. The model doesn't know how they relate to each other. This is where the power of the Transformer architecture is leveraged.

The sequence of visual tokens (along with added positional embeddings to retain spatial information) is fed into a series of Transformer encoder layers. Just as with text, the self-attention mechanism in these layers allows each patch to "look at" every other patch, calculating attention scores to build a contextualized representation. Early layers might learn low-level features like colors and textures, while deeper layers learn to recognize high-level concepts and object parts.

Let's train Vision Language Models (VLM) from scratch using just Text-Only LLMs!

The Neural Breakdown video provides an excellent explanation of how a pre-trained Vision Transformer performs this contextualization step.

Watch from 05:39 to 08:19. The key concept here is how self-attention transforms independent patch embeddings into a set of 'context-aware' visual tokens.

This process transforms raw pixels into a sequence of rich, contextualized vector embeddings that carry visual meaning.

Integrating Visual Tokens with a Language Model

Now that we have a sequence of visual tokens, how do we get the LLM to process them? The final step involves making these visual tokens compatible with the LLM's own text embeddings and feeding them into the model.

Let's consult a technical survey for a precise, architectural overview.

Vision Encoders in Vision-Language Models: A Survey

This survey paper from Jina AI provides a canonical diagram and a formal definition of the vision encoder's role.

First, carefully examine Figure 1, which shows the complete architecture. Then, read Section 2.1 ('Scope and Definitions') to understand the formal definition of a vision encoder and visual tokens. Finally, read Section 2.2 ('Vision Transformer Architecture') for a mathematical summary of the process we've just discussed.

As illustrated in the paper, the standard approach involves two final steps:

Projection: The visual tokens from the vision encoder (e.g., dimension 1024 from a CLIP model) are passed through a projection module, typically a simple Multi-Layer Perceptron (MLP). This MLP maps the visual embeddings from the vision encoder's dimension to the language model's embedding dimension (e.g., 4096 for Llama).
Concatenation: These projected visual tokens are now treated as if they were word tokens. They are concatenated with the embeddings of the text prompt to form a single, unified input sequence that is fed into the LLM.

The LLM then performs self-attention across this combined sequence of text and image tokens, allowing it to reason about the visual information in the context of the textual query.

Beyond Simple Projection: Advanced Connectors and Tokenizers

While a simple MLP projector is surprisingly effective, it's not the only method. The design of this "connector" module is an active area of research.

Q-Former: Architectures like BLIP-2 use a more complex connector called a Q-Former. It uses a small set of learnable query vectors (e.g., 32) that use cross-attention to "pull" the most relevant information from the larger set of visual tokens produced by the ViT. This acts as an intelligent compression step, creating a short, fixed-length sequence of visual tokens to pass to the LLM.
Discrete Visual Tokens (VQ-VAE): The methods above produce continuous vector embeddings. An alternative approach, which is highly relevant to the DAP planner we will study later, is to create discrete visual tokens. A Vector-Quantized Variational Autoencoder (VQ-VAE) is used for this.

Vector Quantized Variational Autoencoder (VQ-VAE) Architecture — This diagram illustrates a VQ-VAE. The encoder maps an image to a latent grid. Each vector in this grid is then "snapped" to the nearest vector in a learned, finite 'Codebook'. The index of that codebook vector becomes the discrete visual token (e.g., token ID 85, 12, 1023, ...). This makes visual tokens structurally identical to text token IDs.

This VQ-VAE approach is used in models like Google's Chameleon and the Discrete-token Autoregressive Planner (DAP), as it allows the model to treat images as just another "language" to be generated autoregressively.

Conclusion

In this lesson, we have demystified the process of how a language model can "see." We've traced the path from a raw image to a sequence of visual tokens that an LLM can process.

Key Takeaways:

A vision encoder is a neural network that transforms an image into a sequence of vector embeddings, known as visual tokens.
The Vision Transformer (ViT) is the standard architecture for this. It works by:
1. Dividing the image into patches.
2. Flattening each patch and using a linear layer to project it into a fixed-dimensional embedding.
3. Using self-attention to contextualize the patch embeddings, creating information-rich visual tokens.
To be processed by an LLM, these visual tokens are passed through a projection module (like an MLP) to match the LLM's embedding dimension.
The final visual tokens are concatenated with text tokens and fed into the LLM as a single input sequence.
Advanced methods exist, such as the Q-Former for compression and VQ-VAE for creating discrete visual tokens, which are crucial for certain generative planning architectures.

Preview of the Next Lesson:
We've seen how visual and text tokens can be brought into a common dimensional space. In our next lesson, we will delve deeper into unified embedding spaces, exploring how models like GPT-4o take this a step further to natively handle not just images and text, but also audio, within a single, seamless latent representation.

Recommended resources