Trajectory Planning as Autoregressive Sequence Modeling

From Points to Sequences: What is a Trajectory?

Before we can model a trajectory, we must first define it. In the context of autonomous driving, a plan is not a single decision but a continuous path over time and space.

To formalize this, let's turn to a clear definition.

As the article explains, a trajectory is a sequence of waypoints that specifies the vehicle's future state (e.g., position, orientation) at a series of fixed time steps. For example, a planner might generate 20 waypoints, each 0.4 seconds apart, to define a plan for the next 8 seconds.

This is the first critical insight: a trajectory is inherently a sequence. This sequential nature makes it a natural candidate for sequence modeling techniques, the same kind used to process language.

The Paradigm Shift: From Predicting Actions to Generating Tokens

So, how do we get a model to generate this sequence? There are two fundamentally different ways to think about this, which are beautifully contrasted in the diagram below from the DAP paper.

Let's break this down:

1. The "Action" Model (Left): This is the more traditional neural network approach. It functions as a regression model. Given sensor inputs, it directly predicts a set of continuous numbers, such as the (x, y) coordinates for each waypoint in the future trajectory. This is analogous to how a CNN you might be familiar with could regress to find the continuous coordinates of a bounding box. The model is trained with a loss function like Mean Squared Error (MSE) to minimize the distance between the predicted path and the ground truth path.

2. The "Token" Model (Right): This is the revolutionary new approach. It reframes the problem as generation, just like a language model. Instead of predicting continuous coordinates, it predicts a sequence of discrete tokens. At each step, it performs a classification task: from a finite "vocabulary" of possible maneuvers, which one should come next?

Why make this shift? The primary motivation is to leverage the phenomenal success and scalability of the decoder-only Transformer architecture. As the introduction to the DAP paper states, these models have proven to scale incredibly well with data and size when trained as next-token predictors. By casting driving as a next-token prediction task, we can harness these powerful, well-understood architectures for robotics.

Defining the "Language" of Driving

If we're treating planning like language modeling, we need a vocabulary. What are the "words" that a car can "say"? This process of converting the continuous world into a discrete vocabulary is called tokenization.

The DAP architecture provides a clear example of how this is done.

The model defines a vocabulary that includes tokens for different aspects of the driving scene:

• Command Tokens: High-level instructions like turn_left, go_straight, or change_lane_right.
• Ego State Tokens: This is how continuous motion is discretized. Instead of using (x, y) coordinates, the model uses physical properties like curvature ($\kappa$) and acceleration ($a$). These continuous values are then quantized into a finite number of bins. For example, "slight left turn with gentle acceleration" might become token 512. This converts the infinite space of possible movements into a manageable, discrete vocabulary.
• Environment Tokens: The surrounding world, represented by a Bird's Eye View (BEV) map, is also converted into a sequence of discrete tokens. (We will explore exactly how this is done using VQ-VAE in a future lesson).

By tokenizing everything, the entire driving problem—the command, the car's past movements, and the surrounding scene—is transformed into a single, unified sequence of integers, just like a sentence in natural language.

Autoregressive Planning: "Speaking" the Future

Once the world is represented as a sequence of tokens, we can use a standard decoder-only Transformer to generate a plan autoregressively.

Autoregressive simply means that the prediction for each step is conditioned on all the previous steps.
$p({\text{token}}_{t+1}|{\text{tokens}}_{<t+1})$
A language model does this to predict the next word in a sentence. A driving model does this to predict the next "action token" in a plan.

The DAP paper explains this process clearly. The model takes the sequence of past tokens as input and generates the future sequence one token at a time. Crucially, it uses an interleaved strategy:

1. Predict the environment tokens for the next time step (Scene_t+1).
2. Then, conditioned on that predicted future scene, predict the ego action token for that time step (Action_t+1).

This is a powerful concept. The model is forced to first "imagine" how the world will evolve (e.g., "that car ahead will brake") before it decides on its own action ("I should slow down"). This effectively embeds a predictive world model directly into the planner.

To see a real-world application of this concept, let's watch a clip about Wayve's Lingo-2 model, which takes this idea a step further.

As the video shows, the Lingo-2 model generates a single output sequence that can contain tokens for both natural language explanations ("Slowing down for the pedestrian") and action tokens (the trajectory itself). This is the ultimate expression of the paradigm: driving commands and descriptive language are just different "words" in the same vocabulary, generated by the same underlying sequence model.

Conclusion

In this lesson, we have demystified how the abstract task of trajectory planning can be practically reformulated as an autoregressive sequence modeling problem. This conceptual leap is what allows the autonomous driving field to build upon the tremendous progress made in large language models.

Key Takeaways:

• Trajectory as a Sequence: A vehicle's plan is a sequence of waypoints, making it suitable for sequence modeling.
• The Power of Discretization: The core idea is to tokenize the continuous world. Instead of regressing continuous values, the model learns to classify the next "action token" from a discrete vocabulary. This allows us to use powerful Transformer architectures.
• The "Language" of Driving: The vocabulary consists of tokens representing commands, discretized ego-motion (like curvature and acceleration), and the environment.
• Autoregressive Generation: A decoder-only Transformer predicts the next token in the sequence conditioned on the past, effectively "writing out" the plan step-by-step. This process can even interleave predictions about the world with actions for the ego-vehicle.

Preview of the Next Lesson:

Now that you understand the high-level framework of reformulating planning as a language modeling task, our next lesson will be a Deep Dive into the DAP architecture. We will build directly on today's concepts to examine its tokenization scheme in greater detail, analyze its interleaved generation strategy, and understand how it functions as a unified world model and planner.