DAP Tokenization for Ego History, Navigation, and Environment

Part 1: The Three Modalities of DAP

As we've discussed, a generative planner like DAP operates by predicting a sequence of tokens, similar to how a large language model predicts the next word. To do this, it must first convert all its inputs into a compatible, discrete token format. DAP considers three primary input modalities, each with its own tokenization scheme.

The following table from your course notes provides a high-level summary, which we will deconstruct in detail.

Let's explore each of these, starting with the simplest and building on what we learned in the last lesson.

1. Environment: Scene Tokens (Recap)

As we covered in detail previously, the environment around the vehicle is captured in a BEV feature map. A VQ-VAE then processes this continuous map, quantizing it into a grid of discrete scene tokens. Each token represents a learned semantic concept about a small patch of the world.

2. Navigation Command: The Command Token

This is the most straightforward tokenization. The planner needs to know its high-level goal, which is provided as a navigation command (e.g., "Go Straight," "Turn Left," "Change Lane Right").

Since there is a small, finite set of possible commands, they are treated as categorical variables.

• Process: Each command is assigned a unique integer ID. For example: 0 for "Go Straight," 1 for "Turn Left," etc.
• Output: This integer ID becomes the command token. When fed into the Transformer, this token ID is mapped to a corresponding embedding vector, just like any other token.

The DAP paper formalizes this by stating that the command c is converted into a one-hot vector, which is a standard way to represent categorical data for neural networks.

Part 2: Ego History - Tokenizing the Vehicle's Trajectory

The most intricate part of DAP's tokenization scheme involves representing the vehicle's own past movements. A trajectory is inherently continuous—a smooth path through space. How can we discretize this effectively?

DAP's approach is to first transform the trajectory from world coordinates into a representation that directly relates to vehicle controls: curvature and acceleration.

• Curvature ($\kappa$): This measures how sharply the vehicle is turning. A straight path has $\kappa =0$, while a tight turn has a high magnitude of $\kappa$. It's directly related to the steering angle.
• Acceleration ($a$): This is the rate of change of the vehicle's speed. Positive values mean speeding up; negative values mean braking.

This κ-a representation is powerful because it captures the vehicle's dynamics in a way that is independent of its absolute position and heading.

The Conversion and Discretization Process

The DAP paper outlines a two-step process to get from raw ego states to discrete action tokens.

1. Conversion to Continuous κ-a:
   The model takes a sequence of ego states $(x_t,y_t,{\psi }_t)$ (position and yaw angle at time $t$) and converts them into continuous curvature ${\kappa }_t$ and acceleration $a_t$ values using the following kinematic equations:

   $$\begin{array}{rlrl}{v}_t& =\frac{\sqrt{(x_{t+1}-x_t{)}^2+(y_{t+1}-y_t{)}^2}}{\Delta t}& & \text{(Velocity)}\\ {\omega }_t& =\frac{\text{wrap}({\psi }_{t+1}-{\psi }_t)}{\Delta t}& & \text{(Angular Velocity)}\\ {\kappa }_t& =\frac{{\omega }_t}{\max (v_t,\epsilon )}& & \text{(Curvature)}\\ a_t& =\frac{v_{t+1}-v_t}{\Delta t}& & \text{(Acceleration)}\end{array}$$

   Here, $\epsilon$ is a small constant to prevent division by zero.

2. Discretization:
   The continuous ${\kappa }_t$ and $a_t$ values are then discretized using a uniform grid. Imagine a 2D grid where the x-axis is acceleration and the y-axis is curvature. The range of possible values for each is divided into a fixed number of bins (e.g., 256 bins for curvature, 128 for acceleration). Each continuous $({\kappa }_t,a_t)$ pair falls into one specific cell of this grid, which is identified by two integer indices: $i_t^{\kappa }$ and $i_t^a$.

3. Packing into a Single Token (Optional but Common):
   These two indices are then "packed" into a single integer token, the action token $A_t$, using the formula:
   $A_t=i_t^{\kappa }\cdot N_a+i_t^a$
   Where $N_a$ is the total number of acceleration bins. This is a standard technique to map a 2D index to a 1D index.

The result is a single integer, $A_t$, that concisely represents the vehicle's motion at that timestep.

This choice of a discrete action representation is a key design decision. To understand the trade-offs involved, the following video discusses the pros and cons of discrete vs. continuous action spaces. While the video advocates for a continuous approach (diffusion), it provides excellent context for why DAP's discrete method is a valid and common alternative.

Part 3: Assembling the Final Token Sequence

Now that we have tokenized all three modalities, we can assemble the final input sequence for the Transformer. DAP arranges the tokens in a specific, interleaved format that encodes the historical context.

The sequence structure is defined as:
${\mathbf{z}}_{t^{\ast }-H,t^{\ast }}=[C_{t^{\ast }},V_{t^{\ast }-H},A_{t^{\ast }-H},\dots ,V_{t^{\ast }},A_{t^{\ast }}]$

Let's break this down:

• ${\mathbf{z}}_{t^{\ast }-H,t^{\ast }}$: The full input sequence, covering a history of $H$ steps up to the current time $t^{\ast }$.
• $C_{t^{\ast }}$: The sequence begins with a single Command Token for the current goal.
• $V_t$: A block of Scene Tokens representing the environment at time $t$.
• $A_t$: A single Action Token representing the ego vehicle's action at time $t$.

The sequence structure is therefore: [Command, Scene_past, Action_past, Scene_past+1, Action_past+1, ..., Scene_present, Action_present].

This format allows the autoregressive Transformer to learn the relationships between the environment, the ego-vehicle's actions, and the overall navigation command. It learns patterns like "When the command is 'turn left' and the scene looks like an intersection, a high-curvature action token is likely to follow."

Conclusion

In this lesson, we have fully detailed the tokenization scheme that underpins the Discrete-token Autoregressive Planner (DAP). By converting all inputs—environment, commands, and ego history—into a unified sequence of discrete tokens, DAP transforms a complex robotics problem into a format solvable by standard sequence modeling techniques.

Key Takeaways:

• DAP uses a multi-modal tokenization scheme to represent the state of the world and the vehicle's actions.
• Navigation Commands are treated as categorical variables and converted into a single Command Token.
• Ego History is first converted from $(x,y,\psi )$ coordinates into a more kinematically meaningful curvature-acceleration ($\kappa -a$) representation. These continuous values are then discretized via uniform binning to produce a single Action Token per timestep.
• The Final Input Sequence for the Transformer interleaves Scene Tokens and Action Tokens over a historical window, preceded by a Command Token. This structure provides rich context for the model to learn the "grammar" of driving.

Preview of the Next Lesson:

We now have a complete picture of how DAP prepares its input. The next logical question is: how does it generate its output? In our next lesson, we will dive into DAP's autoregressive interleaving strategy. You will see how the model predicts future scene tokens before generating its own action token for that step, a process that effectively embeds a predictive world model directly within the planner.