This first module builds the habit of turning claims about a model’s internals into claims that can be checked. Before investigating whether a feature represents a concept or whether a component causes a behavior, you need a precise map of where information can travel. This lesson traces one transformer block from its incoming residual stream, through attention and an MLP, to its outgoing stream and logits.
Allow about 40 minutes. The aim is not to identify what a particular head means yet. It is to distinguish the vector each component writes from the computation that produced it—and from the effect that changing it might have.
The residual stream is the running state
A transformer maintains a vector at each token position. Collect those vectors into a matrix , with one row per position. The model starts with token embeddings, usually alongside positional information. Each block receives the current residual stream, computes updates, and adds them to that stream rather than replacing it.
Use the supplied diagram as a map of one block:
The vertical line in the diagram is the residual stream. At its first addition point, it retains the incoming vector while receiving the attention output. At its second, it retains that combined vector while receiving the MLP output. The diagram’s labels the final state at the unembedding; it is not a subtraction from .
The video offers a visual account of what an attention head contributes. Treat its adjective-and-noun example as an illustration of a possible learned operation, not an interpretation established for a real head.
Attention in transformers, step-by-step | Deep Learning Chapter 6
Watch “Attention in transformers, step-by-step” by 3Blue1Brown for a visual account of how attention produces updates at token positions and how multiple heads contribute.
Watch value updates, focusing on the distinction between selecting source positions and producing an update at a destination position. Then watch parallel heads, focusing on how their suggested modifications are combined. The first passage simplifies a head’s value mapping; the equations below make the output projection explicit.
For the architectural account, read the opening and two short passages of A Mathematical Framework for Transformer Circuits. Its additive view is especially useful when you later need to decide exactly which activation a hook captures.
A Mathematical Framework for Transformer Circuits
Read the architecture overview in “A Mathematical Framework for Transformer Circuits” by the Transformer Circuits authors to connect the diagram to the residual-stream equations.
At the start of the article, before “Virtual Weights and the Residual Stream as a Communication Channel,” read the block overview. In “Virtual Weights and the Residual Stream as a Communication Channel,” locate the paragraph beginning “The residual stream is simply the sum”; read the stream description. Finally, in “Attention Heads are Independent and Additive,” read from the discussion beginning “In the original Vaswani” through the head decomposition. Follow the equality between concatenating head results and summing their projected outputs.
Trace the two additions
Let be the stream entering a block. For each attention head , let mean its output after projection back into residual-stream dimensions. The block’s attention update is the sum of those head outputs:
All heads in this attention layer read the same incoming stream and run in parallel. They do not read one another’s newly written outputs within this layer. A conventional implementation may concatenate their smaller results and apply one output matrix; splitting that matrix into head-specific pieces gives the additive expression above.
At destination position , an idealized head output can be written as
Here is the attention weight assigned by destination to source . The head reads source-position vectors, combines their value projections, and writes one resulting vector at . In a causal language model, the sum excludes future positions. Crucially, an attention pattern shows where a head reads, not by itself what information it moves or what its write does to the prediction.
Next the MLP reads the post-attention stream, not the original block input. If denotes its output in residual-stream dimensions, the second addition is
That last form is the trace to keep in mind. The MLP adds another vector, but its vector can depend on what the heads just wrote. The MLP operates separately at each token position; unlike attention, this step does not itself select information from other positions.
In a model with more blocks, becomes the next block’s incoming stream. Earlier writes remain in the running sum unless subsequent components modify or counteract them.
From a component write to a logit
If this is the final block, an idealized unembedding applies a linear map to the residual vector at the prediction position . For candidate token ,
Substituting the block trace gives a direct accounting of that logit:
Thus a component’s direct logit contribution is its actual write, projected onto a candidate token’s unembedding direction. The contributions add exactly in this idealized final-block calculation. A useful alternative is the logit difference between candidates and :
Projecting each write onto shows whether it directly favors over , even when it raises or lowers both individual logits.
The TransformerLens Exploratory Analysis Demo develops this accounting under the name direct logit attribution. Read its qualification about final layer normalization as carefully as its additive argument.
Exploratory Analysis Demo - TransformerLens Documentation
Read the “Direct Logit Attribution” discussion in the TransformerLens exploratory analysis demo to see what additive residual writes permit researchers to measure—and where the simple unembedding equation needs qualification.
In “Direct Logit Attribution,” begin in the “Background” paragraph at “The central object of a transformer” and read the attribution explanation, stopping before the task-specific metric discussion. Then find the “Ignoring LayerNorm” subsection and read the scaling caveat. Focus on why fixing the final normalization scale permits a linear decomposition, rather than treating normalization as inherently linear.
Real models may apply a final LayerNorm before unembedding, and implementations may have biases or other architectural details. Because a normalization scale depends on the combined stream, applying the unembedding independently to every unscaled write is not generally an exact decomposition of the real logits. The simplified equation is exact for the stated idealized block; practical attribution must account for the actual model’s final computation.
A complete trace with numbers
Consider one token position in a toy final block. Its incoming vector and two head writes are
After attention, . Suppose the MLP computes
It therefore writes , leaving . Let the unembedding directions for two candidate tokens be and , with zero biases.
| Residual-stream term | Vector | Contribution to | Contribution to |
|---|---|---|---|
| Incoming stream | |||
| Head 1 write | |||
| Head 2 write | |||
| MLP write | |||
| Total |
The final logit difference is . Head 1’s direct contribution to that difference is , and the MLP’s is . Those figures correctly describe the writes observed in this run.
They do not mean that disabling head 1 would change the difference by only . If its write were removed while the rest of this toy computation were rerun, the MLP would receive and write . The resulting logit difference would be , a change of . The arithmetic makes the distinction precise: decomposing the original output holds the observed MLP write fixed; changing a head and recomputing can change that downstream write too. In an earlier block, still later blocks could respond as well.
This is why a large projected write is a useful clue, not yet a complete causal account of what a component does. Conversely, a small direct logit contribution does not rule out an important effect through downstream components.
What this trace establishes
You can now follow a component through a supplied block: identify the incoming stream, locate each head’s projected write, sum the parallel head writes with the incoming stream, evaluate the MLP on that post-attention state, and add its write. At the final position, an unembedding converts the resulting state into logits.
The residual stream’s additive structure makes direct contributions inspectable. It does not make the computations producing those contributions independent, nor does it turn an attractive visualization into evidence that a component implements a proposed concept. The next lesson compares neurons, activation directions, sparse-autoencoder latents, and circuits as different possible units for making such an explanation.