Welcome back. Last lesson, you used naturally occurring activations as replacements and learned to keep causal claims conditional on the donor, recipient, and behavior tested. This lesson changes the intervention: instead of swapping in a recorded activation, you will add a chosen direction to an activation. The central question is not merely whether the output changes, but whether that direction produces a more targeted change than a sham intervention or an equally large perturbation in a random direction.
From a contrast to an intervention
A common way to propose a direction is to collect prompts associated with opposite sides of a property, record activations at the same layer and token position, and subtract their means. The result is a candidate direction, not yet an established representation of the property.
Watch the short difference-in-means explanation in Causal Mechanistic Interpretability (Stanford lecture 1), presented by Atticus Geiger on Goodfire. Focus on the distinction between constructing a vector from contrasting examples and testing what happens when it is injected.
Causal Mechanistic Interpretability (Stanford lecture 1) - Atticus Geiger
Watch the difference-in-means steering example in Atticus Geiger’s lecture to see how contrasting prompt groups yield an intervention direction.
In the opening activation-steering segment, watch the contrast method. Follow where the activations are recorded, how their group means are compared, and what is subsequently added to the model. Treat the claim that the vector captures the intended property as a hypothesis to test.
The supplied Activation Addition diagram shows the corresponding single-pair version: activations from two contrasting prompts are differenced, and that difference is injected during a separate user-prompt run. The diagram depicts the proposed intervention, not its controls.

Read the mechanism in Activation Addition: Steering Language Models Without Optimization. It specifies both the point of injection and the coefficient controlling the added vector.
Activation Addition: Steering Language Models Without Optimization
Read the paper’s description of activation addition to pin down what is changed in the forward pass.
In the main text’s “Activation engineering” discussion, find the passage beginning “Activation addition.” near Algorithm 1 and Figure 1. Read the construction and injection. Stop after the explanation of the injection coefficient. Notice that the recorded difference is added at a selected location; it does not replace the entire activation as in last lesson’s patch.
Let be the activation at a prespecified site for evaluation prompt . Given positive and negative construction prompts, define
Here is the norm of the added perturbation. Normalizing first makes its magnitude explicit: an injection coefficient of applied to an unnormalized vector from one experiment need not be comparable to a coefficient of applied to a different vector. The construction prompts should be separate from evaluation prompts; otherwise a direction chosen because it works on particular examples can look more reliable than it is.
Suppose the intended behavior is a higher logit margin for a designated answer than for a specified alternative. For each held-out prompt, use the same scoring rule under every condition:
Using logits on a fixed prompt avoids mistaking sampling variation in generated text for a change caused by the hook. If the claim concerns longer generated responses, evaluate those too, with generation settings held fixed and an explicit target-behavior measure.
What each control rules out
A sham run installs the same hook at the same site but adds a zero vector. It checks whether the apparent effect requires a nonzero addition, rather than a difference in execution, scoring, or hook handling. The sham should agree with a no-hook run up to expected numerical variation. It does not tell you whether the chosen direction matters: almost any sufficiently large nonzero addition could differ from zero.
For that comparison, draw random directions , normalize each one, and add the same norm :
Use many draws, not one fortunate or unfortunate random vector. For a candidate direction applied consistently across prompts, keep each sampled random direction fixed across the same evaluation prompts before drawing the next one. Match the layer, tensor site, token position, timing, number of injections, and perturbation norm. During generation, “once at the prompt’s last token” and “at every generated token” are different interventions; the controls must use whichever policy the candidate uses.
This comparison asks whether the candidate produces an unusual effect relative to isotropically sampled directions of equal size. It does not automatically control for every property of a learned direction. The Circuits Updates – April 2024 discussion makes that limitation concrete using preliminary feature-intervention experiments. Sparse-autoencoder features are a later topic in this course; for now, the relevant point is why a random vector in the raw activation space can be a weak comparator.
Read Jack Lindsey’s comparison of learned feature interventions with magnitude-matched perturbations to understand both the value and the weakness of a random-direction baseline.
In “How Strongly do Dictionary Learning Features Influence Model Behavior?”, begin with the paragraph explaining the experiment and read the control design. Stop after the bullet describing perturbations in feature-activation space. Focus on the stated concern that activation variability is not necessarily isotropic: equal-length raw-space vectors need not perturb equally consequential parts of the model.
A learned candidate may point into directions where ordinary model activations vary substantially, while a uniformly random direction may largely occupy less consequential dimensions. Beating random vectors then supports a narrower claim—special relative to that sampling procedure—not uniqueness to the proposed concept. A stronger follow-up might draw controls matched to the activation distribution or compare against other meaningful, similarly selected directions. Those are additional tests, not substitutes for reporting the simple random baseline honestly.
The Activation Addition paper uses a norm-matched random-vector experiment. Its result also illustrates why “changed the model” and “changed the intended behavior” must be measured separately.
Activation Addition: Steering Language Models Without Optimization
Read Appendix H to see an actual random-vector comparison and the difference between broad distributional change and a targeted steering effect.
In “Appendix H Investigating random ActAdd vectors,” begin where the authors introduce injections of random vectors with similar magnitudes. Read the comparison, ending after the discussion of Figure 11. Track two separate observations: what happens to the qualitative output and how much the next-token distribution changes. A larger distributional shift is not, by itself, better steering.
A matched evaluation in PyTorch
Given your familiarity with activation hooks, the crucial implementation detail is to make the hook identical across conditions except for delta. The following scaffold assumes block returns a tensor of shape [1, sequence, hidden], prompts are run individually without padding, and the intervention targets the last prompt position. A module returning a tuple needs an adapted hook that changes only the intended tensor.
def score_with_delta(ids, delta, block, model, desired_id, alternative_id):
def intervene(_module, _inputs, output):
changed = output.clone()
changed[:, -1, :] += delta.to(output)
return changed
handle = block.register_forward_hook(intervene)
try:
with torch.inference_mode():
logits = model(ids).logits[:, -1, :]
margin = logits[:, desired_id] - logits[:, alternative_id]
return margin.item()
finally:
handle.remove()
Set model.eval() before comparing runs. Use torch.zeros_like(v) for the sham, for the candidate, and for each random control. If you batch variable-length prompts, replace -1 with each prompt’s actual selected position; a padded final position would test a different intervention. Confirm that a zero-addition hook reproduces the no-hook score before trusting any contrast.
For each held-out prompt, record the candidate change relative to sham,
and repeat the same calculation for every random direction. Aggregate over the same prompts for every condition. The random controls then give a distribution of aggregate effects, against which you can place the candidate’s aggregate effect. Report the number of prompts and random draws, not just a favorable example.
A compact result might look like this; the numbers are illustrative:
| Condition | Mean target margin | Off-target task accuracy |
|---|---|---|
| Sham | ||
| Candidate direction | ||
| Random directions: median | ||
| Random directions: upper 95th percentile of target margin | — |
At this site and strength, the candidate raises the target margin well beyond most sampled equal-norm random directions, while the reported off-target task changes little. That is evidence of selective controllability relative to these controls. It would be weaker evidence if random directions routinely raised the target margin just as much, or if the candidate’s apparent gain accompanied a severe loss of ordinary task performance. An off-target metric matters because a direction that makes the model favor one answer on every question can improve a narrow target margin without performing the intended discrimination.
Keep both axes of evaluation visible. A random direction could shift the overall next-token distribution more than the candidate yet produce little of the desired behavior. Conversely, a candidate could produce the desired shift while disrupting coherence or instruction-following elsewhere. Neither a dramatic completion nor a single distribution-distance statistic resolves that trade-off.
The causal claim you can carry forward
A successful, controlled addition establishes that forcing the model’s activation along this direction at this site can alter its downstream behavior, and that the effect is not well explained by a zero-addition run or by the tested equal-norm random directions. It does not show that the unmodified model normally uses that direction to generate the behavior, that the direction represents only the named concept, or that a suspected shortcut has been removed. Contrast prompts can differ in wording and other cues; those differences may also enter . And adding a vector can create an internal state the model rarely reaches naturally.
For a shortcut hypothesis, keep the evaluation distribution particularly disciplined: preserve task-relevant evidence while varying the nuisance cue, and assess whether steering changes reliance on that cue rather than simply shifting all answers. The directional control establishes a baseline for specificity of intervention; matched input counterfactuals and other causal tests are needed for a fuller account of reliance.
The key takeaway is an experimental hierarchy: sham isolates the effect of adding something rather than running the hook; magnitude-matched random directions test whether the candidate is more targeted than generic additions of the same size. Both comparisons depend on matched sites, prompts, scoring, and injection policy. Next lesson, you will vary the intervention strength rather than testing only one norm, tracing how target effects and off-target costs change together.