Hello. An interpretability signal can be useful at one checkpoint and misleading at the next. Fine-tuning changes the model that produces the activations, even if the probe, sparse autoencoder (SAE), or steering direction used to inspect them stays fixed. This lesson develops a way to assess that change: compare an unchanged readout with newly fitted readouts, then repeat the relevant causal intervention at each checkpoint. The aim is to tell apart a disappearing behavior, a changing representation, and a signal that was never a reliable mechanism in the first place.
What does a fixed signal measure?
Suppose you have checkpoints . For a specified layer and token position, let be the activation produced by checkpoint on input . You fit a probe at , then freeze its weights and apply it to later activations:
A change in this score tells you that the old readout responds differently to activations from the later model. It does not, by itself, tell you that the concept, behavior, or mechanism has disappeared.
The supplied schematic depicts a related setup: a series of reinforcement-learning checkpoints is examined with one fixed SAE. It shows feature activations being compared over training, followed by attempts to interpret features and steer behavior.

The separation matters. A shift in a feature’s mean activation can result from a change in how the model represents a concept, but it can also reflect which inputs were sampled, a change in activation scale, or a feature responding to something other than the intended concept. The feature labels and performance figures drawn in the schematic are illustrations of analyses to perform, not evidence that measuring drift establishes a mechanism.
Learn MI gives a concise account of the risk when a training objective targets an internal coordinate.
Interpretability-Guided Training | Learn MI
Read Learn MI’s discussion to see why keeping an old signal small is not the same as keeping the associated behavior small.
First, in the passage immediately before “Turning Features into Rewards,” read from the question beginning “Suppose a projection penalty stays near zero” through the warning. Then go to “Failure Modes and Evaluation.” Read the five checks, focusing on “Refresh the readout” and “Keep behavioral holdouts.”
If training penalizes projection onto one direction, the optimizer can satisfy that constraint while preserving a shortcut in another direction or layer. A near-zero penalty is evidence about the penalized coordinate. Whether shortcut reliance fell is a separate behavioral question.
Compare old and new readouts without moving the benchmark
A refreshed readout is fitted using activations from checkpoint . It asks whether the property remains readable somewhere in the chosen activation site, even if the old mapping no longer reads it well:
For a comparison you can trust, hold the evaluation task fixed while fitting the readout anew. Use the same held-out, labeled examples at every checkpoint, with the same layer, token-position rule, target definition, and metric. Fit each only on a separate training split; use validation data for choices such as regularization strength. Neither the old nor the fresh probe should be fitted on the final comparison set.
It helps to record both discrimination and score scale. Probe AUROC, for example, asks whether positive examples tend to outrank negative ones; score distributions or calibration show whether an old numerical threshold still means what it used to. Raw mean scores alone are especially fragile: an overall increase can occur even when positives and negatives remain just as separable—or become less separable. Keep concept prevalence and relevant context groups visible rather than collapsing everything into one average.
Consider a hypothetical shortcut cue in a question-answering task. At , the fixed probe distinguishes cue-present from cue-absent examples well. At , it performs near chance, while a fresh probe at the same site performs well on held-out examples. That pattern supports changed readout geometry, not shortcut removal. If a fresh probe succeeds only at a different layer, the evidence instead suggests possible rerouting across sites. In either case, probe success establishes that information is decodable, not that the model uses it to answer.
For SAE latents, refreshing requires extra care. A fixed SAE assigns the same encoder and latent index to activations at every checkpoint, making a longitudinal score easy to compute. A newly trained SAE may produce a latent with similar examples but a different index or decoder direction. Do not equate latent number across two SAEs merely because the numbers match. Check held-out activation examples, counterexamples, and decoder-direction similarity when proposing a correspondence.
The following research study provides a concrete reason not to treat a persistent semantic label as a stationary direction.
Tracking the Feature Dynamics in LLM Training: A Mechanistic Study
Read the study’s account of how SAE feature meanings and feature directions evolve across checkpoints. Its tracked SAEs are not identical to a frozen SAE; that distinction is useful here.
In Section 3, “Semantic Evolution of Features,” start at the definition of a concept-level feature and read the evolution patterns through the “Grouping” paragraph, stopping before Section 4. Then read Section 5, “Analysis of Feature Drift,” including subsections 5.1 “Decoder Vector Evolution” and 5.2 “Trajectory Analysis”: begin at the drift analysis, ending just before Section 6. Focus on the distinction between stable semantic associations and changing decoder-vector directions; you need not work through the intervening geometric measures in Section 4.
The study follows SAEs across training using successive initialization and reports that feature directions can continue drifting after a feature has acquired recognizable meaning. Its cosine-similarity analysis measures geometric alignment of decoder vectors. That is informative about a proposed feature’s stability, but it does not establish that the model uses the feature for a behavior. Conversely, an unchanged fixed-probe score need not mean an unchanged mechanism: downstream computations can change while that score remains stable.
Repeat the causal test at every checkpoint
A readout comparison diagnoses what is accessible to the measurement. To assess a mechanism, repeat the intervention that originally supported the claim. If the claim is “this representation contributes to shortcut-driven answers,” use matched inputs that preserve task evidence while changing the shortcut cue. At each checkpoint, measure the unmodified model first, then apply the intervention and measure its effect on the same kind of examples.
For a reminder of what makes an internal intervention a test of a particular computational claim, watch this short example.
Causal Mechanistic Interpretability (Stanford lecture 1) - Atticus Geiger
Watch “Causal Mechanistic Interpretability (Stanford lecture 1)” by Atticus Geiger on Goodfire for its carry-variable example. It illustrates why an intervention needs a predicted output, not merely a noticeable effect.
Watch the carry test. Attend to the distinction between locating a value that can be changed and testing whether the change produces the output predicted by the proposed mechanism across counterfactual cases. The example concerns one model; here you will repeat that logic across checkpoints.
Here is a practical checkpoint protocol. Let be a preselected measure of shortcut-driven behavior for —for example, a logit preference for the answer favored by the nuisance cue over the answer supported by the task evidence. For intervention , where denotes the fixed or fresh target, report the paired effect
The sign and size of have meaning only alongside the unmodified behavior , the exact replacement baseline, and off-target performance. For example, removing a shortcut-associated coordinate could lower the shortcut metric because it damages answer quality generally. Track task accuracy on cue-neutral examples as a check.
Keep the kind of intervention consistent across checkpoints. If you tested a probe direction by replacing its component of an activation with a reference value, repeat that operation for both the frozen direction and each refreshed direction; a probe weight is not automatically a clean causal feature. If you tested an SAE latent, preserve the original activation’s reconstruction residual when removing that latent, so effects are not confounded with wholesale SAE reconstruction error. Use suitable sham or magnitude-matched controls, and compare effects on held-out matched inputs rather than selected examples that made the feature look promising.
Absolute intervention magnitudes also need attention. A numerical dose copied from may be negligible or extreme at if activation scales have changed. Report the fixed-dose result and, if comparing efficacy at comparable relative strength, a separately specified scale-matched result. Do not silently retune the intervention until it works: that would answer a different question.
Read the three lines of evidence together
The most informative record is a checkpoint-by-checkpoint panel: external behavior, fixed versus fresh readability, and fixed versus fresh intervention effects. These observations constrain different claims.
| Observed pattern on held-out examples | Reasonable interpretation | What remains unresolved |
|---|---|---|
| Old probe weakens; fresh probe succeeds; shortcut behavior persists | The old signal is no longer a dependable monitor; the information is still readable at the tested site. | Whether the refreshed representation drives the shortcut. |
| Old intervention weakens; fresh intervention has a targeted effect | The original causal handle may have lost effectiveness while a new one works in this test. | Whether either handle captures the full mechanism. |
| Both probes succeed; interventions have little targeted effect | The cue is readable but may not be used through the tested coordinates. | Other pathways, or whether the intervention was well designed. |
| Shortcut behavior falls; both readouts and tested effects weaken | Evidence consistent with reduced shortcut reliance on the tested distribution. | Unmeasured representations and behavior outside that distribution. |
These are interpretations, not automatic verdicts. A fresh linear probe failing does not prove that no nonlinear or distributed encoding exists. A successful intervention shows a causal handle under its specified conditions, not that the feature is unique, necessary on every input, or complete. Inspect uncertainty across held-out examples and relevant subgroups; an average effect can hide a mechanism that has moved for some contexts but not others.
The central design principle is to avoid using an internal signal as its own final examiner. If training was guided by , evaluate target behavior with independent labels or matched counterfactuals, not just a lower score. Refreshing a readout tests for a changed encoding; repeating causal tests asks whether the proposed connection to behavior still holds. Neither can substitute for the other.
Takeaways
Across checkpoints, a frozen probe or SAE provides a consistent measurement rule, not a guarantee of consistent meaning or causal role. Hold the evaluation examples and definitions fixed, compare the old readout with newly fitted ones, and repeat matched interventions while monitoring unmodified behavior and collateral damage. When those measurements disagree, report the disagreement: it is often the clearest evidence of representation drift or rerouting.
The next lesson turns this audit into a controlled feature-guided fine-tuning experiment, with a matched training baseline and an independent behavioral evaluation.