Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 27

Estimating Latent Effects via Residual-Preserving Ablation

Recommended resources

Hello. A sparse autoencoder (SAE) can assign a nameable latent to part of a transformer’s activation, but a name and a high activation are not evidence that the latent affects the model’s behavior. This lesson turns that interpretation into a targeted intervention: set one latent to zero, keep the SAE’s original reconstruction error, and measure what changes downstream. The distinction matters because an SAE can reconstruct an activation closely while missing information that is crucial to the behavior you are testing.

Plan on about 40 minutes: roughly 10 minutes for the readings, 10 for the derivation and example, and 20 for the PyTorch implementation and interpretation.


What, exactly, are you removing?

At a chosen layer and token position, let be the activation passed to the rest of the transformer. The SAE encodes it as and decodes that code to an approximation . The information not captured by the decoder is the reconstruction residual:

For an affine SAE decoder, we can write:

where is latent ’s decoder direction. The SAE is a separate model of ; the transformer normally continues from , not from .

Read the worked example below before considering the intervention. Its output metric depends on a direction that the SAE reconstruction misses.

ai-safety-interpretability-circuits/docs/modules/07-sparse-autoencoders.md at main · Danny0951/ai-safety-interpretability-circuits · GitHub

Read the worked example in Danny0951’s sparse-autoencoder notes to see why a small reconstruction error can have a large effect on a particular output metric.

In “3. Worked example: TopK reconstruction and residual error,” begin with the three decoder directions immediately above the encoder preactivations. Continue through the worked example, including the box titled “Error-preserving feature ablation.” Stop before “4. Failure modes in learned dictionaries.” Focus on the contrast between geometric reconstruction quality and the task-relevant signal left in the residual.

To remove latent , first compute from the original activation . Let be a copy with only set to zero. Feed the following activation into the transformer’s downstream computation:

This keeps the original residual fixed while changing the SAE-decoded contribution assigned to latent . With an affine decoder, the decoder bias and every other latent’s contribution cancel:

That final expression is convenient to implement, but the longer one makes the experimental choice explicit. Do not replace with and call the resulting effect an effect of latent . That replacement removes both and . Likewise, do not re-encode the patched activation and zero the latent again: that would define a different, potentially iterative intervention.

The reading’s two-dimensional example shows the confound sharply. The SAE approximates by a diagonal reconstruction of about , leaving . For the output direction , the reconstruction contributes zero to , while the residual contributes the entire value . If the diagonal latent is the only active latent, a residual-preserving ablation leaves that output measure at ; replacing the activation with the empty partial reconstruction would instead take it to zero. The apparent “latent effect” in the latter experiment would actually be the effect of discarding the residual.



Run the ablation at the model’s activation site

The following scaffold assumes a trained sae with encode and decode methods, a transformer model, and a hook_point module whose tensor output is the same activation site on which the SAE was trained. It intervenes at one shared token position in each batch item. In a model whose hook point returns a tuple, adapt the hook to replace only the relevant tensor and preserve the rest of the return structure.

import torch

model.eval()
sae.eval()

# Supply these for your model and held-out inputs:
# inputs: dict accepted by model(**inputs)
# hook_point: module returning activations of shape [batch, seq, d_model]
# token_position: integer position shared across this batch
# latent_index: SAE latent to ablate
# desired_id, contrast_id: token IDs defining the output metric

def run_with_ablation(preserve_residual: bool):
    def hook(_module, _args, x):
        x_pos = x[:, token_position, :]       # [batch, d_model]
        z = sae.encode(x_pos)                  # [batch, n_latents]

        z_off = z.clone()
        z_off[:, latent_index] = 0.0

        partial_reconstruction = sae.decode(z_off)

        if preserve_residual:
            original_reconstruction = sae.decode(z)
            replacement = (
                x_pos
                + partial_reconstruction
                - original_reconstruction
            )
        else:
            # Diagnostic comparison, NOT an isolated latent ablation.
            replacement = partial_reconstruction

        patched = x.clone()
        patched[:, token_position, :] = replacement
        return patched

    handle = hook_point.register_forward_hook(hook)
    try:
        return model(**inputs).logits
    finally:
        handle.remove()


def logit_difference(logits):
    next_token_logits = logits[:, -1, :]
    return (
        next_token_logits[:, desired_id]
        - next_token_logits[:, contrast_id]
    )


with torch.inference_mode():
    clean_logits = model(**inputs).logits
    ablated_logits = run_with_ablation(preserve_residual=True)
    partial_logits = run_with_ablation(preserve_residual=False)

clean_score = logit_difference(clean_logits)
ablated_score = logit_difference(ablated_logits)
partial_score = logit_difference(partial_logits)

effect = clean_score - ablated_score  # one value per example
print("Mean residual-preserving effect:", effect.mean().item())
print(
    "Mean partial-reconstruction change:",
    (clean_score - partial_score).mean().item(),
)

A positive effect means that, on these inputs, removing the latent’s decoded contribution decreased the desired-versus-contrast logit difference. It does not mean that the model would never produce the desired answer without the latent: the metric is a particular next-token comparison at a particular intervention site. The partial_score run is a diagnostic for how much your conclusion would change under the confounded replacement; its difference from the clean run is not an isolated latent effect.

Before interpreting results, check the mechanics. Verify that the chosen latent is active on the examples where you expect an effect; zeroing an already-zero coordinate makes no change. Verify that the hook executes at the intended layer and position, that the SAE was trained for that exact activation site, and that a hook returning unchanged reproduces the clean output. For a variable-length batch, select each example’s intended token position rather than silently using one position for all examples. Keep the model and SAE fixed during these inference runs.



What the behavioral change does—and does not—establish

The same residual-preserving construction appears in published feature-steering experiments. The methodological account below is worth reading for its explicit decomposition of the activation and its caution about intervention strength. It discusses clamping to various values; setting a naturally active latent to zero is the ablation considered here.

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

Read the “Feature Steering” methodology in Scaling Monosemanticity to compare the residual-preserving intervention with one used in model experiments.

In the appendix’s “Feature Steering” subsection, read the implementation and caveat. Start at the description of decomposing residual-stream activity into an SAE reconstruction and an error term; finish after the discussion of extreme clamp values causing nonsensical behavior. Focus on what the intervention leaves unchanged and why an effect from an out-of-range clamp needs a different interpretation from an ordinary zero ablation.

On held-out examples, report the distribution of effect, not just one memorable completion. Specify the prompts, layer, token position, latent, target and contrast tokens, and whether the latent was active. Pair the target metric with an off-target measure—such as next-token loss or accuracy on examples where the proposed concept should not matter. An ablation that changes the target behavior while broadly damaging predictions supports a weaker interpretation than one with a selective effect. You can also compare with an appropriately magnitude-matched control direction, since changing an activation by a large vector can affect outputs without vindicating that vector’s semantic label.

A residual-preserving effect is causal evidence about this intervention: at this site, on these inputs, removing while retaining the original SAE error changed downstream computation. It is not, by itself, proof that the latent uniquely represents the named concept, that it is the model’s only route to the behavior, or that the same effect holds on other prompts. Correlated or split SAE features may still carry related information. Conversely, a small effect can reflect redundancy, a poorly chosen output metric, or testing where the latent is inactive—not necessarily an irrelevant feature.

The key discipline is to keep the claim as precise as the manipulation. If the hypothesis concerns a shortcut cue, test the ablation on matched examples with and without that cue while preserving the task evidence. A selective change in shortcut-sensitive behavior is more informative than a change on examples where cue and correct answer always coincide.



Conclusion

An SAE latent ablation should modify the original activation by the difference between two SAE decodings, not replace it with a partial reconstruction. That difference removes the chosen latent’s decoded contribution while preserving the original reconstruction residual, which may contain behaviorally important information. Measure the resulting downstream change against a clean run on held-out examples, and interpret it as a conditional intervention effect rather than a complete explanation of a concept.

Next, the course examines cases where SAE features split or absorb one another—important reasons why the effect of ablating any single latent may not match the boundaries of a human-described concept.