Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 10

Linear Decodability Without Causal Influence

Recommended resources

Hello. This final probing lesson addresses a distinction that matters throughout interpretability: a model can make a concept easy to read from an activation without using that concept to produce its answer. You will build a small network where that distinction is exact, check it in PyTorch, and then compare it with what probing and intervention studies can establish in less controlled models. Allow about 40 minutes.


Readable information is not necessarily used information

A linear probe is a separate readout trained to predict a label from an activation. If it succeeds on held-out data, it tells you that the label is linearly decodable at that location. The model’s own output computation might read the same information, read something else in that activation, or ignore the activation altogether. Probe accuracy alone does not distinguish those possibilities.

This short segment frames the difference in terms of what a computation contains and what its output depends on.

Causal Mechanistic Interpretability (Stanford lecture 1) - Atticus Geiger

Watch the closing Q&A in “Causal Mechanistic Interpretability (Stanford lecture 1)” by Atticus Geiger, published by Goodfire. It contrasts a map of decodable information with a map of information that affects subsequent computation.

Watch probing versus use. Focus on the example of a representation that remains rich in decodable information despite having no influence on the relevant subsequent output, and on why an intervention asks a different question from a probe.

The diagram below captures the ambiguity of an impressive probe score. The probe receives both model activations and supervision for the concept-prediction task. Its success does not, by itself, identify a route by which the original model uses the concept.

Model representations and concept-label supervision both contribute to a probe’s high F1 score; that score alone does not show that the original model’s output depends on the decoded concept.

To remove this ambiguity, we can construct a model whose entire computation is visible.



A concept mixed into every hidden coordinate—and absent from the answer

Let an input have two independent binary properties, each equally likely to be or :

  • is task evidence. The correct output class is determined by its sign.
  • is a candidate concept. It carries no information about the correct class.

Suppose a network computes the two-dimensional hidden activation

Neither coordinate is simply “the concept coordinate”: both mix and . Nevertheless, a linear probe can recover perfectly:

Now define the model’s own output logit using a different linear readout:

The model predicts the positive class when . It is correct on every possible input, yet its logit is independent of . The four cases make both claims visible:

Task evidence Concept Hidden activation Probe score Model logit

The probe can classify the concept with 100% accuracy. More strongly, we can prove that the model does not use it in this specified computation. Holding fixed while changing changes the activation in the direction . The model’s readout weights are , whose dot product with that change is zero. Thus the logit stays fixed for either value of .

Notice the precise scope of “unused.” The concept is not absent from the hidden state: changing changes both hidden coordinates. Nor would zeroing just one coordinate be a clean test of concept use, because that also damages the task signal. What is absent is a contribution of to this model’s output.



Verify the distinction with a matched intervention

The calculation is short enough to check without training either readout. The probe weights are an explicit witness that linear decoding is possible; they are not weights used by the model.

import itertools
import torch

# All four equally weighted combinations: columns are task evidence and concept.
x = torch.tensor(list(itertools.product([-1.0, 1.0], repeat=2)))
t, c = x.T

h = torch.stack((t + c, t - c), dim=1)
probe_score = (h[:, 0] - h[:, 1]) / 2
task_logit = (h[:, 0] + h[:, 1]) / 2

# Change only c. The replacement is exactly the hidden state for (t, -c).
h_c_flipped = h + (-2 * c).unsqueeze(1) * torch.tensor([1.0, -1.0])
probe_after = (h_c_flipped[:, 0] - h_c_flipped[:, 1]) / 2
logit_after = (h_c_flipped[:, 0] + h_c_flipped[:, 1]) / 2

# A comparison: change task evidence while holding c fixed.
h_t_flipped = h + (-2 * t).unsqueeze(1) * torch.tensor([1.0, 1.0])
task_logit_t_flipped = h_t_flipped.sum(dim=1) / 2

assert torch.equal(probe_score, c)
assert torch.equal(task_logit, t)
assert torch.equal(probe_after, -c)
assert torch.equal(logit_after, task_logit)
assert torch.equal(task_logit_t_flipped, -t)

print("Concept probe accuracy:",
      ((probe_score > 0) == (c > 0)).float().mean().item())
print("Cue flip leaves model logits unchanged:",
      torch.equal(logit_after, task_logit))
print("Task-evidence flip changes model logits:",
      torch.equal(task_logit_t_flipped, -task_logit))

The cue flip is a particularly clean test: it replaces each hidden state with one produced by another valid input that has the same task evidence and the opposite concept value. The probe reverses its prediction, confirming that the targeted information changed; the model logit does not move. Flipping while preserving provides a check that the output is responsive to the task property rather than being constant.

This is stronger than observing a low or high probe score. Because we know the complete equations, we can verify both decoding and output invariance for every possible input in this toy domain.



Relating the proof to empirical probing results

Real networks rarely provide such a complete algebraic proof. A useful comparison is a synthetic language task in which a character can appear in an input without being needed to solve the main task.

Does Probing Accuracy Entail Task Relevance?

Read the synthetic experiment in this EACL paper to see how a probe can recover a property that is unnecessary for the target task.

On printed pp. 3369–3370, read Section 5.1, “Synthetic Task,” beginning with the task construction and initial result through the opening discussion of Section 5.2, “Results.” Identify what determines the entailment label, how the character c is introduced in the NOISE condition, and what the probe predicts. You can stop after the initial result; the subsequent representation-size and probe-capacity comparisons are not needed here.

The paper’s NOISE condition illustrates an important warning: a property need not be predictive of the main-task label for a probe to recover it from learned representations. There is also an important difference from our construction. The paper’s reported probing classifier is an MLP, not the explicit linear probe above. And showing that a property is unnecessary for the task does not, on its own, prove that a particular trained model never uses it. The toy model warrants that stronger claim because we can inspect its output function exactly.

For a closer empirical counterpart to the linear decoding question, consider subject-number information in contextual word representations:

The Functional Relevance of Probed Information: A Case ...

Read this paper’s paired probing and intervention results. They show why finding information across many representations is different from finding where it affects a particular prediction.

In Section 2, “Probing for Plurality” (printed p. 836), begin at the sentence setup and probe results; focus on the linear probes and their accuracy at different word positions. Then read Section 3, “Causal Interventions” (printed pp. 837–838), from its opening through the intervention results. Track the contrast between positions where subject number is decodable and positions whose modification does not change the tested agreement behavior.

In that study, linear probes find subject-number information at many word positions. Targeted interventions, however, have little or no effect on the tested verb-agreement behavior at some of those positions, while interventions at others do affect it. That supports a behavior- and intervention-specific claim about functional relevance—not a universal proof that the information at an unaffected position can never matter for any output. Our toy example supplies the exact proof; the empirical study shows what analogous evidence looks like when the mechanism is not fully known.



Takeaway

A successful linear probe establishes a readout: in the toy network, recovers the concept perfectly. A claim about model use instead concerns the model’s own computation: its readout cancels that concept exactly. Holding task evidence fixed and changing the concept separates those claims without relying on probe accuracy as a proxy for causation.

The next module turns this distinction into experimental practice. Its activation interventions will test how estimated effects depend on what replaces an activation—a choice our exact, matched toy replacement made unusually straightforward.