A probe can predict a label accurately from cached activations and still fail when the context changes. The key question in this lesson is not simply “Can the probe read the label?” but “Does its readout survive when a nuisance cue stops being a reliable stand-in for the label?”
Assume you have already trained a linear probe on activations from a frozen model. You will learn how to evaluate that same, unchanged probe on held-out contexts with a different nuisance–label relationship, report the results by group, and interpret what those results do—and do not—say about the model’s representations. Plan on roughly 40 minutes, including the readings.
What a shifted-context test isolates
Let be the target label, a nuisance cue, the input, and the frozen model’s activation at your chosen site. Your trained probe is a readout of , not a retraining of the base model. Suppose the training context makes agree with on 90% of examples. A probe that reads only can achieve approximately 90% target-label accuracy there, even if it cannot reliably read task-relevant evidence.
A useful test context changes that relationship—for example, so that agrees with on only 10% of examples—while keeping the definition of and the task-relevant evidence intact. If the probe primarily reads the nuisance cue, its apparent success should deteriorate. The reversal is deliberately stark; a balanced context, in which agreement is about 50%, is another useful test.
The following paper gives a compact example of why this matters. Its synthetic task has an available stable feature, yet the learned predictor can favor a shortcut and perform poorly when the shortcut correlation flips. The paper evaluates predictors, not trained activation probes; here, use its distribution design and group analysis as a model for probe evaluation.
Don't blame Dataset Shift! Shortcut Learning due to Gradients and ...
Read the paper's synthetic shift and its group-based evaluation to see why average accuracy in the original context can conceal shortcut dependence.
In Section 2, “Shortcut learning in perception tasks due to maximizing margins,” and Section 2.1, “Shortcut learning in linear perception tasks” (pp. 2–3), read the setup and result. Focus on which feature remains predictive and which label relationship changes between training and test; the typesetting of the data-generating equation may be awkward in the PDF. Then go to Section 5, “Vision and language experiments” (pp. 7–8). From its “Datasets” paragraph, read the groups and metric, including the short “Metrics, model selection, and hyperparameters” subsection after Table 1. Notice that groups are defined jointly by target label and nuisance cue.
The distinction between a change in group proportions and a change within groups is important. If the only change is how often each combination occurs, then performance within each group might remain constant while overall accuracy changes sharply. In a real deployment shift, examples within a group can change too. Measure on genuinely held-out contexts rather than assuming that a reweighted old test set captures every aspect of the new one.
What to hold fixed—and what to measure
Consider images whose label is the foreground object and whose nuisance cue is the background. Hold the feature extractor, activation site, probe weights, activation normalization, and decision threshold fixed. Choose any threshold and model-selection settings using training and validation data before examining the shifted test results. If the images come from related sources or have multiple edited versions, split by underlying source example so that near-duplicates cannot cross from probe training into test.
Your test data should retain both labels: for the target and for the nuisance. For binary and , report accuracy in all four groups, not just one aggregate. In particular, inspect the groups where conflicts with . They are uncommon in the original correlated context but common under a reversal.
The Waterbirds results below provide a useful distinction. The paper first tests an original classifier on images with the background removed. Separately, it trains a logistic-regression readout on internal features and assesses how well the target can be decoded. These are related observations, but neither should be silently substituted for the other.
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
Read how the paper separates a classifier's behavior from information available in its learned representation.
In Section 4.1, “Feature learning on Waterbirds data,” read the Waterbirds comparison. Compare performance on the Original and “FG-Only” test data; note that foreground-only images remove the original background cue rather than merely changing its frequency. In Section 4.2, “Simplicity bias,” find the paragraph beginning “Decoding feature representations.” Read the decoding procedure. Focus on what is trained on the features, which set is group-balanced, and which set supplies the reported worst-group result.

This distinction is central to your evaluation. A new linear probe may read foreground information that the original classification head underweights. Strong probe performance on conflicting-background examples supports linear availability of target information at the probed site. It does not establish that the original head uses that information. Conversely, weak probe performance does not establish that the information is absent: it may be encoded nonlinearly, poorly captured at that site, or hard to estimate from the available probe-training data.
A concrete evaluation report
Suppose you have 100 held-out examples in each group. The table shows illustrative accuracies for one fixed probe:
| Target | Nuisance | Probe accuracy |
|---|---|---|
| 0 | 0 | 95% |
| 0 | 1 | 40% |
| 1 | 0 | 83% |
| 1 | 1 | 88% |
The agreement groups average 91.5% accuracy; the conflicting groups average 61.5%. The worst-group accuracy is 40%. Those figures convey more than a single overall score: the probe’s errors are concentrated where one particular nuisance value conflicts with one particular label.
To see how context affects an aggregate, imagine only changing group frequencies while holding these four accuracies fixed. With balanced target classes and 90% agreement, weighted overall accuracy would be 88.5%. With 10% agreement, it would be 64.5%. The probe did not change; the mixture it faced did. This calculation explains why a reassuring original-context average is insufficient. It is not a replacement for measuring on an independently held-out shifted context, where within-group accuracies might also change.
With cached activations, the mechanics of reporting are straightforward. The example assumes binary labels, a probe returning one logit per example, and a threshold fixed on validation data:
import torch
@torch.no_grad()
def evaluate_probe(probe, activations, y, z, threshold):
probe.eval()
logits = probe(activations).reshape(-1)
predictions = (logits >= threshold).long()
correct = predictions.eq(y.long())
groups = {}
for label in (0, 1):
for nuisance in (0, 1):
mask = (y == label) & (z == nuisance)
if not mask.any():
raise ValueError(f"Missing held-out group {(label, nuisance)}")
groups[(label, nuisance)] = correct[mask].float().mean().item()
return {
"overall": correct.float().mean().item(),
"worst_group": min(groups.values()),
"groups": groups,
}
Run the same function on the original-context and shifted-context test activations without updating either model. Record the number of examples per group alongside each accuracy: a worst-group estimate based on very few examples can be unstable. If you compare several probes, keep their evaluation sets and selection protocol identical, and avoid selecting the “best” probe after inspecting the shifted test scores.
A foreground-only test answers a slightly different question from a correlation reversal. Removing a background asks how the probe behaves when that cue is absent; reversing the correlation asks how it behaves when the cue remains present but becomes misleading. Foreground-only processing can also introduce visual changes beyond cue removal. Both tests are informative, provided their conclusions are stated at the level each test warrants.
Reading the result without overclaiming
A probe that stays accurate across held-out nuisance contexts, including conflicting groups, provides stronger evidence that its target readout is not merely exploiting the original nuisance correlation. The conclusion is bounded by the contexts tested: another unmeasured cue could still explain its success.
A probe that fails chiefly on conflicting groups suggests reliance on a context-dependent correlate, but does not by itself identify the exact feature used. And comparing a successful retrained readout with a failing original classifier supports a distinction between information present in a representation and information used by the deployed readout—not a complete mechanistic account of how either computation works.
The essential workflow is therefore to freeze the trained probe, test it on independent contexts with a deliberately altered nuisance–label relationship, and report both overall and -group performance. In the next lesson, you will examine how differences in readout capacity, activation dimension, and sample size can make comparisons between probes unfair even when their test sets are well designed.