Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 7

Validating Probe Accuracy Against Control-Label Baselines

Recommended resources

Welcome back to probing. In the previous lesson, you fitted a linear readout to frozen activations and evaluated it on held-out groups. That procedure tests whether labels are readable from a representation. It does not yet tell you whether an impressive score reflects the structure you intended to study, rather than a readout that has learned a collection of arbitrary associations.

This lesson adds a control-label baseline. You will see why the choice of what receives a random label matters, run a target probe and control probes on the same activations, and interpret the gap without treating it as proof of a mechanism. Allow about 40 minutes, including the reading and code.


The confound: a probe can learn the task

Consider probing token activations for part of speech. A token’s activation may expose its word identity. That alone gives a supervised probe a useful lookup key: after seeing enough labeled occurrences of a word, the probe can learn what to predict for that word. A held-out sentence does not rule this out if it contains familiar words.

This is not necessarily bad prediction. Word identity really is informative about part of speech. The interpretability issue is narrower: high probe accuracy alone cannot separate a representation of the proposed linguistic distinction from the probe’s ability to associate accessible word identities with labels. Even a linear probe can learn many such associations when its input is high-dimensional.

Read John Hewitt’s account of this problem before looking at the control. It connects the probe’s learned decision rule to the information available in its input.

Designing and Interpreting Probes · John Hewitt

Read John Hewitt’s explanation of the probe confounder problem. Focus on why a correct prediction can depend jointly on the frozen representation and decisions learned through probe supervision.

In “The Probe Confounder Problem,” begin with the toy example. Continue through the opening of “Control tasks and selectivity,” ending at the two properties called “structure” and “randomness.” Pay particular attention to the passage where accessible word type lets the probe memorize distinctions.



Random labels that preserve the opportunity to memorize

A useful control makes the probe perform an arbitrary but learnable task. For the part-of-speech example, sample one random control label for each word type, then give every occurrence of that type the same control label. Fit a fresh probe to those labels, using the same frozen activations, split, readout family, and evaluation procedure as the real probe.

The supplied illustration makes the construction concrete:

The upper panel assigns arbitrary numerical labels to word types; the lower panels show those assignments reused across two sentences beside the actual part-of-speech tags. “The” and “ran” both receive control label 10, while “dog” and “quickly” both receive 15 despite having different grammatical roles.

The assignments do not express a grammatical rule. Their value as a control comes from two properties held together:

  • Random across types: the particular assignment was not a linguistic distinction the frozen model had reason to learn.
  • Stable within a type: a probe that can identify a word type can learn its arbitrary label from training occurrences and apply that association in held-out sentences.

That second property is easy to lose. If you independently shuffle every token occurrence’s label, the same word may receive conflicting labels in training and testing. A probe that could memorize stable word-to-label associations will then score near chance on the control. Such a result would understate the readout’s capacity for the memorization you meant to test.

Read the construction and its proposed comparison in the same article.

Designing and Interpreting Probes · John Hewitt

Continue Hewitt’s article to see how a type-level control is constructed and how its score is compared with the real task score.

In “Control tasks and selectivity,” start at the control construction. Read through “Constructing a control task for part-of-speech tagging,” “Properties of control tasks,” and “Selectivity.” Notice both why test-set word-type overlap permits control-task success and why the comparison uses held-out accuracy, not training accuracy.

Hewitt calls the difference selectivity:

where each accuracy is measured on its corresponding held-out labels. For example, a real-task accuracy of and control accuracy of give a selectivity gap of , not . The control says that this probe can achieve much of that apparent success on arbitrary, type-consistent labels.

A class-frequency baseline answers a different question. Predicting the most common label tests whether a score beats a trivial constant prediction; a type-level control tests whether the readout can learn arbitrary distinctions using information such as identity. Report both, especially if label frequencies are uneven.



Run the comparison on one activation cache

The following self-contained PyTorch example uses synthetic token activations. Each vector contains a one-hot word-type signal and a noisy signal for a binary real label. The synthetic design is deliberate: the real label is present, but word identity is also readily available to the probe. Sentences, not individual token rows, are assigned to train, validation, and test partitions. Because word types recur across sentences, a held-out token can still have a familiar type.

The code trains the same linear-probe procedure on real labels and on five independently sampled type-level control mappings. It reports the held-out majority-class baselines as well as the probe scores. Running it and inspecting the outputs should take about 12–15 minutes.

import torch
import torch.nn.functional as F

torch.set_num_threads(1)
torch.manual_seed(3)

# Synthetic frozen activation cache: one row per token.
n_types, n_groups, tokens_per_group = 64, 120, 10
n = n_groups * tokens_per_group
types = torch.randint(n_types, (n,))
groups = torch.arange(n_groups).repeat_interleave(tokens_per_group)

# Real labels vary across occurrences; the last activation coordinate
# carries a noisy signal about them. The first 64 reveal word type.
y_real = torch.randint(2, (n,))
real_signal = (
    2 * y_real.float() - 1 + 0.65 * torch.randn(n)
).unsqueeze(1)
X = torch.cat(
    [F.one_hot(types, n_types).float(), real_signal], dim=1
).detach()

# Hold out entire sentence groups, while allowing types to recur.
order = torch.randperm(n_groups)
train_groups = order[:72]
val_groups = order[72:96]
test_groups = order[96:]

train = torch.isin(groups, train_groups)
val = torch.isin(groups, val_groups)
test = torch.isin(groups, test_groups)

def fit_and_score(labels):
    # Identical probe family, initialization, and selection rule.
    torch.manual_seed(19)
    probe = torch.nn.Linear(X.shape[1], 1)
    optimizer = torch.optim.Adam(probe.parameters(), lr=0.05)
    best_val_loss = float("inf")
    best_state = None

    for _ in range(180):
        probe.train()
        logits = probe(X[train]).squeeze(1)
        loss = F.binary_cross_entropy_with_logits(
            logits, labels[train].float()
        ) + 0.001 * probe.weight.square().sum()

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        probe.eval()
        with torch.no_grad():
            val_logits = probe(X[val]).squeeze(1)
            val_loss = F.binary_cross_entropy_with_logits(
                val_logits, labels[val].float()
            ).item()

        if val_loss < best_val_loss:
            best_val_loss = val_loss
            best_state = {
                key: value.detach().clone()
                for key, value in probe.state_dict().items()
            }

    probe.load_state_dict(best_state)
    with torch.no_grad():
        predictions = (probe(X[test]).squeeze(1) >= 0).long()
        accuracy = (predictions == labels[test]).float().mean().item()

    majority = torch.bincount(
        labels[train], minlength=2
    ).argmax()
    majority_accuracy = (
        (labels[test] == majority).float().mean().item()
    )
    return accuracy, majority_accuracy

real_accuracy, real_majority = fit_and_score(y_real)

control_results = []
for seed in range(5):
    generator = torch.Generator().manual_seed(100 + seed)

    # Sample once PER TYPE, then reuse the mapping for all its tokens
    # in training, validation, and test.
    type_to_control = torch.randint(
        2, (n_types,), generator=generator
    )
    y_control = type_to_control[types]
    control_results.append(fit_and_score(y_control))

control_accuracies = [result[0] for result in control_results]
control_majorities = [result[1] for result in control_results]
seen_type_fraction = torch.isin(
    types[test], types[train].unique()
).float().mean().item()

print("Test tokens with a type seen in training:",
      round(seen_type_fraction, 3))
print("Real accuracy / majority baseline:",
      round(real_accuracy, 3), round(real_majority, 3))
print("Control accuracies:",
      [round(score, 3) for score in control_accuracies])
print("Control majority baselines:",
      [round(score, 3) for score in control_majorities])
print("Real minus mean control accuracy:",
      round(real_accuracy - sum(control_accuracies) / 5, 3))

Inspect the seen-type fraction alongside the control scores. A type-level control tests memorization most directly when evaluation includes types whose arbitrary assignments could have been learned during training. If nearly all test types are new, low control accuracy tells you little about the probe’s ability to memorize assignments for recurring types.

This example also guards against a tempting inference. The real-label signal was explicitly built into the final activation coordinate. Yet the control can score well because the other coordinates make types accessible. A small selectivity gap therefore does not prove that real-label information is absent. It says that raw real-task accuracy, with this readout and evaluation setup, is weak evidence for distinguishing that information from what the readout can learn about arbitrary type labels.



What the gap warrants

Compare held-out real accuracy, control accuracy, and their respective majority baselines, rather than interpreting selectivity alone. If real accuracy substantially exceeds control accuracy across several sampled mappings, the result supports a limited conclusion: this probe predicts the actual labels better than it predicts arbitrary labels constructed to offer a similar type-memorization opportunity. If both scores are high, accessible identity and readout capacity complicate an interpretation based solely on real-task accuracy. If both are low, neither task was reliably decoded by this procedure.

The comparison remains conditional on the control you chose. A word-type control is apt when repeated word identity is a plausible route to memorization; it does not test every nuisance cue. It also cannot establish that the original model uses the probed information in its own output computation. Strong selectivity is evidence about a readout’s discrimination between two labeling tasks, not a causal intervention on the model.


A control-label baseline keeps the probe honest by giving it an arbitrary task that remains learnable from repeated cues. Sample labels per cue type, keep them stable across splits, fit the same probe procedure, and compare held-out scores and class-frequency baselines. Interpret the resulting gap as a relative diagnostic, not as a certificate that a representation contains—or a model uses—a particular concept. In the next lesson, you will test a different vulnerability: whether a probe trained in one set of contexts still works when a nuisance correlation changes.