Hello. This module examines shortcut claims from the outside in: first establish what the model does when a suspected cue changes, then investigate whether a proposed internal mechanism explains that behavior. In this lesson, you will estimate shortcut reliance with matched input counterfactuals. The central requirement is that the task-relevant evidence—and therefore the correct answer—stays fixed while a suspected nuisance cue changes.
Why ordinary accuracy can conceal a shortcut
Suppose a sentiment classifier receives a review and a source header. In its training data, reviews from Source A were usually positive and reviews from Source B were usually negative. The source may help predict labels in that dataset, but it is not evidence about what an individual review says. A classifier can achieve respectable test accuracy while partly using the header instead of reading the review.
The Clever Hans example gives a compact illustration of this broader problem: a system can produce the expected answer for a reason different from the one we intended.
Why AI is like a (Clever Hans) Horse - Computerphile
Watch “Why AI is like a (Clever Hans) Horse” from Computerphile for the distinction between successful prediction and reliance on the intended evidence.
Watch Clever Hans for the motivating example, then the classifier for its AI analogue. Focus on why accuracy alone cannot identify which cue produced an answer.
A useful response is to evaluate the model on nearby inputs, rather than only on another random sample from the same distribution. The contrast-sets paper explains how systematic gaps in a dataset can let a simple decision rule look strong on an ordinary test set.
Evaluating Models' Local Decision Boundaries via Contrast Sets
Read this paper’s account of dataset gaps and locally matched examples. It motivates the design used here, while making an important distinction about which edits change the correct label.
In Section 2.1, “The Problem,” read the toy setting; use Figure 2 to see why ordinary test points can miss an incorrect local decision boundary. Then read Section 2.2, “Definitions,” and Section 2.3, “Contrast sets in practice,” from the definitions and construction discussion. Pay particular attention to the final distinction between a model-changing edit and a gold-label-changing edit.
The distinction for this lesson: the paper’s contrast sets typically change the gold label to test whether a model responds to meaningful evidence. To estimate reliance on a nuisance cue, we instead change the cue while holding the gold label fixed. Both methods examine behavior near an original input, but they test different expectations.
Make the comparison genuinely matched
Let denote the task-relevant evidence in a held-out example, the candidate shortcut cue, and a complete input assembled from that evidence and cue. For the source-header example, is the review text and is either “Source A” or “Source B.” The task specifies that the correct sentiment depends on the review, not its source:
A negative pair might consist of the same review—“The pacing dragged and the ending made little sense”—presented once under each source header. If the classifier’s answer changes, that change cannot be attributed to different review evidence: there is none. It is evidence that the input edit affects this model’s behavior.
This is related to, but not identical to, the usual practice of generating a counterfactual example that flips a prediction. The survey below introduces that conventional objective and the practical role of human review. Read it to clarify what must be adapted for a label-preserving shortcut test.
[PDF] A Survey on Natural Language Counterfactual Generation
Read the survey’s definition of a counterfactual example and its account of manual editing. The conventional desired prediction change is not a requirement for our nuisance-cue pairs.
In Section 2, “Definition of Counterfactual Example” (p. 4799), read from the definition, continuing through the description of a desired new prediction. Then go to Section 3.1, “Manual Generation” (p. 4800), and read the editing principles and the immediately following sentence on cross-validation. Focus on minimal, task-aware edits and independent checking—not on inducing a label flip.
The image shows a sampled datapoint, an expert-created counterfactual, and an approval step before the new datapoint joins an augmented dataset. For measurement, retain the original and edited inputs as a linked evaluation pair. Approval must verify that the nuisance cue changed but the task label did not; adding either member to training would turn this into a different experiment.

A practical construction protocol is:
- Specify the task boundary. State why the candidate cue is nuisance for this task. If source is legitimately part of the target definition, changing it need not preserve the answer.
- Sample held-out base examples before inspecting the model’s responses. Include multiple labels, contexts, and cue-compatible examples; avoid drawing conclusions from only memorable failures.
- Create both cue versions for each base example. Keep the review, question, or other relevant evidence identical. Keep formatting and all unrelated fields fixed as far as possible.
- Check each pair. Verify the intended cue difference, unchanged gold label, understandable inputs, and no accidental extra clue. Human task judgments are especially useful when an edit could subtly alter meaning.
- Record the paired outputs together. An unpaired comparison of different Source A and Source B reviews cannot isolate the header: their contents also differ.
Minimal text distance is helpful, but it is not the criterion that matters most. A one-word edit can change the answer; a longer header substitution can leave the task evidence untouched. Matching is defined by the task and the proposed nuisance cue, not by an edit-distance threshold alone.
Turn paired predictions into an estimate
Assume Source A was correlated with positive labels in the training data. Let be the model’s reported probability for positive sentiment. For each held-out review, calculate the within-pair change:
A positive means the tested Source A header tends to raise the positive score relative to Source B on these matched examples. To report changes in final decisions as well, use the paired flip rate:
These measures answer different questions. The score difference detects shifts that never cross a decision threshold; the flip rate measures how often the final classification changes. Neither is the model’s ordinary test accuracy.
Consider this deliberately small illustration, with a positive prediction when exceeds :
| Held-out review | Gold label | with A | with B | Decision flips? |
|---|---|---|---|---|
| 1 | Positive | 0.92 | 0.62 | No |
| 2 | Positive | 0.81 | 0.48 | Yes |
| 3 | Negative | 0.68 | 0.18 | Yes |
| 4 | Negative | 0.44 | 0.12 | No |
The four paired differences are , , , and . Thus , and the flip rate is . Yet accuracy is three out of four under A and three out of four under B. The equal aggregate accuracies hide substantial dependence on which header accompanies the same review.
Report the paired differences separately by gold label as well. A signed average can conceal opposing effects if a cue helps one class but hurts another; error rates under each cue reveal whether sensitivity translates into failures. With a realistic sample, report the number of independent base examples and uncertainty for the paired estimate. If several edits share one original review, treat that review—not each edited copy—as the independent unit when assessing uncertainty. The four-row illustration establishes no population-wide rate.
What the result establishes—and what it does not
Changing only the specified header gives a controlled input-level intervention. If the model’s output changes reliably across checked pairs, the model is behaviorally sensitive to that edit in the tested contexts. When the task definition excludes the header as evidence, that sensitivity supports a shortcut-reliance claim.
The claim still needs precise boundaries. A pair may inadvertently change typography, tokenization, plausibility, or another cue along with the intended source identity. Review these possibilities and, where useful, compare with a similarly small task-preserving control edit that does not change the hypothesized cue. Check performance across both labels and varied review styles, rather than extrapolating from one template. If pairs were selected because they already made the model fail, report that selection and evaluate the estimated rate on a separately selected held-out set.
Most importantly, input sensitivity does not locate a feature or circuit inside the model. Nor does a lack of flips establish that the model never uses the cue: its effect may shift scores without changing decisions, or appear only in other contexts. State the conclusion at the scale of the tested cue, input distribution, model, and metric.
Takeaway
A shortcut-reliance estimate compares the same task evidence under different nuisance-cue values, verifies that the gold label remains fixed, and summarizes paired score changes and decision flips. This can expose cue dependence that aggregate accuracy misses, while leaving the internal mechanism an open question. The next lesson brings input-level results together with targeted internal interventions to test whether a proposed mechanism actually explains that dependence.