Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 33

Controlled Feature-Guided Fine-Tuning and Behavioral Evaluation

Recommended resources

Hello. In the previous lesson, you compared fixed and refreshed feature readouts across checkpoints and repeated causal tests rather than assuming that an internal signal kept the same meaning during training. Now the question becomes experimental: does using that signal to guide fine-tuning improve the behavior you care about, beyond what ordinary fine-tuning would achieve?

This lesson develops a controlled design around a candidate shortcut feature. The central separation is between the training signal, the behavioral outcome, and the mechanistic interpretation. A feature score can guide updates, but it cannot serve as the sole evidence that the updates worked.


Define the claim before choosing a loss

Consider a question-answering model that receives a passage and an irrelevant badge suggesting one of two answers. In the available training examples, the badge often agrees with the passage. A model might learn to favor the badge even when the passage says otherwise. Suppose earlier matched-input tests and activation interventions identified a candidate SAE feature associated with this behavior.

The experiment is not simply “can we make the feature quieter?” A more useful claim is:

Compared with ordinary supervised fine-tuning on the same examples, fine-tuning that penalizes this feature will reduce badge-driven wrong answers on new passages, without materially reducing the model’s ability to answer from the passage.

That statement identifies a matched baseline, an externally observable target, and a potential cost. It also limits the claim to the tested setting. The SAE feature may respond to several things besides shortcut use; penalizing it is an experimental treatment, not a guarantee that the model will stop using the badge.

Prepare three data partitions before training. Use a discovery partition to select and test the feature, a training and validation partition to fit models and choose training settings, and a locked behavioral test partition that plays no role in feature selection or tuning. Keep paired badge variants and closely related passages or templates in the same partition. Otherwise, near-duplicates can make generalization look stronger than it is.

The test partition should include matched pairs whose passage, question, and correct answer are unchanged while the badge changes. Include cases where the badge conflicts with the passage, cases where it agrees, and cases with no badge. To test collateral damage, also include a separate setting where a similar indicator is genuinely relevant to the task. A model that learns to ignore every indicator has not necessarily learned when an indicator is appropriate evidence.



Change one training ingredient at a time

One way to use an SAE feature during supervised fine-tuning is to add a penalty to the usual answer loss. Let be the student model’s activation at a prespecified layer and token position. Let be the selected feature score from a frozen SAE encoder. A possible objective is

Here, contains training examples for which the badge is not legitimate answer evidence. The threshold and penalty weight are chosen without consulting the locked test set. Gradients pass through the frozen encoder to the student activation; the encoder itself is not refitted. The supervised answer loss remains in place so that suppressing the score is not the model’s only objective.

This is a design choice, not a universally correct loss. In particular, penalizing the feature on all inputs could damage legitimate uses of whatever the feature encodes. Even restricting the penalty to does not establish that the feature is specific to shortcut use.

The following study provides a concrete example of combining supervised fine-tuning with a feature penalty, then comparing against other training methods.

SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs

Read the SASFT paper for its experimental structure: what the feature-guided objective changes, what its baselines hold constant, and how behavior is evaluated outside the feature score.

In Section 4.2, “SASFT,” read the objective through the combined training loss. Focus on which examples receive the auxiliary penalty and why its threshold is not simply zero. In Section 5.1, “Experimental settings,” read the baselines, including the matched SFT comparison. In Section 5.2, “Main results,” read the evaluation plan. Skim Tables 1 and 2 for the contrast between target behavior and other capabilities, then read the authors’ interpretation. Treat claims of preserved capability as something to check against each reported benchmark, not as a substitute for that comparison.

For the badge experiment, the primary control is ordinary supervised fine-tuning from the same initial checkpoint, on the same examples and answer targets, with the same optimizer, batch sequence, number of updates, and otherwise matched hyperparameters. The guided arm differs by the auxiliary feature penalty. Run multiple paired seeds, using the same seed and data order within each baseline–guided pair. Report the extra computational cost of the feature-guided arm even if its optimizer-step budget is matched.

An additional specificity control can replace the selected feature with a prespecified unrelated feature or an activation direction whose scale and penalty magnitude are matched. If both auxiliary penalties help equally, the result supports “an extra regularizer helped” more readily than “this particular interpreted feature was useful.” Choose control directions and the permitted hyperparameter search before examining test outcomes. Give the baseline a comparable validation and tuning budget; do not select a successful guided run from many attempts while reporting a single untuned baseline.



Evaluate answers independently of the feature

For each held-out conflict example, record whether the model chooses the badge-favored answer even though the passage supports another answer. Let be the proportion of such errors. The primary comparison is

A negative value favors feature-guided training on this specified outcome. Because the arms answer the same held-out examples, compare their results pairwise. Report variation across passages and training seeds rather than treating every generated token as an independent replicate.

A lower alone is insufficient. The model might abstain more often, produce invalid answers, or become less capable of using the passage. Report, alongside , accuracy on both members of each matched badge pair, the frequency with which swapping only the badge changes the answer, abstention or invalid-output rates, and accuracy on badge-neutral examples. Measure performance where an indicator is legitimate evidence. Specify an acceptable off-target decline before opening the test results; otherwise, “minimal degradation” can be decided after seeing the numbers.

These measurements must be independent of the optimized feature score. Passage-supported answers can be checked against held-out labels under a fixed scoring rule. For open-ended answers, use a separate, specified evaluation process that was not the training reward. Inspect a sample of failures: the same aggregate accuracy can hide unsupported answers, blanket refusals, or changes in answer format.

A second feature-guided approach illustrates why separating the trained policy from its evaluation harness matters. Here, probes help supply a reward for training, but the authors also distinguish changes in the policy itself from improvements due to monitoring and intervention at test time.

Features as Rewards: Using Interpretability to Reduce Hallucinations

Read this study as a contrasting feature-guided training design. Pay particular attention to which model supplies the probe-based reward and which reported improvements require a test-time harness.

Under “Our approach,” begin in “Probe pipeline: detecting hallucinations in complex long-form responses” and read the probe pipeline through the opening paragraph of “RL setup.” Then, under “Results: reducing hallucinations with RLFR,” read the results breakdown. Distinguish the reported overall reduction with test-time assistance from the portion the authors attribute to a change in the policy itself.

Running reward probes on a frozen model prevents gradients from directly changing that monitor’s activations. It does not make output-level reward gaming impossible: a student can still learn outputs that score well without improving the intended behavior. Likewise, a successful test-time correction pipeline is valuable, but it is not by itself evidence that the unassisted fine-tuned model improved. For the badge experiment, evaluate both arms first without steering, filtering, or selective sampling; report any additional test-time system as a separate condition.



Decide what the result would—and would not—establish

A compact report should show, for the initial model and every trained arm, the held-out shortcut metric, matched-pair accuracy, off-target metrics, and uncertainty. Keep feature scores in a separate diagnostic panel. This prevents a drop in the optimized score from being mistaken for the main result.

The previous lesson’s checkpoint audit remains useful. Apply the original feature readout across training, fit refreshed readouts on separate data, and repeat the targeted intervention on held-out inputs at selected checkpoints. If the guided model’s original score falls but a fresh readout still detects badge information and badge-driven errors persist, the objective may have changed the monitored representation without fixing the behavior. If errors fall but the fixed score does not, the training may have improved behavior through another route. Neither disagreement should be concealed.

The strongest result for this experiment would be a reproducible improvement over matched SFT on independent, held-out badge counterfactuals, with acceptable off-target performance and a weaker or absent benefit from the specificity control. That would support the practical value of this feature-guided training treatment under the tested conditions. It would not, on its own, prove that the SAE feature uniquely represents the shortcut or that the shortcut mechanism has been removed everywhere.



Takeaways

A controlled feature-guided fine-tuning experiment changes a specified training signal while matching the starting model, examples, update budget, and selection procedure to a baseline. Its main examiner is independent behavior on held-out matched inputs, accompanied by off-target checks—not the internal score it optimized. Feature diagnostics and repeated interventions can explain or challenge the outcome, but they should not replace it.

The next lesson audits a stronger claim of shortcut removal: whether apparent success reflects proxy gaming, migration to unmonitored representations, or a result confined to the tested distribution.