Optimizing PEFT Hyperparameters

1. LoRA's Main Controls: Rank and Alpha

As a quick refresher, LoRA (Low-Rank Adaptation) avoids updating the massive number of weights in a pre-trained model. Instead, it injects small, trainable "adapter" matrices into the model's layers. The beauty of this approach lies in its efficiency, but its power comes from configuring these adapters correctly. The two most fundamental hyperparameters are rank and alpha.

To get a clear, visual intuition for how these work, let's start with a video.

Now, let's formalize these concepts, drawing on the deep dive provided by the LoRA Hyperparameters article. This will be our primary reference for the technical details.

The rank (r): The Capacity Knob

As you've just read, the rank (r) is the single most important LoRA hyperparameter. It defines the inner dimension of the low-rank matrices A and B, directly controlling the number of trainable parameters.

• Function: It sets the "capacity" or "expressiveness" of the adaptation.
• Intuition: Think of rank as the number of independent patterns the adapter can learn.
  • A low rank (e.g., 4, 8) constrains the model to learn simple, broad adjustments. This is a form of regularization and is effective for simple tasks or when you have little data, helping to prevent overfitting.
  • A high rank (e.g., 64, 128) gives the model more freedom to learn complex, nuanced changes. This is necessary for tasks that are very different from the model's pre-training data but increases the risk of overfitting.

The alpha (α): The Strength Knob

The alpha (α) parameter acts as a scaling factor for the LoRA update. The crucial part isn't alpha itself, but the ratio alpha / r.

• Function: This ratio determines the magnitude of the adapter's output before it's added back to the original, frozen weights.
• Intuition: If rank is the capacity, alpha helps you set the "volume" of that capacity.
  • alpha = r: The scaling factor is 1. The LoRA update is added with its "natural" magnitude. This is a neutral, common baseline.
  • alpha = 2 * r: The scaling factor is 2. The LoRA update is amplified. This is a very common heuristic, often used to make the adapter's contribution more significant without needing a higher learning rate.
  • alpha < r: The scaling factor is less than 1. The LoRA update is dampened, resulting in a more conservative change that preserves more of the base model's behavior.

The reason alpha exists is to decouple the choice of rank from the learning rate. By using the alpha/r scaling, you can change the rank without having to drastically re-tune your learning rate, as the magnitude of the updates remains more consistent.

2. Where to Adapt: Target Module Selection

LoRA adapters don't have to be applied to every layer. You can choose which parts of the Transformer architecture to modify. This choice has a significant impact on both performance and efficiency.

Let's return to our main textual resource to understand this.

To summarize the strategies:

1. Attention-Only (e.g., q_proj, v_proj): This is the most common starting point. It's highly parameter-efficient and focuses on changing how the model routes information. For many instruction-following tasks, modifying the query and value matrices is sufficient.
2. All Attention (q_proj, k_proj, v_proj, o_proj): A slightly more comprehensive approach that still focuses on the attention mechanism.
3. All Linear Layers (Attention + FFN): This is the most powerful but also most parameter-heavy PEFT approach. The Feed-Forward Networks (FFNs) are thought to store factual knowledge. Adapting them is crucial when you need to teach the model new facts or adapt it to a domain with specialized knowledge.

An important finding from the QLoRA paper, as discussed in the video below, is that to match the performance of full fine-tuning, adapting all linear layers is often necessary.

This gives you a clear trade-off: start with attention-only for maximum efficiency, but if performance is lacking, expand to all linear layers for maximum capacity, memory permitting.

3. Preventing Overfitting: LoRA Dropout

Just like in traditional deep learning, dropout is a powerful regularization technique. In LoRA, it's applied specifically to the adapter pathway.

• Function: During training, it randomly sets some of the activations within the LoRA adapter to zero.
• Intuition: It forces the adapter to learn redundant representations, preventing it from becoming too reliant on any single pathway. This makes the learned adaptation more robust and less likely to memorize the training data.
• Practical Use: A lora_dropout of 0.05 or 0.1 is a common choice. It's especially useful when you have a small dataset or are using a high rank, both of which increase the risk of overfitting.

4. Practical Recipes and Optimization Strategy

Now that we understand the individual components, how do we combine them? Hyperparameter tuning can be a daunting search problem. Fortunately, community experience has produced some reliable starting points.

Let's look at a summary of practical recipes and an image that shows the real-world impact of these choices.

The table below, from Sebastian Raschka's experiments, shows how different r and alpha settings on a 7B model affect performance on various benchmarks. Notice how there is no single "best" setting; performance varies by task. For instance, r=256, alpha=512 does well on MMLU, but r=256, alpha=128 is better on TruthfulQA.

A General Hyperparameter Search Strategy

When you need to find the optimal settings for your specific task, don't search randomly. Prioritize your search budget based on impact:

1. Target Modules: Decide if you can get away with attention-only or if you need to include FFNs. This is your biggest lever for capacity.
2. Rank (r): This is your next most important knob. Search in powers of 2 (e.g., 8, 16, 32, 64). Observe the trade-off between performance and training time/memory.
3. Alpha (α): Instead of searching for alpha independently, search for the alpha/r ratio. A common approach is to simply set alpha = 2 * r and focus on tuning the rank.
4. Dropout: Tune this last. A small value like 0.05 is often a safe bet. If you see clear signs of overfitting, you can increase it.

Conclusion

You've now demystified the key control knobs for LoRA. You've moved beyond just running fine-tuning jobs to understanding how to intelligently configure them, directly connecting the diagnostics from our last lesson to concrete, actionable treatments.

Key Takeaways:

• rank (r) is Capacity: It controls the number of trainable parameters. Higher ranks allow for more complex adaptations but risk overfitting. Lower ranks act as a regularizer.
• alpha (α) is Strength: It controls the magnitude of the adaptation. The key is the alpha / r ratio, with alpha = 2 * r being a common and effective heuristic.
• Target Modules are Location: Applying LoRA to attention layers modifies information flow, while applying it to FFNs modifies stored knowledge. Targeting all linear layers offers the most power at the cost of more parameters.
• LoRA Dropout is Regularization: It's a targeted tool to combat overfitting within the adapter itself, especially useful for small datasets or high ranks.
• Optimize Systematically: When tuning, prioritize your search: Target Modules → Rank → Alpha → Dropout.

What's Next?

So far, we've focused on Supervised Fine-Tuning (SFT), where we teach a model to mimic the style and format of a curated dataset. However, SFT doesn't guarantee the model will be helpful, harmless, or aligned with complex human preferences. In our next lesson, we'll explore the next frontier: explaining the conceptual difference between SFT and alignment techniques like DPO and RLHF, which are designed to shape a model's behavior in more nuanced ways.