Merge LoRA Adapter for Inference

Hello! Welcome to the final lesson in our module on practical fine-tuning.

In our last session, we took the important step of launching and monitoring the fine-tuning process. We saw how to kick off the training with trainer.train() and how to interpret the training and validation loss curves to diagnose issues like overfitting. You now understand what happens during training.

This lesson focuses on what comes next: taking the result of that training—the LoRA adapter—and preparing it for actual use. We will cover how to save the adapter and then explore the two primary strategies for using it for inference: loading it on top of the base model dynamically or merging it into the base model to create a new, standalone model.

1. The Output of Fine-Tuning: The LoRA Adapter

After your trainer.train() call completes, the output_dir you specified in TrainingArguments (e.g., ./food-extract-finetuned) will contain the results. Because we used LoRA, the trainer doesn't save a new, multi-billion parameter model. Instead, it saves the small set of adapter weights you trained. This is the key to parameter-efficient fine-tuning.

Typically, you'll find a few files, but the most important ones for the adapter itself are:

• adapter_model.safetensors (or adapter_model.bin): This file contains the trained LoRA weights (the A and B matrices).
• adapter_config.json: This configuration file tells the PEFT library how to load and apply the weights—which modules they target, their rank (r), alpha (lora_alpha), etc.

The Hugging Face Trainer handles saving automatically based on your save_strategy. You can also save the final adapter manually after training is complete.

# Assuming 'trainer' is your trained SFTTrainer object
output_dir = "./final_adapter"
trainer.save_model(output_dir)




# The tokenizer is also important for inference, so save it too
tokenizer = trainer.tokenizer
tokenizer.save_pretrained(output_dir)

This simple command packages the adapter and its configuration, making it ready for deployment.

2. Preparing for Inference: Two Strategies

Now that you have your adapter, you have two main paths to use it for generating text:

1. Load the adapter on top of the base model: This keeps the base model and the adapter separate. It's flexible, allowing you to switch between different adapters without reloading the large base model.
2. Merge the adapter into the base model: This combines the adapter and base model weights into a new, single model. This is often preferred for production because it simplifies deployment and removes any inference-time overhead.

Let's explore both.

Strategy A: Using a Separate Adapter for Inference

This approach is ideal when you want to serve multiple fine-tuned tasks from a single base model instance. You can load the base model once and then dynamically apply different adapters as needed.

The peft library from Hugging Face makes this straightforward. The process is:

1. Load the original, pre-trained base model (e.g., microsoft/Phi-3-mini-4k-instruct).
2. Use the PeftModel class to load your adapter and apply it to the base model.

Here is what that looks like in practice:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch




# Define model and adapter paths
base_model_id = "microsoft/Phi-3-mini-4k-instruct"
adapter_path = "./final_adapter" # Path where you saved your adapter




# 1. Load the base model
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=BitsAndBytesConfig(load_in_4bit=True), # Can still use quantization
    device_map="auto",
)




# 2. Load and apply the LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_path)
tokenizer = AutoTokenizer.from_pretrained(adapter_path)

print("Model loaded with adapter ready for inference.")

Your model object is now ready to be used for generation, with the fine-tuned behavior from your adapter.

Strategy B: Merging the Adapter and Base Model

For many production scenarios, you want a single, self-contained model artifact. Merging the LoRA weights into the base model achieves this.

Conceptually, merging is the process of taking the original weight matrix $W$ and the LoRA matrices $A$ and $B$, and computing a new, permanent weight matrix: $W^{\prime }=W+BA$. This is done once, offline. The resulting model is a standard Transformer model that no longer requires the peft library for inference.

The peft library provides a simple method for this: .merge_and_unload().

After merging, you can save the complete model using the standard .save_pretrained() method from the transformers library.

3. End-to-End Workflow: Merge, Save, and Infer

Let's tie everything together with a complete script. This workflow takes your trained adapter, merges it with the base model, saves the result as a new standalone model, and then uses it for inference.

This is a very common pattern for preparing a fine-tuned model for deployment.

Here's a consolidated script based on that pattern, adapted for our Phi-3 model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel




# --- 1. Define Paths ---
base_model_id = "microsoft/Phi-3-mini-4k-instruct"
adapter_path = "./food-extract-finetuned/checkpoint-50" # Using a checkpoint as our adapter
merged_model_path = "./phi3-food-extract-merged"




# --- 2. Load the base model and apply the adapter ---
# Load base model in high precision for merging
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16, # Use the model's native dtype for merging
    device_map="cpu", # Load on CPU to avoid VRAM issues during merge
)




# Load the PeftModel by combining base model and adapter
model = PeftModel.from_pretrained(base_model, adapter_path)




# --- 3. Merge the adapter into the base model ---
merged_model = model.merge_and_unload()
print("Model merged successfully.")


```grasp
{
  "type": "exercise",
  "id": "068e2acf-ecd7-4aa6-a5ee-8ee3edb3021a"
}

--- 4. Save the merged model and tokenizer ---

merged_model.save_pretrained(merged_model_path)
tokenizer = AutoTokenizer.from_pretrained(adapter_path)
tokenizer.save_pretrained(merged_model_path)
print(f"Merged model and tokenizer saved to {merged_model_path}")

--- 5. Test the merged model for inference ---

Clear memory before loading the new model

del model, base_model, merged_model
torch.cuda.empty_cache()

Load the merged model for inference (can be quantized now)

from transformers import pipeline

You can load the merged model like any other standard transformer model

merged_model_for_inference = AutoModelForCausalLM.from_pretrained(
merged_model_path,
device_map="auto"
)
tokenizer_for_inference = AutoTokenizer.from_pretrained(merged_model_path)

Create a pipeline

pipe = pipeline("text-generation", model=merged_model_for_inference, tokenizer=tokenizer_for_inference)

Generate a response

messages = [
{"role": "user", "content": "Extract the food entities from the following text:\n\nText: I had a delicious burger with fries and a vanilla shake for lunch."},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.1)
print(outputs[0]["generated_text"])

This script provides a reusable template for finalizing your fine-tuned model, making it a portable and performant artifact ready for any standard inference environment.




### Conclusion

Congratulations on completing the module on practical fine-tuning! You have progressed from understanding the 'why' of fine-tuning to performing the entire workflow: configuring, training, monitoring, and now, preparing the model for deployment.

**Key Takeaways:**

*   LoRA fine-tuning produces a small **adapter**, consisting of weights and a config file, not a full model.
*   You can save an adapter using `trainer.save_model()` or `model.save_pretrained()`.
*   For inference, you can either load the adapter on-the-fly with `PeftModel` for flexibility or **merge** it into the base model using `.merge_and_unload()` for performance and simplicity.
*   A merged model is a standard Transformer model that can be saved and loaded anywhere, without dependency on the PEFT library.

This concludes our deep dive into modifying a single model's behavior. In the next module, we'll zoom out and explore how multiple models, or a single model with access to tools, can collaborate to solve even more complex problems. Our first lesson will introduce the concept of **multi-agent collaboration patterns**, setting the stage for building sophisticated AI agent systems.