Dataset Preparation for Supervised Fine-tuning

1. What is Supervised Instruction Fine-Tuning (SFT)?

Before we can prepare a dataset, we need to be crystal clear on what it's for. Supervised Fine-Tuning (SFT) is the process of teaching a pre-trained language model to follow instructions or adopt a specific persona by training it on a dataset of high-quality examples. Each example is essentially a prompt-and-response pair.

Think of it this way: a base pre-trained model (like Llama 3) has completed its general education by reading a huge portion of the internet. It's incredibly knowledgeable but might not be a great conversationalist or assistant. SFT is like an apprenticeship where you provide this knowledgeable model with specific examples of the tasks you want it to perform, along with the ideal way to complete them.

2. Sourcing and Structuring Your Data

The first step in preparing a dataset is acquiring the raw data. This data generally falls into three categories.

Regardless of the source, the goal is to structure this data into a consistent format. For instruction tuning, this usually means creating records that contain, at a minimum, an instruction and the desired output.

The two most common structures are:

1. Instruction-Following Format (e.g., Alpaca): This is ideal for single-turn, question-answer, or instruction-completion tasks. Each record is typically a JSON object with fields like instruction, input (optional context), and output.

2. Conversational Format (e.g., ShareGPT): This is used for training chat models. The data is structured as a list of turns in a conversation, with each turn having a role (e.g., "user", "assistant") and content.

3. From Structured Data to a Model-Ready String: Prompt Templating

An LLM does not take a JSON object as input; it takes a single, continuous string of text. The process of converting your structured data fields (instruction, input, output) into this single string is called prompt templating.

Different models are fine-tuned with different templates. Using the correct template is critical because the model learns to expect and respond to these specific formats and special tokens.

Let's explore two common templating methods.

Method 1: The Alpaca Format

A widely used format is the one popularized by the Stanford Alpaca project. It combines the instruction, input, and response fields into a standard template.

As you saw, the Alpaca format might look like this:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input}

### Response:
{output}

Method 2: Model-Specific Chat Templates

Modern instruction-tuned models like Llama 3-Instruct or Mistral-Instruct are trained not just on single instructions but on conversations. Their tokenizers come with built-in chat templates that automatically format a list of conversational turns into the exact string format the model expects, including special tokens like [INST], [/INST], <|start_header_id|>, etc.

Using the tokenizer's apply_chat_template method is now the standard best practice, as it ensures you are perfectly matching the format the model was aligned with.

sample = {
    "instruction": "Translate the following to French.",
    "input": "Hello, how are you?",
    "output": "Bonjour, comment ça va?"
}

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Translate the following to French.

### Input:
Hello, how are you?

### Response:
Bonjour, comment ça va?

4. From String to Tensors: Tokenization and Loss Masking

The final and most technical step is to convert the formatted string into tensors that the model can train on. This involves two key concepts you'll use constantly in practice.

The Training Objective Revisited

As we've discussed, a causal language model is trained to predict the next token. When we fine-tune, we give the model the entire formatted string (prompt + response) as input. The model's task is to predict the next token at each position.

So, if the input is [t1, t2, t3, ...], the target labels will be [t2, t3, t4, ...].

The Crucial Trick: Loss Masking

Here's the key insight for instruction fine-tuning: we only want to penalize the model for getting the response part wrong. We don't care if it can't predict the tokens in the prompt we gave it. We want it to learn to generate the output given the instruction and input.

To achieve this, we use loss masking. We construct a labels tensor that is identical to the input_ids tensor, but we replace the token IDs corresponding to the prompt with a special value (usually -100). The PyTorch cross-entropy loss function is designed to ignore labels with this value.

This ensures that the loss is only calculated on the response tokens, and therefore, the gradients are only backpropagated based on the model's ability to generate the desired output.

5. Final Quality Checks

Before you invest expensive GPU hours into a fine-tuning job, a final data-quality pass is essential.

Here is a practical checklist based on industry best practices:

• Structural Integrity: Ensure all your data files are valid (e.g., properly formatted JSON) and contain the required fields.
• Remove Duplicates: Deduplicating your dataset prevents the model from overweighting certain examples and can improve generalization.
• Content Filtering: Scan for and remove toxic content or Personally Identifiable Information (PII) unless it's a specific goal of the fine-tuning.
• Length Constraints: Most examples should fit within your model's context window. Longer examples will be truncated, leading to incomplete training samples.
• Split Your Data: Partition your final dataset into training, validation, and test sets (e.g., 85% train, 10% validation, 5% test) to monitor for overfitting and evaluate the final model's performance.

Conclusion

You have now learned the end-to-end process of preparing a dataset for supervised instruction fine-tuning. This is a foundational skill that moves you from being a user of LLMs to a builder.

Key Takeaways:

• SFT requires structured data, typically in an instruction-following (Alpaca) or conversational (ShareGPT) format.
• Data can be sourced from public datasets, internal company data, or generated synthetically by a "teacher" model.
• Prompt templating is the critical step of converting structured data into the single-string format the model expects, using either a standard format like Alpaca's or a model-specific chat template.
• Loss masking is a vital technique where prompt tokens in the label tensor are replaced with -100 to ensure the model is only trained to predict the desired response.
• Rigorous data quality checks before training are essential to avoid wasting time and compute resources.

What's Next?

We've covered the "how" of fine-tuning algorithms and the "what" of the data they need. In the next lesson, we will finally bring these two pieces together. You will learn how to implement PEFT (e.g., LoRA) on a pre-trained model using the HuggingFace PEFT library, taking the dataset we've just designed and launching your first fine-tuning job.