Advanced Prompting Techniques for Image Generation

Part 1: Negative Prompts and Classifier-Free Guidance (CFG)

At its simplest, a negative prompt is a list of concepts you want to exclude from your generated image. Common examples include terms like "blurry, low quality, distorted, extra limbs" to avoid artifacts, or "trees, buildings" to control the background of a scene.

Let's start by seeing why this is so important, especially in modern models.

The improvement is undeniable. But how does this work? The model doesn't have a separate "negative prompt" input. The magic lies in a technique called Classifier-Free Guidance (CFG).

The Mechanism of Classifier-Free Guidance

CFG is a clever method that allows a conditional diffusion model to improve its adherence to the conditioning signal (e.g., a text prompt) without needing a separate classifier network.

The core idea involves two key phases:

1. Training: The model is trained on conditional inputs (e.g., an image and its text description). However, a certain percentage of the time (e.g., 10-20%), the conditioning signal is randomly dropped and replaced with a generic "unconditional" token. This forces the model to learn to denoise both with and without guidance.

2. Inference (Sampling): At each denoising step, we perform two forward passes through the U-Net:

   • One pass conditioned on our text prompt to get a noise prediction, let's call it ${ϵ}_{\text{cond}}$.
   • A second pass conditioned on the unconditional token (e.g., an empty string "") to get an unconditional noise prediction, ${ϵ}_{\text{uncond}}$.

The final noise prediction, ${ϵ}_{\text{pred}}$, is an extrapolation. We take the unconditional prediction and push it in the direction of the conditional one:

Here, $w$ is the guidance scale (or CFG scale).

• If $w=0$, we get the unconditional prediction.
• If $w=1$, we get the conditional prediction.
• If $w>1$ (e.g., a common value is 7.5), we are extrapolating, pushing the result further in the direction of our prompt than what the model saw during training. This amplifies the prompt's influence, leading to images that align more strongly with the text.

The following video explains this mechanism in the context of class-conditioning, but the principle is identical for text-conditioning.

From CFG to Negative Prompts

The leap to negative prompts is surprisingly simple. Instead of using a generic empty string for the "unconditional" pass, we use our negative prompt.

The process becomes:

1. Get the text embeddings for the positive prompt ("A majestic lion"). This gives us context_pos.
2. Get the text embeddings for the negative prompt ("blurry, cartoon, low-resolution"). This gives us context_neg.
3. At each sampling step, do two forward passes:
   • One with context_pos to get ${ϵ}_{\text{pos}}$.
   • One with context_neg to get ${ϵ}_{\text{neg}}$.
4. Calculate the final noise prediction:

This formula intuitively tells the model: "Start with what I don't want, and move away from it towards what I do want."

Implementation in the Sampling Loop

Given your background, you can see how this maps to a PyTorch implementation. Here's the pseudocode for a single sampling step:

def sample_step(noisy_latents, t, positive_prompt, negative_prompt, guidance_scale):
    # 1. Get text embeddings from the encoder
    c_pos = text_encoder(positive_prompt)
    c_neg = text_encoder(negative_prompt)

    # 2. Duplicate latents and concatenate contexts for a single batch forward pass
    latent_model_input = torch.cat([noisy_latents, noisy_latents])
    context = torch.cat([c_neg, c_pos])
    
    # Add time embeddings
    time_emb = get_time_embedding(t)
    time_emb = torch.cat([time_emb, time_emb])

    # 3. Perform two forward passes in one go
    # The U-Net's forward signature is forward(x, context, time)
    noise_preds = unet(latent_model_input, context, time_emb)
    
    # 4. Separate the predictions
    noise_pred_neg, noise_pred_pos = noise_preds.chunk(2)

    # 5. Apply the CFG formula
    final_noise_pred = noise_pred_neg + guidance_scale * (noise_pred_pos - noise_pred_neg)
    
    # 6. Use the final prediction to denoise the latents (as in a standard DDPM/DDIM step)
    # ...
    
    return denoised_latents

This is a highly efficient implementation as it leverages batching to perform both forward passes simultaneously.

Part 2: Prompt Re-weighting

Negative prompting is about excluding concepts entirely. Prompt re-weighting provides more subtle, fine-grained control by increasing or decreasing the influence of specific words within your positive prompt.

For example, in the prompt "A painting of a red car", you might want to make the car more red, or ensure the "painting" style is strongly expressed.

As the article shows, a common syntax is (a majestic castle:1.3). This instructs the model to pay 30% more attention to "a majestic castle". How do we implement this? There are two main approaches.

Approach 1: Heuristic (Embedding-level Manipulation)

Many popular implementations (like Automatic1111's web UI) parse this syntax and manipulate the text embeddings before they are sent to the U-Net. This is handled by libraries that sit between the user's prompt string and the text encoder. For example, a weight of 1.3 might be implemented by duplicating the corresponding text tokens in the sequence fed to the text encoder, effectively making them appear more often. This is a practical but indirect way of influencing the model.

Approach 2: Principled (Attention-level Manipulation)

A more direct and powerful method is to manipulate the cross-attention maps themselves during the U-Net's forward pass. This was explored in the influential "Prompt-to-Prompt" paper.

The core idea: in the previous lesson, you learned that cross-attention calculates a score for how much each pixel should "attend" to each word token. By directly scaling these scores for a specific word, we can precisely control its influence on the final image.

The implementation of this principled approach involves intercepting the attention calculation within your CrossAttention module.

Here's the logic:

1. In your CrossAttention module, after calculating the attention scores but before applying softmax: $$\text{attention\_scores}=\frac{Q\cdot K^T}{\sqrt{d_k}}$$
2. Identify the index j of the token you want to re-weight (e.g., the token for "red").
3. Multiply the scores corresponding to that token by your desired weight c.

   # Inside CrossAttention forward pass
   # attention_scores has shape (batch, heads, seq_len_q, seq_len_kv)
   
   weight = 1.3
   token_index_to_reweight = 4 # Example: index for 'red'
   
   attention_scores[:, :, :, token_index_to_reweight] *= weight
   

4. Apply the softmax function to the modified scores. $$\text{attention\_probs}=\text{softmax}(\text{attention\_scores})$$
5. Proceed with the rest of the attention calculation (attention_probs @ V).

This method gives you surgical control over the influence of each token. It requires modifying your CrossAttention module or using PyTorch hooks to inject this logic, which is a very powerful technique for an ML engineer to have in their toolkit.

Conclusion

Today, you've learned how to implement two of the most important techniques for controlling text-to-image diffusion models. These methods are essential for practical applications, allowing you to debug artifacts and steer generation with high precision.

Key Takeaways:

• Negative Prompting is powered by Classifier-Free Guidance (CFG): By replacing the unconditional prediction with one conditioned on a negative prompt, we can steer the generation away from unwanted concepts.
• CFG Implementation: This involves a dual forward pass during inference (one for the positive prompt, one for the negative) and combining the results with a guidance scale.
• Prompt Re-weighting offers fine-grained control: It allows you to amplify or suppress the influence of specific words within your prompt.
• Re-weighting Implementation: This can be done at the attention level by directly scaling the attention scores for specific tokens before the softmax operation, offering a principled way to control semantic influence.

Next Lesson Preview:

We've just discussed manipulating cross-attention maps to control the output. This raises a crucial question: how can we see what the model is attending to? In the next lesson, we will dive into exactly that. We will learn how to analyze cross-attention maps to debug prompt alignment failures, giving you a powerful diagnostic tool to understand why a model might be ignoring or misinterpreting parts of your prompt.