Evaluating LLM-Generated Content with Evaluation Prompts

The "LLM-as-a-Judge" Pattern

The idea of using one AI to evaluate another's output is a powerful and increasingly common technique known as "LLM-as-a-Judge." It's based on a simple but effective premise: it is often easier to critique work against a set of rules than it is to create that work from scratch. In your experience as an engineer, this is analogous to code review—it's a different, more focused skill than writing the initial implementation.

To get a formal introduction to this concept, let's watch a short video.

As the video explains, we aren't asking an LLM to re-do its work; we're giving it a simpler, more constrained task: classify a piece of text against specific criteria.

Designing Effective Evaluation Prompts

An LLM-as-a-Judge is only as good as the instructions it's given. The "art" of this technique lies in designing precise, unambiguous evaluation prompts. Let's explore some best practices and then apply them to our specific criteria.

With these principles in mind, let's design prompts for our three key criteria: brand voice consistency, clarity, and coherence.

1. Evaluating Brand Voice Consistency

In Module 1, you created a brand voice guide. Now, we'll translate that guide into a prompt an LLM can use for evaluation. Your brand voice determines your content's personality and is crucial for building a recognizable presence.

Let's assume your brand voice guide defined a persona that is "expert but approachable," a tone that is "analytical and insightful," and language that is "clear, concise, and avoids excessive jargon."

Here is an example of an evaluation prompt for this criterion:

You are a brand strategy expert evaluating a piece of content for brand voice alignment.

**Brand Voice Guidelines:**
- **Persona:** Expert but approachable.
- **Tone:** Analytical and insightful.
- **Language:** Clear, concise, and avoids unnecessary technical jargon.

**Content to Evaluate:**
"""
{generated_article_text}
"""

**Task:**
Evaluate the content against the brand voice guidelines. Respond with a JSON object containing two fields:
1. `score`: A boolean value (`true` if the content aligns with the brand voice, `false` otherwise).
2. `explanation`: A brief explanation for your score, detailing which aspects of the brand voice were met or missed.

This prompt is effective because it:

• Establishes a clear role for the LLM ("brand strategy expert").
• Provides the specific evaluation rubric (the brand voice guidelines).
• Requests a structured output (JSON) that can be easily parsed in our TypeScript code.

2. Evaluating Clarity and Coherence

While related to brand voice, clarity and coherence are distinct structural qualities.

• Clarity: Is the writing easy to understand? Are sentences unambiguous?
• Coherence: Does the text flow logically from one point to the next? Is there a clear argument or narrative?

We can combine these into a single, focused evaluation. The "G-Eval" framework offers excellent examples of how to define such metrics.

Drawing inspiration from this, here is an example prompt for clarity and coherence:

You are an expert editor evaluating the clarity and coherence of a technical article.

**Content to Evaluate:**
"""
{generated_article_text}
"""

**Task:**
Evaluate the content on a scale of 1 to 5 for both clarity and coherence. Respond with a JSON object containing the following fields:
1. `clarity_score`: An integer from 1 (very confusing) to 5 (very clear).
2. `coherence_score`: An integer from 1 (disjointed) to 5 (flows logically).
3. `explanation`: A brief explanation for your scores, pointing out specific examples of well-structured or confusing passages.

From Prompts to a Structured Evaluator Node

Designing prompts is the first step. To make this operational in your LangGraph agent, you need to implement a node that executes these evaluations and provides a structured output. The concept of using Pydantic models in Python to enforce this structure is a powerful one. We can achieve the same result in TypeScript using interfaces.

This next resource provides a fantastic, code-heavy walkthrough of building structured evaluators. While the code is in Python, the pattern is directly translatable to TypeScript and the OpenAI SDK.

Implementing the critiqueNode in TypeScript

Let's translate the pattern from the LangSmith article into a TypeScript function for our LangGraph workflow.

First, define the structure for your evaluation results using an interface.

// in a types/evaluation.ts file

interface BrandVoiceEvaluation {
  score: boolean;
  explanation: string;
}

interface ClarityCoherenceEvaluation {
  clarity_score: number;
  coherence_score: number;
  explanation: string;
}

// This will be added to our main AgentState
export interface ContentEvaluation {
  brand_voice?: BrandVoiceEvaluation;
  clarity_coherence?: ClarityCoherenceEvaluation;
  is_approved: boolean;
  feedback_for_rewrite?: string;
}

Now, you can create the critiqueNode function. This node will take the generated content from the state, run one or more evaluation prompts, and update the state with the structured results.

// in your nodes.ts file
import { OpenAI } from 'openai';
import { AgentState } from './state'; // Assuming your state is defined here
import { ContentEvaluation } from './types/evaluation';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

// A helper function for a single evaluation
async function evaluateBrandVoice(text: string): Promise<BrandVoiceEvaluation> {
  const system_message = `
    You are a brand strategy expert... // The full prompt from earlier
  `;

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: system_message },
      { role: "user", content: text }
    ],
    response_format: { type: "json_object" },
  });

  try {
    const result = JSON.parse(response.choices[0].message.content || '{}');
    // Simple validation
    if (typeof result.score === 'boolean' && typeof result.explanation === 'string') {
      return result as BrandVoiceEvaluation;
    }
    throw new Error("Invalid JSON structure from brand voice evaluator.");
  } catch (e) {
    console.error("Failed to parse brand voice evaluation:", e);
    return { score: false, explanation: "Evaluation failed due to parsing error." };
  }
}

// The main critique node for the graph
export async function critiqueNode(state: AgentState): Promise<Partial<AgentState>> {
  console.log("--- Critiquing Content ---");
  if (!state.article) {
    throw new Error("Article not found in state for critique.");
  }

  // You can run evaluations in parallel
  const [brandVoiceEval /*, clarityEval, ...*/] = await Promise.all([
    evaluateBrandVoice(state.article),
    // Call other evaluator functions here
  ]);

  const evaluation: ContentEvaluation = {
    brand_voice: brandVoiceEval,
    // clarity_coherence: clarityEval,
    is_approved: brandVoiceEval.score, // Simple logic: approve if brand voice is OK
    feedback_for_rewrite: !brandVoiceEval.score ? brandVoiceEval.explanation : undefined,
  };
  
  console.log("Critique complete:", evaluation);

  return { evaluation };
}

This critiqueNode can now be added to your LangGraph workflow. A conditional edge following this node would check state.evaluation.is_approved. If true, it proceeds to the human approval step. If false, it can loop back to the generation node, using the feedback_for_rewrite to guide a better attempt.

Conclusion

You have now learned how to construct a sophisticated, automated quality assurance process for your content generation pipeline. By moving beyond simple error handling to agentic self-critique, you are building a system that can not only create content but also refine it to meet high standards of quality and brand consistency.

Key Takeaways:

• LLM-as-a-Judge: This pattern uses an LLM with a constrained task (evaluation) to enforce quality on the output of another LLM.
• Prompt Design is Key: Effective evaluation requires prompts that define a clear role, provide a specific rubric, and request a structured output (like JSON).
• Deconstruct Criteria: Break down abstract goals like "good content" into specific, measurable criteria like brand_voice, clarity, and coherence.
• Structured Output for Automation: Use interfaces (like Pydantic models in Python) and the LLM's JSON mode to get evaluation results in a format your agent can programmatically act upon.

Preview of the next lesson:
We've built a powerful quality gate for style and structure. But what about the substance? Is the content factually accurate? The next lesson addresses this head-on as we move to the next learning outcome: Implement a claim extraction and verification pipeline that checks generated content against trusted data sources. We will add another crucial guardrail to our system to prevent it from generating plausible-sounding but incorrect information.