Building a Claim Verification Pipeline

The Anatomy of an Automated Fact-Checker

Fact-checking a piece of text isn't a single action but a sequence of distinct tasks. We can model our pipeline on a logical, multi-step process that mirrors how a human researcher would work.

A very effective way to structure this is by thinking in terms of specialized agents or nodes, each with one job.

As the article highlights, this process of grounding the LLM's output in external, verifiable data is what transforms it from a creative text generator into a reliable assistant. We will implement these three logical steps within our LangGraph workflow.

Step 1: Claim Extraction

Our first task is to parse the generated article and extract a list of all statements that present themselves as facts. These are sentences that contain numbers, statistics, product names, or specific assertions that can be proven true or false.

To do this programmatically, we'll use a technique we've touched on before: instructing an LLM to return a structured JSON output. We define the "shape" of the data we want, and the LLM acts as a powerful, flexible parser.

Implementing the extractClaimsNode in TypeScript

Let's translate this concept into code for our project. We'll start by defining the TypeScript interfaces for our structured output.

// in a types/verification.ts file

export interface VerifiableClaim {
  claim: string; // The specific, verifiable statement
}

export interface ClaimVerificationResult extends VerifiableClaim {
  status: 'Supported' | 'Contradicted' | 'Unsubstantiated';
  evidence: string; // Summary of evidence found
  reasoning?: string; // Explanation for Contradicted/Unsubstantiated
}

Now, we can write the prompt and the function for our extractClaimsNode. This node will take the generated article from the agent's state and populate a new field, claims_to_verify.

// in your nodes.ts file
import { OpenAI } from 'openai';
import { AgentState } from './state';
import { VerifiableClaim } from './types/verification';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function extractClaimsNode(state: AgentState): Promise<Partial<AgentState>> {
  console.log("--- EXTRACTING CLAIMS ---");
  if (!state.article) {
    throw new Error("Article not found in state for claim extraction.");
  }

  const system_message = `
    You are a meticulous editor tasked with identifying verifiable claims in a technical article.
    Analyze the user-provided text and extract all specific, factual statements that can be independently verified.
    Ignore subjective opinions, marketing fluff, and general statements. Focus on data, statistics, technical specifications, and concrete events.

    Respond with ONLY a valid JSON object with a single key "claims", which is an array of objects. Each object should have a single key "claim" containing the extracted statement.

    Example Input: "Our new 'Quantum' processor is 50% faster and uses the Foxtrot architecture. We think it's the best on the market."
    Example Output:
    {
      "claims": [
        { "claim": "The 'Quantum' processor is 50% faster." },
        { "claim": "The 'Quantum' processor uses the Foxtrot architecture." }
      ]
    }
  `;

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: system_message },
      { role: "user", content: state.article }
    ],
    response_format: { type: "json_object" },
  });

  try {
    const result = JSON.parse(response.choices[0].message.content || '{}');
    const claimsToVerify: VerifiableClaim[] = result.claims || [];
    console.log(`Extracted ${claimsToVerify.length} claims to verify.`);
    return { claims_to_verify: claimsToVerify };
  } catch (e) {
    console.error("Failed to parse claims from LLM response:", e);
    return { claims_to_verify: [] };
  }
}

Steps 2 & 3: Evidence Gathering and Verification

With a structured list of claims, we now move to the research phase. For each claim, we need to find evidence and then make a judgment. This is a perfect use case for a pattern called Corrective-RAG (CRAG), a form of "active" retrieval where the agent reasons about the information it finds.

The CRAG workflow is simple but powerful:

1. Retrieve: Look for evidence in your primary, trusted data sources (in our case, the tech articles we collected in Module 2).
2. Grade: Assess if the retrieved information is relevant to the claim.
3. Act:
   • If relevant, proceed to verification.
   • If not relevant or insufficient, trigger a corrective action: a web search to find external evidence.
4. Verify: Compare the original claim against all the gathered evidence to determine if it's supported.

To implement the "grading" and "verifying" steps, we'll once again use the LLM-as-a-Judge pattern, but with prompts specifically designed for factual assessment.

Integrating the Verification Pipeline in LangGraph

We now have the building blocks for a complete verification pipeline. We can implement this as a verifyClaimsNode that runs after the extractClaimsNode. This new node will orchestrate the entire CRAG process for each claim.

Here is a conceptual outline for the verifyClaimsNode. It's more complex because it contains an inner loop.

// in your nodes.ts file
// Assume we have a `webSearchTool` and a `vectorstore` retriever available

export async function verifyClaimsNode(state: AgentState): Promise<Partial<AgentState>> {
  console.log("--- VERIFYING CLAIMS ---");
  if (!state.claims_to_verify || state.claims_to_verify.length === 0) {
    console.log("No claims to verify.");
    return { verified_claims: [] };
  }

  const verificationResults: ClaimVerificationResult[] = [];

  for (const claim of state.claims_to_verify) {
    // This is a simplified representation of the logic.
    // In a real implementation, each of these would be an async LLM call with specific prompts from the Self-RAG tutorial.
    
    // 1. Retrieve from internal sources
    const internalDocs = await retriever.invoke(claim.claim);
    
    // 2. Grade relevance of internal docs
    const isRelevant = await gradeRelevance(claim.claim, internalDocs); // LLM call

    let evidence = internalDocs;
    if (!isRelevant) {
      console.log(`Internal docs not relevant for "${claim.claim}". Performing web search.`);
      // 3. Corrective action: web search
      const webResults = await webSearchTool.invoke(claim.claim);
      evidence.push(...webResults);
    }
    
    // 4. Verify claim against all gathered evidence
    const verification = await verifyWithEvidence(claim.claim, evidence); // LLM call using Hallucination Grader prompt
    
    verificationResults.push({
      claim: claim.claim,
      status: verification.status, // e.g., 'Supported'
      evidence: verification.evidenceSummary,
      reasoning: verification.reasoning
    });
  }

  console.log("Verification complete.");
  return { verified_claims: verificationResults };
}

This node would be added to your StateGraph. A conditional edge following it would then inspect state.verified_claims to decide the next step. If all claims are 'Supported', the graph proceeds to the final human approval step. If any are 'Contradicted' or 'Unsubstantiated', it can route to a new reviseArticleNode or simply flag the article for manual review, passing along the detailed verification results.

Conclusion

You have now designed a sophisticated automated fact-checking pipeline. By separating the process into distinct stages—extraction, evidence gathering, and verification—and by using corrective patterns like CRAG, you are building a powerful guardrail against factual errors. This moves your system from just generating fluent text to generating trustworthy, credible content.

Key Takeaways:

• Fact-checking is a multi-step process: Break it down into Extract, Search, and Verify.
• Structured Extraction is Key: Use an LLM with a JSON schema to turn unstructured text into a clean list of verifiable claims.
• CRAG provides a robust verification pattern: Retrieve from trusted sources, grade the relevance of what you find, and use a web search as a fallback.
• Graders are specialized LLM prompts: Use Retrieval Graders to check for relevance and Hallucination Graders to check for factual support.
• This pipeline is a critical guardrail: The results of this verification process will determine the next steps in your LangGraph workflow.

Preview of the next lesson:
We've designed the pipeline and can now get a verdict on each claim. But what do we do with that verdict? In our next lesson, we will focus on the final part of this learning outcome: "Integrate the verification pipeline as a guardrail to flag or revise inaccurate content." We'll implement the conditional logic and nodes that act on the output of our fact-checker, making our agent capable of self-correction.