Structuring Data for Risk Models

Introduction

In our last lesson, we designed a pipeline to ingest and extract data from a variety of financial documents, transforming them from unstructured files into structured JSON objects. While a critical first step, this extracted data is still "raw"—it's inconsistent, potentially inaccurate, and not yet fit for the rigorous demands of a financial risk model.

This lesson addresses that gap directly. Your objective is to specify data normalization and validation rules to structure this extracted information for risk models. We are moving from mere extraction to ensuring data integrity. This process is analogous to the rules of evidence in law or the due diligence process in finance: we must programmatically verify the facts before they can be used to build a case or make a decision. This corresponds to the "Enrichment" and "Review/Validation" stages of the Intelligent Document Processing (IDP) framework we discussed previously.

The Core Challenge: Inconsistent and Unreliable Data

The output from our extraction agent, while structured, is not yet standardized. An LLM might extract the same conceptual value in multiple ways:

• Dates: Jan 5, 2023, 05/01/2023, 2023-01-05
• Currency: $1,500.00, 1,500, 1500.0
• Labels: Gross Pay, gross_pay, Total Earnings
• Missing Values: null, "", "N/A", or the field might be omitted entirely.

A risk model, much like a compiled program, is unforgiving. It expects a predictable, rigid schema. Feeding it inconsistent data will lead to silent failures, incorrect calculations, and ultimately, flawed risk assessments. Our task is to define a system of rules to clean, standardize, and validate this data before it ever reaches the risk analysis agent.

The Solution: Defining a "Golden Schema" with Pydantic

The most robust way to enforce data quality is to define a "golden schema"—a strict, explicit definition of what the final, clean data must look like. We then parse the raw, extracted JSON against this schema.

In the Python ecosystem, the definitive tool for this is Pydantic. It uses Python's type hints to perform runtime data validation, parsing, and serialization. For an agentic system, Pydantic is not just a convenience; it is a fundamental component for ensuring reliability.

Let's explore how to use Pydantic to build our validation layer. This video provides a comprehensive introduction.

By defining a Pydantic model, we are creating a contract. For instance, for a pay stub, our initial model might look like this:

from pydantic import BaseModel
from datetime import date
from typing import Optional

class PayStubData(BaseModel):
    employer_name: str
    pay_date: date
    gross_pay: float
    net_pay: float
    ytd_gross_pay: Optional[float] = None # This field is optional

This simple class already enforces several rules:

• employer_name must be a string.
• pay_date must be a valid date. Pydantic will automatically try to parse common date string formats into a date object.
• gross_pay and net_pay must be floating-point numbers. Pydantic can often coerce strings like "1500.00" into the correct numeric type.
• ytd_gross_pay is optional and can be missing.

The Validation Workflow in Practice

The process is straightforward: take the raw JSON dictionary from the extraction pipeline and attempt to load it into our Pydantic model.

If the raw data {"pay_date": "2023-03-15", "gross_pay": "3,500.00"} is parsed, Pydantic will automatically attempt to convert "2023-03-15" to a date object and "3,500.00" to a float. If a field is missing or has an invalid type (e.g., gross_pay: "three thousand"), Pydantic will raise a ValidationError with a precise description of the error. This error itself is valuable data, forming an audit trail of data quality issues.

Specifying Advanced Rules and Business Logic

Type checking is only the first step. For underwriting, we need to enforce more specific business logic. This is where we define the core normalization and validation rules.

1. Field-Level Constraints

Pydantic's Field allows us to add constraints directly to our model definitions.

from pydantic import BaseModel, Field

class LoanApplication(BaseModel):
    applicant_age: int = Field(gt=18, description="Applicant must be over 18.")
    loan_amount: float = Field(gt=0, description="Loan amount must be positive.")
    email: str = Field(pattern=r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$")

Here, we've specified rules that are fundamental to risk assessment:

• An applicant's age must be greater than 18.
• The loan amount must be positive.
• The email must match a standard format.

2. Custom Validation and Normalization Logic

For logic that goes beyond simple constraints, we can create our own custom validator functions. This is where we can implement powerful normalization rules.

For example, imagine we extract a pay_frequency field that could be "Bi-weekly", "biweekly", or "semi-monthly". We can normalize this to a standard enum.

from pydantic import BaseModel, field_validator
from enum import Enum

class PayFrequency(str, Enum):
    WEEKLY = "weekly"
    BIWEEKLY = "biweekly"
    SEMIMONTHLY = "semimonthly"
    MONTHLY = "monthly"

class EmploymentInfo(BaseModel):
    pay_frequency: PayFrequency

    @field_validator('pay_frequency', mode='before')
    @classmethod
    def standardize_pay_frequency(cls, v: str) -> str:
        v = v.lower().strip()
        if "bi-weekly" in v or "biweekly" in v:
            return "biweekly"
        if "semi-monthly" in v:
            return "semimonthly"



        # ... add other mappings ...
        return v

In this example, the @field_validator runs before the standard type validation, cleaning the raw string so it can be correctly parsed into our PayFrequency enum. This combination of normalization and strict typing is extremely powerful.

Structuring Complex and Nested Data

Financial documents are rarely flat. A tax form has sections, a bank statement has a list of transactions, and an application has nested objects for personal, employment, and financial details. Pydantic excels at modeling these hierarchical structures.

This allows us to create a single, comprehensive Pydantic model for an entire underwriting package, with nested models for each document type, ensuring end-to-end data integrity.

From Validation to Structured LLM Output

This concept of defining a schema and validating data against it is so critical that it's now a core pattern for working with LLMs. Instead of asking an LLM to just extract text and hoping for the best, we can provide it with our Pydantic schema and instruct it to return a JSON object that conforms to that schema. Libraries like Instructor are designed for exactly this.

This creates a powerful feedback loop:

1. Agent -> LLM: "Extract the data from this document and format it according to this PayStubData Pydantic model."
2. LLM -> Agent: Provides a JSON output.
3. Agent: The Instructor library automatically parses the JSON into the PayStubData model.
4. Result:
   • Success: The data is valid and can proceed to the next step.
   • Failure: A ValidationError is raised. The agent can then automatically send the error message back to the LLM and ask it to correct its output.

An Enterprise-Grade Quality Assurance System

For a production financial system, we can formalize these rules into a comprehensive Quality Assurance (QA) framework. This moves beyond just data types to encompass logical consistency and reasonability.

A key architectural pattern for high-stakes validation, described in the LAVA research paper, is to create a bifurcated system:

1. Symbolic Reasoner: An LLM handles semantic validation (e.g., "Does this text imply the applicant has a stable job?").
2. Arithmetic Processor: For any numerical calculation (e.g., "Does Gross Pay - Deductions = Net Pay?"), the LLM generates the formula, but a deterministic tool (like a Python interpreter) executes it. This prevents numerical hallucinations. A second LLM can even act as a "checker" to ensure the generated formula correctly represents the rule. This separation of concerns is a hallmark of a robust, auditable system.

Conclusion

In this lesson, you have learned how to specify the critical data normalization and validation rules needed to transform raw, extracted data into a reliable format suitable for risk models. This is not a mere data cleaning exercise; it is a fundamental aspect of building trustworthy and auditable agentic systems in finance.

Key Takeaways:

• Define a "Golden Schema": Use a tool like Pydantic to create a strict, explicit model of your desired data structure, including data types, optionality, and constraints.
• Validate and Normalize: Parse raw extracted data against this schema. Use field-level constraints (Field) and custom validators (@field_validator) to enforce business logic and normalize inconsistent values.
• Embrace Structured Output: Leverage your Pydantic schemas to instruct LLMs to return structured, pre-validated data, creating a self-correcting extraction loop.
• Build a Layered QA System: For enterprise-grade reliability, layer multiple validation strategies: type checks, reasonability checks (e.g., historical comparison), and logical consistency checks (e.g., using an Arithmetic Processor for calculations).

Preview of the Next Lesson:

We now have a pipeline that produces clean, validated, and structured data representing the contents of the underwriting documents. The next logical step is to use this data for analysis. In the upcoming lesson, you will design a process for an agent to perform integrated risk analysis using both this structured data and the original unstructured text. We will explore how an agent can synthesize these different forms of information to reason about risk and begin formulating a credit recommendation.