ML Data Validation & Quality Monitoring

The Unique Challenge of Data Quality in ML

Data validation is not a new problem, but it takes on unique dimensions in the context of machine learning. Unlike traditional software where bad data might cause a single transaction to fail, in ML, poor data quality can silently degrade model performance over time, introduce subtle biases, or, in the worst cases, cause feedback loops that amplify errors throughout the system.

The following paper from Google researchers, which describes the data validation component of their TFX platform, provides an excellent real-world perspective on these challenges.

As the paper illustrates, several factors make data validation in ML particularly challenging:

• Decoupled Systems: The teams generating the data are often different from the teams consuming it to train models. This leads to a lack of visibility and a high potential for mismatched assumptions.
• Silent Failures: ML models are often robust to a certain amount of noise. This means they might continue to produce predictions without crashing, even if the input data is corrupt, masking the problem until performance has significantly degraded.
• Feedback Loops: In many systems, model predictions are logged and eventually become part of future training sets. An error that affects predictions can thus become permanently embedded in the training data, creating a vicious cycle.

A robust data validation strategy, therefore, must be designed to catch errors early, provide clear and actionable alerts, and account for the entire ML lifecycle.

A Layered Strategy for Data Validation

A comprehensive data validation strategy can be broken down into two main layers:

1. Static Validation (Intra-batch): Validating a single batch of data against a set of predefined expectations or a schema.
2. Dynamic Validation (Inter-batch): Comparing different datasets to each other to detect drift or skew, such as comparing the current production data against the training data.

Let's explore each layer in detail.

Layer 1: Schema-Based Validation

The cornerstone of a data quality strategy is a schema. A schema is a formal description of your data's expected properties, going far beyond simple data types. It acts as a contract that your data must adhere to. A production-grade system checks every incoming batch of data against this schema.

The TFX paper provides an excellent blueprint for what such a schema should contain.

Key takeaways from this approach include:

• Schema as Code: The schema should be treated as a production asset, version-controlled alongside your model training code.
• Comprehensive Constraints: A good schema validates not just data types (INT, FLOAT), but also:
  • Presence: Is a feature required or optional? What percentage of examples must have it? This catches bugs where a feature is silently dropped.
  • Valency: Is the feature single-valued or multi-valued? This prevents errors in how data is parsed into tensors.
  • Domain: For categorical features, what are the allowed string values? For numerical features, what is the expected range (e.g., age must be > 0)?
• Schema Lifecycle: Manually creating a schema for thousands of features is impractical. A robust system should support:
  1. Inference: Automatically generating an initial schema by analyzing existing data.
  2. Curation: Allowing users to manually review and refine the inferred schema.
  3. Evolution: When new data violates the schema (e.g., a new category appears), the system should alert the user and suggest a schema update, which can then be formally approved and versioned.

Layer 2: Detecting Drift and Skew

While schema validation catches many errors within a single dataset, some of the most critical issues in ML arise from differences between datasets. This family of problems is broadly known as ML drift.

The following video provides a clear, high-level overview of the different types of drift.

To summarize the key concepts:

• Concept Drift: The fundamental relationship between features and the target variable changes (i.e., $P(y|X)$ changes). For example, during a recession, a person's income level may no longer predict creditworthiness in the same way it used to. This is often the hardest type of drift to detect and usually requires labeled data.
• Data Drift: The statistical distribution of the input features changes (i.e., $P(X)$ changes). For example, a marketing campaign in a new region might change the distribution of user ages or locations.
• Training-Serving Skew: This is a critical and common form of data drift where the data distribution at serving time is different from the distribution of the training data. This can be caused by different code paths for feature generation, data sampling bugs, or time-travel issues (e.g., using data for training that wasn't available at serving time).

Detecting Data Drift and Skew

Detecting these changes requires comparing the distribution of your "current" data (e.g., live production data) against a "reference" baseline (e.g., the training dataset).

When ground truth labels are available in near real-time, you can monitor for drift by tracking model performance metrics (accuracy, precision, recall, etc.). A drop in performance is a strong signal of drift.

However, in many cases, labels are delayed or unavailable. In this scenario, you must rely on statistical methods to compare the feature distributions directly. The TFX paper discusses a practical approach to this.

Common statistical measures for detecting distribution drift include:

• Population Stability Index (PSI): Widely used in finance, it measures the change in distribution of a variable between two populations.
• Kolmogorov-Smirnov (K-S) Test: A non-parametric test that compares the cumulative distribution functions of two samples.
• Kullback-Leibler (KL) Divergence: Measures how one probability distribution is different from a second, reference probability distribution.
• Jensen-Shannon (J-S) Divergence: A symmetric and smoothed version of KL divergence, often preferred in practice.

The choice of metric often involves a trade-off between statistical power and interpretability. The key is to establish a baseline, set a threshold for alerts, and provide actionable context when drift is detected (e.g., which feature drifted the most?).

Practical Implementation: Tools and Workflows

These concepts are operationalized within MLOps workflows using a combination of orchestration tools and specialized libraries.

Open-source libraries like Evidently AI, Great Expectations, and the TensorFlow Data Validation (TFDV) library (which is based on the paper we studied) are designed specifically for this purpose. They provide the machinery to:

1. Define a schema or a set of expectations.
2. Profile new data.
3. Compare data profiles against the schema or against a reference dataset.
4. Generate detailed reports on data quality, drift, and model performance.

The following article demonstrates how a tool like Evidently fits into an automated pipeline.

This practical workflow of comparing "current" vs. "reference" data is the central paradigm for all drift and skew detection systems.

Conclusion

In this lesson, we have constructed a comprehensive strategy for data validation and quality monitoring. You are now prepared to design a system that protects ML pipelines from the pervasive threat of bad data.

Key Takeaways:

• Data validation is a first-class citizen in MLOps, just as important as the model or infrastructure.
• A robust strategy is layered, combining schema-based validation for static checks within a dataset and drift/skew detection for dynamic checks between datasets.
• A schema should be treated as code—inferred, curated, and versioned—and should define expectations for feature presence, valency, and domains.
• Be able to clearly distinguish between data drift (change in $P(X)$), concept drift (change in $P(y|X)$), and training-serving skew.
• Drift detection relies on comparing a current dataset to a reference dataset, using either model performance metrics (if labels are available) or statistical distance measures (if labels are absent).

In our next lesson, we will build directly on today's topic. Having established the importance of a data schema, we will explore how to design a strategy for data and schema versioning. This is crucial for ensuring reproducibility, tracking data lineage, and enabling the "time travel" capabilities that make modern data platforms like the Lakehouse so powerful for ML.