In our last lesson, we focused on safeguarding the integrity of our ML pipelines by designing robust data validation and monitoring strategies. A central piece of that puzzle was the schema, which acts as a formal contract for your data. This naturally leads to the question: what happens when that contract needs to change? Or when the data itself is updated?
This lesson addresses that question directly, focusing on the learning outcome: Design a strategy for data and schema versioning to ensure reproducibility and track lineage. We will explore how to manage changes to every component of an ML system—data, schemas, code, and models—to ensure that any result, whether a model artifact or a single prediction, can be reliably reproduced, debugged, and audited. This capability is not a luxury; it is a fundamental requirement for building production-grade ML systems and a core competency tested in system design interviews.
The Foundation: Versioning, Provenance, and Reproducibility
Before diving into specific strategies, it's crucial to understand three interconnected concepts that form the "why" behind our designs. A system might fail, a model might exhibit bias, or a regulatory body might demand an explanation for a decision. In all these cases, a simple "we don't know" is not an acceptable answer.
The following chapter from Machine Learning in Production provides a clear definition of these concepts and uses a compelling scenario—debugging a potentially discriminatory loan decision—to illustrate their importance.
Versioning, Provenance, and Reproducibility
This reading sets the stage by defining the core concepts and motivating their importance through a realistic example.
Please read the introductory section "Versioning, Provenance, and Reproducibility", and then continue with the motivating scenario. As you read, consider how you would begin to investigate the loan decision problem without the ability to know which model version, which data, and which code were used.
To summarize the key terms:
- Versioning: The practice of tracking and managing changes to artifacts (code, data, models) over time. This allows you to retrieve any historical state.
- Provenance (or Lineage): The record of where your data came from and the sequence of transformations it has undergone. It's the "family tree" of your data and models.
- Reproducibility: The ability to recreate an outcome, such as a trained model or a specific prediction, by using the exact same versioned components and processes.
Without these three pillars, debugging complex ML systems becomes an exercise in guesswork, and accountability is impossible.
Designing a Data Versioning Strategy
Software engineers have long used tools like Git to version code. However, these tools were designed for text-based source code and struggle with the scale of typical ML datasets. Pushing a 100GB dataset into a Git repository is simply not feasible. We need different strategies.
The Machine Learning in Production chapter outlines five common approaches to versioning large datasets.
Versioning, Provenance, and Reproducibility
This section provides a conceptual framework for data versioning by detailing the common strategies and their respective trade-offs.
Please read the section on versioning large datasets. As you read, focus on the fundamental trade-off that appears in most of these strategies: storage space vs. computational effort for retrieval.
The five strategies you just read about offer a spectrum of solutions:
- Storing Copies: Simple and fast retrieval, but high storage cost.
- Storing Deltas: Space-efficient if changes are small, but computationally expensive to reconstruct old versions.
- Offsets in Append-Only Data: Highly efficient for immutable data like logs and event streams. This is the core idea behind event sourcing.
- Versioning Individual Records: Works well for key-value stores or document databases where records are updated independently (e.g., a customer profile).
- Versioning the Pipeline: Instead of storing derived data (like feature sets), you version the code and raw data needed to recreate it on demand. This is powerful but requires fully deterministic pipelines.
These are the conceptual building blocks. Now, let's see how they are implemented in practice.
Practical Data Versioning with DVC
A popular approach is to use a tool that integrates with Git but offloads the storage of large files. Data Version Control (DVC) is a prime example of this philosophy.
The core idea is simple yet powerful:
- Your code lives in Git, as usual.
- Your large data files live in remote storage (like AWS S3, Google Cloud Storage, or even a local network drive).
- DVC creates small metadata files (pointers) that contain information (like an MD5 hash) about the large data files. These tiny pointer files are checked into Git.
This way, Git does what it does best (versioning code and small text files), while your large data is managed separately. A git checkout of a specific branch or commit gives you the code and the pointers to the correct version of your data, which you can then retrieve using a DVC command.
This video from a Databricks presentation provides an excellent walkthrough of the fundamental DVC workflow.
Data Versioning and Reproducible ML with DVC and MLflow
This demonstration shows the core commands and concepts of DVC in action.
Watch the demo segment from "how it works". Pay close attention to the sequence of commands: dvc add: Tells DVC to start tracking a data file. This creates the .dvc pointer file. git commit: Commits the small .dvc pointer file, not the large data file. dvc push: Pushes the large data file to the configured remote storage. dvc pull: Retrieves the data version corresponding to the current pointer file.
DVC is a powerful and popular tool, but it's not the only one. Other tools tackle this problem with different approaches, such as providing Git-like semantics directly on top of databases or data lakes.
Designing a Schema Versioning Strategy
Just as data changes, so does its structure. The schema you defined for data validation in the previous lesson will inevitably evolve as business requirements change, new features are added, or data sources are updated. Managing this evolution is critical to prevent downstream applications from breaking.
This video provides a concise overview of schema evolution, its causes, and best practices for managing it.
How to Handle Schema Evolution: Best Practices for Adapting Your Database!
This video covers the fundamentals of managing schema changes in a production environment.
Please watch the first two parts of the video: The introduction, which explains what schema evolution is and why it happens. The section on best practices for handling it, such as backups, versioning, and using staging environments.
As the video highlights, a key practice is schema versioning. Instead of overwriting a schema, you create new versions (e.g., customer_v1, customer_v2). This allows different applications to migrate to the new schema at their own pace.
A common pattern for managing this, especially in streaming systems, involves a Schema Registry.
- Producers of data tag their messages with a schema version ID.
- Before sending data, they register the schema with a central registry (like Confluent Schema Registry). The registry checks if the new schema is compatible with previous versions (e.g., backward or forward compatible).
- Consumers can then fetch the correct schema version from the registry to correctly deserialize the data.
Tools like Apache Avro and Protobuf are designed with this in mind, as they embed schema information within their data format, making them ideal for systems that require robust schema evolution management.
Tracking Lineage: From Raw Data to Final Prediction
Versioning gives you snapshots in time. Lineage connects those snapshots into a coherent story. It answers the question: "To produce this specific model, what version of the raw data was used, what version of the feature engineering code was run, and with what hyperparameters?"
There are several ways to track lineage, ranging from manual to fully automated.
Versioning, Provenance, and Reproducibility
This section details the common strategies for capturing lineage information within an ML pipeline.
Please read the section on provenance strategies. Note the progression from manual, error-prone methods to automated, framework-based approaches.
The most robust approaches use pipeline frameworks that capture lineage automatically. DVC, which we saw earlier, does this by defining pipelines as a Directed Acyclic Graph (DAG). In a dvc.yaml file, each stage declares its dependencies (deps) and its outputs (outs). DVC uses this information to build a complete graph of your workflow, automatically creating the data lineage.
The following resource provides a detailed, practical example of a multi-stage dvc.yaml file. You don't need to absorb every line of code, but scanning it will give you a concrete understanding of how a pipeline's lineage is declared as code.
How to Build End-to-End Machine Learning Lineage
This tutorial walks through building an entire ML pipeline with DVC, demonstrating how lineage is captured at each step.
Please scan through "Step 2: The ML Lineage". For each of the six stages (ETL, data drift, preprocess, etc.), observe the structure of the dvc.yaml configuration. Notice how the deps of one stage often include the outs of a previous stage, explicitly linking them together to form the lineage graph.
Putting It All Together: A Fully Reproducible Experiment
The ultimate goal is to tie everything together. When you run an experiment, you need to log not only the results (e.g., accuracy) and hyperparameters but also the exact version of the data that was used.
This is where DVC and an experiment tracking tool like MLflow work in concert.
- DVC versions the data.
- MLflow tracks the experiment, and you log the DVC data version as one of the experiment's parameters.
The final segment of the Databricks video demonstrates exactly how this is done. A Python script uses the DVC API to get the URL of a specific data version (identified by a Git tag) and passes that URL to MLflow to be logged with the experiment run.
Data Versioning and Reproducible ML with DVC and MLflow
This final part of the demo shows how to connect data versions to experiment runs, achieving full reproducibility.
Watch the segment from "access an exact version". Focus on how the script uses dvc.api.get_url() to retrieve a data path and mlflow.log_param() to record that path, linking the model's metrics directly to the data it was trained on.
This combination of tools gives you a complete, auditable record. Given a model in MLflow, you can find the exact data version pointer, use git and dvc to retrieve that data, and perfectly reproduce the training run.
Conclusion
In this lesson, we established a complete framework for versioning and lineage in ML systems. You are now equipped to design systems where every artifact and every result is traceable and reproducible.
Key Takeaways:
- Versioning is essential for reproducibility, debugging, and auditing. You must version code, data, schemas, and models.
- Data versioning requires specialized tools like DVC, Dolt, or LakeFS that can handle large files, as Git alone is insufficient. DVC's approach is to store metadata in Git and large files in remote storage.
- Schema evolution is inevitable and must be managed through versioning (e.g.,
schema_v1,schema_v2), compatibility rules, and tools like Schema Registries, often used with Avro or Protobuf. - Data lineage tracks the entire lifecycle of data and models. It can be captured automatically by pipeline frameworks like DVC, which define the workflow as a DAG of dependencies and outputs.
- True reproducibility is achieved by combining data versioning (DVC) with experiment tracking (MLflow) to create an unbreakable link between a trained model and the exact data used to create it.
In our next lesson, we will continue building out the operational components of our ML system. Having established how to version and track our pipeline artifacts, we will focus on how to design a logging and event collection architecture to monitor what happens when these pipelines run in a live production environment.