Automated Evaluation Gates in CI/CD

Welcome back to your journey into LLM Operations. In the previous lesson, we built a Python script from the ground up to serve as our evaluation suite. That script is capable of capturing crucial quality and performance metrics for any given LLM, forming the analytical core of our MLOps pipeline.

Today, we transition from scripting to automation. Your goal is to integrate the evaluation script into a CI/CD workflow as an automated pass/fail gate. Drawing on your extensive background in infrastructure automation and SRE, you will configure a CI/CD pipeline that automatically runs these evaluations and makes a critical decision: should this model change proceed toward production, or does it represent a regression that must be blocked? This lesson brings us one step closer to a fully automated, end-to-end deployment pipeline for LLMs.

1. The Concept of an Automated Evaluation Gate

In traditional software engineering, CI/CD pipelines use unit tests and integration tests as quality gates. For LLMs, we need a similar, but more specialized, mechanism. An LLM evaluation gate is a stage in your CI/CD pipeline that:

1. Triggers automatically, typically when a change is proposed (e.g., a new prompt, a fine-tuned model, or a change in the RAG pipeline).
2. Executes the evaluation suite against a "golden dataset" of test cases.
3. Analyzes the results, comparing key metrics (like accuracy, faithfulness, or latency) against predefined thresholds.
4. Decides to either PASS the pipeline, allowing the change to be merged or deployed, or FAIL the pipeline, blocking the change and alerting the team.

This automated process is the primary defense against deploying a model that is less accurate, more prone to hallucination, or slower than its predecessor.

2. Choosing the Right Tools for CI/CD

In the last lesson, we used DeepEval within a custom Python script to understand the fundamentals of metric calculation. While powerful, integrating custom scripts into CI/CD requires boilerplate code for parsing outputs and handling exit codes.

For production CI/CD workflows, it's often more efficient to use a tool designed for this purpose. Promptfoo is an excellent open-source framework that excels at CI/CD integration. It uses a simple YAML configuration, has a robust command-line interface (CLI), and can easily be configured to fail a build if quality thresholds are not met.

Let's get a feel for how promptfoo works.

The key takeaway is that promptfoo lets us define our entire evaluation suite—prompts, test cases, and pass/fail criteria—in a single, version-controllable YAML file.

3. Setting Up the Jenkins CI/CD Environment

Given your background with enterprise systems, we'll use Jenkins, a stalwart of CI/CD automation, to build our pipeline. Your experience with Docker will make the setup straightforward. We will run Jenkins inside a Docker container, giving us a clean, isolated environment.

After completing these steps, you should have a Jenkins server running and a pipeline job ready to be configured.

4. Implementing the Evaluation Gate in a Jenkinsfile

Now we will write the Jenkinsfile. This file defines the stages of our CI/CD pipeline using Groovy syntax. We will create a stage specifically for evaluation that runs promptfoo and acts as our quality gate.

The promptfoo documentation provides an excellent template for this.

Here is a complete Jenkinsfile that you can place in the root of your project's Git repository. It defines a multi-stage pipeline that checks out your code, runs the promptfoo evaluation, and implements the pass/fail gate.

Jenkinsfile

pipeline {
    agent {
        // We need a Docker agent that has Node.js installed to run promptfoo
        docker {
            image 'node:20'
        }
    }

    environment {
        // Make your LLM API key available to the pipeline
        // Assumes you've stored it as a "Secret text" credential in Jenkins with the ID 'openai-api-key'
        OPENAI_API_KEY = credentials('openai-api-key')
        // Define a cache path to speed up subsequent runs
        PROMPTFOO_CACHE_PATH = "${WORKSPACE}/.cache/promptfoo"
    }

    stages {
        stage('Checkout') {
            steps {
                // Clones the repository code into the workspace
                checkout scm
            }
        }

        stage('Run LLM Evaluation') {
            steps {
                sh '''



                    # Install promptfoo and run the evaluation
                    # The -o flag outputs the results to a JSON file
                    # This assumes you have a promptfooconfig.yaml in your repo
                    npx promptfoo@latest eval \
                        -c promptfooconfig.yaml \
                        -o results.json
                '''
            }
        }

        stage('Quality Gate') {
            steps {
                script {
                    // Read the JSON file produced by promptfoo
                    def results = readJSON file: 'results.json'
                    // Extract the number of failures
                    def failures = results.results.stats.failures

                    echo "Evaluation complete. Successes: ${results.results.stats.successes}, Failures: ${failures}"

                    // This is the gate: if there are any failures, fail the pipeline
                    if (failures > 0) {
                        error("Evaluation FAILED with ${failures} failures. Blocking merge/deployment.")
                    } else {
                        echo "Evaluation PASSED. All quality checks met."
                    }
                }
            }
        }
    }
    
    post {
        // This block always runs, regardless of pipeline status
        always {
            // Archive the results so they can be reviewed from the Jenkins UI
            archiveArtifacts artifacts: 'results.json', allowEmptyArchive: true
        }
    }
}

How it works:

1. Agent: The agent { docker { image 'node:20' } } block tells Jenkins to run the pipeline inside a Docker container using the official Node.js 20 image. This provides the npx command needed for promptfoo.
2. Environment: We securely load the OPENAI_API_KEY from Jenkins's built-in credential manager. You'll need to add your API key there.
3. Run LLM Evaluation: The sh step executes a shell command to run promptfoo. It points to your promptfooconfig.yaml and saves the output to results.json.
4. Quality Gate: This is the heart of our automation.
   • The script block allows for more complex Groovy code.
   • readJSON is a Jenkins pipeline utility that parses the results.json file into a Groovy object.
   • We extract the failures count from the stats.
   • The if (failures > 0) condition checks if any test assertion failed. If so, the error() function immediately stops the pipeline and marks it as FAILED.
5. Post-build Action: The archiveArtifacts step ensures that the results.json file is saved with the build, allowing you to inspect the detailed results later.

To make this work, commit this Jenkinsfile along with a promptfooconfig.yaml to your repository. When you run the pipeline in Jenkins, it will automatically execute these stages, providing a clear pass or fail signal based on your model's performance.

Conclusion

In this lesson, you have successfully bridged the gap between evaluation scripting and MLOps automation. By integrating an evaluation suite into a Jenkins pipeline, you've built an essential component of a reliable release process for LLMs. This automated gate acts as a vigilant guardian, ensuring that only high-quality, performant model changes can proceed.

Key Takeaways:

• An automated evaluation gate is a critical CI/CD stage that prevents regressions in LLM quality and performance.
• Tools like Promptfoo are designed for CI/CD, simplifying the definition of tests and pass/fail criteria through YAML configuration.
• A Jenkinsfile allows you to orchestrate the entire evaluation process, from checking out code to running tests and making a pass/fail decision.
• The core logic of the gate involves running the evaluation, parsing the structured output (e.g., JSON), and using a conditional statement to fail the pipeline if metrics fall below a set threshold.

In our next lesson, we will address the question of what happens after a model passes this CI gate. We will move into the realm of Continuous Deployment by learning how to implement canary deployments for new model versions on Kubernetes with traffic splitting, a crucial technique for safely releasing new models to production.