Introduction
Welcome back! In our last lesson, you sketched the architectural blueprint for your MVP—a block diagram of the end-to-end data and model pipeline. This gave you a clear picture of how the components of your tool will connect and transform data from input to output.
This lesson addresses the final, crucial question before we start coding: "How will we know if it's successful?" Your learning outcome is to define measurable success criteria for your tool. These criteria act as your "definition of done" for the project. They transform a vague goal like "build a good flake detector" into a concrete target like "build a detector that is correct 80% of the time," providing a clear benchmark against which you can evaluate your work.
Why "It Works" Isn't Enough: From Function to Value
In your QA experience, you know that a test isn't just about whether the code runs without crashing; it's about verifying that the code produces the expected output for a given input. Similarly, the success of your ML-powered tool isn't just about it running without errors. It's about whether it provides real, measurable value to the user.
A tool can be technically impressive but practically useless. For instance, a flake detector that flags 50% of all tests as flaky might have high recall (it finds all the flaky tests), but the noise from false positives makes it unusable for a developer.
This is where we move from measuring simple accuracy to measuring things that build user trust. A trustworthy AI tool is reliable, predictable, and its value is clear.
Evaluation Metrics for AI Products That Drive Trust
To build a successful tool, we need to think beyond just traditional ML accuracy. This article from Product School provides an excellent framework for evaluating AI products on dimensions that build user trust.
Please read the section 'Model Evaluation Metrics Taxonomy for AI Products'. Pay attention to the different categories, from 'Accuracy and core performance metrics' to 'Human evaluation'. This will give you a broad vocabulary for defining success.
As the article highlights, metrics fall into several categories, including core performance, latency, fairness, and robustness. For your MVP, we will focus on defining a few core metrics that are tightly aligned with the primary goal of your tool.
Aligning Metrics with Your Project Goal
The success criteria you define must directly reflect the problem your MVP aims to solve. Let's consider how to frame this for each of your potential project directions.
If your project is Flake Detection...
The user's goal is to efficiently find and fix genuinely flaky tests without wasting time investigating false alarms. This immediately brings up a classic trade-off in machine learning classification.
- High Precision: When the tool flags a test as "flaky," it's very likely to be correct. This builds trust.
- High Recall: The tool finds most, if not all, of the flaky tests in the suite. This ensures comprehensiveness.
For a developer, time is a critical resource. A tool that "cries wolf" too often (low precision) will quickly be abandoned. Therefore, prioritizing precision is often the better starting point.
To go deeper on this, let's look at a research paper focused on this exact problem.
[PDF] An Evaluation of Machine Learning Methods for Predicting Flaky Tests
For a classification task like flake detection, understanding the trade-off between precision and recall is critical. This research paper evaluates ML models for this exact task and provides an excellent discussion on the relevant metrics.
Please read section '2.2. Performance Metrics and Parameters Tuning' and the short subsection near the end of the paper titled 'Precision Vs Recall'. Focus on how the authors describe the cost of false positives versus false negatives in the context of flaky tests.
As the paper argues, a false negative (missing a flaky test) is undesirable, but a false positive (flagging a stable test as flaky) is "obnoxious" because it wastes developer effort. This justifies prioritizing precision:
A secondary metric would be Recall, which measures how many of the actual flaky tests you found:
If your project is Test Case Generation...
The user's goal is to generate tests that are immediately useful. A "useful" test is one that is:
- Syntactically correct and executable.
- Logically sound (tests a valid aspect of the code).
- Ideally, adds new value (e.g., increases code coverage).
For an MVP, the most critical success criterion is that the generated code works out-of-the-box.
- Primary Metric: Execution Success Rate. What percentage of the generated test cases run without syntax errors or immediate runtime exceptions?
- Secondary Metric: Code Coverage Gain. For the tests that execute successfully, what is the average increase in line or branch coverage for the function under test?
If your project is Technical Debt Prioritization...
The user's goal is to get a ranked list of files or modules that helps them focus their refactoring efforts on the most problematic areas first. Perfect ranking is difficult, but an MVP should at least be able to elevate known "bad" files to the top.
- Primary Metric: Top-k Agreement. When run on a repository with a few files known to be complex or historically buggy, do these files appear in the top
k(e.g., top 5) of the ranked list? This is a practical, qualitative check that is easy to measure. - Secondary Metric: In later stages, you could explore more formal ranking metrics like Normalized Discounted Cumulative Gain (nDCG), which you saw in the syllabus. For an MVP, however, a simpler heuristic is sufficient.
Your Task: Define Your MVP's Success Criteria
Now it's time to formalize the "definition of done" for your project. Based on the MVP scope and pipeline you created in the previous lessons, define two or three measurable success criteria.
For each criterion, please provide:
- Metric: The name of the metric.
- Target: The specific, quantifiable value you are aiming for.
- Justification: A brief sentence explaining why this metric is important for your MVP and how it reflects value to the end-user.
Use the examples below as a guide.
Example for FlakeFinder:
- Criterion 1:
- Metric: Precision
- Target:
> 0.85 - Justification: To build developer trust, it is critical that when the tool identifies a test as flaky, it is correct at least 85% of the time, minimizing wasted investigation time on false positives.
- Criterion 2:
- Metric: Recall
- Target:
> 0.20 - Justification: While prioritizing precision, the tool should still aim to find at least 20% of all known flaky tests in a project, providing a useful starting point for developers.
Example for TestGenie:
- Criterion 1:
- Metric: Execution Success Rate
- Target:
> 0.90 - Justification: The generated test code must be syntactically correct and runnable 90% of the time to save the developer effort, not create more debugging work.
- Criterion 2:
- Metric: Non-zero Code Coverage
- Target:
> 0.75 - Justification: At least 75% of the successfully executed tests should cover at least one line of the target function, ensuring they are not just empty or trivial tests.
Example for DebtRanker:
- Criterion 1:
- Metric: Top-5 Agreement
- Target: On a sample project, at least 2 of the 3 files pre-identified as "high-debt" must appear in the top 5 of the tool's ranked list.
- Justification: The core value of the tool is its ability to surface the most critical items, so its top rankings must align with known problem areas.
- Criterion 2:
- Metric: Latency
- Target:
< 60 seconds - Justification: The analysis for a medium-sized repository should complete in under a minute to be practical for a developer to run locally.
Conclusion
Congratulations on completing the planning phase of your project! You have now defined not only what you're building (MVP scope), and how you'll build it (pipeline diagram), but also how you'll measure its success (your criteria). This complete blueprint will be your guide throughout the implementation process.
Key Takeaways:
- Success criteria must be measurable and aligned with user value.
- Metrics like precision and recall are essential for classification tasks (e.g., flake detection) and involve a trade-off that is a product decision.
- For generation tasks, execution success rate and code coverage are strong indicators of quality.
- For ranking tasks, a simple heuristic like Top-k Agreement can be a very effective MVP metric.
Preview of the Next Module:
With your plan finalized, it's time to start writing code. The next module, Implementation: Building the ML Pipeline, is where your ideas will turn into a working tool. Your very first lesson will be to create a script to fetch and version the primary raw data source for your chosen project—the first block in your pipeline diagram and the first practical step toward hitting the targets you've just set.