Defining Scope and Success Criteria for an AI Quality-Engineering Project

Hello. In the previous lesson, you shortlisted prior-work case studies that you could potentially discuss without exposing confidential details. This lesson takes a different route: you will define an independent, public AI quality-engineering project whose architecture, decisions, and results you can show directly in an interview.

By the end, you should have a problem statement, a firm boundary around what the project will and will not do, and success criteria that could reveal failure as readily as success. Allow about 40 minutes, including the short video and two readings.

Start with a decision worth improving

A portfolio project is more convincing when it answers an engineering question than when it merely connects several technologies. Here is a suitable question for your target roles:

Can an evidence-backed AI service select fewer automated tests after a code change while retaining the tests needed to detect likely regressions—and fall back safely when its recommendation cannot be trusted?

The setting is a fictional appointment-scheduling application built solely for this portfolio. Its small Java API and browser interface have a TypeScript Playwright suite. Changes to booking rules, availability, notifications, or authentication create different testing needs. Running every test after every change provides a simple, reliable reference point, but it may spend time on unrelated tests. Selecting tests by fixed path-to-test rules may be faster, yet can miss relationships not represented in those rules.

That tension justifies investigating an AI-assisted selector; it does not prove AI is necessary. A deterministic selector might meet the same goals with less cost and uncertainty. Your project should make that comparison visible rather than assume the AI approach wins.

This is specifically an AI quality-engineering problem because a plausible recommendation is not the same as a correct one. A selected test identifier must exist; the evidence offered for selecting it must relate to the change; and an omitted test may matter far more than an extra test. Unlike an ordinary unit test with one expected output, the recommendation also needs assessment across varied changes. Watch how the following demonstration distinguishes pass/fail assertions from measurements and traces across runs.

The project’s problem statement can now be specific:

For a fictional application with a bounded automated test suite, build and evaluate a service that recommends executable test IDs for a submitted change. Compare its selections with running the full suite and with a deterministic rules baseline. Preserve test coverage for designated high-risk changes, make each accepted recommendation inspectable, and use a defined fallback when the AI output is invalid or insufficiently supported.

Notice what this statement does not promise: faster real-world releases, fewer production defects, or superior AI performance. Those would require evidence beyond this portfolio’s synthetic setting.

Draw the boundary before choosing the architecture

“Medium-sized” should mean one complete, inspectable decision workflow, not a miniature enterprise platform. For each selection request, the proposed system receives changed paths, a short change description, and permitted component metadata. It consults a small corpus of original test descriptions and synthetic historical evidence. It returns a set of test IDs with supporting evidence and uncertainty information. A separate, deterministic control validates the result and decides what the worker may execute.

Your existing strengths in Java platform design, TypeScript test integration, and Python AI work make a three-part implementation useful if each part demonstrates a real boundary. The Java service can own requests and execution authority; the Python service can produce evidence-backed recommendations; and the TypeScript worker can execute known Playwright test IDs. This is a proposed scope, not a mandate to build every production-platform feature around those components.

This boundary keeps the most important architectural distinction visible: a model may recommend, but it does not have execution authority. If it returns an unknown ID, malformed output, or unsupported recommendation, the control layer must reject or safely replace that result according to a documented policy. For the first version, a full-suite fallback is easy to explain and conservative, though its cost must count against the project’s efficiency results.

NIST’s AI Risk Management Framework offers a useful discipline here. Its Map function asks you to specify intended use and context before measuring risk; its Measure function asks for methods and benchmarks tied to that context. The diagram places governance across the functions rather than treating risk review as a final inspection.

One practical consequence of that reading: label the project as a portfolio prototype evaluated on original, synthetic cases. Do not describe it as a validated selector for an organization’s production releases. A public repository gives you freedom to show the architecture, but it does not give your synthetic results external validity.

Define success so the system can lose

The unit of evaluation is one change request. For each request, a versioned dataset should record the eligible test IDs and a human-reviewed set of expected test IDs. Mark some expected IDs as must-run for especially consequential changes—for example, a change affecting double-booking prevention. Include ordinary changes, cross-component changes, ambiguous descriptions, unknown paths, and deliberately misleading repository text. Keep the examples and labels original, and record how you decided what belonged in the expected set.

Use three comparators:

1. Full suite: every eligible test runs. It has no selection reduction and serves as the coverage reference for this bounded suite.
2. Deterministic rules: a documented path/component-to-test mapping selects tests without an LLM.
3. AI-assisted selector: the proposed workflow produces a recommendation that passes—or fails—deterministic checks.

The rules baseline is essential. If it matches or beats the AI-assisted approach at comparable coverage, that is an architecturally valuable finding, not a failed portfolio.

For a dataset of change requests, let $E_i$ be the expected test set, $S_i$ the final set actually selected after validation and any fallback, and $T_i$ the eligible full-suite set for request $i$. Useful aggregate measures are:

Recall asks how many expected tests were retained. Precision asks how many selected tests were expected. Reduction measures fewer selected tests, not elapsed-time savings: tests have different runtimes, and execution overhead exists. Report actual runtime separately if you measure it. Exclude empty-denominator cases from a ratio only under a documented rule, and report how many there were.

Here is an illustrative result, not a claim about the future project. Across 40 change requests, suppose there are 1,200 eligible test opportunities, 200 expected-test opportunities, and 600 final selections. If 184 selections match expected tests, recall is 92%, precision is about 31%, and reduction is 50%. The headline reduction looks strong, but the low precision says substantial unnecessary work remains. A rules baseline retaining more expected tests with fewer selections would be preferable despite the AI system’s plausible explanations.

Before implementation, set provisional acceptance criteria such as these:

These numbers are design targets, not NIST requirements or measured achievements. Review them after an initial dataset and baseline run, documenting any change rather than quietly adjusting thresholds to fit a result. If a model-backed run is nondeterministic, record configuration and repeat runs; distinguish stable control-layer assertions from variable selection quality.

OpenAI’s evaluation guidance makes a complementary point: numerical scores are useful, but they need the right cases and human judgment. Read its evaluator and edge-case discussion with this change-to-test decision in mind, rather than treating its examples as a prescribed toolchain.

A labelled dataset is still an imperfect representation of what could break. The final report should show individual misses, disagreements about labels, and cases where the model abstained or fell back. “Passed the threshold” is narrower than “safe to trust for real releases.” That distinction is one of the strongest signals of quality-engineering judgment you can demonstrate.

The charter you will carry forward

The resulting charter is compact enough to guide later work:

Project: An original, public prototype for evidence-backed change-to-test selection in a fictional appointment-scheduling application.
Decision supported: Which known automated tests to run for a submitted change. The AI proposes; deterministic controls validate and authorize execution.
Comparison: Full suite and documented rules baseline on the same versioned, human-reviewed change cases.
Primary success conditions: Retain expected and must-run tests, reduce final selections without hiding fallbacks, reject unsafe outputs, and make each decision inspectable.
Limit: Results apply to the stated synthetic application and evaluation set; they do not establish production defect-risk reduction.

This is substantial enough to expose service boundaries, executable integration, evaluation design, and failure handling—useful evidence for both AI Quality Engineering Architect and Principal SDET interviews. It is also bounded enough to finish and explain. If time becomes constrained, preserve the baseline, evaluation dataset, control boundary, and end-to-end demonstration before adding platform features.

Key takeaways

Your portfolio project now has a testable question, not just a technology stack. Its boundary separates AI recommendation from execution authority and separates public, synthetic evidence from prior confidential work. Its success criteria consider coverage, reduction, baseline comparison, safe failure, and inspectability together; no single attractive score can stand in for the whole result.

Next, you will prepare the public repository that will hold this project, including checks designed to prevent accidental disclosure before any implementation is published.