Defining Pilot Acceptance Criteria

Hello again. The previous lesson recorded a proposed choice for video ingestion and made clear that accepting an architectural direction is not the same as accepting a pilot. Now the question changes: what evidence would show that the delivered system works well enough for its agreed purpose?

This lesson turns a customer goal into pilot acceptance criteria. You will define a baseline, target metrics, test conditions, and exclusions so that your team and the customer can interpret the same results in the same way. For an edge-AI pilot, that means judging both the customer’s workflow and the behavior of the complete system—not just whether a model runs on a device.

Agree on success before observing results

“Show that the system detects people reliably” sounds like a goal, but it does not say how reliable, under which conditions, or who decides. If those details are settled after a demonstration, each side can reasonably interpret the result differently.

Watch Dreamit’s 5 Rules for Successful Startup Pilots, POCs, Trials, concentrating on its advice about jointly approved, numerical benchmarks and a fixed evaluation period. You need not adopt every commercial definition in the video; the useful principle here is to agree on what success means before the pilot begins.

A practical agreement names one primary outcome and a small number of supporting conditions. The primary outcome answers, “Did this help with the customer’s job?” Supporting conditions answer, “Did it do so with acceptable errors and operational behavior?” A fast detector that produces constant false alerts may worsen the job it was meant to improve.

A pilot criterion is stronger than a feature test. A feature test might say, “Given a camera stream, when a person crosses the line, an alert appears.” Pilot acceptance asks how often that happens, how often alerts are wrong, whether the result holds over representative operating hours, and whether staff benefit. Individual test cases help produce evidence; they do not replace the agreement about how to judge the evidence.

Build each criterion from a baseline and a decision rule

Read the opening objective-and-hypothesis material and Sections 2. Scope & Boundaries and 3. Success metrics & SLA (Acceptance Criteria Template) of Pilot Plan & Acceptance Criteria Template from The Hunger Engine. Then look at Section 9. Rollback & stop criteria. Focus on the fields that make a result interpretable, rather than adopting the template’s example metrics for a media system.

The baseline is the comparison point, usually measured from the current workflow under comparable conditions. The target is the result required of the pilot. If the goal is to reduce time spent reviewing video, “review time fell” is insufficient: record how long review took before the pilot, what staff activity counts as review, and the improvement required.

For each metric, specify enough detail that someone outside the implementation team could reproduce the calculation:

1. What is counted? Define the event and the numerator and denominator, where applicable.
2. Where and when? Specify the devices, inputs, operating conditions, observation window, and minimum usable sample.
3. How is it measured? Name the log, annotation, observation sheet, or other evidence—and who checks it.
4. What is the rule? State the threshold and whether every supporting condition must also pass.

A baseline need not always be a number. An alert-latency metric may have no equivalent in a manual workflow. Mark its baseline not applicable, explain why, and agree on a target based on the user’s need and a preliminary measurement. Do not invent a “before” latency to make the table look complete. Conversely, if you claim that staff time will improve, measure staff time before locking the comparison.

Apply the rule to a complete edge-AI pilot

Consider an illustrative pilot: one fixed camera observes a doorway, and an edge device sends a countable alert when a person crosses a marked line in one specified direction. Staff currently review footage to produce a count. The customer’s proposed benefit is less review work without missing too many crossings.

First fix the event definition. A reference crossing occurs when a person’s torso centre passes the agreed line in the specified direction. Reviewers independently mark the time of each visible crossing in authorised recordings. Match at most one alert to each reference crossing, within two seconds of its marked time; extra alerts remain false alerts. Without this rule, a system could emit repeated alerts and still appear to have detected every person.

Next specify the test setting: one agreed camera position and line, the selected device and software versions, ten four-hour shifts covering both busy and quiet periods, and at least 200 independently labelled reference crossings. Normal variation in lighting and partial occlusion is included. These quantities are illustrative planning thresholds, not claims that this is a statistically sufficient sample for every deployment.

The following is an example acceptance sheet. Every number must be checked with the customer against the real workflow before use.

Recall is the share of reference events that received a matching alert:

The false-alert rate divides unmatched alerts by the forty scheduled camera-hours, not merely by the hours when the system happened to stay healthy. Otherwise, an unreliable system could improve its reported rate by going offline. Outages must also be reported directly; if the agreed observation or labelling minimum cannot be met, the team must not declare a pass from a smaller convenient subset. Agree in advance whether such an interruption causes a restart, an inconclusive result, or a failure.

The latency measure is an end-to-end measure: time from the crossing frame to the emitted alert, not just model-inference time. A model benchmark alone cannot establish this criterion because capture, buffering, tracking, and alert delivery also take time. Similarly, the staff-time measure needs observation of the actual workflow; device logs cannot tell you how long a person spent resolving ambiguous alerts.

Why AI criteria need distributions, not perfect-case demos

An AI system will not classify every real-world input correctly. Acceptance can still be a clear pass-or-fail decision, but that decision is often based on a measured rate or percentile rather than a promise that every prediction is correct.

Read Section 2.2, “Acceptance Criteria for AI-Based Systems,” and the beginning of Section 2.2.1 on pages 24–25 of the ISTQB Certified Tester AI-Testing Syllabus. Focus on its distinction between functional and non-functional characteristics and on how threshold-based examples differ from “the model works.”

For the doorway pilot, recall describes detection quality; false-alert rate describes the burden imposed by errors; latency describes a system property; and staff time tests whether those properties add up to customer value. A result just above 90% recall is still an estimate from the observed sample, not a guarantee of 90% on every future day. Record the event count and the conditions represented alongside the percentage.

State exclusions without hiding ordinary failures

Exclusions define what the pilot does not claim to prove. For this example, reasonable exclusions might be other doorways, a moved camera, identity recognition, audio analysis, and conditions in which the doorway is completely unobservable. They should be agreed before data collection. Partial occlusion, normal lighting changes, and ordinary busy periods remain in scope; excluding them after they cause poor results would make the evaluation misleading.

Separate exclusions from constraints. “Raw video must remain on the edge device during this pilot” is not an inconvenient case to exclude; it is a condition of operating the pilot. The team might verify the approved configuration and inspect recorded network activity for unauthorised raw-video transfers. A detected violation triggers a pause and investigation even if the recall and staff-time targets look good. The exact media-handling rule and verification method require agreement with the customer before deployment.

Finally, write down what happens when evidence is incomplete. An external camera failure may prevent a valid measurement; a failure caused by your own ingestion component is evidence about your system, not an excuse to discard a difficult shift. Agree on attribution, documentation, and any restart rule beforehand. Keep raw observations and annotate changes to the camera, model, software, or test conditions; otherwise two weeks of numbers may not describe the same system.

The team responsible for measurement should review this sheet with the customer representative who can accept the pilot. Sign-off before testing does not force anyone to approve a poor result. It makes the later decision—pass, fail, or inconclusive—defensible.

Key takeaways

A pilot acceptance criterion connects a baseline to a target, then fixes the measurement method, window, conditions, and pass rule. One primary customer outcome keeps the pilot focused; supporting measures prevent a superficially good outcome from concealing missed events, false alerts, or excessive latency. Exclusions limit what the result claims to cover, but must not quietly remove ordinary operating conditions or system failures.

In the next lesson, you will create a technical risk register. The unresolved threats to these criteria—such as an unreliable camera stream, insufficient labelled events, or a media-handling violation—will become risks with owners, mitigations, and observable triggers.