Defining Measurable MVP Success Criteria

Welcome back. In the last lesson, you turned Harborline’s returns-guidance journey into user stories with acceptance criteria. Those criteria tell the team whether a particular capability works—for example, whether an agent can open a citation that supports a suggested claim. They do not, by themselves, tell Harborline whether the MVP is worth using.

Today we will define success at three levels: business value, the agent’s experience, and technical performance. You will see how to give each measure a population, a method, a baseline, a target, and a time window—without presenting an unverified estimate as a fact.

Start with the decision, not the dashboard

Imagine an MVP that produces well-cited answers in seconds, yet agents still spend just as long resolving returns cases. Its technical results may be encouraging, but its business case remains unproven. Conversely, an assistant that makes cases faster by confidently citing withdrawn guidance has not succeeded.

A useful scorecard therefore answers three different questions:

1. Business: Does the workflow produce a worthwhile improvement without damaging service quality?
2. User experience: Can agents complete the task effectively, and is the assistant practical to use?
3. Technical: Does the system produce supported, access-appropriate responses reliably enough for the pilot?

Bernard Marr’s short video makes the distinction between an AI capability and the organizational outcome it is meant to serve.

A metric name alone is not a success criterion. Consider this photographed service-metrics board:

“Time to completion” is a promising measure, but whose time, completing what, over which cases? For Harborline, every proposed criterion needs enough detail that two people could collect comparable evidence:

Metric contract: Define the task and eligible cases; the numerator and denominator or start and stop points; the data source; the baseline and target; the measurement window; and who must approve the interpretation.

We can draft that contract now. The numbers below are illustrative pilot targets for Harborline to negotiate, not claims about its current performance.

Define business success around the whole task

Harborline’s business question is not “Does the assistant answer quickly?” It is closer to: Does it reduce the effort needed to reach a defensible returns-guidance decision? Agents may search the shared drive, check a PDF’s status, contact a specialist, and revise an answer. Measuring only time inside the assistant would miss much of that work.

One candidate business criterion is:

During a four-week pilot, median agent active time per eligible returns-guidance case is at least 20% lower than the comparable pre-pilot baseline, with no increase in policy-related rework.

This needs an operational definition before it can be approved. “Agent active time” could begin when the agent starts investigating the case and end when they record either a supported response or an appropriate referral. It should include searching and checking sources outside the assistant. Waiting for a specialist’s reply may need a separate elapsed-time measure, so differences in specialist queues do not masquerade as agent effort saved. Harborline would need to decide which cases are eligible and compare similar case types and complexity across periods.

If the verified baseline median is $B$ minutes and the pilot median is $P$, the proposed improvement test is:

The earlier brief’s 12-minute estimate is not a verified baseline. If a study later established a 12-minute median under the agreed definition, this illustrative target would be 9.6 minutes or less. Until then, record the baseline as unknown, rather than reporting a reduction against 12 minutes.

The companion rework guardrail matters. A faster initial response is not a saving if agents or specialists must correct more cases afterward. Harborline could define rework as a case reopened or corrected because guidance was unsupported or wrong, then compare that rate across similar cases. A policy owner must help classify those incidents; raw reopen counts may include unrelated causes.

Finally, avoid converting every minute apparently saved into cash. Agent capacity, case volumes, adoption, specialist effort, tool costs, and rework all affect financial value. For this MVP, measured effort and quality can establish a credible business signal. A rollout cost-and-benefit estimate can follow when those inputs are known.

Define user success at the decision point

The agent’s task is not merely to receive an answer. In the stories from last lesson, success sometimes means finding approved, inspectable guidance; in other cases it means recognizing that available guidance is absent or conflicting and taking the agreed escalation route. A user-experience test must count appropriate escalation as success when that is the right outcome—not reward an assistant for answering every question.

Nielsen Norman Group’s benchmarking guidance is useful here because it begins with the user group and task, then chooses measures and gathers a baseline.

For Harborline, a task-success criterion might be:

In a moderated pilot test of 40 pre-agreed returns-case attempts, at least 85% end with the agent identifying applicable approved evidence or choosing the correct escalation path, as judged against a policy-owner-approved rubric.

The denominator is all 40 attempts, including abandoned or incorrectly completed tasks. Forty attempts and 85% are illustrative planning choices, not statistically conclusive proof of production performance. The cases should include straightforward guidance, missing support, conflicting documents, and access-restricted material. Report results by case type as well as overall: strong performance on easy lookups must not conceal failures on conflicts.

An accompanying efficiency measure could be median time from reading the case to identifying usable evidence or the appropriate escalation, compared with the same task measured in the current workflow. Record failures and timeouts separately; calculating time only for successful attempts can make a system that fails often look deceptively fast. A brief post-task ease rating could add the agent’s perspective, but an agent’s confidence is not proof that the cited policy is correct.

Notice the difference from a user story’s acceptance criterion. “The citation opens its supporting passage” tests one delivered behavior. “Agents successfully resolve or escalate at least 85% of representative tasks” tests whether the assembled MVP helps people perform the workflow.

Define technical success without rewarding unsafe shortcuts

The assistant has its own failure modes. It may retrieve the right-looking but withdrawn PDF, produce a claim its citation does not support, refuse cases it could answer, or reveal information from a restricted document. A single “answer accuracy” score would obscure these differences.

The following short segment introduces the distinction between a response’s faithfulness to retrieved material, its relevance to the question, and its correctness against an expected answer.

Microsoft’s guidance adds an important caution: an answer can be based on retrieved context yet still misinterpret it, and a model may respond differently to repeated prompts.

For an initial Harborline evaluation set, a compact technical scorecard might specify these provisional criteria:

• Supported policy claims: At least 95% of affirmative policy claims in evaluated answers are supported by their cited, accessible passages from documents whose approved status has been verified. Reviewers count claims, not merely answers with a citation link. A link that opens but does not support the claim fails.
• Useful answers and appropriate restraint: On answerable test cases, at least 85% receive a supported response that addresses the task. On cases designated by the policy owner as unanswerable or conflicting, at least 95% avoid a definitive unsupported policy answer and indicate the agreed next step. Report these separately: an assistant that refuses everything should not pass.
• Access-control release gate: In every pre-agreed restricted-document test, the unauthorized agent receives no restricted content, including through summaries or citation previews. Zero observed disclosures in a finite test set is a release requirement, not a guarantee that no vulnerability exists.
• Response time: At an agreed pilot load—for example, 10 concurrent users—the 95th-percentile time from submitting a question to receiving the complete response is no more than six seconds. Report errors and timeouts alongside latency; dropping slow requests must not improve the statistic artificially.

These are candidates, not universally appropriate thresholds. The team first needs a question set with policy-owner-approved expected evidence and correct escalation outcomes, plus users with specified document permissions. It should record the test-set mix and run conditions. When the documents or questions change, the results need revisiting. Later lessons will cover constructing that evaluation set and diagnosing retrieval and answer failures; here the objective is to state what evidence would count as MVP success.

Turn candidate targets into an agreed pilot decision

A practical success-criteria document should keep targets, unknowns, and release gates visibly separate. For Harborline, the 20% effort reduction, 85% task success, and answer-quality thresholds are targets to validate with stakeholders. The current active-time baseline, the definition of “approved,” the escalation route, and the document-permission model remain open. Unauthorized disclosure is different: it is a gate that should block release even if median handling time improves.

Before the pilot, agree on who records time, who judges task outcomes, who verifies policy support, and who reviews access-control results. Fix the measurement definitions before seeing the pilot results; otherwise, changing which cases count can turn a disappointing result into a misleading success. Review both aggregate figures and failures from the difficult branches of the journey.

Conclusion

MVP success needs more than a collection of attractive numbers. Business criteria test whether Harborline gains a worthwhile workflow improvement; user-experience criteria test whether agents can reach the right decision or escalation; technical criteria and release gates test whether the assistant’s behavior is supported, appropriately restricted, and dependable. Each needs a defined task or population, collection method, baseline where relevant, target, and time window.

These criteria are still a proposal until the people who own the workflow, policy, data, and security agree to them. In the next lesson, we will identify those stakeholders and specify what each must validate before Harborline can trust the MVP assessment.