Defining Measurable Automation Success Criteria

Hello! In the previous lesson, you scored routine request routing as a candidate for a supervised pilot. The estimated benefit was six gross staff-hours per month, but that number depended on two untested assumptions: that 90 requests would be eligible and that each would save four minutes.

This lesson turns that estimate into a measurable success criterion. You’ll define what would count as a successful pilot, how you would measure it, and what would stop you from calling it successful even if it saved time. That skill will help you judge automation projects by observed results rather than by whether a workflow merely runs.

A working automation is not necessarily a successful one

“Build a workflow that routes requests” describes a deliverable. “Reduce the time people spend routing requests without increasing wrong assignments” describes the result the deliverable is meant to achieve. The workflow could run exactly as designed yet save little time because staff must correct its assignments. Conversely, it could save time by routing everything immediately while quietly sending difficult requests to the wrong team.

Watch these parts of How to Set SMART Goals and Get Results from Grow with Google. The distinction between launching a project and checking its results after launch is especially useful for automation.

For our request-routing pilot, “the automation is live” is a launch check. It does not answer whether people spend less time handling eligible requests or whether requests reach the right destination. A success criterion needs an observable outcome and a rule for deciding whether the observed result is good enough.

Choose a measure that matches the reason for automating

Start with the problem, not with whatever the software makes easiest to count. If the problem is repetitive manual effort, measure human handling time. If the problem is slow service, measure elapsed time from submission to assignment or response. Those are different quantities: an automatic route could reduce elapsed time dramatically while leaving staff with substantial checking work.

Microsoft’s Measure and communicate the business value of Power Platform distinguishes measurable outcomes from less easily quantified benefits and suggests practical process measures. The examples apply beyond any particular platform.

For the request-routing pilot, average human handling minutes per eligible request is a sensible primary measure because reducing that effort was the reason the opportunity scored well. A wrong-route rate is a companion measure: time savings are not valuable if they come from skipping necessary judgment. You need not create a large dashboard for a small pilot. One primary outcome and a few checks against harmful shortcuts are often more informative than a dozen loosely related numbers.

Benefits that are harder to measure can still matter. Staff might find a recorded review queue easier to use than an email inbox. You can gather feedback about that experience, but do not convert it into an invented financial saving. Keep the claim proportional to the evidence.

Write the criterion before running the pilot

A measurable criterion specifies what is counted, the target, the comparison, the observation period, and the data source. The target should use the same unit as the measure. Otherwise, “improve routing by 30%” could mean less staff time, faster assignment, or fewer mistakes.

How to Develop Key Performance Indicators from OnStrategy gives a compact check for a usable measure: define it clearly, set a numeric target, identify its data source, and decide how often to review it.

Here is a proposed criterion, using the previous lesson’s request-routing figures. The six-minute manual handling time was provisional; before treating it as a baseline, suppose you time a sample of 30 eligible requests handled manually and confirm an average of six minutes. The pilot target below is a decision threshold, not a result already achieved:

During a four-week supervised pilot with at least 60 eligible requests, reduce average human handling time per eligible request from a confirmed pre-pilot baseline of 6 minutes to no more than 3 minutes, including checking and correction time. Calculate the average from a handling-time record for every eligible pilot request.

The wording does several jobs. “Eligible” limits the comparison to requests that have the required information and a sufficiently clear topic for the routine rules. “Human handling time” includes work after the automation runs; it is not the workflow’s execution time. “Every eligible pilot request” prevents selecting only the quickest cases. The four-week period and minimum count prevent a result from being declared on the strength of a few convenient examples.

The earlier forecast assumed 2 minutes of remaining human work. A target of no more than 3 minutes is less ambitious than that forecast: at 90 eligible requests in a month, it would correspond to at least 4.5 gross staff-hours saved, rather than the forecast six. This distinction is deliberate. A forecast estimates a possible benefit; a success criterion states the result you agree is sufficient to continue. Both figures remain provisional until measured.

Protect the result with quality checks

A speed criterion alone can reward bad routing. Before the pilot, define a wrong route as an eligible request initially sent to a destination that does not match the agreed routing rules. Then calculate:

For this illustrative pilot, you might require a wrong-route rate of no more than 2%, checked by reviewing every eligible pilot assignment. Record who conducts the review and how disagreements about the correct destination are resolved. If the rules themselves are disputed, an apparently precise error rate will not be trustworthy.

Also reconcile all submissions, not only eligible ones, against the workflow’s recorded outcomes. Every submission should have an assignment, a follow-up status, or an intake-review status. A request with missing information or an unclear topic should not disappear simply because it was excluded from the time-saving calculation. This is a separate check against silent failures, not a claim that exception handling has been automated away.

Together, these statements give the pilot a usable decision rule:

• Primary result: average human handling time is no more than 3 minutes per eligible request, against the confirmed 6-minute baseline.
• Quality condition: wrong-route rate is no more than 2% of eligible pilot requests.
• Completeness condition: every submitted request has a recorded outcome, including those sent for human review.
• Evidence requirement: observe four weeks and at least 60 eligible requests; otherwise, extend the measurement rather than declaring a pass from too little data.

The thresholds are example choices to agree with the people responsible for the process, not universal standards. If a wrong assignment has more serious consequences than a delayed response, the quality condition would need to be stricter.

The value cards pictured below show how results can be presented as prominent numbers. But a headline amount and percentage change alone do not tell a reviewer which cases were counted, the comparison period, or how the amount was calculated. Preserve those details behind any summary of your own pilot.

Make the comparison credible

Measurement can fail even when the arithmetic is correct. If the pre-pilot sample contains mostly straightforward requests but the pilot includes many borderline topics, the two average times may not be comparable. Record the eligibility rule before the pilot, apply it consistently, and note meaningful changes in request volume or mix. Keep the baseline’s sample size and timing method alongside the result.

For each pilot case, a simple record can contain the submission ID, whether it was eligible, its final status, human handling minutes, initial destination, and reviewed correct destination. The ID lets you check for missing or repeated cases without relying on memory. A designated process owner can review the figures weekly and make the pass-or-fail assessment at the end of the observation period.

Finally, describe what the outcome does and does not establish. Meeting a three-minute target demonstrates reduced gross handling effort for the measured cases. It does not, by itself, prove a cash saving or a positive return after setup and maintenance effort. If the pilot misses the target, the records should help distinguish a flawed automation from an inaccurate original estimate—for example, checking might take longer than the two minutes forecast.

Key takeaways

A measurable success criterion is a pre-agreed test of the outcome, not a statement that the automation was built. For the request-routing pilot, that means a defined eligible population, a confirmed baseline, a handling-time target, a measurement window, and quality checks that prevent speed from hiding errors or lost requests.

Next, you’ll examine where human review or approval should remain in the workflow—a design choice that directly affects both the remaining handling time and the risks your success criterion must detect.