Recommended resources
The freely available resources this lesson will direct you to.
Good to see you again. Last lesson, you framed Maya’s problem as preparing a post-meeting follow-up that preserves the difference between confirmed and conditional information. You also kept the proposed AI role bounded: it can help draft, but Maya checks the notes and decides what is ready to send.
Today you’ll define what success would look like for that experience. The aim is not to collect every possible product metric. It is to choose a small set of observable measures that could tell you whether the design helps Maya complete her task without turning uncertainty into a confident-sounding mistake. Allow about 35–40 minutes, including the video and readings.
Measure the user’s outcome, not the amount of AI
Suppose a prototype generates a follow-up in three seconds and Maya accepts it. That sounds efficient, but it does not answer the important question: did the update accurately represent the meeting? A polished sentence such as “Mina will send a draft Friday” would be a poor outcome if the notes say, “Mina might send a draft Friday if legal approves it.”
A success criterion needs three parts: an observable outcome, a way to measure it, and a decision rule. For example, “Maya can prepare an accurate follow-up” is a goal. “In a test, the completed draft retains the approval condition on Mina’s possible Friday draft” is something you can observe. A decision rule specifies what result would prompt you to proceed, investigate, or redesign.
Watch these portions of Eleken’s 6 Key UX Metrics Explained. They introduce task success, partial success, time, and perceived ease—useful measures, provided you define what success means for this particular task.
Watch Eleken’s “6 Key UX Metrics Explained” to see how common usability measures are collected during a task.
Watch task success, focusing on why partial outcomes deserve their own description rather than an arbitrary fractional score. Then watch task time and perceived ease. Consider what each measure would reveal—and what it could miss—if Maya produced a fast but inaccurate follow-up.
For Maya, “completed a draft” is too weak a definition of task success. Before testing, specify a short scoring rule: the draft should distinguish confirmed items from conditional ones, preserve relevant conditions, avoid adding unsupported commitments, and be usable as a follow-up after Maya’s review. A draft that looks finished but states an unapproved commitment is not a complete success.
NNgroup’s 7 Steps to Benchmark Your Product’s UX provides a way to narrow the measurement plan. Read its guidance on choosing a task, selecting a few complementary metrics, and establishing a baseline.
7 Steps to Benchmark Your Product's UX
Read this Nielsen Norman Group article for a practical way to choose measures tied to a specific user task rather than starting with a long list of available numbers.
In “Step 1: Choose What to Measure,” read the “Tasks” and “Metrics” subsections, beginning with choosing metrics; continue through the advice to use a small number of measures. In “Step 2: Decide How to Measure,” read measurement methods. Finally, in “Step 3: Collect First Measurement: Establish Baseline,” read from the pilot-study advice. Focus on why a time or success-rate target needs a clearly defined task and a comparable first measurement.
The article’s HEART categories are a menu, not a checklist. Task effectiveness and efficiency are immediately relevant here. Adoption or retention might matter after launch, but neither can tell you whether an individual follow-up misstates a commitment. A satisfaction rating can add context, but someone might find a draft easy to use while overlooking an error.
Make the cost of a mistake visible
AI introduces a measurement problem that a simple completion rate can conceal. If Maya corrects an unsafe AI draft before marking it ready, the final outcome may be sound—but the system still created review work and exposed her to a plausible false claim. Conversely, counting only errors in the initial AI output would ignore whether the interface helped her catch and correct them. Keep the initial suggestion, Maya’s review behavior, and the reviewed result distinct.
Trust is part of this distinction. The aim is not to maximize trust; it is to help users expect what the feature can actually do.
For the course scenario, you know what the supplied notes say, but you do not know whether legal approval happened elsewhere. That makes “Mina will send a draft Friday” unsupported by the supplied notes. A good draft could keep the condition, mark the item for checking, or leave it out pending confirmation. Your criterion should not require the AI to discover a fact it was never given.
The People + AI Guidebook’s discussion of negative impact is helpful here: a measure becomes more useful when a concerning result has a planned response.
User Needs + Defining Success - People + AI Research
Read this People + AI Guidebook passage for its approach to monitoring harmful outcomes and linking a threshold to an action.
In “Account for negative impact,” read negative-impact guidance. Continue to “Key concept” and its template ending with an action. Focus on the distinction between noticing a bad result and deciding what the team will do about it.
In Maya’s case, an unsupported commitment is more consequential than an awkward sentence. It warrants its own guardrail, rather than being averaged into a general quality score. A useful question is: If this number worsens, what design decision would we revisit? If the answer is unclear, the metric may be too vague to guide the project.
A compact success-criteria set for Maya
Here is a provisional plan, not a claim about how real coordinators perform. Imagine testing a Figma prototype with five people doing Maya’s task, using prepared meeting notes that contain at least one clearly conditional item. These four measures cover the task result, a serious failure, user understanding, and effort.
| Criterion | Exactly what to record | Provisional pilot decision rule |
|---|---|---|
| Accurate task completion | Number of reviewed follow-ups that meet the predefined scoring rule, out of all task attempts. Record partial outcomes separately. | Aim for at least 4 of 5 completed attempts; inspect every miss. |
| No unsupported firm commitments | Number of reviewed follow-ups that state a conditional item as confirmed, out of all reviewed follow-ups. Also log whether the error first appeared in the AI suggestion. | Treat any occurrence as a reason to investigate and revise before proceeding. |
| Recognition of uncertainty | Number of participants who, before finishing, identify the conditional item and choose to preserve, check, or omit its uncertain status, out of all participants exposed to it. | Aim for at least 4 of 5; examine where anyone missed it. |
| Time to a reviewed draft | Time from receiving the notes to marking a reviewed draft ready, with unfinished attempts reported separately. Compare with a comparable manual task. | Aim for no slower than the manual baseline, without weakening the first three criteria. |
The thresholds are design hypotheses for an early pilot, not established industry standards. Five participants can reveal important breakdowns, but “zero errors in five attempts” does not prove the feature is safe, and a difference of one participant changes a five-person percentage substantially. Report the counts and observed behaviors; do not present them as reliable population rates.
Notice how the criteria work together. Accurate completion asks whether Maya achieves her goal. The guardrail isolates the mistake that could most damage that goal. Recognition checks whether the experience supports Maya’s judgment rather than merely producing a good-looking answer. Time tests the productivity benefit, but speed cannot compensate for an inaccurate update.
A tempting alternative would be “percentage of AI suggestions accepted.” Acceptance alone is ambiguous: Maya may accept because the suggestion is useful, or because it sounds authoritative. It is better treated as a clue to investigate alongside accuracy and review behavior, not as the definition of success.
Turn the criteria into a credible measurement plan
To measure these criteria later, prepare the test materials before observing participants. Write the source notes and an answer guide that identifies what is confirmed, what remains conditional, and what cannot be determined from the notes. Define how to score an omission or a corrected AI error. Otherwise, the meaning of “success” may shift after you see the results.
During a prototype test, ask participants to prepare the follow-up as they normally would. Observe their edits and decisions without prompting them to look for Mina’s condition. When they mark a draft ready, score the reviewed draft against the answer guide; separately log errors in the initial AI suggestion and whether they were caught. Ask a short follow-up question if a choice is unclear. Observation tells you what happened; the conversation can help explain why.
For the time comparison, first measure a comparable manual version of the task. Use note sets of similar difficulty and vary which version participants encounter first, so familiarity with the notes does not masquerade as a design improvement. Record unfinished tasks rather than making a fast failure look like efficient work. If the first pilot shows that the notes, scoring rule, or timing method are confusing, fix the method before treating its numbers as a benchmark.
Finally, connect results to decisions. If an unsupported commitment survives review, inspect both the generated wording and the cues available to Maya. If drafts are accurate but review takes longer than doing the task manually, investigate where the extra effort occurs. If people finish quickly yet fail to notice conditional language, do not declare victory on efficiency: the success definition needs all four measures to be read together.
Key takeaways
A measurable UX success criterion names what users must accomplish, how you will observe it, and what result calls for action. For Maya, a small balanced set is more informative than a single “AI accuracy” or “time saved” number: measure the reviewed task outcome, guard against unsupported commitments, observe recognition of uncertainty, and compare effort against a baseline. Keep pilot targets provisional and pair the numbers with the behavior behind them.
You now have a problem statement and a way to judge whether an AI-assisted experience addresses it. In the next module, you’ll begin choosing interaction patterns that give Maya useful assistance while making review and correction possible.