Recommended resources
The freely available resources this lesson will direct you to.
Welcome back. Last lesson, you classified specific productivity tasks by how well an LLM could assist with them. A meeting summary looked promising because someone could compare it with the notes; automatically assigning work without review was much less dependable.
Today the question is what to do when an AI result looks plausible but cannot be trusted. You’ll learn to distinguish what went wrong in the output from what may have caused it, then use that distinction to diagnose examples from meeting notes, policy questions, and workplace metrics. This is a useful skill when deciding whether an AI product needs better information, clearer instructions, a different tool, or a narrower job.
A convincing answer is not necessarily a reliable one
Suppose a meeting assistant writes: “Mina will send the client proposal on Friday.” That sentence is polished and actionable. But the notes say only, “Mina might send a draft Friday if legal approves it.” The assistant has turned a possibility into a commitment.
That is an observable symptom: the output contradicts or overstates its source. It does not yet tell us the cause. Perhaps the assistant never received the complete notes. Perhaps it received them but misinterpreted “might.” Perhaps its instructions encouraged it to produce definite action items even when the discussion was tentative. A good diagnosis keeps those explanations separate until there is evidence for one.
Watch IBM Technology’s Why Large Language Models Hallucinate for a vocabulary of output problems and an introduction to why they occur. You do not need the details of the generation algorithms; focus on the difference between an error you can observe and a cause you must investigate.
Why Large Language Models Hallucinate
IBM Technology’s Why Large Language Models Hallucinate distinguishes invented facts, contradictions, and responses that conflict with the request, then discusses several possible contributors.
Watch output symptoms for examples of contradictions and false claims. Continue with possible causes, paying particular attention to information quality and the context supplied with the request. The discussion of decoding methods is background, not something you need to diagnose from a product screenshot.
Hallucination is often used for a fabricated or unsupported claim, but not every unreliable result is a hallucination. A summary can be factually accurate yet omit the only decision the user needed. An answer can be correct for last year’s policy but wrong for the current one. A generated number can happen to be right even though the system had no access to the data needed to establish it. Diagnosing reliability means looking beyond whether the prose sounds right—or even whether one answer turned out right by chance.
Separate facts, context, and expectations
Google’s Errors + Graceful Failure introduces a distinction especially useful for product design: sometimes an AI system produces the kind of result it was built to produce, but its assumptions about the user’s situation are wrong. The result may be technically valid and still fail the person’s task.
Read Google’s Errors + Graceful Failure to distinguish errors in the information or prediction from errors about what is relevant to the user.
In “Identify user, system, and context errors,” read context errors, from their definition through the advice about inspecting signals. Then go to “Identify error sources.” Read “Discover prediction & training data errors,” beginning with the source overview, and continue through its three named examples: “Mislabeled or misclassified results,” “Poor inference or incorrect model,” and “Missing or incomplete data.” Finally, in “Check output quality for relevance errors,” read from relevance errors through the “Low confidence” and “Irrelevance” examples. As you read, ask whether each problem concerns the answer’s truth, the system’s available information, or its fit to the user’s situation.
The reading discusses AI systems broadly, including classifiers such as a plant-identification app. Its examples help name possible classes of cause; they do not let you conclude that a particular LLM error was caused by mislabeled training data. From a single bad meeting summary, you usually cannot inspect what the model learned during training.
For productivity experiences, a practical first pass is to sort the problem into three questions:
- Is the claim supported? “Legal approved the proposal” is unreliable if the notes say approval is pending. The issue may involve missing source material or a mistake in interpreting material that was available.
- Is the result relevant to this situation? A generally sound travel suggestion may be unusable if it ignores the company’s lodging limit. A helpful response for one user or moment may be inappropriate for another.
- Is the product implying more certainty or completion than it has earned? “Your follow-up is sent” is a different claim from “Here is a draft follow-up.” Check whether an action actually happened, not just whether the model said it did.
These questions can overlap. A policy answer based on an old document may be well supported by that document and still be wrong for the user’s current decision.
Trace the information the assistant actually had
A user may see a document open beside an AI feature and assume the assistant has read it. That assumption is not always warranted. An AI product may supply selected text, retrieve a passage, pass along a conversation history, or call a tool for current data. The model can only work with information the system actually makes available to it, and even supplied information can be incomplete or misused.
IBM Technology’s Context Engineering vs. Prompt Engineering: Smarter AI with RAG & Agents makes this distinction concrete. Notice how a request can be understandable to a person yet still lack a location, a policy, or confirmation that an external action succeeded.
Context Engineering vs. Prompt Engineering: Smarter AI with RAG & Agents
IBM Technology’s Context Engineering vs. Prompt Engineering: Smarter AI with RAG & Agents shows why better wording alone cannot compensate for missing documents, current data, or workflow state.
Watch the travel example for the mistaken destination and missing company policy. Then watch state and tools for how a system keeps track of a task, retrieves relevant material, and accesses external information. Focus on what information or confirmation the assistant would need; you do not need to design that architecture yet.
This gives you several distinct hypotheses when an answer is wrong:
- The needed information was never supplied. An assistant asked about “our leave policy” has not necessarily been given the organization’s policy.
- The system supplied the wrong or an outdated source. Access to a policy document is not the same as access to the approved, current one.
- Relevant material was supplied but lost or overlooked. A long set of notes might contain a qualification that the summary omits.
- The request or task instructions left an important choice ambiguous. “List the owners” does not say whether to include people merely proposed as owners.
- A lookup, calculation, or action was not verified. If the job requires an exact metric or confirmation that a message was sent, fluent generated text is not a substitute for checking the relevant system.
Do not pick one solely because it is a familiar AI failure. Ask what evidence would distinguish the hypotheses. If the current policy was not among the materials provided, missing context is a strong explanation. If it was provided and the answer cites an obsolete paragraph instead, investigate source selection. If the correct paragraph was provided and cited but the conclusion contradicts it, investigate interpretation.
Diagnose the failure in the workflow
Consider four situations a product designer might encounter. In each, the useful diagnosis identifies both an observed failure and the next thing to check.
Meeting notes: an invented commitment. The source says, “Mina can probably prepare a draft by Friday, pending legal review.” The assistant outputs, “Mina owns delivery of the final proposal on Friday.” The observable problem is not just a wrong name or date: it has removed probably, draft, and pending legal review. If those qualifications were in the text the assistant received, an overconfident interpretation is likely. If they were absent from an automatically shortened transcript, incomplete input is the better initial explanation. Inspect the supplied excerpt before blaming the model’s interpretation.
Policy answer: confident but stale. A user asks whether meals are reimbursable under the current travel policy. The assistant says yes and points to a policy page that was replaced last month. The citation shows where the claim came from, but it does not establish that the source is current. Check which document the product selected and whether its date and status were available. Changing the wording of the user’s question alone would not fix a system that consistently supplies an obsolete policy.
Usage metric: precision without a data check. A manager asks how many people used a feature last month; the assistant returns “326” with no indication that it accessed usage data. The defect is not proven by the number’s precision. The problem is that there is no visible basis for an exact, company-specific claim. Check whether an appropriate data source was queried, whether “used” and “last month” were defined, and whether any calculation completed. Even if 326 happens to match the real figure, an unsupported guess is not a dependable way to answer the next query.
Feedback summary: accurate but unhelpful. A designer asks for concerns about onboarding from a collection of customer comments. The assistant correctly summarizes recurring praise for the product’s appearance while omitting the few comments describing onboarding failures. This may not be a fabricated fact. It may be a relevance or coverage failure: the answer favors the most common theme over the requested concern. Compare it with the full comments and inspect whether the instruction emphasized onboarding and whether those comments were included in the material supplied.
Each scenario suggests a different investigation. “Try a more powerful model” is not a diagnosis. Neither is “write a better prompt” when the assistant lacks the current policy or has no way to verify a metric.
Make a diagnosis that can be checked
Anthropic’s Reduce hallucinations offers two useful principles for investigating unsupported answers: permit the system to acknowledge missing information, and connect substantive claims to source passages. Treat these as ways to test and constrain a result, not guarantees that every cited or quoted answer is correct.
Reduce hallucinations - Claude Platform Docs
Read Anthropic’s Reduce hallucinations for examples of acknowledging uncertainty and checking claims against provided material.
In “Basic hallucination minimization strategies,” read the “Analyzing a merger & acquisition report” example, from its request to analyze a supplied report through the instruction beginning admit uncertainty. Then read the “Auditing a data privacy policy” example, from the policy request through its instruction to base the analysis on extracted quotes. Notice that the first example addresses absent evidence, while the second helps reveal whether an interpretation follows from evidence that is present.
You can express a product-facing diagnosis in four parts, without claiming access to the model’s hidden reasoning:
| Part | What to record |
|---|---|
| Observed output | The specific claim, omission, or action status that seems unreliable. |
| Comparison point | The source passage, user request, current record, or tool result against which to check it. |
| Likely cause | The best-supported hypothesis, with uncertainty where evidence is missing. |
| Discriminating check | What to inspect next to separate that cause from a plausible alternative. |
For the meeting example, a careful note might read: The output presents Friday delivery as a confirmed commitment, while the original note makes it a tentative draft subject to legal review. The assistant may have dropped the qualifications during summarization; check whether the full note, including those phrases, was provided to it. If it was not, investigate how the notes were selected or shortened.
That diagnosis is more useful than “the AI hallucinated.” It tells a team what failed for the user and what evidence to gather before choosing a remedy. It also avoids a common trap: asking for citations or quotes can expose an unsupported claim, but a quoted source can itself be outdated, irrelevant, or incorrectly interpreted.
Key takeaways
When AI output is unreliable, start with the observable mismatch: an unsupported claim, an omitted qualification, an irrelevant answer, or an unverified assertion that an action is complete. Then trace what the assistant was asked to do, what information it actually received, and what external data or action the task required. Treat a cause as a hypothesis until a check distinguishes it from alternatives.
This diagnostic habit will help you frame credible AI product opportunities: you can specify not only what assistance is valuable, but also what must be available and checkable for it to work. Next, you’ll trace a user request from the interface through model processing and back to the product experience, making the places where these failures can arise more explicit.