Defining Success Criteria for a Live, Data-Flowing Deployment

Hello again. In the previous lesson, you used six workstreams to see why a working integration is not the same thing as a ready deployment. This lesson gives those workstreams a shared finish line. You will define what someone should be able to observe before calling a deployment live, data-flowing, and users-logged-in—and what still cannot be concluded at that point.

The distinction matters in onboarding. “The API works” is an engineering result. “The customer can use an approved production service with current, trustworthy data” is a delivery result. You need criteria that let both teams tell the difference.

Start with the customer’s intended use, not the launch announcement

Suppose a customer wants designated reviewers to examine operational records for emerging risks. A deployment could be announced as live while those reviewers see no recent records. Equally, the records could arrive correctly while reviewers cannot sign in. Neither condition meets the intended use.

The ClientSuccess webinar makes a useful starting point: success must be defined with the customer, because an internal definition of “deployed” may not match what the buyer or daily users need. Watch the discussion of customer outcomes and the later exchange about differing stakeholder views.

For this kind of onboarding, define three related but distinct claims:

1. Live: The agreed service is released for its intended production use, under the required approvals and with an operating owner.
2. Data-flowing: The agreed source data reaches the intended destination repeatedly, within agreed timing and quality limits—not merely in one successful test.
3. Users-logged-in: Authorized, representative users can authenticate and reach what their roles permit. At least one intended workflow has been demonstrated; a login count alone does not establish usable access.

These are launch success criteria, not a promise that the software has already discovered a risk or improved a safety outcome. Those longer-term results depend on use and evaluation after launch. Keeping the claims separate makes it possible to report honestly: “Production is enabled and the first load passed, but we have not yet observed the regular feed.”

Turn each claim into a testable statement

A useful criterion answers four questions: Under what conditions? What must be observed? What counts as passing? Who accepts the evidence? Record where the evidence will come from as well. “SSO complete” answers none of these reliably; “a designated reviewer signed in through the customer’s identity provider, reached the agreed view, and had only the approved permissions” can be checked.

Here is an illustrative set for a deployment with a scheduled file feed. The numbers are examples for discussion, not standard targets or implied contractual commitments.

The owner who produces evidence need not be the person authorized to accept it. You may verify a file arrival; the customer’s data owner may need to accept its meaning and completeness. You may coordinate the launch decision without granting a privacy approval on someone else’s behalf.

Notice the precision in “each expected file.” If a source produces no new records during a quiet period, an empty batch might be legitimate. The criterion therefore needs an agreed delivery schedule or other sign of source activity, plus a rule for distinguishing “nothing happened” from “the feed failed silently.”

Readiness evidence is broader than a successful test

A single successful end-to-end run is important, but it does not establish that production can be operated safely. Microsoft Learn’s go-live checklist is a useful reference for the difference between test results, stakeholder acceptance, and support readiness. It describes a general enterprise deployment; use its categories to sharpen your criteria rather than copying every item into a smaller onboarding.

This suggests a useful rule: a criterion should test the claim you intend to make. A test-environment transfer demonstrates feasibility, not a live production feed. A signed acceptance note records a decision, but it does not substitute for the test results on which that decision rests. Monitoring configured in advance is readiness evidence; a monitored production delivery supplies operating evidence.

A brief video on go-live readiness reinforces the point that technical tests and user capability answer different questions.

“Data-flowing” needs a clock, a count, and a failure signal

Your experience with integrations will help you separate transport success from delivery success. A successful API response or received file proves that something crossed a boundary. For an ongoing feed, the customer also needs to know whether the right data arrived, how current it is, and whether a failure would be noticed.

The image below illustrates several possible points at which delay can accumulate. Its Kafka, stream-processing, and data-lake components are one example architecture, not an assumed design for the deployment you will own.

If the customer cares when a record becomes available for review, measuring only when it was ingested can conceal later processing delays. An end-to-end freshness criterion would state the starting event, the point at which the result is queryable, the acceptable delay, and the measurement period. A scheduled daily export might instead be judged against an agreed delivery cutoff; use the clock that matches the source and the intended workflow.

Also distinguish a snapshot from an agreed service level. The image shows a current freshness value, a previous-day percentile, and a last-hour peak. These describe different windows and statistics. A peak above a percentile target does not, by itself, prove that the percentile service level was breached. Treat all pictured targets as illustrative until the customer agrees on definitions and thresholds.

Freshness is only one dimension. For a launch decision, pair it with a completeness check, an account of rejected or excluded records, and a named response when deliveries are late or invalid. If historical records are also in scope, assess that load separately from the recurring feed: yesterday’s successful migration does not prove tomorrow’s scheduled delivery.

“Users-logged-in” means appropriate access to a real workflow

SSO can be technically configured and still fail the launch purpose. An identity administrator might be able to sign in while the designated reviewer cannot. A reviewer might authenticate successfully but receive no useful role, see the wrong data, or lack permission to perform the intended task.

For launch, representative access evidence should therefore cover three boundaries:

• Authentication: Do the intended users reach the production service using the agreed sign-in method?
• Authorization: Do their roles expose the right capabilities and data, while restricting access that was not approved?
• Usability for the initial purpose: Can a representative user complete an agreed task with the available data, such as locating and reviewing a record?

The third boundary is deliberately modest. Observing a reviewer complete one representative workflow establishes initial usability, not sustained adoption. The number of people actively using the service, the quality of their decisions, and the longer-term value of risk discovery require observation after launch. This distinction prevents an onboarding team from claiming a business outcome on the strength of a login screenshot.

For sensitive records, a negative permission check matters as much as a positive login check. “Reviewer A can open the intended records” is incomplete if “User B, who lacks that role, cannot open them” has not been verified. The particular roles and permitted views must come from the agreed customer access model, not from a generic template.

Agree the finish line before announcing it

Bring the three claims back to the six workstreams from the previous lesson. The commercial view confirms which sources, users, and initial workflows are in scope. Technical and data evidence show that production behavior works and that its contents can be trusted for initial use. Security establishes permitted handling and access. Organizational evidence identifies who can approve launch and respond to faults. Adoption evidence shows that intended users can begin the workflow, while leaving longer-term usage outcomes for follow-up.

Before treating a criterion as agreed, specify its scope, threshold, observation period, evidence source, and accepting owner. If one is unknown, mark it as to be agreed, not as implicitly passed. For example, “data arrives promptly” cannot be verified until “promptly” is tied to a source schedule and a point at which the data must be available.

Finally, a launch decision should state any exceptions plainly. A minor, understood defect with a workaround and an authorized acceptance decision differs from an unexamined gap in data permissions or an invisible production feed. The aim is not to claim a flawless system. It is to make a defensible statement about what is working, what remains uncertain, who has accepted the remaining risk, and who will notice if the service deteriorates.

Key takeaways

A successful onboarding finish line is more specific than “built and switched on.” Live requires an approved, operable production service; data-flowing requires repeated, timely, checked delivery; users-logged-in requires appropriate access and a demonstrated initial workflow. Each claim needs conditions, observable evidence, a pass standard, and someone authorized to accept it. None alone proves sustained adoption or a safety benefit.

Next, you will learn to classify onboarding information as a risk, issue, action, decision, assumption, or dependency—a way to keep unresolved launch criteria visible rather than letting them disappear inside a general status update.