Classifying Service Failures by Impact

Welcome back. In the previous lesson, you mapped a web request from DNS and a public load balancer through private application instances to data stores and downstream services. That map identified where a request can fail. This lesson focuses on a different question: what did the failure do to the user or their data?

For an SRE or DevSecOps practitioner, separating the cause of an incident from its effect is essential. A database problem, bad deployment, cloud-network rule, or compromised credential can each produce very different user-visible outcomes. By the end of this lesson, you will be able to classify a failure primarily as an availability, latency, correctness, or durability problem—and recognize when one incident has more than one effect.

---

Classify by the broken promise, not the broken component

A service makes several promises at once. A user expects it to be reachable, reasonably quick, accurate, and trustworthy with previously accepted data. The four classifications in this lesson describe which promise was broken.

The four labels are:

Failure effect	Broken user promise	Typical user observation
Availability	“I can complete this valid action.”	The service is unavailable, rejects the request, or never provides a usable response.
Latency	“I can complete this action quickly enough.”	The correct response eventually arrives, but exceeds an agreed time threshold.
Correctness	“The result or action is right.”	The service returns wrong, stale, duplicated, unauthorized, or otherwise invalid data or behavior.
Durability	“Data you accepted will remain intact and retrievable.”	Previously confirmed data is later lost, corrupted, or cannot be read when required.

Two related terms deserve care:

• Throughput measures how much work a service can handle, such as requests per second. It is often a cause or capacity signal, rather than the primary user-facing failure category in this lesson. Insufficient throughput may lead to latency or availability failures.
• A security incident is not automatically a separate reliability category. For example, a broken authorization rule that exposes another customer’s order is primarily a correctness failure for the user journey and a confidentiality incident for security response. Both classifications matter.

The central habit is this:

First describe what a valid user operation experienced. Then investigate which component or change caused that effect.

A database outage is not itself a classification. If it produces HTTP 503 responses, it is causing an availability failure. If requests still succeed but take twelve seconds, it is causing latency failures. If it returns the wrong customer record, it is causing correctness failures. If committed records disappear after the outage, it is causing a durability failure.

---

The four failure effects in detail

Availability: no usable completion

An availability failure occurs when a well-formed, valid request cannot complete successfully from the user’s point of view.

Examples include:

• DNS cannot resolve the service hostname.
• The load balancer has no healthy targets and returns an error.
• A security-group change blocks load balancer traffic to the application.
• The application crashes and returns HTTP 500 errors.
• A database is unavailable, so an order lookup or checkout request fails.
• A client waits until its deadline expires and receives no usable response.

Availability is usually assessed over a population of valid requests:

$$\text{Availability}=\frac{\text{successful valid requests}}{\text{eligible valid requests}}$$

The word valid is important. A request rejected because it is malformed or unauthenticated may be an expected product behavior, not a reliability failure. Teams must state the denominator explicitly instead of treating every non-success status code as an outage.

A timeout can be subtle. At the server, the work may eventually finish; at the client, the user has received no usable outcome. For the user journey, that is normally an availability failure. The underlying slow operation is likely also a latency problem worth investigating.

Latency: completion, but too slowly

A latency failure occurs when the service completes the intended operation, returns an acceptable result, but does not do so within the time users or dependent systems require.

For example:

• GET /orders/123 returns the correct order after eight seconds, while the intended limit is one second.
• Checkout confirmation succeeds after the user’s browser displays a long spinner.
• An internal API responds in 900 milliseconds when its callers need a response within 200 milliseconds to avoid exhausting their own request capacity.

Latency is not “any response that feels slow.” It requires a defined threshold for an operation. A common SLI asks for the proportion of requests that finish within that threshold:

$$\text{Latency success rate}=\frac{\text{eligible requests completed within the threshold}}{\text{eligible requests measured}}$$

The threshold varies with the journey. A health check, interactive search, report generation, and asynchronous batch job have very different expectations.

It is also useful to examine the distribution, not only the average. A service whose average latency is 100 milliseconds can still be painful if a meaningful tail of requests takes ten seconds. Percentiles such as the 95th or 99th percentile help expose that long tail.

Correctness: a response that should not have been returned

A correctness failure happens when the system returns the wrong answer or performs the wrong business action—even when the HTTP request succeeded quickly.

Examples include:

• An order endpoint returns another customer’s order.
• A pricing deployment applies a discount twice.
• A cache key collision causes one user to receive another user’s account balance.
• A payment is charged twice after an unsafe retry.
• A reporting pipeline computes incorrect totals.
• A user sees an account balance that is too stale for the product’s stated behavior.

This is why HTTP status alone cannot establish health. A 200 OK response may still represent a severe incident. The protocol says the server successfully handled an HTTP exchange; it does not prove that the business result is true, authorized, or complete.

Correctness often crosses the SRE and DevSecOps boundary:

• Application logic, schema changes, and pipeline transformations can introduce incorrect data.
• Authorization defects can return data to the wrong identity.
• Excessive cloud permissions can permit an unintended modification.
• A rollback that restores service availability might not automatically repair incorrect writes already made.

Measuring correctness is more difficult than counting errors because the system must have a trustworthy basis for knowing what “right” means. Teams commonly use known-good test cases, reconciliation against an independent calculation, integrity constraints, or carefully chosen business invariants.

Durability: accepted data no longer survives

A durability failure occurs when data that the service represented as successfully stored is later unavailable, irretrievably corrupted, or missing when it is needed.

Examples include:

• Checkout confirms an order, but a storage fault later loses the order record.
• A successful file upload is acknowledged, but the object is deleted and cannot be restored.
• A database restore completes, but committed data from the promised recovery period is absent.
• Data corruption makes archived records unreadable.

Durability differs from temporary availability:

Situation	Primary classification	Why
A database is down for ten minutes, but all data returns afterward.	Availability	Data was temporarily inaccessible, not lost.
A read request eventually returns the right record after eight seconds.	Latency	The data remains accessible, but response time is unacceptable.
The record returns with the wrong amount or owner.	Correctness	The system returned an invalid result.
The record was acknowledged as saved but is later gone permanently.	Durability	The system failed to retain accepted data.

A useful operational question is: Did the system promise that the write succeeded? If it rejected a write with an explicit error, the user can retry and the event is generally availability-related. If it confirmed the write and later cannot produce that data, the system has violated a durability promise.

---

Read the service effect through the user journey

The Google SRE book connects these categories to the kinds of systems users interact with. This reading establishes the terminology and, importantly, notes that correctness matters even when it is not owned solely by an SRE team.

A request-driven service such as the orders API from the previous lesson usually emphasizes availability and latency. A storage service has the additional long-term obligation of durability. A pipeline may complete work at high volume and still fail if the transformed output is wrong.

The system type provides a clue, not a limitation:

• An API may expose correctness failures when it returns the wrong data.
• A storage service can be slow or unavailable, not only non-durable.
• A pipeline can be available to operators but produce incorrect or stale output.
• Every production system should care about correctness, because a fast, reachable wrong answer is still a failure.

---

A practical classification method

During an incident, avoid beginning with labels such as “the database is broken” or “Kubernetes is down.” Start from one specific user operation, such as view order, create payment, upload file, or generate invoice.

Use this four-step method.

1. State the expected contract.
   Specify the operation, expected result, and any time or retention promise. For example: “An authenticated customer can retrieve only their own order within one second,” or “A confirmed invoice remains retrievable for seven years.”

2. Record the observed outcome.
   Use evidence from the client, load balancer, application logs, traces, database records, and audit logs. Record status code, duration, returned value, and whether the operation was acknowledged as complete.

3. Choose the primary effect.
   Identify the first broken promise:

   • No usable completion for a valid operation: availability
   • Valid completion, but beyond the time target: latency
   • Completion with an invalid result or action: correctness
   • Previously acknowledged data lost or irrecoverable: durability

4. Record secondary effects separately.
   A real incident can have several effects. Do not force a single label if evidence shows otherwise. State the primary customer impact, then list associated effects, security implications, and likely causes.

Consider a familiar cloud service:

Incident observation	Primary classification	Possible cause to investigate
ALB returns 503 because no targets pass health checks.	Availability	Failed deployment, overloaded Pods, bad health-check path, blocked port.
The orders API returns accurate results in 6 seconds rather than under 500 milliseconds.	Latency	Slow database query, exhausted connection pool, saturated CPU, remote dependency delay.
GET /orders/123 returns order 124 with HTTP 200.	Correctness	Authorization defect, caching bug, query error, corrupted mapping.
An order was confirmed yesterday but cannot be retrieved from the primary database or backups.	Durability	Data deletion, replication failure, failed restore design, corruption.

A single incident can evolve across categories. Suppose a database becomes overloaded:

• Initially, queries return the right data slowly. The observed effect is latency.
• As connection pools fill, requests time out. The user-visible effect becomes availability.
• If emergency retry logic creates duplicate charges, the incident has also produced correctness failures.
• If an unsafe failover loses acknowledged writes, it has added durability failures.

This is why incident updates should distinguish cause, symptoms, and impact. “Database saturation” is a likely cause. “Thirty percent of checkout requests time out” is an availability impact. “Some successful checkouts may have been charged twice” is a potential correctness impact that needs separate verification.

---

From classification to evidence

Each failure dimension points toward different evidence. You do not need every telemetry system to begin classifying an incident, but you do need enough evidence to avoid guessing.

Dimension	Strong evidence	Weak or misleading shortcut
Availability	Valid request success rate, user-facing synthetic checks, HTTP error counts, client timeouts	“The instance is running.” A running process may still be unusable.
Latency	Client or edge duration, latency histograms, percentiles, distributed traces	Average latency alone. It can hide slow tail requests.
Correctness	Reconciliation, known-good test inputs, audit logs, domain invariants, user reports confirmed by data	HTTP 200 responses or healthy infrastructure.
Durability	Successful-write records compared with later reads, backup-restore tests, replication and integrity checks	“The database is currently reachable.” Reachability does not prove retention.

The IBM Technology video gives a short view of latency, errors, and diagnosis using the Golden Signals. Treat its “errors” discussion as useful evidence for availability analysis, not as a replacement for the broader availability definition used in this lesson.

---

Keep the categories distinct when systems overlap

The SRE Workbook offers a useful service-type framework. It places availability and latency naturally with request-driven services, correctness with pipelines, and durability with storage—while recognizing that real architectures contain all three kinds of components.

For the order service request path, one architecture contains all three:

• The GET /orders/123 API is request-driven. It must be available, fast enough, and correct.
• A nightly sales aggregation job is a pipeline. It must transform records correctly and keep the data sufficiently current.
• The orders database and backups are storage. They must preserve accepted records and make them retrievable when needed.

That perspective prevents a common operational mistake: declaring the system healthy because the API returns 200 responses while the pipeline is silently producing incorrect reporting data or backups cannot restore a critical record.

A concise incident classification statement might read:

From 14:05 to 14:18 UTC, authenticated GET /orders/{id} requests were available but breached the one-second latency target for 18% of users. Responses were verified as correct. The initial hypothesis is database connection-pool saturation; no evidence currently indicates data loss or unauthorized access.

This wording separates observable impact from hypothesis, identifies what was checked, and makes clear what remains unknown.

---

Key takeaways

A service failure should be classified by the promise broken for a user or their data:

• Availability: a valid operation cannot produce a usable successful outcome.
• Latency: the operation succeeds, but not within its required time.
• Correctness: the service returns or performs the wrong result, even if it is fast and returns HTTP 200.
• Durability: data that was accepted as stored is later missing, corrupted, or irrecoverable.

The same technical cause can create different effects, and one incident may create several effects at once. Classify the primary user impact first, then record secondary impacts, security implications, and suspected causes separately.

In the next lesson, you will turn one of these user journeys into a measurable service-level indicator, defining exactly what counts as a good event, a bad event, and the evidence used to calculate it.