Processing Semantics Under Retries

Welcome back. In the previous lesson, you separated what a distributed system does during a network partition from what it optimizes when connectivity is healthy. The same discipline matters for message processing: a timeout does not reveal whether an operation failed or whether it succeeded but its acknowledgement was lost.

This lesson traces that uncertainty through a producer, broker, consumer, and sink. By the end, you should be able to inspect a retry scenario and determine whether it produces loss, duplicates, or an effectively-once result.

---

Retries create an information problem

Consider a pipeline that receives an OrderPaid event and writes a row to an analytics table. A producer sends a message, or a consumer processes one, and then waits for an acknowledgement. If that acknowledgement never arrives, there are two possible realities:

1. The receiver never performed the operation.
2. The receiver performed the operation, but the response was delayed, lost, or arrived after the sender timed out.

From the sender’s perspective, these cases look identical. Retrying protects against the first case, but can repeat work in the second. Refusing to retry prevents duplicates, but can lose work.

This is why delivery semantics are not merely Kafka configuration labels. They are explicit choices about which error is acceptable when the system cannot know whether a prior attempt took effect.

A useful distinction is:

Question	Example
How many times was the message delivered?	Did the consumer receive OrderPaid(order-17) zero, one, or two times?
How many times did business logic run?	Did the consumer run its insert or update twice?
How many times did the intended effect occur?	Did the analytics fact, balance update, or shipment request occur once?

The first is a transport-level property. The third is the one that usually matters to a data engineer.

Read Confluent’s compact overview before tracing the failure windows in detail.

The three guarantees

The terms describe bounds on the number of deliveries or processing attempts for a particular logical message.

Guarantee	What failures can cause	Typical retry policy
At-most-once	A message or its effect may be lost	Do not retry after uncertainty, or record progress before work
At-least-once	A message may be delivered and processed repeatedly	Retry until success is confirmed
Effectively-once	Physical retries may occur, but the intended durable effect appears once	Retry, then make duplicate work harmless or reject it atomically

“Exactly-once” is frequently used in system documentation, but it needs a scope. A broker may provide exactly-once behavior for a consume-transform-produce workflow within that broker. That does not automatically mean that an external database update, email, payment request, or object-store write happened exactly once.

For architecture reasoning, effectively-once processing is usually the more useful target:

A message may be delivered and attempted more than once, but its intended business or data-state effect is committed no more than once.

---

Trace 1: producer retries and duplicate records

Suppose a producer has an event:

message_id = 8f31
type       = OrderPaid
order_id   = 17
amount     = 49.99

It sends this event to a broker.

At-most-once at the producer

With at-most-once behavior, the producer sends once and moves on without retrying when confirmation is missing.

A possible sequence is:

1. The producer sends message 8f31.
2. A network fault drops the request before the broker durably records it.
3. The producer does not retry.
4. No consumer ever sees the event.

This avoids a duplicate caused by that producer retrying. The price is loss. This may be defensible for high-volume telemetry where an occasional missing click or CPU sample does not alter an important business decision. It is generally unsuitable for financial events, CDC records, or data needed for reconciliation.

At-least-once at the producer

With at-least-once behavior, the producer waits for a confirmation and retries if it does not receive one.

Now consider the more subtle sequence:

1. The producer sends message 8f31.
2. The broker durably appends it.
3. The acknowledgement is lost or arrives after the producer timeout.
4. The producer retries 8f31.
5. The broker appends it again unless it can recognize the retry as a duplicate.

The producer’s retry was rational: it did not know the first attempt succeeded. Yet consumers may now read two log records representing one logical business event.

An idempotent producer addresses this specific broker-write window. Kafka can associate a producer identity and sequence information with writes so that retrying the same send does not create a second log entry. This is valuable, but its scope is limited:

• It protects against duplicates from retrying a producer write to Kafka.
• It does not automatically make downstream consumer code idempotent.
• It does not deduplicate two independently created messages that represent the same business event but carry different identities.

A stable event or message ID remains essential throughout the pipeline.

---

Trace 2: consumer offset commits decide loss versus duplication

Producer delivery is only one boundary. A consumer has its own acknowledgement: it commits its position in a source log.

In Kafka, that position is the offset. For a given consumer group and partition, a committed offset tells the next consumer instance where processing should resume. The ordering of two actions determines the failure mode:

• saving progress, and
• committing the output of processing.

Commit the offset before processing: at-most-once processing

Suppose the consumer reads offset 420, containing OrderPaid(order-17).

1. The consumer reads offset 420.
2. It commits its next position, indicating that offset 420 is complete.
3. It crashes before writing the analytics row.
4. Another consumer instance resumes after offset 420.

The event has been skipped. Since progress was recorded first, the source will not redeliver it. This is at-most-once processing: no duplicate execution, but an accepted message can disappear before its effect is persisted.

Process before committing the offset: at-least-once processing

Now reverse the order:

1. The consumer reads offset 420.
2. It writes the analytics row.
3. It crashes before committing the offset.
4. Another consumer resumes from offset 420.
5. It writes the row again unless the sink or consumer prevents it.

This is at-least-once processing. It favors completing the durable effect, but a crash in the gap between processing and acknowledgement causes reprocessing.

The central rule is simple:

If you record progress before the effect, you risk loss. If you record progress after the effect, you risk duplicates.

The correct choice depends on whether the sink operation is safe to repeat.

---

Making at-least-once processing safe

At-least-once delivery is common because it is often the safer default for valuable data: uncertainty leads to a retry rather than silent loss. The engineering task then becomes ensuring that repeated delivery does not produce repeated effects.

Watch the failure scenario and the transactional deduplication technique in the following video. The implementation is application-oriented, but the same pattern applies to data-pipeline consumers writing into relational sinks.

The general solution is an idempotent consumer. Idempotent means that processing the same logical message repeatedly has the same final result as processing it once.

Some operations are naturally idempotent:

SET order_status = 'CANCELLED'

Repeating this update leaves the state as CANCELLED.

Other operations are not naturally idempotent:

balance = balance - debit_amount
daily_total = daily_total + amount
INSERT a new shipment row

Repeating an increment, debit, or unconstrained insert changes the result each time. These need an explicit deduplication mechanism.

Read the explanation of this pattern, including its crucial atomicity requirement.

The processed-message pattern

A common implementation holds two writes in one sink transaction:

1. Insert a record such as (subscriber_id, message_id) into a PROCESSED_MESSAGE table with a unique constraint.
2. Apply the business update or write the output row.
3. Commit the transaction.
4. Acknowledge or commit the source offset only after the sink transaction succeeds.

Conceptually:

BEGIN TRANSACTION

INSERT INTO processed_message (subscriber_id, message_id)
VALUES ('analytics-order-paid-consumer', '8f31');

If the insert succeeds:
    write the order-paid fact row
    update the daily revenue aggregate

COMMIT TRANSACTION

acknowledge the source message

The actual implementation should treat a unique-key conflict as “already processed,” not as a business failure. The consumer can acknowledge that redelivery without performing the update again.

The important detail is not the preliminary check alone. A separate SELECT followed later by an INSERT is vulnerable to concurrent consumers both seeing “not processed yet.” The database’s unique constraint, combined with one transaction covering both the deduplication record and the data update, supplies the race-safe decision.

Retry trace with an idempotent consumer

Return to offset 420 and message ID 8f31.

Failure point	What is durable?	What happens on retry?	Final effect
Consumer crashes before the transaction commits	Neither deduplication record nor business update	Retry performs the transaction	One effect
Transaction commits, but acknowledgement fails	Deduplication record and business update	Retry hits the unique constraint and skips the update	One effect
Acknowledgement succeeds	Transaction and source progress are durable	No retry normally occurs	One effect

This gives effectively-once behavior for the database effect even though the broker may deliver the same message several times.

The deduplication identity should normally include both:

• the message ID, identifying the logical event; and
• the consumer or subscriber ID, identifying which handler has applied it.

One message may legitimately be handled once by billing, once by shipping, and once by an analytics projection. A global “processed” marker would incorrectly block the other consumers.

---

Kafka transactions: powerful, but scoped

Kafka can coordinate a specific form of exactly-once behavior when an application:

1. reads records from Kafka,
2. transforms them,
3. writes results to Kafka topics, and
4. commits the consumed offsets.

Kafka transactions can commit the produced output records and the consumed offsets together. If the transaction aborts, downstream consumers configured to read only committed data do not see its output, and the source position is not advanced as successfully processed.

Read the consumer-receipt section to connect offset timing with the three semantics, then focus on the transactional consume-produce case.

This is extremely useful for Kafka-to-Kafka pipelines. For example, a streaming job may read raw events, produce a cleaned topic, and atomically commit its consumed offsets. A crash does not expose half of a transactional output to properly configured downstream consumers.

But an external side effect changes the problem:

Kafka event
    consumer
        external database update
        offset commit

Kafka cannot by itself atomically commit an arbitrary external database transaction, an email request, or an HTTP call. If the database update succeeds and the consumer fails before offset commit, Kafka can redeliver the message. The database side still needs idempotency or another transactionally coordinated protocol.

For calls to an external service, an equivalent safeguard is often an idempotency key: the receiver persists the key and returns the prior result when it sees the same request again. Without a transactional boundary or receiver-side deduplication, claiming end-to-end exactly-once is usually unjustified.

---

A practical method for analyzing any retry path

When reviewing a pipeline design or debugging duplicate rows, trace one named message through the system. Do not begin with the broad claim “the pipeline is exactly once.”

Use this checklist:

1. Identify the logical message.
   What stable identifier says two attempts represent the same event? Kafka offset alone is often useful within a source partition, but an explicit event ID is more portable across systems.

2. Locate the durable effect.
   Is the important outcome a Kafka record, a Delta table row, a database update, an object-store file, or an API request?

3. Locate every acknowledgement.
   Producer acknowledgements, consumer offset commits, database commits, checkpoint completion, and HTTP success responses each create distinct failure windows.

4. Choose the retry policy.
   If an acknowledgement is absent, will the system retry? If it does, which completed operations might run again?

5. Test whether the effect is idempotent.
   Re-run the same message mentally. Does the final state stay correct? Be careful: “set status” may be safe for duplicates but still require versioning or ordering rules when older events can arrive after newer ones.

6. Verify atomicity of deduplication and mutation.
   A deduplication marker must be committed with the state change it protects. Otherwise a crash can leave one without the other.

7. State the guarantee and its boundary precisely.
   For example: “Kafka-to-Kafka transformation is exactly-once for consumers using committed-read isolation; writes to the reporting database are effectively-once through a transactional processed-message table.”

That final sentence is far more operationally useful than a blanket “exactly-once pipeline” claim.

---

Key takeaways

• A missing acknowledgement creates uncertainty: the prior attempt may have failed, or it may have succeeded invisibly.
• At-most-once avoids duplicate retries but can lose messages or processing effects.
• At-least-once retries after uncertainty, protecting against loss while allowing duplicate delivery and duplicate execution.
• In a consumer, committing offsets before processing risks loss; committing offsets after processing risks reprocessing on failure.
• Effectively-once means retries may occur, but duplicate attempts cannot create a second intended effect.
• An idempotent consumer commonly achieves this by storing a unique (subscriber_id, message_id) marker in the same transaction as the business update.
• Kafka idempotent producers and transactions are valuable, but their guarantees have boundaries. External databases and side effects still require an explicit idempotency or coordination design.

Next, we will move from individual retry paths to broader system impact: identifying failure domains and blast radius in a distributed data architecture.