Recommended resources
The freely available resources this lesson will direct you to.
Hello again. Last lesson, we framed a decision at the invoice intake–reporting boundary without choosing a design. We now have two comparable candidates: direct delivery to a reporting API with persisted retry, and a durable asynchronous channel. The question today is how to evaluate them without treating either as inherently faster, safer, or easier to operate.
We will test both against the pilot’s customer objective—at least 95% of submitted invoices searchable within 15 minutes—and against scalability, reliability, security, cost, operability, and delivery constraints. The aim is a defensible comparison, not yet a final ADR. In the next lesson, that comparison will become a recorded decision.
Establish what cannot be traded away
A useful architecture review separates requirements an option must satisfy from advantages that help distinguish acceptable options. If the applicable tenant-isolation controls cannot be met, a low implementation estimate does not rescue the design. Conversely, if both options satisfy those controls, their different operating costs and delivery risks deserve comparison.
The Azure Well-Architected Framework image provides a compact reminder that a cloud workload has several qualities to protect at once. Its five pillars are cost optimization, security, reliability, operational excellence, and performance efficiency. For our decision, delivery feasibility needs explicit attention alongside them: a design the teams cannot integrate and validate for the pilot is not a viable pilot design.
AWS’s OPS01-BP06 Evaluate tradeoffs while managing benefits and risks adds a decision-making discipline: identify benefits and risks in relation to business goals, then consider the likelihood and impact of risks with the relevant stakeholders. Read the indicated portion as a method for weighing options—not as permission to offset a mandatory security requirement with a faster launch.
OPS01-BP06 Evaluate tradeoffs while managing benefits and risks - AWS Well-Architected Framework
Read AWS’s guidance to distinguish a desirable benefit from a risk that could prevent the business objective.
In the guidance beneath the opening discussion of decision authority, find the bold labels “Identify benefits,” “Identify risks,” and “Assess benefits against risks and make informed decisions.” Read the comparison guidance, from “Understand and manage benefits and risks” through the example contrasting speed to market with reliability. Notice that both benefit and risk are judged against stakeholder goals, not against a universal preference for one technology.
For the invoice pilot, the review should therefore proceed in two passes:
- Confirm gates. Security and privacy owners specify applicable tenant-isolation and audit requirements. Product confirms the start event, denominator, and authorized query used to measure “searchable within 15 minutes.” Platform and team owners confirm any approved-service, spending, and delivery limits. These are inputs to verify, not numbers to invent.
- Compare options that can pass. Examine how each handles bursts, failures, investigation, ongoing expense, and cross-team implementation. Mark an answer unknown when it depends on an untested capability.
The 95% timeliness target is not permission to lose or ignore the remaining 5% of invoices. The teams need to define what happens to invoices that miss the time target, including how they are found, recovered, and reported.
Test the handoff under load and failure
A channel can absorb incoming work temporarily; it cannot make the reporting service process invoices faster. Direct delivery can also buffer work if intake persists a pending state before retrying. The relevant scalability question is not “Which option has a queue?” It is where pending work accumulates, how quickly it can drain, and whether intake remains safe while reporting falls behind.
Consider an illustrative test case, not a forecast for these customers. Suppose 6,000 invoices arrive uniformly over five minutes while reporting can finish processing only 10 per second. Arrival averages 20 per second. At the end of the burst, roughly 3,000 invoices remain to be processed; at 10 per second, clearing that backlog takes another five minutes, before any additional delay to make results searchable. Neither handoff design fixes that downstream capacity limit. The test should measure the full journey for each invoice, particularly those near the tail, rather than report only average API or publication latency.
ByteByteGo’s System Design Was HARD - Until You Knew the Trade-Offs contrasts scheduled batch work with continuous event processing. Watch this short segment for the latency-versus-complexity question it raises. Treat its generalizations as prompts for investigation: neither a batch job nor an event channel is automatically easy to recover or timely enough for this pilot.
System Design Was HARD - Until You Knew the Trade-Offs
Watch ByteByteGo’s comparison of batch and stream processing to sharpen the timeliness question for invoice reporting.
Watch batch versus streams. As it contrasts scheduled work with continuous processing, consider how an existing reporting import—if one exists—would have to demonstrate its actual schedule, processing capacity, and recovery time before becoming a credible third option.
Reliability requires a second, more precise distinction: an acknowledgment is not the same as a searchable invoice. A successful call to a reporting API might mean only “accepted for processing.” Likewise, successful publication to a channel says nothing by itself about whether reporting consumed, indexed, and exposed the invoice.
Walk each option through the same failure cases:
- Failure immediately after intake acceptance. With direct delivery, is the pending invoice durably recorded as part of acceptance so it can be retried after a process crash? With an asynchronous channel, how do we prevent an accepted invoice from disappearing between the intake database commit and publication? A transactional outbox is one possible mechanism to investigate; merely calling “save” and “publish” in sequence is not an atomic handoff.
- Reporting unavailable for several minutes. Where does work wait? What limits retries or consumption? Can intake continue accepting invoices without exhausting its own storage or resources? What alerts before the waiting time consumes the customer-visible budget?
- Delivery or processing repeated. A timeout may leave intake uncertain whether reporting received an invoice, and a channel may redeliver work. Both designs need reporting-side duplicate handling based on stable invoice and tenant identity. An idempotent handler is a practical safeguard; it is not proof that every step of the customer journey happens exactly once.
- One invoice cannot be processed. Can operators identify that invoice, see why it is stuck, repair it, and retry safely without reprocessing an entire customer batch?
These cases make reliability testable. “Supports retries” is weaker evidence than a demonstrated crash-and-recovery test showing that accepted work remains discoverable and ultimately becomes searchable.
Compare the whole operating arrangement
Now place both candidates side by side. The entries below are evaluation hypotheses, not findings about the existing systems. In particular, a managed channel does not remove the need for application logic, and a direct API does not eliminate durable storage or support work.
| Dimension | Direct API with persisted retry | Durable asynchronous channel |
|---|---|---|
| Scalability and timeliness | Intake must store and schedule pending deliveries without letting retries overwhelm reporting. Reporting API capacity and indexing still determine the tail of the journey. | The channel can buffer a burst separately from intake, but consumer capacity, backlog age, and indexing still determine the tail. |
| Reliability | A durable pending record and clear acknowledgment semantics are essential; timeouts create uncertainty about whether reporting received work. | Durable publication and consumption require a safe acceptance-to-publication boundary; redelivery and repeatedly failing messages need explicit handling. |
| Security | Reporting’s API needs authenticated, authorized service access; tenant identity must be validated rather than trusted merely because it arrived from intake. | Producer and consumer permissions, message contents, retention, and replay access need review; tenant identity must also be checked by reporting. |
| Cost | May use existing service and database capacity, but pending-work storage, retry traffic, API capacity, and support time still cost money. | Adds channel and potentially outbox, storage, and transfer costs; a managed service may reduce some infrastructure work without removing operating effort. |
| Operability | Teams need to see pending and failed deliveries, retry age, API responses, and who can initiate repair. | Teams need to see publication failures, backlog age, consumer failures, and who owns replay and channel configuration. |
| Delivery feasibility | Depends on a reporting-owned API contract, its availability for the pilot, and joint failure testing. | Depends on an approved channel, agreed ownership, publication and consumer work, permissions, and joint failure testing. |
Security is a particularly poor place for shorthand. “Private network” or “encrypted channel” does not, by itself, establish tenant isolation. For either option, reviewers should trace an invoice across the boundary and ask: Which identity is authorized to send it? How is its tenant established and checked by reporting? Who may view or replay its contents? What evidence lets an auditor reconstruct what happened? The appropriate security and privacy owners must judge the applicable controls.
Cost also extends beyond the cloud bill. A small pilot might have low message volume but substantial integration and on-call cost. Conversely, reusing existing infrastructure can look inexpensive while placing significant retry and recovery work on the intake team. Estimate build effort, recurring service charges, and expected operating effort over a stated horizon; keep estimates as ranges where usage or support effort is uncertain.
Operability is where team ownership becomes concrete. A dashboard that says “invoice accepted” is insufficient when a customer asks why it is absent from reports. Both teams need a shared correlation key and a way to establish the last confirmed state: accepted by intake, handed off, received by reporting, and searchable through the authorized query. Before preferring an option, identify which team monitors each failure state and which team has the authority to retry or repair it.
Reach a conditional assessment, then seek discriminating evidence
A comparison should end with a judgment about what would change the answer. At present, neither candidate is proven to meet the pilot objective. A reasonable conditional assessment is:
Keep both options viable. Prefer direct delivery for the pilot if reporting already has an appropriate API, intake can durably retain pending work, and joint tests show acceptable recovery and customer-visible latency within the confirmed limits. Prefer the asynchronous channel if an approved, supportable channel is available and its separation of intake from reporting materially improves burst handling or recovery without unacceptable delivery and operating cost.
This is deliberately not the ADR decision. It says which evidence has decision value. The Tech Lead can coordinate a short validation plan with named owners:
- Product and customer integration: confirm what counts as “submitted,” the denominator and measurement window, and the query that proves searchability.
- Intake and reporting: demonstrate a correlated invoice journey under normal load, a representative burst, a reporting outage, a process crash, and a duplicate delivery.
- Security and privacy: review service permissions, tenant checks, retained data, replay access, and audit evidence for each candidate.
- Platform and team leads: confirm channel availability, service limits, cost ranges, implementation dependencies, and who will support each component.
If a scheduled import is found during discovery, evaluate it against these same gates and scenarios rather than excluding it because it is batch-based. Its schedule might make the 15-minute objective implausible—or existing capacity and a sufficiently frequent, reliable import might make it viable. Evidence decides.
Conclusion
Architecture evaluation is not a vote for a pattern. For this handoff, compare the complete path from submission to searchability, distinguish non-negotiable gates from tradeable benefits, and expose where each design stores work, fails, recovers, and requires human ownership. Both candidates need proof under bursts, outages, duplicates, and tenant-boundary checks; both carry delivery and operating costs.
The next lesson will turn the validated comparison into an ADR: a clear decision, its rationale, the trade-offs accepted, and the consequences the teams must own.