Event-Driven Architecture on AWS: where the flow breaks
Services that started as independent modules quickly become overgrown with synchronous calls: order checkout triggers payment, inventory, notifications, and analytics right in the request handler.
One failed receiver stops the entire chain, and under peak load some events do not arrive — the publisher gets a timeout, retries the call, and multiplies pressure on an already overloaded service.
Investigating such an incident stretches over days because there is no single bus and no flow schema. It is unclear where an event was lost, whether the receiver processed it, and in what order retries occurred.
Service coupling grows quadratically: each new consumer requires a separate integration with each publisher, and a contract change turns into coordination between teams.
Event-driven architecture reverses this scheme: the publisher publishes an event once, and the bus handles routing.
In EventBridge, selection rules are built by event source, type, and content — receivers do not know about each other, and a new consumer connects with a separate rule, without changes to the publisher's code.
Event loss moves from the "sometimes disappears" class into managed failures: the bus buffers the stream at peak, retries delivery on a schedule, and routes unprocessed messages to a separate queue for review.
An event archive allows replaying the stream for the needed period, and a schema registry fixes the message format — an incompatible change is caught at publication, not at the moment of receiver failure.
What event exchange delivers: measurable effects
The effect of event exchange is checked not by a presentation but by a checklist: time to react to a business event, number of changes in the source service when connecting a new consumer, share of failures that did not reach the user, and time to restore the stream after a failure.
Time to react to a business event instead of overnight export
The classic scheme is a scheduled daily export: an order goes to the warehouse when the batch job runs, and the status in the customer portal updates by morning.
Event routing in Amazon EventBridge cuts the gap to seconds: publication to the bus and delivery to the subscriber happen in the same cycle, and filtering rules cut excess before it reaches the consumer queue.
We check the effect by the time from customer action to a change in adjacent systems — it drops from hours to seconds.
Loose coupling: adding a consumer without changing the source
The source publishes an event and does not know who reads it. A new consumer — analytics, anti-fraud, notifications — connects via a subscription, not by modifying the source service.
- The event contract describes type, version, and payload.
- Routing and filters are configured on the bus side.
- Undelivered messages go to a retry queue instead of being silently lost.
Along with batch exports, a separate class of failures goes away — data desync between systems: an event is either delivered or recorded as an error with retry.
Industry reviews call broken exchange between systems one of the most expensive causes of downtime; we measure it not in money but in the share of operations stalled within an hour and the time to restore the flow.
Routing architecture: options and limitations
The choice of routing mode determines what happens to an event when the receiver is unavailable or rejected the message. Below are three working schemes and their behavior under failures.
Selection rules and attribute-based filtering
Routing rules are set declaratively — by event source and attributes. Filtering on the bus side cuts excess traffic before invoking the receiver, so the subscriber gets only its slice, not the whole stream.
Subscription modes differ: targeted delivery, broadcast, and competing reads when several instances of a service share the stream.
What to do with a message that could not be delivered
A delivery guarantee does not mean "we always deliver" — it means the fate of every message is known. After retries are exhausted, the event goes to a dead-letter queue, from where it is replayed manually or the failure cause is analyzed. Without this loop, receiver failure becomes silent data loss.
| Option | When it fits | Limitations |
|---|---|---|
| Direct point-to-point delivery | One producer, one consumer, moderate flow | No buffer when the receiver is unavailable; a new consumer requires changes at the sender |
| Central bus with selection rules | Many consumers, each with its own event slice | Requires event schema discipline; an error in the rule silently cuts off the subscriber |
| Hybrid: bus plus dead-letter queue | Scenarios where event loss is unacceptable | More components: need a process for queue review and replay |
Implementation stages: from flow diagram to handover
We start the transition to event-driven architecture on AWS not with bus setup but with flow inventory: first we capture the current state, then design the event schema, and only then enable routing in production.
A skipped audit costs more than the migration itself — hidden links between services surface under load when rollback is expensive.
What a flow audit looks like
This is not a developer survey but tracing: we look at which service publishes an event, who reads it, and what happens on redelivery.
The output is a flow map with the owner, frequency, and criticality of each event plus a list of "dead" subscriptions and duplicate routes that we disable before migration.
- Event flow inventory — a map of sources, subscribers, and owners, domain boundaries.
- Event schema design — a single contract, required fields, versioning, and backward compatibility.
- Bus and routing rules setup — filters, target queues, retries, dead-letter queue.
- Access separation — separate roles for publish and subscribe, environment isolation, no broad permissions.
- Failure and retry testing — shut down the subscriber, break the connection, duplicate the event, and see how the system recovers.
- Handover with training — documentation, incident review, on-call shift learns to read the flow map.
Acceptance checks before launch
Launch is ready when an event retry does not create a second record, undelivered messages return to work, and a new service connects to the bus without changes in other teams' repositories. Separately, we run node failure: the system should degrade predictably, not silently lose events.
Configurations, event schemas, and handover procedures
The package left with the customer after launch is working documentation, not a presentation: the on-call shift uses it to investigate an incident without the integration author.
In a bus based on Amazon EventBridge, configuration is stored separately from service code, so routing and permissions can be changed without releasing the consumers themselves.
Access policies and service roles
- Bus configuration: routing rules, filters by event type and source, separate channels for critical and background flows.
- Event schema description: field composition, contract version, and backward compatibility rule — adding a field does not break existing consumers.
- Access policies: which service role publishes and reads what, channel restrictions, environment separation, and separate permissions for debug subscriptions.
Documentation and support procedures
- Flow map: who consumes whose events, where buffering sits, and at which point retry processing engages.
- Retry processing procedure: number of attempts, interval between them, conditions for moving an event to the dead-letter queue, and manual restart order.
- Failure investigation order and instructions for the on-call shift: how to disable a consumer and replay the flow without losing events.
- Integration documentation: how to connect a new service and which checks to pass before production.
Case study: how order flow between systems was smoothed
Before: manual reconciliation and nightly exports
The online store, warehouse, and billing lived on different platforms, and exchange went through nightly CSV exports.
An order could hang for hours: if the file did not assemble, the discrepancy was learned from the customer, not the system — support started investigating before engineers.
Losses reached up to 4% of items per week, and each discrepancy was reconciled manually across two tables. During a sales peak, exports could not keep up, and some orders waited for the next window — the delay depended on the calendar, not on service load.
After: metrics and remaining limitations
We replaced batch exchange with an event bus: the system publishes a fact — "order created", "payment confirmed", "shipment assembled" — and subscribers receive it immediately.
The bus became AWS EventBridge with routing rules and a dead-letter queue: an event either reaches the recipient or goes to investigation, but does not disappear silently.
Metrics after: lost event share — 0 per quarter vs 4%; order reaction time — minutes instead of hours; manual investigations — 2 per week instead of 30, and those due to dirty source data, not delivery.
The limitations remained honest. Global event order is not guaranteed — handlers were made idempotent by order number, so a retry does not create a duplicate. The accounting system, which does not publish changes itself, is still polled on a schedule.
The event schema is versioned: without this, a format change breaks subscribers. We described a similar project publicly — the same pitfalls exactly.
How does event exchange differ from batch export?
Batch export works on a schedule: the source once an hour or once a day forms a data snapshot and transmits it whole.
Event exchange is structured differently — the source publishes the fact of change at the moment it occurs, and recipients fetch it themselves, without polling and without a shared schedule.
The difference in architecture immediately shows in three places: reaction latency, cost of schema change, and behavior on receiver failure.
Reaction latency in batch mode equals the export period: a change that arrives right after the run waits for the next one.
Data marts and reports live with a lag of hours, and scenarios where fast reaction matters — recalculating stock, notifications, payment verification — have to be covered by separate synchronous logic on top.
Real-time mode cuts latency to seconds: event routing between services is convenient to keep on a managed bus like AWS EventBridge, where subscription rules are described declaratively, not by code on each consumer side.
A data schema change affects both modes, but differently. In batch, it is enough to fix one query — all consumers see the new field set simultaneously, but they are tightly coupled: the change breaks reporting until it is coordinated with everyone.
In event exchange, the event contract becomes a public interface: fields are added with versioning, old subscribers continue working, but each handler must be checked for compatibility.
Receiver failure is the most telling difference. Batch export is idempotent by nature: if intake fails, the next run brings the same snapshot.
An event stream gives no such luxury — you need idempotent handlers, redeliveries with growing delay, and a dead-letter queue, otherwise events are silently lost.
Batch mode remains a reasonable choice where reaction is not critical: nightly reconciliations, heavy analytics, regular exports to a warehouse, exchange with systems that fundamentally cannot accept a stream. It is simpler to operate and more predictable in load.
Event exchange wins where a business process runs in minutes, not days, and where a lost change costs more than infrastructure complexity.
Technical concerns: reliability, retries, access
An event bus delivers a message at least once — retry is the price of resilience to receiver failures, not a defect. Below are five questions teams ask us first.
What happens if the receiver is unavailable?
The event is retried with a growing delay until the receiver comes up, then goes to the dead-letter queue. From there it is returned manually, and queue growth is visible on the monitoring dashboard.
How to avoid processing an event twice?
Handlers are made idempotent by event key: a retry with the same key is recognized and does not change state. The log of applied keys is stored where a status change or payment matters.
Can a running system be connected?
Yes, with a transition layer: the bus receives events in parallel with the current flow, and handlers run in observation mode. After several days without discrepancies, we disable the old flow.
How to separate access to individual flows?
We attach permissions to bus rules and application roles: each service publishes and reads only its event types. Secrets are issued per service, not per team.
Who is responsible for failure investigation?
An archive with search by key and time allows finding a specific event and replaying it precisely. Latency and error-rate metrics are brought to one screen so the on-call sees deviation before the business; we handle incident investigation.
Let's discuss your event schema and come back with a plan
To come back with an integration assessment, not generic wording, we need your environment as it is: system composition, event format, and places where it currently breaks. Describe the task in free form — this is enough to start the review.
What to send for assessment:
- list of source systems and event consumers, including external services and queues;
- example of a real event or log export — format, attributes, daily volume;
- bottlenecks already noticeable: delays, duplicates, lost messages, manual failure investigation;
- delivery latency and processing order requirements for critical flows;
- filtering and routing rules, environment separation, and access between environments;
- bus support mode: who is on call and how we react to a rule failure or poisoned event.
In response you will get a flow diagram with failure points and buffers, a stage plan — from migrating the first routes to decommissioning the old ones and handing the bus over for support — and a timeline assessment.
Then the decision is yours: we are ready to start with the area where the effect is visible fastest.