Implementing Saga Pattern for Distributed Transactions

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: order-saga-events
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: "604800000"
    min.insync.replicas: "2"
    cleanup.policy: compact

Producer must include enable.idempotence=True — Kafka guarantees exactly-once write on retry. This is critical for Saga, where duplicate events lead to repeated execution of steps.

How to implement handler idempotency?

Kafka at-least-once means: an event may arrive twice. Each handler in a Saga must be idempotent — re-execution with the same event ID does not change state. Pattern: check if event was already processed → execute local transaction → publish next event.

async def handle_order_created(event: OrderCreatedEvent):
    if await redis.get(f"processed:payment:{event.saga_id}"):
        logger.info(f"Event {event.saga_id} already processed")
        return
    async with db.transaction():
        payment = await payment_service.charge(
            customer_id=event.customer_id,
            amount=event.total_amount,
            idempotency_key=event.saga_id
        )
        await db.execute(
            "INSERT INTO saga_steps (saga_id, step, status) VALUES ($1, $2, $3)",
            event.saga_id,
            "payment",
            "completed"
        )
        await redis.setex(f"processed:payment:{event.saga_id}", 86400, "1")
        await kafka_producer.produce("payment.completed", PaymentCompletedEvent(...))

If step N fails, steps 1..N-1 must be undone. Compensating transactions must also be idempotent — use an idempotency_key with a refund- prefix.

Why orchestration is preferable?

Temporal is a workflow engine with persistent state, automatic retry, and full history of each execution. When a worker crashes, the process resumes from the stop point. Infrastructure: Temporal Server (Docker or Kubernetes), PostgreSQL for persistence.

@workflow.defn
class OrderSagaWorkflow:
    @workflow.run
    async def run(self, order: OrderRequest) -> OrderResult:
        try:
            payment = await workflow.execute_activity(
                process_payment,
                order.payment_details,
                start_to_close_timeout=timedelta(seconds=30))
        except ActivityError:
            raise
        try:
            reservation = await workflow.execute_activity(
                reserve_inventory,
                order.items,
                start_to_close_timeout=timedelta(seconds=30))
        except ActivityError:
            await workflow.execute_activity(refund_payment, payment.payment_id)
            raise
        try:
            shipment = await workflow.execute_activity(
                create_shipment,
                order,
                start_to_close_timeout=timedelta(seconds=60))
        except ActivityError:
            await workflow.execute_activity(cancel_reservation, reservation.id)
            await workflow.execute_activity(refund_payment, payment.payment_id)
            raise
        return OrderResult(order_id=order.id, payment_id=payment.payment_id, shipment_id=shipment.id)

Monitoring Saga: what to watch

DLQ (Dead Letter Queue) is critical. Events not processed after 3 retries go to DLQ. They need to be handled automatically or manually.

How to implement Saga pattern: step-by-step guide

1. Architecture audit: identify failure points in current microservices.
2. Saga design: describe steps, compensations, event schema.
3. Infrastructure deployment: set up Kafka/Temporal cluster, topics, persistence.
4. Handler implementation: implement idempotency, compensating transactions, DLQ.
5. Monitoring and alerts: set up Prometheus/Grafana dashboards, alerts on compensations and DLQ.

What is included in our work

• Architecture audit: analysis of current microservices and failure points.
• Saga design: defining steps, compensations, event schema.
• Infrastructure deployment: Kafka/Temporal cluster, topic configuration, persistence setup.
• Handler implementation: idempotency, compensating transactions, DLQ.
• Monitoring and alerts: Prometheus/Grafana dashboards, alerts on compensations and DLQ.
• Documentation and training: runbook, postmortem templates, team training.

Implementation process

Cost is calculated individually after a microservice architecture audit. Contact us for an assessment of your project.

Order turnkey Saga implementation — get a stable distributed system.