Event-Driven Architecture: Patterns That Scale and Anti-Patterns That Burn

# Event-Driven Architecture: Patterns That Scale and Anti-Patterns That Burn

I've watched teams pull the trigger on event-driven architecture for the wrong reasons more times than I've watched it succeed. "We need Kafka because our microservices should be loosely coupled!" — said every team that then spent six months debugging message ordering and duplicate events.

The truth is more nuanced. Event-driven architecture is a power tool, not a solution. Used well, it delivers decoupling, resilience, and auditability that request-response systems can't touch. Used poorly, it creates an invisible dependency graph where a single malformed event cascades into four different service outages before anyone notices.

This post covers the patterns I've seen work at scale, the anti-patterns that keep burning teams, and the concrete code and infra decisions that separate the two.

The One Decision That Determines Everything

Before you pick Kafka, RabbitMQ, or the kitchen sink, you need to answer one question:

The answer dictates your entire architecture. And the answer is almost never "exactly-once" even though everyone wants it.

Here's the uncomfortable truth: exactly-once at the broker level doesn't mean exactly-once in your system. Kafka's exactly-once semantics (EOS) guarantees that a producer won't duplicate sends and a consumer won't duplicate reads within a single transaction. But your downstream database call? Your external API webhook? Those can still fire twice if the consumer crashes after the commit but before the side effect.

For 95% of systems, at-least-once + idempotent consumers is the right choice. Accept duplicates at the broker level, eliminate them at the business level.

Pattern 1: The Transactional Outbox

This is the most important pattern in event-driven architecture, and almost nobody implements it on their first try.

The problem: you write to your database and publish an event in a single request. The database write succeeds but the publish fails. Now your data is inconsistent — the order is saved but the billing service never heard about it.

The solution: write the event to an outbox table in the same database transaction as your business data. A separate process reads the outbox and publishes events reliably.

// ❌ The naive approach that WILL lose events
async function createOrder(input: CreateOrderInput): Promise<void> {
  const order = await db.query(
    "INSERT INTO orders (user_id, total, status) VALUES ($1, $2, 'pending') RETURNING *",
    [input.userId, input.total]
  );

  // If this fails, the order is saved but the event is lost
  await kafka.produce("orders.created", {
    orderId: order.rows[0].id,
    userId: input.userId,
    total: input.total,
  });
}

// ✅ The transactional outbox pattern
async function createOrder(input: CreateOrderInput): Promise<void> {
  await db.transaction(async (tx) => {
    const order = await tx.query(
      "INSERT INTO orders (user_id, total, status) VALUES ($1, $2, 'pending') RETURNING *",
      [input.userId, input.total]
    );

    // Write to outbox in the SAME transaction
    await tx.query(
      `INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
       VALUES ('order', $1, 'order.created', $2)`,
      [
        order.rows[0].id,
        JSON.stringify({
          orderId: order.rows[0].id,
          userId: input.userId,
          total: input.total,
        }),
      ]
    );
  });
  // Transaction commits → outbox row is persisted alongside the order
}

// The outbox publisher — runs as a background worker
async function pollOutbox(broker: KafkaProducer): Promise<void> {
  while (true) {
    const messages = await db.query(
      `SELECT id, aggregate_type, aggregate_id, event_type, payload
       FROM outbox
       WHERE processed_at IS NULL
       ORDER BY created_at ASC
       LIMIT 100
       FOR UPDATE SKIP LOCKED`
    );

    for (const msg of messages.rows) {
      try {
        await broker.produce(msg.event_type, JSON.parse(msg.payload), {
          key: msg.aggregate_id,
        });

        await db.query(
          "UPDATE outbox SET processed_at = NOW() WHERE id = $1",
          [msg.id]
        );
      } catch (err) {
        console.error(`Failed to publish outbox event ${msg.id}:`, err);
        // Don't crash — retry on next poll
      }
    }

    await sleep(1000); // Poll interval
  }
}

• FOR UPDATE SKIP LOCKED — multiple outbox workers don't stomp on each other
• ORDER BY created_at ASC — preserve event ordering
• processed_at — allows monitoring of outbox lag and dead-letter visibility

The transactional outbox gives you at-least-once delivery from the database side. Pair it with idempotent consumers and you've solved the hardest part of event-driven reliability.

Pattern 2: Idempotent Consumers

If your broker guarantees at-least-once delivery (and it should), your consumers must handle duplicates. The mechanism is simple: deduplicate by event ID.

// ✅ Idempotent consumer with a dedup table
async function handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
  await db.transaction(async (tx) => {
    // Check dedup first — same transaction as the business logic
    const existing = await tx.query(
      "SELECT 1 FROM processed_events WHERE event_id = $1",
      [event.id]
    );

    if (existing.rows.length > 0) {
      console.log(`Skipping duplicate event ${event.id}`);
      return;
    }

    // Business logic
    await tx.query(
      "INSERT INTO invoices (order_id, user_id, amount, status) VALUES ($1, $2, $3, 'pending')",
      [event.orderId, event.userId, event.total]
    );

    // Record the event ID
    await tx.query(
      "INSERT INTO processed_events (event_id, processed_at) VALUES ($1, NOW())",
      [event.id]
    );
  });
}

-- Cleanup processed events older than 7 days
DELETE FROM processed_events WHERE processed_at < NOW() - INTERVAL '7 days';

Anti-Pattern 1: The God Event

The most common mistake I see: one massive event type that carries everything.

// ❌ God Event — everything in one message
interface OrderUpdatedEvent {
  orderId: string;
  userId: string;
  email: string;
  shippingAddress: Address;
  billingAddress: Address;
  items: OrderItem[];
  totals: {
    subtotal: number;
    tax: number;
    shipping: number;
    discount: number;
    grandTotal: number;
  };
  paymentMethod: string;
  paymentStatus: string;
  fulfillmentStatus: string;
  notes: string;
  createdAt: string;
  updatedAt: string;
  // ... 40 more fields
}

1. Schema coupling — Every service that subscribes to order.updated receives the full payload, even if it only needs orderId + status. A change to billing logic now forces a schema change across 8 downstream services.
2. No semantic meaning — Is this event about "payment completed" or "shipping address changed"? You can't tell without comparing fields. Downstream services end up writing comparison logic (if old.status !== new.status) instead of responding to named events.
3. Data bloat — Events accumulate unnecessary data. Each event sat in Kafka for 7 days at 50KB instead of 500 bytes. That's real storage and network cost.

// ✅ Specific events for specific state changes
interface OrderPlaced {
  orderId: string;
  userId: string;
  items: { sku: string; quantity: number; price: number }[];
  total: number;
}

interface PaymentCompleted {
  orderId: string;
  paymentId: string;
  amount: number;
  method: "card" | "invoice" | "swish";
}

interface OrderShipped {
  orderId: string;
  trackingNumber: string;
  carrier: "postnord" | "dhl" | "bring";
}

interface OrderDelivered {
  orderId: string;
  confirmedAt: string;
}
``$$

The rule: **emit an event for each meaningful business-state transition, not for every attribute change.** If you can't name what happened in past tense (OrderPlaced, PaymentCompleted), your event is too vague.

## Anti-Pattern 2: The Implicit Dependency Graph

I've debugged outages where Service A fails, which causes a backlog in its event queue, which causes Service B to not receive events, which causes Service C to fall behind by 4 hours. Nobody noticed until the customer-facing dashboard started showing stale data.

The problem? **Event-driven architecture hides dependencies.** In a REST system, every API call is an explicit dependency. In an event system, a consumer that processes `order.created` creates an implicit dependency on the producer of that event. And when there are 15 event types flowing through 8 services, nobody can trace the dependency graph.

**Mitigation strategies:**

1. **Document your event contracts in a schema registry** — Confluent Schema Registry, or even a simple `events.ts` file that's shared across services. Every event type gets a version number.

2. **Define SLAs for every consumer** — "The invoicing service must process `order.created` events within 30 seconds at P99." This tells you when to page.

3. **Monitor consumer lag as a first-class metric** — Not just Kafka broker metrics. Per-consumer-Group, per-topic lag with alerting thresholds.

// Consumer health check — export as Prometheus metrics

export function trackConsumerLag(consumerGroup: string, lag: number): void {

consumerLagGauge.labels({ consumer_group: consumerGroup }).set(lag);

if (lag > 10_000) {

// 10 seconds behind

alertManager.sendWarning({

title: Consumer lag high: ${consumerGroup},

description: Lag is ${lag} messages (≈${msToHumanReadable(lag * avgProcessingTime)}),

severity: "warning",

});

}

if (lag > 100_000) {

alertManager.sendCritical({

title: Consumer critical: ${consumerGroup},

severity: "critical",

});

}

}

`$

async function processOrderEvent(event: OrderEvent): Promise<void> {
  const MAX_RETRIES = 3;
  const retryCount = event.metadata?.retryCount ?? 0;

  try {
    await handleOrder(event);
  } catch (err) {
    if (retryCount < MAX_RETRIES) {
      // Retry with exponential backoff
      await publishDelayed(event, Math.pow(2, retryCount) * 1000, {
        retryCount: retryCount + 1,
        lastError: (err as Error).message,
      });
    } else {
      // Send to DLQ
      await dlqProducer.produce("orders.dlq", {
        originalEvent: event,
        error: (err as Error).message,
        stack: (err as Error).stack,
        failedAt: new Date().toISOString(),
        consumerGroup: "order-processor",
      });
    }
  }
}
``$

**The DLQ is not a trash can.** A DLQ without monitoring is just deferred pain. You need:

- **Alerting on DLQ writes** — If events land in the DLQ, someone needs to investigate.
- **A replay mechanism** — The ability to fix the issue and replay DLQ'ed events back into the main topic.
- **Categorisation** — Separate transient errors (retry) from permanent errors (schema violations, invalid data).

At Rrezvin, we route DLQ events into a Slack channel. Every Monday, we review the batch and either replay or archive each one.

## Anti-Pattern 3: Events as RPC

This is the most insidious anti-pattern on the list. It happens when teams use a message broker as an async RPC mechanism.

$$

Event sourcing stores state as a sequence of events. Current state is derived by replaying them. It's powerful, but it's also the most over-engineered solution to problems that a updated_at column would solve.

// Minimal event-sourced aggregate
type AccountEvent =
  | { type: "AccountOpened"; accountId: string; owner: string; initialDeposit: number }
  | { type: "DepositMade"; accountId: string; amount: number }
  | { type: "WithdrawalMade"; accountId: string; amount: number }
  | { type: "AccountFrozen"; accountId: string; reason: string };

class BankAccount {
  private balance: number = 0;
  private isFrozen: boolean = false;
  private version: number = 0;

  // Rebuild state from event stream
  static loadFromHistory(events: AccountEvent[]): BankAccount {
    const account = new BankAccount();
    for (const event of events) {
      account.apply(event);
    }
    return account;
  }

  private apply(event: AccountEvent): void {
    switch (event.type) {
      case "AccountOpened":
        this.balance = event.initialDeposit;
        break;
      case "DepositMade":
        this.balance += event.amount;
        break;
      case "WithdrawalMade":
        if (!this.isFrozen) this.balance -= event.amount;
        break;
      case "AccountFrozen":
        this.isFrozen = true;
        break;
    }
    this.version++;
  }

  withdraw(amount: number): AccountEvent {
    if (this.isFrozen) throw new Error("Account frozen");
    if (this.balance < amount) throw new Error("Insufficient funds");
    return { type: "WithdrawalMade", accountId: "…", amount };
  }
}
``$$

**The hard parts nobody tells you:**
- **Schema evolution of events** — Old events in the store have old formats. Every time you replay, you must handle every schema version. This isn't hard with planning, but it's a tax you pay forever.
- **Snapshots** — Replaying 10 million events to rebuild state is slow. You need periodic snapshots (current state at version N) so you only replay from the last snapshot.
- **Deleting data** — Event sourcing makes it hard to delete data. GDPR right-to-erasure becomes an architectural challenge.

## Anti-Pattern 4: No Schema Validation

"Events are just JSON" — until someone adds a required field without telling anyone, or changes a field type, or the `customerId` field is sometimes a string and sometimes a number.

$$