Event Sourcing at Scale: The Parts Nobody Talks About

# Event Sourcing at Scale: The Parts Nobody Talks About

Last year, our finance team flagged a discrepancy: $43,000 in subscription charges that didn't match customer invoices. The billing service's current state showed everything was fine—the database rows had been updated correctly for months. Everyone pointed fingers at the payment processor.

Then someone remembered we'd built the billing system with event sourcing. We pulled the event stream for those accounts and replayed six months of billing events. Twenty minutes later, we found a one-line bug in a price-migration script that had been silently overwriting plan IDs for three weeks. The fix took an hour. Without that event log, we'd still be arguing with Stripe.

That's the promise of event sourcing: an append-only ledger of everything that happened, giving you an audit trail, time-travel debugging, and the ability to rebuild state from scratch. Conferences love it. Blog posts love it. What they don't love is the morning you wake up and your read-model projection is 14 hours behind, your snapshots are silently corrupt, and your "simple" event schema has 17 versions with three incompatible migrations.

Here's what actually happens when you run event sourcing at scale.

---

The Read Model Lag Death Spiral

The standard advice: "Projections are eventually consistent. Just accept a few seconds of lag." This works until it doesn't.

We had a checkout service that projected order events into a read model used by the customer-facing dashboard. During a Black Friday sale, order volume 4x'd. The projection worker—a single-threaded Go consumer pulling from Kafka—started falling behind. Five seconds became thirty seconds. Thirty seconds became two minutes. Customers refreshed their dashboard, saw no order, and placed duplicate orders. Support tickets spiked.

The root cause wasn't throughput. It was a projection that joined across three event types—OrderPlaced, PaymentAuthorized, InventoryReserved—and blocked on a missing PaymentAuthorized event because the payment processor was slow under load. The projection worker sat idle waiting for one event type while thousands of other events stacked up behind it.

The fix was three-fold:

First, we split the monolithic projection into independent projections per use case. The "show me my orders" view doesn't need payment status. The "show me my payment" view doesn't need inventory data. Each projection processes only the events it needs.

Second, we added a dead-letter timeout. If a projection can't resolve a join within 30 seconds, it writes an "incomplete" state and moves on. The dashboard shows "Processing..." instead of nothing. Better than a blank page.

Third, and most importantly, we built a synchronous projection path for the critical user-facing query: "did my order go through?" After appending the OrderPlaced event, the write side synchronously updates a thin cache (Redis, TTL 5 minutes) with the essential fields. The dashboard checks the cache first, then falls back to the projection.

// Write side: after appending event, update cache synchronously
func (s *OrderService) PlaceOrder(ctx context.Context, cmd PlaceOrderCommand) (*Order, error) {
    event := OrderPlaced{
        OrderID:   xid.New().String(),
        AccountID: cmd.AccountID,
        Items:     cmd.Items,
        PlacedAt:  time.Now(),
    }

    if err := s.eventStore.Append(ctx, "order", event.OrderID, []Event{event}); err != nil {
        return nil, fmt.Errorf("append event: %w", err)
    }

    // Synchronous cache for immediate read-your-writes
    cacheKey := fmt.Sprintf("order:latest:%s", event.OrderID)
    s.cache.Set(ctx, cacheKey, OrderSummary{
        ID:     event.OrderID,
        Status: "placed",
        Total:  calculateTotal(event.Items),
    }, 5*time.Minute)

    return &Order{ID: event.OrderID, Status: "placed"}, nil
}

The lesson: eventual consistency is a spectrum, not a binary. Identify the queries where "eventually" actually means "right now," and build synchronous read paths for those. For everything else, staleness is acceptable—as long as users know it.

---

Snapshots Are a Feature You'll Build Twice

Every event sourcing guide tells you: "Take periodic snapshots to avoid replaying the entire event stream." What they don't tell you is that snapshot corruption is a ticking time bomb.

Our first snapshot implementation was straightforward: every 1,000 events, serialize the aggregate state as JSON and store it alongside the event stream. Restoring meant loading the latest snapshot, then replaying events since the snapshot's sequence number. It worked for six months.

Then we deployed a schema migration that added a TaxRate field to the InvoiceGenerated event. Old snapshots had aggregate state without tax information. New events referenced tax rates. When a projection replayed events after restoring an old snapshot, the aggregate state was missing the TaxRate field, and the projection logic panicked on a nil pointer.

We'd built snapshots assuming the serialized state would always be compatible. It wasn't.

The rebuild:

type Snapshot struct {
    AggregateID   string    `json:"aggregate_id"`
    AggregateType string    `json:"aggregate_type"`
    Version       int       `json:"version"`       // schema version
    SequenceNum   int64     `json:"sequence_num"`   // event position
    State         []byte    `json:"state"`          // protobuf-encoded, versioned
    Checksum      string    `json:"checksum"`       // SHA-256 of State
    CreatedAt     time.Time `json:"created_at"`
}

func (r *Repository) Load(ctx context.Context, aggregateID string) (*Aggregate, error) {
    snap, err := r.snapshotStore.Latest(ctx, aggregateID)
    if err != nil && !errors.Is(err, ErrNoSnapshot) {
        return nil, err
    }

    var agg *Aggregate
    var startSeq int64

    if snap != nil {
        // Validate snapshot integrity before trusting it
        if !snap.ValidateChecksum() {
            r.metrics.SnapshotCorruption.Inc()
            log.Warn("snapshot checksum mismatch, discarding", "id", aggregateID)
            snap = nil // fall through to full replay
        } else {
            agg, err = r.hydrateFromSnapshot(snap)
            if err != nil {
                log.Warn("snapshot hydration failed", "id", aggregateID, "err", err)
                snap = nil // schema changed, fall back to replay
            }
            startSeq = snap.SequenceNum + 1
        }
    }

    if agg == nil {
        agg = NewAggregate(aggregateID)
        startSeq = 0
    }

    events, err := r.eventStore.Read(ctx, aggregateID, startSeq)
    if err != nil {
        return nil, err
    }

    for _, evt := range events {
        if err := agg.Apply(evt); err != nil {
            return nil, fmt.Errorf("apply event %d: %w", evt.SequenceNum, err)
        }
    }

    return agg, nil
}

The key changes:

1. Version your snapshots. The serialized state blob carries a schema version so hydration logic can detect incompatibility.
2. Checksum everything. A SHA-256 of the serialized state catches bit-rot and storage-layer corruption before it poisons your aggregate.
3. Fall back gracefully. If a snapshot is corrupt or incompatible, discard it and replay from event zero. Yes, it's slower. It's also correct.

We also started running a nightly integrity job that loads every snapshot, validates checksums, replays the subsequent events, and compares the result against a fresh full replay. It caught two corrupted snapshots in the first week. The storage layer (an S3-compatible object store) had a silent data corruption issue we'd never have found otherwise.

---

Event Schema Evolution: Your Events Will Break

The most dangerous lie in event sourcing is: "Events are immutable, so schema evolution is simple—just add fields." Events are immutable. Your understanding of events is not.

Here's a real migration we had to handle:

v1: OrderPlaced { order_id, customer_id, items[] } v2: Added currency field. Fine, additive change. v3: Changed items[] from { sku, quantity } to { sku, quantity, unit_price }. Now v1 events replay with missing unit_price and our revenue projections are wrong. v4: Split customer_id into customer_id (UUID) and customer_external_id (legacy system). Old events only have one field. Which one was it?

The solution that actually works: explicit upcasters. Every event carries a schema version. Before an event hits any projection or aggregate, it passes through a chain of upcasters that transform it to the current version.

// Each upcaster transforms one version to the next.
// The chain runs v1→v2, v2→v3, v3→v4, etc.

interface Upcaster {
  sourceVersion: number;
  targetVersion: number;
  upcast(event: Record<string, unknown>): Record<string, unknown>;
}

const upcasters: Upcaster[] = [
  {
    sourceVersion: 1,
    targetVersion: 2,
    upcast: (e) => ({ ...e, currency: e.currency ?? 'USD' }),
  },
  {
    sourceVersion: 2,
    targetVersion: 3,
    upcast: (e) => ({
      ...e,
      items: (e.items as any[]).map((item: any) => ({
        ...item,
        unit_price: item.unit_price ?? 0, // v1 events had no pricing
      })),
    }),
  },
  {
    sourceVersion: 3,
    targetVersion: 4,
    upcast: (e) => {
      const id = e.customer_id as string;
      const isUUID = /^[0-9a-f-]{36}$/i.test(id);
      return {
        ...e,
        customer_id: isUUID ? id : '00000000-0000-0000-0000-000000000000',
        customer_external_id: isUUID ? null : id,
      };
    },
  },
];

function upcastToLatest(
  event: { version: number; payload: Record<string, unknown> }
): Record<string, unknown> {
  let payload = { ...event.payload };
  let version = event.version;

  while (version < CURRENT_VERSION) {
    const upcaster = upcasters.find(
      (u) => u.sourceVersion === version
    );
    if (!upcaster) {
      throw new Error(`No upcaster from version ${version}`);
    }
    payload = upcaster.upcast(payload);
    version = upcaster.targetVersion;
  }

  return payload;
}

Three rules we now enforce:

1. Never mutate an upcaster. Once deployed, it's immutable—just like the events it transforms. If you got the migration wrong, write a new upcaster that corrects it.
2. Test upcasters against production event samples. We snapshot 10,000 real events from production and run every new upcaster against them in CI. A migration that works on synthetic data fails on real data surprisingly often.
3. Version your read models, not just your events. A projection built for v4 events should refuse to consume v3 events. If the upcaster chain can't produce v4, fail loudly—don't silently produce wrong results.

---

Projections That Outgrew Their Database

We started with PostgreSQL for everything: event store, read models, projections. It worked until our largest projection table hit 400 million rows.

The problem wasn't storage—PostgreSQL handled that fine. It was the shape of our queries. The dashboard projection, which denormalized order events into a flat table for fast querying, needed to support these access patterns:

The aggregate queries were killing us. We'd built a normalized projection that was great for CRUD and terrible for analytics. Adding more indexes made writes slower without fixing the fundamental problem: one projection was trying to serve too many access patterns.

The solution: projection per query pattern.

We split the single Order read model into three:

1. PostgreSQL — transactional queries: "show me my orders," "get order by ID." Normalized, indexed for PK lookups.
2. ClickHouse — analytical queries: revenue by category, cohort retention, monthly trends. Columnar storage, purpose-built for aggregation.
3. Redis — hot-path cache: "did this order go through?" TTL'd, eventually invalidated by the projection.

Each projection subscribes to the same event stream but builds a different shape optimized for its queries:

// Each projection consumes the same events, produces different shapes
func (p *AnalyticsProjection) Handle(ctx context.Context, evt Event) error {
    switch e := evt.(type) {
    case OrderPlaced:
        return p.ch.Insert(ctx, OrderPlacedRow{
            OrderID:   e.OrderID,
            AccountID: e.AccountID,
            Total:     calculateTotal(e.Items),
            Category:  categorize(e.Items),
            PlacedAt:  e.PlacedAt,
        })
    case OrderShipped:
        return p.ch.Update(ctx, OrderShippedRow{
            OrderID:   e.OrderID,
            ShippedAt: e.ShippedAt,
        })
    default:
        return nil // this projection ignores events it doesn't need
    }
}

The cost: more infrastructure, more moving parts, more failure modes. The benefit: queries that took 12 seconds now take 80 milliseconds. For a customer-facing analytics dashboard, that's the difference between usable and abandoned.

Don't do this on day one. Start with a single projection store. When—and only when—a specific query pattern becomes painful, spin out a dedicated projection. Premature optimization here means maintaining three projection workers, three connection pools, and three failure modes for a system with ten users.

---

When NOT to Use Event Sourcing

I've spent 2,000 words on how to make event sourcing work. Here's when you shouldn't use it at all:

CRUD with no audit requirement. If you're building a settings page, a blog, or an internal tool where nobody cares what the value was three weeks ago, you don't need event sourcing. A row in PostgreSQL with an updated_at column is fine. You're not building a bank. Teams smaller than five engineers. Event sourcing adds operational complexity: event store maintenance, projection monitoring, upcaster testing, snapshot integrity, replay procedures. A three-person team cannot afford this overhead while also shipping product features. When the domain model changes rapidly. If your product is in discovery mode and you're rethinking the data model every sprint, event sourcing will slow you down. Every schema change becomes an upcaster. Every deprecated field lives forever in the event stream. Wait until the domain stabilizes. When strong consistency is non-negotiable. If your use case requires ACID transactions across multiple aggregates—transferring money between two accounts, reserving inventory while charging a card—event sourcing with eventual consistency adds complexity without benefit. Use a relational database with proper transactions. Event sourcing can model these scenarios with sagas and compensating transactions, but the implementation complexity is rarely worth it.

The honest truth: 80% of applications are better served by a relational database and a transaction log than by full event sourcing. Use event sourcing when you need the audit trail, the time-travel debugging, or the ability to build multiple read models from the same truth. If you just want "microservices that communicate asynchronously," use a message queue.

---

What to Do Next

If you're considering event sourcing, start small:

1. Pick one bounded context. Don't event-source your entire system. Pick the domain where audit trails matter most—billing, compliance, inventory—and apply event sourcing there. The rest stays CRUD.

1. Build the replay from day one. The first feature you should build after AppendEvent is a full replay. If you can't rebuild every read model from scratch, you don't have event sourcing—you have a complicated message queue.

1. Monitor projection lag as an SLO. If your projections are 30 seconds behind, your users are seeing stale data. Track lag per projection, alert on it, and build dashboards. This is not optional.

1. Version your events from event zero, not event 1,000. Adding schema versioning to an existing event stream is a migration you don't want to do. version: 1 on every event from the start.

1. Test your upcasters against production data. Synthetic test data won't catch the edge cases. Snapshot real events and run migrations in CI.

Event sourcing is not a silver bullet. It's a tool that solves specific problems—auditability, temporal queries, complex read models—at the cost of significant operational complexity. Use it when the problems it solves are worth the problems it creates. For billing systems, compliance domains, and anything where "what happened and when?" matters more than "what's the current state?"—it's the best tool we have. For everything else, a last_updated_at column works just fine.

---

Have you run event sourcing in production? Hit hard-to-debug projection lag or schema migration nightmares? I'd love to hear about it—especially the parts the conference talks don't mention.