Scaling projections

A partitioned projection may also react to something delivered only live — a read model re-published by another projection, say — never persisted to the event store the bucket poll reads. See Delivery to a partitioned projection below for how those are handled.

Every event carries an event key — the value it is routed by. At write time a partition_key is stamped on it from a hash of that key into a fixed number of buckets (spring.ddd.partitioning.partitions, default 256 — see Configuration properties). For an ordinary domain event the key is its aggregate id — the field you mark with @AggregateId. Because the hash is a pure function of the key, every event with the same key always lands in the same bucket. So an aggregate’s whole event stream stays together in one bucket — which is what makes bucket-level parallelism safe: an aggregate’s events are never split across two instances, so per-aggregate order is preserved no matter which instance owns the bucket at the time.

Each application instance runs a lightweight coordinator that:

This all happens automatically once the annotation is present — there is nothing to wire up. Adding or removing instances triggers a rebalance on the next pass; only the buckets that actually changed owner move, everything else keeps draining undisturbed.

@Partitioned is guarded at startup, not at runtime — a projection that cannot be safely split fails loud with an actionable message instead of silently corrupting or duplicating data.

When a partitioned read model is fed by more than one aggregate — for example an AccountStatement row built from MoneyDeposited (keyed on accountId) and CustomerRenamed (keyed on customerId) — the two event families hash to different buckets on different instances. The framework runs such a projection in index mode: a single-owner indexer tails the event log and records, for each event, which read-model row(s) it updates, co-locating a row’s events in one bucket applied in total order.

What index mode buys you is scale-out for a cross-aggregate read model — the same horizontal scaling a native aggregate-keyed partitioned projection gets. The expensive per-row work — running the listeners and writing the read model — is sharded across buckets and applied in parallel by owners on different instances, instead of being pinned to a single owner. Only the cheap indexing step (key evaluation + a pointer write, no payload copy) is serial, so the part that actually bottlenecks a busy projection is the part that parallelizes — and you get this with total order per row and exactly-once apply preserved. Reach for it (it is automatic) when a cross-aggregate read model’s apply throughput on a single owner is the bottleneck.

Index mode is derived automatically from the read model’s key; there is no flag. The scale-out has a cost, paid only by index-mode projections (native aggregate-keyed projections pay none of it):

Disabling partitioning always runs the same projection correctly on a single owner in global order — the simplest choice, and the right one when a cross-aggregate read model is low-volume enough that scale-out is not needed.

A projection fed only by an upstream read model — see Projection chaining — has no event-store trigger of its own to partition by. It can still carry @Partitioned, but only in one narrow, provable shape: the downstream must be co-keyed with its upstream, meaning a change to an upstream row always re-derives the downstream row of the same id, so the two land in the same bucket by construction.

This is feed mode, the third partitioning mode alongside native and index. Instead of tailing the event store, the downstream tails a durable, partitioned feed of the upstream’s own read-model changes and re-derives its row from the upstream’s current state — cross-instance, without requiring the two projections' buckets to be owned by the same node.

A projection-fed target that is not co-keyed — one that resolves its id through a @ReadModelKey onto the upstream, rather than reading the upstream’s own @AggregateId directly — is admitted too, just not through this direct drain: the framework cannot prove from metadata alone that such a mapping lands in the same bucket as its upstream, so reading the upstream’s feed for this bucket the way a co-keyed edge does would risk a cross-node double-write. Instead it is routed through a per-edge index — see Re-key edges below.

A @Partitioned projection may also consume an upstream read model through a re-defining @ReadModelKey — one whose downstream id does not resolve to the upstream row’s own id, so the two land in different buckets, owned by different instances. This is a re-key edge: a cascade source, not a partition mode of its own — it composes on top of whichever primary mode the downstream already runs in (native, index, or feed), adding one more routed input alongside its event triggers and any co-keyed feed edges.

Because the affecting upstream row can live in any bucket, a re-key edge cannot be drained directly the way a co-keyed one is (Feed mode above). Instead, a stateless single-owner indexer — one per edge — reads the upstream’s whole read-model feed, re-keys each change onto the downstream id, and appends a routing pointer to the downstream’s own partition. The downstream’s own owner then applies that pointer in place, exactly once, in the same bucket its native events already use — so the row stays correctly bucketed and its feed stays coherent no matter how the upstream and downstream ids relate.

What a @ProjectionListener should do depends on what it consumes: a read model carries current state, so a listener consuming one should set the field it copies — totals.totalBalance = summary.balance; an event carries a delta, so a listener consuming one should fold it — totals.total += event.amount. Getting this backwards on a read model is a real mistake, not a style choice: publishCurrentState re-publishes the upstream’s whole current value on every live apply, not the change since the last one, so folding it re-adds the same current value again each time it is re-published. When the re-keyed relationship is genuinely 1:1 — an upstream row that always resolves to one unique downstream row — setting is fully correct, however many times the same value is re-delivered.

branchId fans many accounts into one BranchTotals row, and that shape is exactly what tempts a listener into the mistake:

The edge above is still delivered soundly — exactly once per observed change, routed to the correct row — but that does not make the aggregation correct; the framework logs a startup WARN naming the projection and the edge. To roll up, aggregate events, not read models: MoneyDeposited/MoneyWithdrawn are genuine deltas, so re-keying the event onto branchId and folding it there is the correct way to build a running total across many accounts — see Index mode (cross-aggregate read models) above, which already supports exactly this fan-in.

Composition is free. A downstream is not limited to one kind of input: the same BranchTotals row can simultaneously fold a re-keyed event (a correct running total), set a field from a co-keyed read model, and set another field from a re-key read-model edge like BranchTotalsProjection above — every contribution lands on that row, applied by the row’s single owner, whichever source it came from.

A re-key edge’s routing index is stateless: it resolves and routes a row’s current key, but it never proactively re-signals a row that a key change vacated. If branchId moves — an account transfers to a new branch, say — the new branch’s BranchTotals row is corrected on the very next routed change, but the old branch’s row is left holding stale numbers until the projection is rebuilt. The same is true after any declaration change: an edited @ReadModelKey, a listener added or removed, or a change to the read model’s shape or computation. None of these self-heal on the in-place path — nothing re-routes to a vacated row until the projection is rebuilt.

replay on a @Partitioned projection is not a single node’s job — the bucket owners across the cluster each re-drain their own buckets. One epoch, cleared together, covers everything the projection owns: its partition cursors, its change-feed counter and published frontier, its re-key routing indexes, and its cascade dedup markers. Nothing from the old epoch survives into the new one.

Before any of that happens the projection is quiesced — every bucket owner stops draining at its next pass — so no in-flight batch from before the reset can still be applying once the cursors and indexes are cleared. The quiesce is not instantaneous and is not meant to be: the projection stays paused for at least one lease duration (spring.ddd.cqrs.projection.lease-duration, one minute by default) before its epoch is reset, which is what guarantees that every owner — including one whose node has since gone away — has observed the pause and stopped. The pause lifts on its own once the reset has run; there is nothing to resume manually.

When a rebuild spans a cascade, the closure is released upstream first: a downstream stays paused until every upstream it consumes has had its own epoch reset. Otherwise the downstream would drain the upstream’s pre-reset change feed, whose sequence numbers the upstream is about to re-mint from the beginning, and the downstream’s cursor would sit permanently beyond everything the new epoch ever publishes. Budget for that ordering: the total wait before the last projection in a cascade is reset is roughly the depth of the closure times one lease duration plus one poll interval, since each level is only released after the level above it has been reset and that release is observed on the next poll.

The very first time a partitioned projection is driven it runs this same cycle by itself, without anyone calling replay: it records a horizon, pauses, resets its epoch and re-drains its history once. Read models are only ever removed at the end of a cycle that actually reset the epoch that produced them, so this one-off pass is what makes the completion prune safe rather than destructive. Expect one quiesce of at least a lease duration and one full re-drain the first time a projection is seen; afterwards the durable progress row exists and the projection catches up incrementally.

Side effects still matter during a rebuild. An @IgnoreOnReplay listener is suppressed for the event-sourced portion of the catch-up, exactly as it is on a non-partitioned rebuild — but a delivery fed from an upstream read model (feed mode or a re-key edge) is not part of that catch-up and is not suppressed, since it is not replaying stored history, it is reacting to the upstream’s current state.

Not every partitioned projection can be rebuilt this way. One with no pollable source — reached only through the live-only path, with no event-store trigger, no co-keyed feed, and no re-key edge — has nothing to re-drain: a rebuild would clear its read models with nothing to repopulate them from. replay rejects that projection with a reason instead of accepting it.

replay likewise rejects a partitioned projection you have deliberately halted. Pausing it for a rebuild would overwrite that stop and then lift it as part of the epoch reset, silently undoing the decision to stop it; resume it first, then trigger the rebuild. An error halt is different — a rebuild is a legitimate remedy for a read model an event poisoned, so it is accepted and supersedes the error halt. In the other direction, an error raised by a batch that was already in flight when the rebuild paused the projection does not displace the pause: absorbing exactly those batches is what the pause is for, and the rebuild proceeds.

Nothing is dropped — every event still reaches the projection. What changes is which path delivers it.

The projection’s ordinary events — the ones stored in the event store — arrive through the bucket poll, in order, exactly as they do for any event-sourced projection. They are not applied from the after-commit signal that fires the instant a command runs: that signal fires on whichever instance ran the command, which is usually not the bucket’s owner, so applying it there would sidestep the owner’s ordering and its ownership fence. Delivering these events through the owner’s poll keeps each aggregate’s events in order. The cost is freshness — updates appear at poll cadence rather than instantly, the same throughput-for-latency bargain a Kafka consumer group makes when it spreads a topic across consumers.

A projection can also react to something that lives only in memory and is never written to the event store — for example a read model re-published by another projection. The bucket poll cannot see it, so it is delivered live (no opt-in needed; the framework logs it at INFO on startup, naming it and this implication). It has no fixed ordering against the stored events, and if a live apply and a poll apply ever touch the same row at once, the write-conflict retry (below) reconciles them. That is fine for an occasional side channel, but not something to lean on for a projection’s main event flow.