Projection errors and halts

A projection handles a message it cannot apply as every consumer does: it retries it, then its error handler decides; see Errors and halts. This page covers what that means for a projection: choosing its error handler, what each outcome does to its read models, the halt a changed declaration causes, and the rebuild, a projection’s own way out.

Configuring the error handler

A projection’s error handler is, strongest first:

  1. its own spring.ddd.cqrs.projections.<name>.error-handler;

  2. the errorHandler of its store annotation, @JpaProjection or @QueryProjection;

  3. spring.ddd.cqrs.projection.error-handler, HaltErrorHandler by default.

Each names an ErrorHandler class, a built-in or one of your own (see Error handlers); the property takes its fully qualified name. The class resolves to the one bean of that class: with none or several, the application does not start, naming the projection and the class.

spring.ddd.cqrs:
  projection:
    error-handler: de.dwittkoetter.ddd.messaging.consumer.HaltErrorHandler
  projections:
    account-summary:
      error-handler: de.dwittkoetter.ddd.messaging.consumer.SkipErrorHandler

Setting the property lets an operator change one projection’s error handler with a restart, without changing its code.

@JpaProjection(name = "account-summary", errorHandler = DeadLetterKeyErrorHandler::class)
class AccountSummaryProjection {

    @ProjectionListener
    fun on(event: MoneyDeposited, summary: AccountSummary) {
        summary.balance += event.amount.value
    }
}

An @InMemoryProjection takes no errorHandler: it always halts; see In-memory projections.

What each outcome does to a read model

HALT

The projection stops on every instance. Every read model keeps what was applied until then, and none changes until the projection is resumed or rebuilt. Other projections keep running.

SKIP

The read model the message was for misses what it would have changed until the projection is rebuilt; every later change still reaches it.

DEAD_LETTER_KEY

Only that read model is held back: the account’s later bookings become dead letters too, behind the first, while every other account’s bookings keep reaching the account summary. Once an operator retries or deletes the first, the account’s later bookings are applied in order behind it.

DEAD_LETTER

The account’s later bookings pass the dead letter and reach the account summary, so for that account the bookings may apply out of order. Use it only for a projection whose applies do not depend on order, a running count of bookings say, or for one that reads another projection’s read models, where a newer state replaces an older one anyway.

A handler of your own can choose a different outcome per message, DEAD_LETTER_KEY for an unknown account and HALT for anything else say; each read model then gets what its message’s outcome does. Order is kept per read model, not across read models, while a message is retried, and once its budget is spent unless its outcome is DEAD_LETTER. A read-model key holds at most 255 characters. An event whose key is longer is held back like an event whose key cannot be resolved: it is retried, then the error handler decides, while the projection’s other read models keep updating.

When the declaration changes

A projection halts before it applies anything when its listeners, keys, read-model shape or store changed since its read models were derived and its on-declaration-change is halt; see When a projection’s declaration changes. Rebuild it, or resume it to carry on with its read models as they are. An instance that starts with the declaration the read models were derived under, a rollback say, lifts the halt. A start under any other declaration halts it again only under halt; under warn or rebuild it lifts the halt, and under fail the application does not start.

Rebuild

A rebuild re-derives the projection’s read models from its topics, with its listeners as they are now. It replaces a halt of any reason, deletes the projection’s undelivered messages, and applies a message the projection skipped if its listener now accepts it.

POST /actuator/consumers/projection/account-summary/rebuild starts a rebuild and answers 202 once it is under way; it needs management.endpoint.consumers.access=unrestricted, see the consumers endpoint. From code, ProjectionRebuild.rebuild("account-summary") returns once the replay has begun, and ProjectionRebuild.start("account-summary") as soon as the projection is rebuilding on every instance.

A rebuild is refused while the projection is paused, inside a transaction, and before the projection has saved any progress. After a halt on missing history it replays what the topic still holds and marks the projection as built from incomplete history. See Rebuild and schema evolution and Rebuilding.

In-memory projections

An @InMemoryProjection always halts on a poison message: HaltErrorHandler decides for it, whatever spring.ddd.cqrs.projection.error-handler says, and its own spring.ddd.cqrs.projections.<name>.error-handler naming another class fails the startup. While its retries run, an in-memory projection records the failing read model’s messages in memory and goes on with the others, as any projection does. A change that would add a read model past in-memory.max-read-models fails like any other apply, and halts the projection once its budget is spent. Each instance keeps an in-memory projection’s state, and so its halt, to itself: the halt ends with that instance’s next start, which rebuilds the projection from its topics and meets the message again. The ways out are:

  • fix the cause and deploy: the rebuild on start applies the message;

  • for the read-model bound, raise spring.ddd.cqrs.projections.<name>.in-memory.max-read-models, or spring.ddd.cqrs.projection.in-memory.max-read-models for every in-memory projection (no bound by default), and restart;

  • move the projection to a durable store.

POST /actuator/consumers/projection/account-summary/resume-past-poison-message (it needs management.endpoint.consumers.access=unrestricted, see the consumers endpoint), or ConsumerLifecycle.resumePastPoisonMessage from code, works on an in-memory projection as a stopgap only: it skips the messages on the instance you call, until that instance’s next start, and every other instance stays halted until it is resumed past them itself. Until then the instances are inconsistent with each other. Past a read-model-bound halt, the instance you called drops that read model while others may hold it.

Projection remedies

The examples use the account-summary projection. A rebuild answers 202: it runs on in the background. For every other situation, see Getting a consumer going again.

Halted for any reason, the read models should be derived again

Rebuild.

POST /actuator/consumers/projection/account-summary/rebuild

From code: ProjectionRebuild.start("account-summary") or rebuild("account-summary").

Halted on a declaration change, keeping the read models as they are

Resume.

POST /actuator/consumers/projection/account-summary/resume

From code: ConsumerLifecycle.resume(ConsumerKey("projection", "account-summary")).

A skipped message that the fixed listener now accepts

Rebuild.

POST /actuator/consumers/projection/account-summary/rebuild

From code: ProjectionRebuild.start("account-summary").

An in-memory projection halted on a poison message

Fix the cause and deploy: the rebuild at every start applies the message. There is no request or call for it.