Errors and halts

A consumer that cannot read or apply a message retries it, and meanwhile the message’s key waits; a message it cannot read waits for every key of its partition, since it cannot tell which is its own. A message it cannot read counts as much as one it cannot apply: no upcaster brings it to its current revision, an upcaster fails on it or moves it to another key, its revision is newer than the running code, or its payload or metadata envelope cannot be deserialized. When retrying does not help, the consumer’s error handler decides what is held back: the consumer, the key, or nothing (see Error handlers). This page describes that for every consumer; Projection errors and halts and route consumer errors say what it means for a read model and for a message that leaves the application, and how each chooses its error handler.

Delivery retry

A write conflict, such as an optimistic-lock conflict or a deadlock, is first retried in place a few times within a fraction of a second, spring.ddd.messaging.consumer.conflict-retry.

Any other failure, and a conflict that persists, is retried on later polls under the delivery budget: the consumer’s error handler’s own retry policy (see A retry policy of its own), else spring.ddd.messaging.consumer.delivery-retry, by default 5 retries, 2 s apart at first and doubling up to 30 s, which absorbs about a minute of failure, such as a restarted database or a failover.

The failing message is retried while other keys go on, and whatever its error handler, its key’s later messages wait behind it until the budget is spent. The message is recorded as undelivered for its key, and the consumer goes on applying every other key in order. The retry count survives restarts and moves to another instance with the partition.

delivery-retry.exclude-exceptions lists causes, by fully qualified class name, that are never retried: a failure caused by one of them spends the budget at once. When delivery-retry.include-exceptions is set, only the causes it lists are retried. A message no upcaster brings to its event’s current revision is never retried, whatever these settings or the error handler’s retry policy say: only a deploy brings the upcaster, so the error handler decides at once.

delivery-retry.policy shapes the delays: exponential, the default, linear or fixed; see Configuration properties.

spring:
  ddd:
    messaging:
      consumer:
        delivery-retry:
          max-retries: 5              # the default
          exponential:
            initial-delay: 2s         # the default
            max-delay: 30s            # the default
          exclude-exceptions:
            - com.example.bank.UnknownAccountException

Error handlers

A message that still fails once the budget is spent is a poison message, and the consumer’s error handler decides what becomes of it. An error handler is a bean implementing ErrorHandler: its decide returns one of four outcomes, ErrorOutcome.HALT, SKIP, DEAD_LETTER or DEAD_LETTER_KEY. The framework registers one built-in handler per outcome, each of which always returns it:

HaltErrorHandler (the default)

Holds back the consumer. The message becomes a dead letter, HALTED, and the consumer halts on every instance. The instance that met the message logs it once at ERROR, with the failure:

Consumer projection 'account-summary' halted: it could not read or apply the message at position 4711 in partition 3 of 'capture-topic' within its delivery budget; the other keys' messages were applied meanwhile. Fix the cause, then resume or restart it to apply the message again, or resume it past the poison message to skip it.

Every other instance stops on its next poll and keeps warning while it stays halted; see Halts. Other consumers keep running.

SkipErrorHandler

Holds back nothing. The consumer skips the poison message for its key, logs that at ERROR with the failure, and applies the messages that waited behind it; it never applies the message for that key. A partition that meets a message the consumer skipped already, on this instance or another, passes over it without a budget of its own. A skip does not make the consumer cross missing history: a topic that later loses history it had not read still halts it (see Missing history).

DeadLetterKeyErrorHandler

Holds back the key: its later messages become dead letters behind it until an operator retries or deletes its first one. The message becomes a dead letter, DEAD_LETTER_KEY, and so does every later message of its key, already waiting behind it or still to come, kept in order behind it, while every other key goes on, and the consumer never halts. A later message is not tried: it fails nothing, so its failure count is 0 and it has no failure reason. Once an operator retries or deletes the key’s first dead letter, its later messages are applied in order behind it; one that fails then is an ordinary failure, retried with a fresh budget and decided by the error handler again. A message whose keys the consumer cannot tell has no key to hold: under DEAD_LETTER_KEY it becomes a DEAD_LETTER dead letter, and its partition goes on. The instance that met the message logs it once at ERROR, with the failure; while dead letters remain, the instances holding their partitions log one ERROR every spring.ddd.messaging.consumer.liveness-warn-interval with how many there are and the oldest. A resume or a restart does not retry a dead letter: an operator does (see Undelivered messages), or a start under retry-dead-letters-on-start.

DeadLetterErrorHandler

Holds back nothing once the budget is spent. The message becomes a dead letter, DEAD_LETTER, logged as under DeadLetterKeyErrorHandler, and its key goes on: the messages that waited behind it are applied in order, and the key’s later messages pass it, so for that key the order no longer holds around it. A dead letter an operator retries holds its key again while it is retried. Nothing waits for a dead letter here, so it is easy to miss: watch for the repeated ERROR.

All four live in de.dwittkoetter.ddd.messaging.consumer. How a consumer names its handler depends on the consumer: see Configuring the error handler for a projection and route consumer errors for a route consumer. The name is a class, resolved to the one bean of that class in the application context; when there is none, or more than one, the application does not start, and the failure names the consumer and the class.

What a consumer does with a message it cannot apply: the message is recorded as undelivered for its key and retried under the delivery budget while its key waits and the other keys go on; once the budget is spent the error handler decides: HALT holds back the consumer until a resume, a restart or a resume past the poison message, SKIP holds back nothing and the message is never applied, DEAD_LETTER_KEY holds back the key, its later messages dead letters behind the first, until an operator retries or deletes the dead letter or a restart with retry-dead-letters-on-start retries it, and DEAD_LETTER holds back nothing while the dead letter waits for the same

On a read-model topic only a read model’s latest state matters: whatever the error handler, an undelivered message whose read model has a newer state on the topic is dropped instead of retried, so a consumer never goes back to an older state.

Order is kept per key while a message is retried, and after its budget is spent too, unless its outcome is DEAD_LETTER, or it is a message whose key the consumer cannot tell that dies under DEAD_LETTER_KEY, which loses order for whichever key it belonged to; while it is retried, such a message holds back more than a key. A consumer cannot tell which key a message it cannot read belongs to, and the same holds for a message whose key fails to resolve. Such a message holds back every later message of its partition, whatever its key, until the delivery budget and then the error handler decide; on a topic with one position space, such as the event store’s, that is every later message of that source, the consumer’s other topics going on. Under DEAD_LETTER and DEAD_LETTER_KEY the partition goes on once the message is dead, and the message is applied late, if it is retried once it can be read: by an operator, or by a start under retry-dead-letters-on-start. Under DEAD_LETTER_KEY too the message is recorded as a DEAD_LETTER, and order is not kept for the key it belonged to. An operator’s retry or delete acts on the message once, however many partitions it was recorded in. A message no upcaster can read is decided at once, without waiting out the delivery budget.

What is not a poison message

Only a failure of the consumer’s own work on a message is a poison message: the exception a projection’s listener or a route consumer’s dispatcher throws, the failure to commit what it wrote, and a failure to read the message itself. A failure to save the checkpoint, to keep a partition or to record a message, like a failure to read a topic, records nothing: the consumer delivers again from its checkpoint on its next poll, for as long as the failure lasts, and never halts or skips.

A lost database connection is waited out the same way, however long the outage lasts, and so is a connection the pool cannot hand out in time. So is any failure whose cause is a refused connection or a connection closed mid-read, whatever system it was to: one that a listener or a dispatcher opens to another service counts too, and is retried on every poll rather than recorded as undelivered.

A route consumer also waits out a destination it cannot reach, which its dispatcher reports as DestinationUnavailableException; see route consumer errors.

A deadlock, a lock timeout or a statement timeout that keeps recurring past the budget is a poison message like any other failure.

Changing a consumer’s error handler

A consumer whose error handler changes keeps its undelivered messages. A dead letter keeps the outcome it was decided with, which its status names; the new handler decides the messages given up on from then on:

Status Under the new handler

FAILED, WAITING

Retried as before, and decided by the new handler once the retries are spent.

HALTED

The restart that brings the new handler lifts the halt and retries the message with a fresh budget; the new handler decides it if it fails again.

DEAD_LETTER_KEY, DEAD_LETTER

Still waits for an operator, or for a start under retry-dead-letters-on-start, and a key’s first DEAD_LETTER_KEY still holds its later messages dead behind it; retried, it is decided by the new handler if it fails again, and so is a later message of its key that fails once applied.

An error handler of your own

A handler of your own decides per message. decide receives a FailedMessage: the consumer, the topic, partition and position, the key it failed for (null when its keys could not be told), the cause of the last attempt, how often it was tried, and the consumer’s copy of the message (null when the copy cannot be read). This one dead-letters a deposit to an account that does not exist yet, so the account’s later bookings become dead letters behind it until an operator acts, skips a booking in a currency the account does not hold, and halts on anything else:

@Component
class BookingErrorHandler : ErrorHandler {

    override fun decide(failure: FailedMessage): ErrorOutcome = when (failure.cause) {
        is UnknownAccountException -> ErrorOutcome.DEAD_LETTER_KEY
        is CurrencyMismatchException -> ErrorOutcome.SKIP
        else -> ErrorOutcome.HALT
    }
}

cause is what the consumer’s own work threw; check its causes too when that work wraps the exception you choose by.

decide runs once per message and key whose retries are spent (a message whose keys cannot be told is decided once in each partition it was read in), outside the consumer’s transaction, which is suspended while it runs: a transaction it opens or joins commits or rolls back on its own. Its side effects are at-least-once all the same. The consumer’s transaction can still roll back after decide returned, on a lost lease or a failed checkpoint write say, and the message is then delivered and decided again. A handler that sends, writes or notifies elsewhere, an alert or a copy to another system, must tolerate doing so twice. A decision made on a message’s first failure runs while the consumer still holds its connection, so a handler that writes to the same DataSource takes a second connection: size the pool accordingly.

A decide that throws decides nothing. The message stays undelivered, FAILED, with its retries spent, and an ERROR names the consumer, the position, the key and the handler class. The consumer asks the handler again once the retry policy’s last delay has passed, and never sooner than one poll interval, so a handler that keeps throwing neither spins nor goes unnoticed; the message’s key waits meanwhile, as it waits during its retries.

A retry policy of its own

A handler’s retryPolicy, a Spring RetryPolicy, decides when a failed message is tried again and when it is given up on; left null, the default, the consumer retries under spring.ddd.messaging.consumer.delivery-retry. A policy whose shouldRetry refuses a cause gives up on the message at its first failure with that cause. A message no upcaster can read is given up on at once, whatever the policy says.

Each built-in takes a retry policy as its constructor argument, and the framework’s bean of a built-in steps back when the application declares a bean of the same class. To have every consumer that dead-letters by key retry for longer, declare your own DeadLetterKeyErrorHandler:

@Configuration
class BookingErrorHandling {

    @Bean
    fun deadLetterKeyErrorHandler() = DeadLetterKeyErrorHandler(
        RetryPolicy.builder()
            .maxRetries(10)
            .delay(Duration.ofSeconds(5))
            .multiplier(2.0)
            .maxDelay(Duration.ofMinutes(2))
            .excludes(UnknownAccountException::class.java) // never retried: decided at its first failure
            .build(),
    )
}

Every consumer whose error handler is DeadLetterKeyErrorHandler now uses this bean; consumers under another handler keep theirs. A handler of your own overrides retryPolicy the same way.

Undelivered messages

An undelivered message has one of five statuses:

Status The message

FAILED

Retried when its next retry is due.

WAITING

Behind an earlier undelivered message of its key, not tried yet.

HALTED

A dead letter, its retries spent, that halted the consumer (ErrorOutcome.HALT); it holds its key, and a resume or a start retries it.

DEAD_LETTER_KEY

A key’s first dead letter (ErrorOutcome.DEAD_LETTER_KEY), or a later message of that key, dead behind it until an operator retries or deletes the first; a later message has a failure count of 0 and no failure reason.

DEAD_LETTER

A dead letter its key’s later messages pass (ErrorOutcome.DEAD_LETTER); it waits for an operator.

The last three are the consumer’s dead letters, each named after the outcome that made it one. The consumer’s report, GET /actuator/consumers/projection/account-summary or ConsumerLifecycle, counts its undelivered messages per status, and the consumer.undelivered-messages gauge does the same with a status tag; see the consumers endpoint. Each keeps a copy of its message, and a retry applies the copy, so retention deletes the message from its topic without waiting for it. The copy costs storage for as long as the message stays undelivered: a message held for several keys is copied once per key, and so is every message of a key waiting behind it, and every later message of a key dead under DEAD_LETTER_KEY.

The consumers endpoint lists and changes a consumer’s undelivered messages, from any instance:

  • GET /actuator/consumers/projection/account-summary/undelivered?status=DEAD_LETTER_KEY lists the dead-lettered keys' first dead letters and the later messages dead behind them, in position order, or of any other status, all of them without status, each with its id and why it failed, at most limit (100 by default);

  • POST /actuator/consumers/projection/account-summary/undelivered/{id} retries one;

  • DELETE /actuator/consumers/projection/account-summary/undelivered/{id} deletes one, which skips it for its key;

  • POST /actuator/consumers/projection/account-summary/retry-dead-letters retries every dead letter, after an outage say.

The writes need management.endpoint.consumers.access=unrestricted; see the consumers endpoint. From code, UndeliveredMessages does the same:

val accountSummary = ConsumerKey("projection", "account-summary")
val dead = undeliveredMessages.list(accountSummary).filter { it.status.isDead } // what is dead, and why: failureReason
undeliveredMessages.retry(accountSummary, dead.first().id)                     // once the cause is fixed
undeliveredMessages.delete(accountSummary, dead.first().id)                    // or skip it for its key
undeliveredMessages.retryAll(accountSummary)                                   // after an outage

A retry gives the message a fresh budget and makes it due at once; its key’s waiting messages follow it once it applies. Only the first undelivered message of a key can be retried, unless the message is a DEAD_LETTER, which holds nothing: the call names the one to retry instead. So of a key dead under DEAD_LETTER_KEY only its first dead letter can be retried; retried or deleted, its key’s later messages follow it in order. Deleting a later message of such a key skips only that message. retryAll and retry-dead-letters-on-start do the same for every dead-lettered key: they retry its first dead letter, and its later messages follow. A delete skips the message for its key, whatever its status, and logs that at WARN; its key’s next undelivered message is first then. The instance holding the message’s partition sees the change at the start of its next poll; an apply of that message already under way is rolled back rather than undoing a delete, where the consumer writes in the transaction that saves its checkpoint. A consumer whose writes do not join that transaction can still apply the message once when it is deleted while being applied, or when a failed poll delivers it again. A message whose key could not be resolved, like a message the consumer cannot read, is recorded as undelivered in every partition, under an empty key, since its keys cannot be known. It is still one message: listing and counting show it once, and one retry or delete through any of its entries acts on all of them. A HALTED one is skipped in every partition by resuming past the poison message. A skip is final for the consumer on every instance until a rebuild: a delete, a resume past the poison message and a skip by ErrorOutcome.SKIP are remembered for the consumer. A partition that meets the message later, on any instance, passes over it for the key it was skipped for, or for every key of a message whose keys could not be told, without recording it, spending a budget on it or halting. Call the writes outside a transaction, which they refuse. Neither retryAll nor delete lifts a halt: ConsumerLifecycle.resume does.

Retrying dead letters on start

An outage longer than the delivery budget that fails every key the consumer touches, a downstream system that answers with errors say (a lost or refused connection records nothing), leaves each of them a dead letter under DeadLetterKeyErrorHandler. Call retryAll once the outage is over.

To have dead letters retried without an operator once a fix is deployed, set spring.ddd.messaging.consumer.retry-dead-letters-on-start=true: each start of a consumer gives its DEAD_LETTER_KEY and DEAD_LETTER dead letters a fresh budget and retries them first, whatever its error handler is now; a HALTED one is retried by every start anyway. Every instance that starts does so, so a rolling deploy over three instances retries them up to three times; a message that still fails dies again after one budget each time. A partition moving to another instance is not a start. A dead letter does not hold the topic’s retention back, but it stays in the consumer’s undelivered messages, with its copy, until an operator retries or deletes it, and for a DEAD_LETTER_KEY so does every later message of its key, each with its own copy. The repeated ERROR is the signal to act on. While a consumer has undelivered messages, each poll reads them once more, so an operator’s change is seen on the next poll; a consumer with nothing undelivered reads nothing extra.

Halts

The framework halts a consumer on a problem, on every instance, for a reason that decides how the halt ends. For a projection, Halt reasons shows how each halt ends and what to do. An @InMemoryProjection is the exception to what follows about sharing: its halt belongs to its one instance; see In-memory projections. Two reasons apply to every consumer:

missing history

A topic no longer holds messages the consumer had not read yet; see Missing history.

apply error

Its error handler decided HALT on a poison message; see Apply error.

A projection can also halt because its declaration changed; see When the declaration changes.

A start of any instance, such as a restart, a rolling deploy or a scaled-out instance, lifts a halt for missing history or an apply error, for every instance; a gap that still stands halts the consumer again on its next poll. Pausing a halted consumer replaces the halt with a pause: you take over, and the halt’s reason no longer applies. The instance that halts a consumer logs it at ERROR; every other instance logs it at WARN when it first sees it, and every instance repeats the WARN every spring.ddd.messaging.consumer.liveness-warn-interval (5 minutes by default), so a halted consumer never falls silent.

A halt is kept with the consumer’s checkpoints, so it is shared by every instance, and one resume of the right kind, on any instance, lets every instance through: a plain resume after an apply error, a resume past the gap after missing history. An instance keeps the consumer’s partitions while it is halted, so it applies again as soon as the consumer is resumed.

Deploying a fix usually resumes the consumer on its own. During a rolling deploy the halt can come back: an instance still running the old code that holds the message’s partition for a whole delivery budget after the last new instance started halts the consumer again. Resume it once the rollout is complete, and the halt is lifted for good: a resume during the rollout meets the same race. A poison message that is still there then fails again: every start costs one delivery budget of retries before the consumer halts again, and under autoscaling every scale event repeats that.

Missing history

Retention never deletes past a consumer’s checkpoint, so a consumer halts on missing history only when an @InMemoryProjection was overtaken by retention or the capture topic’s history was deleted outside the framework. A consumer whose checkpoint lies behind history the topic no longer holds halts rather than read across the hole, on every instance that runs it. The instance whose poll met the gap logs it once at ERROR:

Consumer projection 'account-summary' halted: capture-topic holds no history after its checkpoint 100; the topic resumes after 250. Resume past the gap, rebuild from what remains, or restore the missing history.

Every other instance stops applying it on its next poll. A poll already in flight finishes first, and it reads its topics up to their high-water mark, so on a large backlog a halt can take as long as that catch-up. Deleting messages a consumer has already read never halts it.

There are three ways forward:

Resume past the gap

POST /actuator/consumers/projection/account-summary/resume-past-gap, or ConsumerLifecycle.resumePastGap(ConsumerKey("projection", "account-summary")) from code, called once on any instance, continues from where the topic resumes on every instance, keeps everything the read model already holds, and marks the consumer as built from incomplete history. It returns the missing history it resumed past. If the consumer’s status changed meanwhile, for example an operator paused it, the call fails and names that status: the consumer stays in it, already marked past the gap, and a ConsumerLifecycle.resume lets it cross. Neither a restart nor a plain resume does this: ConsumerLifecycle.resume refuses a consumer halted on missing history. The endpoint write needs management.endpoint.consumers.access=unrestricted; see the consumers endpoint.

Start again from what remains

A consumer that starts without a checkpoint never halts on missing history: it starts where the topic resumes and is marked. For a projection a rebuild does that; see Rebuild. For any other consumer, take it out of the application, unsubscribe it from the capture topic (see A consumer that is gone), empty what it built, then put it back: it starts without a checkpoint.

Restore the missing history

Put the deleted messages back from a backup, and with them the capture topic’s start, which retention only ever raises; then restart the consumer: the restart lifts the halt, and nothing is marked.

Apply error

A poison message its error handler decides HALT on halts the consumer until someone acts. The endpoint writes here need management.endpoint.consumers.access=unrestricted; see the consumers endpoint. Once the cause is fixed, resume it, POST /actuator/consumers/projection/account-summary/resume, or restart it: the message is applied again with a fresh budget. When the message can never be applied, resume past it, POST /actuator/consumers/projection/account-summary/resume-past-poison-message. In code:

Fix and resume

Fix the cause (deploy the corrected listener, or bring back the system it calls), then call ConsumerLifecycle.resume(ConsumerKey("projection", "account-summary")) on any instance, or restart the application. The consumer applies the message again, with a fresh budget, and then the messages that waited behind it. A consumer that was paused when its halt was recorded gets a fresh budget on resume too; a plain pause during failing retries keeps the budget already spent.

Resume past the poison message

When the message can never be applied, ConsumerLifecycle.resumePastPoisonMessage(ConsumerKey("projection", "account-summary")), called once on any instance, skips every message the halt waits on and resumes the consumer on every instance; the messages that waited behind them are applied next. It returns the messages it skipped and logs each at WARN. A message whose keys cannot be told, which waits in every partition, is skipped in all of them and returned once; an instance that reaches it later passes over it. If a resume, a start or a pause got there first, the call fails and skips nothing. Call it outside a transaction, which it refuses.

Getting a consumer going again

Each situation has an actuator request and a code equivalent; key stands for ConsumerKey(type, name). The examples use the account-summary projection; a route consumer’s requests use route and the name of its destination. Every endpoint write needs management.endpoint.consumers.access=unrestricted; see the consumers endpoint. A projection can also be rebuilt; see Projection remedies.

Halted on an apply error, the cause is fixed

Resume: the message is applied again with a fresh budget. A restart does the same.

POST /actuator/consumers/projection/account-summary/resume

From code: ConsumerLifecycle.resume(key).

Halted on an apply error, the message can never be applied

Resume past the poison message: its dead letters are deleted, and the consumer runs on.

POST /actuator/consumers/projection/account-summary/resume-past-poison-message

From code: ConsumerLifecycle.resumePastPoisonMessage(key).

Halted on missing history

Resume past the gap: the consumer continues where the topic resumes, marked as built from incomplete history.

POST /actuator/consumers/projection/account-summary/resume-past-gap

From code: ConsumerLifecycle.resumePastGap(key).

Dead letters after an outage

Retry every dead letter, DEAD_LETTER_KEY and DEAD_LETTER ones waiting for an operator included.

POST /actuator/consumers/projection/account-summary/retry-dead-letters

From code: UndeliveredMessages.retryAll(key).

One undelivered message to retry or give up on

List the undelivered messages, then retry or delete one. <id> is the id of an undelivered message from the list.

GET /actuator/consumers/projection/account-summary/undelivered?status=DEAD_LETTER_KEY
POST /actuator/consumers/projection/account-summary/undelivered/<id>
DELETE /actuator/consumers/projection/account-summary/undelivered/<id>

From code: UndeliveredMessages.list(key, UndeliveredStatus.DEAD_LETTER_KEY), retry(key, id) and delete(key, id).

A consumer to stop for a while

Pause it, and resume it later.

POST /actuator/consumers/projection/account-summary/pause
POST /actuator/consumers/projection/account-summary/resume

From code: ConsumerLifecycle.pause(key) and resume(key).

A consumer taken out of the application still holds a topic’s retention

Unsubscribe it from the topic. <topic> is the name of the topic.

POST /actuator/consumers/projection/account-summary/unsubscribe/<topic>

From code: ConsumerLifecycle.unsubscribe(key, topic).