Running projections

This page covers what an operator sees and does with a running projection: its states, the reasons it halts, and how to pause, resume, rebuild and monitor it. Every action works from any instance and holds on every instance; see Running on multiple instances for how the instances share a projection.

Route consumers run the same way. Their requests use route and the destination’s name, POST /actuator/consumers/route/audit/pause say, and everything on this page applies to them except rebuilding and the declaration-change halt. See Routing errors.

States

A projection is in one of four states, and it is the same state on every instance.

RUNNING

It applies messages. Readers see its read models follow the events, with eventual consistency.

PAUSED

An operator paused it, and it applies nothing until it is resumed. Its read models stay readable and stop changing.

HALTED

The framework stopped it on a problem, for a reason; see Halt reasons. Its read models stay readable and stop changing.

REBUILDING

A rebuild holds the projection from the moment it begins until it has reset the projection’s progress. It applies nothing, and pause and resume are refused. Once reset, it is RUNNING again and replays its topics.

A change of state made on any instance takes effect on every instance at its next poll. A paused or halted projection keeps its partitions and polls on, but applies nothing. An @InMemoryProjection is the exception: its state, its halts and every action on it are the instance’s that answers, and every start runs it again.

The four states of a projection: a first start leaves it RUNNING; pause moves RUNNING or HALTED to PAUSED and resume moves PAUSED back to RUNNING; RUNNING halts on missing history, an apply error or a declaration change; a resume, a resume past the gap or the poison message, or a start moves HALTED back to RUNNING; rebuild moves RUNNING or HALTED to REBUILDING, a rebuild called again continues it, and once the rebuild has reset the projection it is RUNNING again while the replay catches up; a rebuild of a PAUSED projection is refused, and so are pause and resume while it is REBUILDING

What happens to a single message the projection cannot apply, before it halts, is shown in Error handlers.

Halt reasons

The instance that halts a projection logs it once at ERROR. Every other instance logs it at WARN when it first sees it, and every instance repeats the WARN every spring.ddd.messaging.consumer.liveness-warn-interval, 5 minutes by default, so a halted projection never falls silent. A halt holds on every instance, except for an in-memory projection’s.

Missing history

MISSING_HISTORY: a topic no longer holds messages the projection had not read yet, so it halts rather than read across the gap.

You can resume past the gap, which keeps the read models and marks the projection as built from incomplete history, rebuild from what remains, or restore the history. A start lifts the halt, but a gap that still stands halts the projection again. See Halted on missing history for the request.

Apply error

APPLY_ERROR: a message could not be applied within its delivery budget, and the error handler decided HALT.

Fix the cause and resume, or restart. When the message can never be applied, resume past the poison message. Rebuilding the projection also ends the halt. See the cause is fixed and the message can never be applied; for an in-memory projection, see In-memory projections.

Declaration changed

DECLARATION_CHANGED: the projection’s declaration changed since its read models were derived, and its on-declaration-change is halt.

Rebuild it, or resume it to carry on with its read models as they are. A start under the declaration the read models were derived under lifts the halt. See Halted on a declaration change and When a projection’s declaration changes.

Pausing and resuming

The requests below need management.endpoint.consumers.access=unrestricted, since the consumers endpoint is read-only by default; see the consumers endpoint. From code, the ConsumerLifecycle bean does the same on any instance. It refuses to run inside a transaction.

Pause

A pause stops the projection on every instance until it is resumed. It holds through restarts, rolling deploys and scaled-out instances. Pausing a halted projection replaces the halt, and pausing a paused one changes nothing. A pause is refused while the projection is REBUILDING and before it has polled once. A poll already under way finishes first, and it reads its topics up to their high-water mark, so on a large backlog a pause can take as long as that catch-up.

POST /actuator/consumers/projection/account-summary/pause

Resume

A resume ends a pause, or an apply-error or declaration halt. It is refused on a missing-history halt, where you resume past the gap instead, and while the projection is REBUILDING. Resuming a running projection changes nothing.

POST /actuator/consumers/projection/account-summary/resume

From code:

@Component
class AccountSummaryMaintenance(private val consumers: ConsumerLifecycle) {
    private val accountSummary = ConsumerKey("projection", "account-summary")

    fun pause() = consumers.pause(accountSummary)

    fun resume() = consumers.resume(accountSummary)
}

A pause, and the resume that ends it, is logged once at INFO on every instance. An in-memory projection is paused on the instance that answers only, until it is resumed or that instance starts again.

Rebuilding

Rebuild a projection when its read models no longer match what the events say: after fixing a listener, to get past a poison message for good, or after a halt on missing history. A rebuild replaces a halt of any reason.

POST /actuator/consumers/projection/account-summary/rebuild

It answers 202 once the projection is REBUILDING on every instance; the rest of the rebuild runs on in the background. Follow it with GET /actuator/consumers/projection/account-summary: the state is REBUILDING until the rebuild has reset the projection, then RUNNING, and the lag of each source shrinks as the replay catches up. A rebuild that fails after it began is logged as a warning and leaves the projection REBUILDING, applying nothing, until the rebuild is called again, which continues it.

From code, ProjectionRebuild.start("account-summary") does the same, and ProjectionRebuild.rebuild("account-summary") returns once the replay has begun, which can take up to the projection’s poll interval plus spring.ddd.messaging.consumer.stale-instance-timeout.

@Component
class AccountSummaryRebuild(private val rebuilds: ProjectionRebuild) {
    fun start() = rebuilds.start("account-summary")

    fun rebuild() = rebuilds.rebuild("account-summary")
}

An @InMemoryProjection is rebuilt on the instance that answers only. Calling the rebuild again while a replay runs starts the rebuild over; the earlier one is superseded. During a rolling deploy, an instance still running the old code can meet the poison message again while the projection replays, and halt it. Rebuild once the rollout is complete.

What readers see

While a rebuild runs, read models stay readable. Each is overwritten when the replay reaches it, so a read model can show an earlier state and then move forward again. Changes made after the reset apply once the replay is complete. Read models the replay did not re-create are deleted at the end, and a read model your application wrote itself, outside the projection, is never deleted. A read model the rebuild cannot delete is kept and named in an ERROR in the log, and the rebuild still finishes; delete it by hand if it should be gone. A rebuild from a topic that no longer holds all its history keeps the read models it did not re-create, logs a warning, and marks the projection as built from incomplete history; the reset clears that mark and the undelivered messages. See What a rebuild does for the concept.

What a rebuild refuses

A refused request answers 409 with the reason, and changes nothing. The refusals are:

  • A paused projection: resume it first.

  • A projection that has not saved any progress yet, and again right after a rebuild’s reset until the replay saves some: call it again shortly.

  • A projection whose read models live where the framework cannot record which ones a rebuild re-created: a JPA persistence unit Spring does not describe, or a database the framework does not support.

  • A route consumer.

A request naming an unknown projection answers 404.

In code, ProjectionRebuild also refuses to run inside a transaction.

Health

The consumers health indicator is DOWN while any consumer is halted, while a running consumer lags further behind in a source than its lag threshold, or while a consumer’s report cannot be read. A paused or rebuilding consumer is deliberate and stays UP; so does one built from incomplete history, which the detail lists. Lag never makes the indicator DOWN until a threshold is set. The health details name each consumer that is not running or is over its threshold as type/name, with its state, its halt reason, since when, the lag of each source over the threshold, incompleteHistory, and an error when its report could not be read. While a rebuild replays, the projection is RUNNING and its lag is what the replay still has to read, so a lag threshold reports it DOWN until the replay catches up.

spring:
  ddd:
    metric:
      consumer:
        lag-threshold: 10000        # every consumer; unset by default
      consumers:
        projection:
          account-summary:
            lag-threshold: 500

Every threshold must be positive. A consumer name containing dots is written in brackets. Once the application is ready, an override that names no consumer is logged as a warning. Lag counts positions, not messages, so choose the threshold from the lag the endpoint shows under normal traffic; see Checkpoint and lag.

A consumer’s state is shared, so every instance reports the same consumers health, and a halted projection makes every instance DOWN at once. Anything that reads the overall health, /actuator/health, then removes every instance at once: a load balancer’s health check, Consul’s default check, Eureka with eureka.client.healthcheck.enabled=true. Point those at a health group without the indicator instead: Boot’s readiness group, /actuator/health/readiness, which exists on Kubernetes or with management.endpoint.health.probes.enabled=true and holds only the readiness state unless you add to it, or a group of your own, such as management.endpoint.health.group.lb.include=ping,db.

What to alert on

  • consumer.state with state=HALTED at 1: a halt needs an operator; see consumer.state.

  • consumer.lag above its usual level for longer than a deploy takes: see consumer.lag.

  • consumer.undelivered-messages with status=DEAD_LETTER_KEY or status=DEAD_LETTER above 0: dead letters wait for an operator; see consumer.undelivered-messages.

  • any consumer.apply with outcome skipped: each is a message the projection never applies for that read model; see Apply outcomes.

consumer.state, consumer.lag and consumer.undelivered-messages report what every instance shares, so aggregate them with max, not sum. consumer.apply is recorded by the instance that applies the message, so sum its counts across instances.