Observability

The endpoint is read-only. It appears with the JDBC event store, spring-ddd-eventsourcing-jdbc; the health indicator and the metrics work without it.

The component appears under /actuator/health/eventStore.

Reports every consumer the application runs — projections, route consumers, and consumers of its own — and runs what an operator does to one. A consumer is named by its type and name: projection and the projection’s name, or route and the destination’s name. Each report is the one ConsumerLifecycle returns: the consumer’s state and since when, how often it was rebuilt, its halt reason, its checkpoint and lag in every source and partition, who runs each partition, its undelivered messages per status, and, for a projection whose store can count them, its read models. See Consumers for what these figures mean, and Errors and halts for what each action does.

The endpoint is read-only by default. Its actions change what the application’s read side shows, so they answer only once allowed explicitly; secure the endpoint as any actuator endpoint that writes:

Lists every instance running a consumer shared across instances, one spread across instances or run by one at a time, with the host it runs on, whether it is live, the consumers it runs and the partitions it holds of each. Every instance answers the same, and the list includes consumers that only another application on the same database runs. A standby runs a consumer but holds none of its partitions: a single-owner consumer’s other instances, say, ready to take over when its owner stops. An instance that stopped stays listed with live false until another instance removes it, and after that for as long as it still holds partitions, then without its host. Those partitions are the orphaned ones, until other instances take them over. See When an instance joins or leaves. A consumer that runs on every instance on its own, an in-memory projection, is not listed. The list is empty while the application runs no consumer shared across instances.

The endpoint is read-only; expose it like any other, by adding instances to management.endpoints.web.exposure.include; see Adding the starter.

It answers:

Each report of the consumers endpoint lists the same under instances: every instance running that consumer, with its host, whether it is live and the partitions it holds.

The component appears under /actuator/health/consumers. What makes it DOWN, the lag threshold, and why to keep it out of a load balancer’s health check are described in Health.

Every meter here is a gauge read when scraped; the consumers are read at most once per max-age, whatever the number of gauges or scrapes. The timers of each apply attempt are described below, under Apply observations. Every meter but consumer.count and instance.count is tagged with the consumer’s type and name. A value that cannot be given is NaN: a lag whose high-water mark could not be read, say, or every meter of a consumer whose report could not be read.

Every meter but the two consumer.instance.* meters reports what all instances share, so every instance exports the same value; aggregate them with max, not sum. instance.count misses an instance that runs none of the application’s consumers shared across instances; the instances endpoint lists it when it runs another application’s. An in-memory projection is the exception: each instance runs it on its own, so all its meters, its state and checkpoints included, are that instance’s own. A consumer started after the registry is bound gets its meters on the next scrape.

Every attempt of a consumer to apply a message is a Micrometer observation named consumer.apply: one transaction applying the records of one position, or one delivery retry of an undelivered message. With Spring Boot’s observation support, which Spring Boot Actuator brings, each attempt is timed by the timer consumer.apply, and the attempts in flight by the long-task timer consumer.apply.active; with a tracing bridge, each attempt is a span named <consumer name> apply. The consumer runtime observes its applies itself, so an application gets these timers without this starter. Each instance records the attempts it makes itself, so aggregate consumer.apply across instances with sum.

Its keys:

How an attempt ended, its outcome:

When the keys of one message end differently, the attempt reports the worst of them: halted, then skipped, then undelivered. Every outcome but applied carries the exception that caused it, where there is one; a span records it as its error. management.observations.enable.consumer.apply=false switches the observation off; management.metrics.enable.consumer=false removes its timers together with the consumer meters. With a tracing bridge, every attempt is a span, and a message without a trace context starts a trace of its own: a rebuild or replay of millions of events emits sampled root traces. Lower management.tracing.sampling.probability, or register an ObservationPredicate bean that rejects consumer.apply, or set management.observations.enable.consumer.apply=false.

A consumer’s apply continues the trace of the work that wrote the message, when the message carries that trace’s context. Messages carry it once the application turns propagation on, which takes Micrometer Tracing with a bridge, such as micrometer-tracing-bridge-otel:

Each event a request appends then carries the request’s trace, and each consumer’s apply of it is a child span of the request’s span. With the property on and no Propagator bean, a warning names the property once the application is ready.

Reading takes no setting: a consumer continues the trace of any event that carries trace-context, whichever application wrote it. A projection that listens to another projection’s read models starts a trace of its own. The metadata key trace-context is the framework’s; a @MessageMetadataProvider of the application cannot use it as well.

Reports every topic, and runs retention on one on demand. A topic is named capture-topic, or read-model: and the projection’s name. Each report holds:

The endpoint is read-only by default; a retention run answers only once allowed explicitly:

Every meter but topic.retention.deleted is a gauge read when scraped; the topics are read at most once per spring.ddd.metric.topic.max-age, one minute by default, since counting a topic’s messages is costly. With max-age at 0 every gauge reads the topics again, so one scrape counts each topic’s messages once per gauge; keep it above 0 outside tests. Every meter but topic.count is tagged with the topic’s name as topic. A gauge whose topic could not be read is NaN.

The gauges report what all instances share, so every instance exports the same value; aggregate them with max. The counter is each instance’s own; aggregate it with sum.

The endpoint is read-only. It reports every saga type the application declares, whether or not the type has an instance yet. A type shows the state and halt reason of the consumer that runs it on this instance, its lag per source (one for each topic the saga’s events come from) and its undelivered messages per status; these are null while no consumer runs the type on this instance. The instance counts active and completed, and pendingDeadlines, appear only where the saga store provides statistics: the JDBC saga store does, a saga store of the application’s own does not, and they are null then. pendingDeadlines is also null when spring.ddd.saga.deadline.enabled=false. Pausing, resuming and the other actions are on the consumers endpoint, under /actuator/consumers/saga/transfer-saga; the lag is the metric consumer.lag.

The component appears under /actuator/health/sagaStore where the saga store provides statistics.

The gauges below, like the health indicator, need the saga store’s statistics; an application’s own saga store provides none. Per-type gauges are snapshotted from the set of known saga types at startup; values are read live on every scrape.