Messaging

Topics, partitions, consumers, error handlers and halts are the framework’s messaging core. Projections and route consumers read the same topics through the same consumer runtime, so this section describes them once, and the pages on projections and routing link here.

What the messaging core does

The framework writes topics in the transaction of the change they record: the capture topic receives each domain event in the transaction that publishes it, and a persistent projection’s read-model topic receives each change to its read models in the transaction that writes it. The event store is read as a topic too.

Consumers read the topics and keep their progress in the database, so they resume where they left off after a restart and on any instance, except an in-memory projection, which keeps its progress in memory. A consumer that cannot apply a message retries it, then records it, skips it or halts, as its error handler decides. The instances that run a consumer share its work by partition.

The actuator shows and steers all of it once spring-ddd-starter-observability is on the classpath; see Observability.

Who consumes topics

Two kinds of consumer read topics today:

Projections keep read models up to date from the event store, the capture topic and other projections' read-model topics.
Route consumers send messages to destinations outside the application, one route consumer per destination.

A consumer is named by its type and its name: projection and the projection’s name, or route and the destination’s name. The actuator endpoints take that name, /actuator/consumers/projection/account-summary, and so does the code, ConsumerKey("projection", "account-summary").

Terms

topic: A durable, ordered sequence of messages that consumers read from where they left off, split into partitions. The capture topic and the read-model topics are topics, and consumers read the event store as one.
position: A message’s place in its topic. Positions only grow.
partition: One of a fixed number of shards that keys hash into, spring.ddd.partitioning.partitions (256 by default). A consumer applies each partition’s messages in order, and the instances running a consumer share it by partition; see Partitions.
high-water mark: How far a topic is safely readable: every message up to it has committed. It is not the topic’s end position, because a position can exist before its transaction commits, and reading past the mark could skip a message that commits later at a lower position. See The high-water mark.
end position: The highest position a topic has handed out, committed or not: at or above the high-water mark. No figure reports it; the projection await in tests waits for it (see Waiting for projections).
checkpoint: How far one consumer has read one partition of a topic. It survives restarts, and the consumer resumes after it on any instance; an in-memory projection’s does not.
lag: The distance between a partition’s high-water mark and the consumer’s checkpoint: what the consumer still has to read, counted in positions, never below zero. A source’s lag is its slowest partition’s.
capture topic: The topic of the domain events the application publishes that no event-sourced aggregate stores, named capture-topic.
read-model topic: A persistent projection’s topic of its read models' changes, named read-model:<projection>, one message per change carrying the read model’s new state. Other projections and route consumers read it.
tombstone: A read-model-topic message saying that a read model was deleted. It carries the read model’s last state.

The actuator shows these figures once spring-ddd-starter-observability is on the classpath. The consumers endpoint reports each consumer’s checkpoint, high-water mark and lag per source and partition. The topics endpoint reports each topic’s high-water mark and every consumer’s checkpoint and lag. The meters consumer.checkpoint, consumer.lag, topic.high-water-mark and topic.lag carry the same figures; see Observability.

In this section

Topics & partitions: The topics an application has, how their messages are split into partitions, and how far a topic can be read.
Capture & read-model topics: What the capture topic and the read-model topics hold, and how long they keep it.
Consumers: How a consumer keeps its progress, what it guarantees, how instances share it, and how to pause it.
Errors & halts: What a consumer does with a message it cannot apply, and how to get it going again.