Capture and read-model topics

The framework writes two kinds of topic itself: the capture topic, which keeps every domain event no event-sourced aggregate stores until its retention deletes it, and one read-model topic per persistent projection, which keeps only the latest state of each read model. This page describes what each holds, how long it keeps it, and what a consumer sees when history is missing.

The capture topic

Every @DomainEvent that no event-sourced aggregate stores is captured, whether or not a projection listens to it yet. An event an event-sourced aggregate stores is never captured: the event store already keeps it. An object published without @DomainEvent is never captured.

Exclude a type with @NotCaptured, which also excludes its subtypes, or by name in spring.ddd.messaging.capture.event.exclude-event-types: a list of canonical event names (<namespace>.<name>) or fully qualified class names, each matched exactly. A type is excluded when either names it.

@NotCaptured
@DomainEvent(namespace = "banking", name = "StatementViewed")
data class StatementViewed(@AggregateId val accountId: AccountId)
spring:
  ddd:
    messaging:
      capture:
        event:
          exclude-event-types:
            - banking.StatementViewed

Capture is on by default; spring.ddd.messaging.capture.event.enabled: false switches it off.

A domain event published inside a transaction is captured in that transaction: it commits or rolls back with the business change. One published outside a transaction is stored in a transaction of its own.

The capture topic stores each event at a position of its own, in the partition of its message key; see Partitions. A projection that listens to an event the capture topic never holds fails at startup; see Projections on captured events.

Read-model topics

Every persistent projection writes a read-model topic, read-model:<projection>: each change to one of its read models adds the read model’s whole new state to the topic, in the transaction that writes the change. An in-memory projection has none. Other projections and route consumers read it; see Projections on projections and Routing read models.

spring.ddd.messaging.capture.read-model.enabled, on by default, writes the read-model topics. With it off none is written, and a projection another projection listens to, a @QueryProjection or a read-model route fails the startup.

Compaction and tombstones

Only a read model’s latest state matters, so the topic compacts itself: it deletes each message that a later message of the same read model supersedes, and keeps each read model’s latest. Compaction is always on and runs on its own schedule, outside every projection’s write: every minute by default, spring.ddd.messaging.capture.read-model.compaction.schedule.interval, or on a compaction.schedule.cron schedule. Each instance compacts on its own schedule; switched off on every instance with compaction.schedule.enabled: false, superseded messages pile up.

A deleted read model leaves a tombstone: a message marking it deleted that carries its last state, so a projection or a route that reads the topic learns of the deletion.

Tombstones are kept for good unless tombstone retention is set. With spring.ddd.messaging.capture.read-model.retention.tombstones.max-age, a tombstone older than that is deleted once compaction and every consumer subscribed to the topic have passed it; it runs every hour by default, retention.schedule.interval. retention.include and retention.exclude limit it to, or keep it from, read-model types named by their fully qualified class name.

spring:
  ddd:
    messaging:
      capture:
        read-model:
          compaction:
            schedule:
              interval: 1m          # the default
          retention:
            tombstones:
              max-age: 30d          # unset by default: tombstones are kept
            schedule:
              interval: 1h          # the default

Keep tombstones.max-age longer than the longest an in-memory projection that reads the topic lags behind: retention cannot see how far such a projection has read.

A read-model topic keeps no other retention, and the event store keeps everything: it is the aggregates' source of truth.

Retention

For an event no event-sourced aggregate stores, the capture topic is the only copy of its history, so pruning it limits how completely the projections it feeds can be rebuilt; see Retention and rebuilds.

By default the capture topic keeps everything. Set an age, a message count or both, and messages outside that window may be deleted:

spring:
  ddd:
    messaging:
      capture:
        event:
          retention:
            max-age: 30d
            max-messages: 5000000

With both set, either limit applies. A set policy runs on its own every retention.schedule.interval (one hour by default) or on a retention.schedule.cron schedule; with retention.schedule.enabled: false it runs only when the application calls CaptureTopicRetention.enforce() itself. Every instance runs its own schedule, and running it on several instances at once is safe.

Retention never deletes past the slowest consumer’s checkpoint, so no consumer loses anything it has not read. A consumer that has subscribed but read nothing yet holds the whole topic. Retention also stops at the topic’s high-water mark. A message a consumer could not apply yet does not hold the topic back: its undelivered message keeps a copy of it, and retries apply the copy. An @InMemoryProjection is the exception: it keeps its checkpoint in memory, so retention never sees it and never holds back for it. Retention can delete messages such a projection has not applied yet, which halts it, and every start rebuilds it from what the capture topic still holds. Keep a policy generous enough for the in-memory projections that read the capture topic; startup logs a warning naming them while a policy is set. Each run reports how far it got and, when a consumer’s checkpoint held it back, which consumer that was: in the log, and in the topics actuator endpoint. POST /actuator/topics/capture-topic/retention runs retention at once, once management.endpoint.topics.access=unrestricted allows the endpoint’s writes; see the topics endpoint. A consumer that reads past messages it has nothing to apply for records that progress with a later save, at most once a minute, so retention may hold back up to a minute’s worth of messages a quiet consumer has already read, for as long as no further message arrives.

A consumer that is gone

A checkpoint never expires. When a projection or a route destination is removed from the application, or a projection no longer reads the capture topic, its checkpoint keeps holding the line until an operator unsubscribes it from the topic; see Unsubscribing a consumer.

Unsubscribing a consumer

Unsubscribing deletes a consumer’s checkpoints on a topic, with their incomplete-history marks and undelivered messages, so it no longer holds that topic’s retention. Once the consumer is unsubscribed from its last topic, its state goes too. The request needs management.endpoint.consumers.access=unrestricted; see the consumers endpoint.

POST /actuator/consumers/projection/account-summary/unsubscribe/capture-topic

Without the topic, POST /actuator/consumers/projection/account-summary/unsubscribe unsubscribes the consumer from every topic it has checkpoints on. From code: ConsumerLifecycle.unsubscribe(ConsumerKey("projection", "account-summary"), "capture-topic"), or without the topic. A route consumer is named route and its destination’s name.

  • It is refused, 409, naming each such topic, while the application still reads the topic under that name, before the consumer first started too. Without a topic, one such topic refuses the whole request, and nothing is unsubscribed. Take the consumer out of the application first.

  • A consumer without checkpoints on the topic, or on any topic, answers 404.

  • An instance that still runs the consumer, an older one during a rolling deploy say, subscribes again at the consumer’s start position and replays: a projection applies its history to its read models again, a route consumer sends to its destination again. Unsubscribe once every instance runs without the consumer, and check that GET /actuator/topics no longer lists it.

  • A consumer that runs again under the same name starts without a checkpoint, as a new one does, and the messages retention deleted in the meantime are missing from its history.

  • An @InMemoryProjection keeps its checkpoints in memory, on each instance: unsubscribing it is always refused, 409.

  • A read-model topic’s compaction is not a consumer and cannot be unsubscribed.

The incomplete-history mark

A consumer that reads a topic which no longer holds all of its history is marked as built from incomplete history, durably; for a projection that means its read models are incomplete:

  • a consumer without a checkpoint that finds history missing starts where the topic resumes;

  • a consumer resumed past the gap continues from there.

Either way one warning names the consumer, the topic and the range that is missing:

Consumer projection 'account-summary' of 'capture-topic' starts after position 250: the topic holds no history after 100 through 250; the consumer is marked as built from incomplete history

The mark shows as incompleteHistory: true for the consumer in the consumers actuator endpoint and in the details of the consumers health indicator. It survives restarts, only ever rises, and goes only when the consumer is unsubscribed from the topic or rebuilt: a rebuild records it again only if its own replay meets missing history. A marked consumer crosses any further missing history with another warning instead of halting: its figures are already known to be incomplete.

  • A consumer is marked even when the deleted messages held nothing it reads. The messages are gone, so the framework cannot tell what they held.

  • A marked consumer does not halt again, even when more history is deleted ahead of it later; it warns instead.

See Which deletions are detected.

Which deletions are detected

Missing history is detected on the capture topic, the only topic whose history retention deletes. On H2, HSQLDB, MySQL, MariaDB and SQL Server the capture topic numbers its positions without gaps, so messages deleted anywhere, by retention or by hand, are detected. On PostgreSQL a rolled-back transaction leaves a gap in the numbering that looks exactly like a deleted message, so only retention’s own deletions are detected there: messages deleted from the capture topic by hand go unnoticed. Delete from the capture topic through retention only.

A new consumer over a capture topic whose only missing history is a hole in the middle, left by a deletion by hand, halts at that hole once, because it has read and recorded history before it. One resume past the gap lets it through; see Missing history.