Topics and partitions

A topic is the sequence of messages that consumers read, and its messages are split into partitions. This page names the topics an application has, shows how a message finds its partition, and explains how far a topic can safely be read.

Three topics

An application has up to three kinds of topic:

  • the event store, event-store, holding the events of event-sourced aggregates; see Event-Sourcing;

  • the capture topic, capture-topic, holding every other domain event the application publishes; see The capture topic;

  • one read-model topic per persistent projection, read-model:<projection>, holding the changes to its read models; see Read-model topics.

The actuator, the meters and an unsubscribe name a topic by these names.

Partitions

A topic numbers its messages with positions that run across the whole topic, and stores each message in one partition: the one its key hashes into, out of spring.ddd.partitioning.partitions partitions (256 by default). An event-sourced event is keyed by its aggregate, a captured event by its message key, and a read model’s change by the read model.

A consumer applies each message in the partition of the key it keeps order by, whatever partition the topic stored it in: a projection by the read model the message changes, a route consumer by its route key. A message without a key is applied once, in partition 0.

A consumer applies the messages of one partition in position order, so the messages of one key keep their order. Messages of different partitions are applied in parallel, in no order relative to each other. The instances running a consumer share it by partition; see Distribution.

One topic split into three partitions: messages without a key hash into partition 0, accounts A and C into partition 1 and account B into partition 2, while positions 1 to 10 run across the whole topic; a consumer keeps one checkpoint per partition, after 3, 4 and 8, and reads up to the high-water mark at 8, so its lag is 5, 4 and 0; position 9 is still in flight, so position 10, already committed, lies beyond the mark, and 10 is the end position

The partition count is fixed

spring.ddd.partitioning.partitions must be a power of two from 1 to 32768, and it is fixed for the life of the data. The event store and the capture topic record the partition count their messages were stored under, and a startup with another count fails, naming both counts.

A consumer whose checkpoints were recorded under another count refuses to go on: it logs a warning naming both counts, keeps retrying, and repeats that it makes no progress every spring.ddd.messaging.consumer.liveness-warn-interval. Going on would apply some messages twice and skip others.

Choose the count before the first deploy: changing it later is not supported. It caps how far a partitioned consumer spreads: each partition is applied by one instance at a time, so instances beyond the partition count stand by.

The high-water mark

A consumer reads a topic up to its high-water mark: the position through which every message has committed. A topic hands out a position before the transaction that writes it commits, so its end position, the highest position handed out, can lie above the mark. A consumer that read past the mark could pass over a position whose transaction is still open, and never see that message once it commits. Reading only up to the mark never skips one.

How quickly the mark follows a commit depends on the database. No setting trades that safety for speed.

How quickly the mark moves

On PostgreSQL the mark of the event store and of the capture topic moves within one polling interval of a transaction’s commit, spring.ddd.messaging.high-water-mark.polling.interval (1 s by default), or within two while another transaction on the database server that has already written or locked rows is still open. It never moves before every such transaction has ended. Transactions are counted per server, not per database: a long-running transaction holds the mark back, and with it every consumer of the topic, even when it runs in another database on the same server, and so does a session left idle in a transaction after writing or a prepared transaction nobody finished. If the mark stops moving while messages commit, look in pg_stat_activity for a long-running or idle-in-transaction session that has written, and in pg_prepared_xacts for a prepared transaction nobody finished.

On H2, HSQLDB, MySQL, MariaDB and SQL Server the mark moves within one polling interval of the commit. There an append holds back every other append, on every instance, until its transaction ends (on MySQL up to innodb_lock_wait_timeout), and a transaction that locks other rows after appending can deadlock with one that locks them before appending. Append late in the transaction and keep transactions that append short; see What one at a time costs.

A read-model topic follows the same rule on PostgreSQL. On the other databases it is readable as soon as its change commits.

Lowering the polling interval makes messages readable sooner, at the cost of more queries against the database. Leave spring.ddd.messaging.high-water-mark.enabled on wherever anything consumes a topic: with it off no mark moves, and no consumer of the event store, the capture topic or, on PostgreSQL, a read-model topic makes progress.

spring:
  ddd:
    partitioning:
      partitions: 256          # the default; fixed for the life of the data
    messaging:
      high-water-mark:
        polling:
          interval: 1s         # the default