Ad-hoc projections

An ad-hoc projection answers on demand. One call replays the projection’s events and hands back the read models they produce, fresh, for that call only. A materialization writes nothing, and it answers while the projection is halted, paused or rebuilding. Changing a returned read model changes only that result.

It solves two problems:

  • A question not worth a projection.

    A monthly report or a one-off investigation needs read models nobody queries the rest of the time. An ad-hoc-only projection stores nothing, tracks no progress and has nothing to rebuild when its code changes.

  • A persistent projection checked or answered without its stored read models.

    Mark it ad-hoc as well, and you can recompute what it holds from the events at any time.

Live projection versus ad-hoc materialization: a live projection applies each event as it arrives and keeps read models the application queries; a materialization replays the events from the first on every call and hands back fresh read models for that call only; nothing is written
Figure 1. The same events feed a live projection as they arrive, and an ad-hoc materialization each time it is called

Declaring

Ad-hoc only

@AdHocProjection declares a projection that is only ever materialized. Its listeners are written as for any other projection; see Writing a projection.

class DepositTotal(var accountId: AccountId? = null, var total: BigDecimal = BigDecimal.ZERO, var deposits: Int = 0)

@AdHocProjection(name = "deposit-totals")
class DepositTotalsProjection {

    @ProjectionListener
    fun on(event: MoneyDeposited, total: DepositTotal) {
        total.accountId = event.accountId
        total.total += event.amount.value
        total.deposits++
    }
}

It is the shorthand for @AdHoc @Projection @Component, and alone it needs a name. It has no store, no background work and no repository. No other projection can listen to its read models, because it never publishes them.

A persistent projection that can also be materialized

Put @AdHoc beside the store annotation. The projection keeps running and storing as before, and gains the on-demand path.

@AdHoc
@JpaProjection(name = "account-summary")
class AccountSummaryProjection {
    // listeners as usual
}

@AdHoc works beside every store annotation. @AdHocProjection beside a store annotation means the same. The name may stand on either annotation; two different names fail the startup.

Materializing

Inject AdHocProjections. materializeOne returns one read model, materialize returns all of them.

@Service
class DepositReport(private val adHoc: AdHocProjections) {

    fun totalOf(accountId: AccountId): DepositTotal? =
        adHoc.materializeOne<DepositTotal>(accountId).readModel

    fun largeDepositors(threshold: BigDecimal): List<DepositTotal> =
        adHoc.materialize<DepositTotal>().findAll { it.total > threshold }
}

The read-model type picks the projection. When several produce it, name the one you mean: adHoc.materialize<DepositTotal>("deposit-totals"). A call fails when several produce the type and none is named, when only projections without @AdHoc do, and when none does. Without the generic shorthand, pass the class: adHoc.materialize(DepositTotal::class).

The projection’s own events are read up to how far they were readable when the call began. An event published while the call runs is not part of its answer.

One read model

The result of materializeOne holds the read model as readModel, or null when no event derives one. incompleteHistory says whether deleted history left it short; see When history has been deleted. The id may be the key itself or a wrapper of it; an id that reduces to no key fails the call before anything is read.

A projection without @ReadModelKey that listens to no other projection reads only that key’s messages. The result is the same as materialize gives for that read model, because an upcaster keeps the key of the event it changes; see Splitting and dropping events. Any other projection reads all its messages and derives only the read model asked for.

Several read models

materialize derives every read model, and its result answers in memory:

  • findAll() lists them all.

  • findAll(predicate) lists those the predicate accepts.

Read models of several keys that must agree, such as both accounts of a transfer, come from one materialize call. Two calls to materializeOne each read the messages as they stand when the call begins, so a deposit stored between them can be in one answer and not the other. A read model holds its key when a listener sets it, as DepositTotal.accountId does above.

Until an instant

Both calls take until, which answers "what did this look like then?", such as an account’s balance at the end of a month.

fun balanceAtMonthEnd(accountId: AccountId, month: YearMonth): BigDecimal? {
    val monthEnd = month.atEndOfMonth().atTime(LocalTime.MAX).toInstant(ZoneOffset.UTC)
    return adHoc.materializeOne<AccountSummary>(accountId, until = monthEnd).readModel?.balance
}

The call applies every message that occurred up to and including the instant, and none after it. When a message occurred is the time on the clock of the instance that published it. An instant still to come gives the present.

The edge is only as exact as the clocks of your instances agree. A message whose transaction commits after the call began is not part of the answer. Treat the result as true for roughly that instant. That is right for reports and analysis, and wrong for anything that needs an exact consistency boundary.

Materializing from other projections

An ad-hoc projection may listen to the read models of a persistent projection, as a live projection does. Here the risk desk rates every account from its summary.

class AccountRisk(var balance: BigDecimal = BigDecimal.ZERO, var overdrawn: Boolean = false)

@AdHocProjection(name = "account-risks")
class AccountRiskProjection {

    @ProjectionListener
    fun on(summary: AccountSummary, risk: AccountRisk) {
        risk.balance = summary.balance
        risk.overdrawn = summary.balance < BigDecimal.ZERO
    }
}

The upstream projection needs no @AdHoc, because being read is not being materialized. It must keep its read models in a store, not in memory, and publish them; see Capture & read-model topics. The application refuses to start otherwise.

An ad-hoc projection fed by another projection: for the present, the call reads the account summaries the summary projection published and derives fresh risks from them, replaying none of the summaries' events; as of a past instant, it rebuilds the summaries from their events up to the instant and derives fresh risks from the rebuilt summaries, reading none of the published ones
Figure 2. For the present, a call reads the summaries as they are published; as of a past instant, it rebuilds them from their events

For the present, a call reads the read models the other projection published, never its events. It reads them as it finds them, so one published while the call runs can be part of the answer. As of a past instant, it rebuilds every projection it reads from, directly or through others, from that projection’s own events. Each is rebuilt once, and those it reads from first. Their stored read models and status play no part, so a halted or rebuilding upstream gives the same answer. Each rebuilt upstream counts against its own memory bound, and sets incompleteHistory when its own history was deleted. materializeOne derives only the read model asked for, but still reads or rebuilds every upstream read model.

While an upstream projection is rebuilt

While the upstream projection is rebuilt, its published read models are a mix of old and rebuilt ones. A call for the present then fails with UpstreamRebuildingException, before it reads anything or as soon as it finds that a rebuild began meanwhile. Retry once the rebuild is complete, or pass until to rebuild from the events.

fun risks(): MaterializedReadModels<AccountRisk> = try {
    adHoc.materialize<AccountRisk>()
} catch (_: UpstreamRebuildingException) {
    adHoc.materialize<AccountRisk>(until = clock.instant())
}

Every instance sees the same rebuilding state, including one that runs none of the projections itself.

Materializing from a listener

A listener can materialize an ad-hoc projection to look up a value where it needs it. Inject AdHocProjections through the projection’s constructor.

@DomainEvent
data class LimitReviewed(@MessageKey val accountId: AccountId)

@Entity
class AccountLimit(
    @EmbeddedId var accountId: AccountId = AccountId(),
    @Version var version: Long? = null,
    var deposited: BigDecimal = BigDecimal.ZERO,
)

@JpaProjection(name = "account-limits")
class AccountLimitProjection(private val adHoc: AdHocProjections) {

    @ProjectionListener
    fun on(event: LimitReviewed, limit: AccountLimit) {
        limit.deposited = adHoc.materializeOne<DepositTotal>(event.accountId).readModel?.total ?: BigDecimal.ZERO
    }
}

A listener may not materialize its own projection, or one that reads from it. The call fails before reading and its message names the cycle. From a consumer’s apply, that failure goes to the projection’s error handler.

A materialization made from a listener needs a database connection of its own, beside the one applying the listener.

Looking up an upstream read model

Instead of calling materializeOne, a listener can declare the read model it needs as a parameter. @AdHocUpstream gives the key as an expression over the event, and the listener receives that key’s read model.

@JpaProjection(name = "account-limits")
class AccountLimitProjection {

    @ProjectionListener
    fun on(event: LimitReviewed, limit: AccountLimit, @AdHocUpstream("accountId") deposits: DepositTotal?) {
        limit.deposited = deposits?.total ?: BigDecimal.ZERO
    }
}

The expression is written as a @ReadModelKey value is: a property such as accountId, or an expression such as transfer.sourceAccountId. It may yield a wrapper of the key. The parameter is nullable: a null key, or no read model for the key, gives null. An expression that yields several keys fails the apply.

As of the event

The read model is materialized as of the instant the listener’s event occurred. An event dated ahead of this instance’s clock is looked up for the present, as an until still to come is. Three things follow:

  • A rebuild of the projection derives what the live projection derived, however long ago the event occurred.

  • A consumer that is behind sees the read model as it was when the event occurred, not changes made since.

  • Materializing the projection as of an instant gives what it held then.

An earlier event that was not yet stored when the lookup ran is missing from the live lookup, and part of a later rebuild’s. When retention has deleted history the lookup needs, it derives from what remains and only a warning shows it; see When history has been deleted.

After changing the declaration of a projection that others look up, rebuild those that look it up. Their read models come from the old declaration, and nothing notices the change.

Only listeners that take an event can declare @AdHocUpstream. A listener that takes a read model has no instant to look anything up as of. Each lookup is a materialization from a listener, so it needs the connection of its own described above.

What startup checks

The application refuses to start, naming the projection, when:

  • a projection listens to an ad-hoc-only projection, which publishes nothing;

  • a projection listens to a projection that does not publish its read models;

  • an @AdHoc projection listens to its own read model;

  • a repository queries a read model that only an ad-hoc-only projection produces;

  • @AdHocProjection alone has no name, or a class names its projection twice, differently;

  • @Projection stands directly on a class without @AdHoc;

  • a name is blank or used twice, ignoring case;

  • an ad-hoc-only projection listens to an event no configured source carries, such as one that is not a @DomainEvent;

  • an ad-hoc-only projection needs a source the application does not configure, such as a capture topic that is switched off.

An @AdHocUpstream parameter fails the startup when:

  • its key is blank or does not parse;

  • it is not nullable, is typed Any or by a type parameter, is the event parameter, or also carries @ReadModel, @MetadataValue or @Value;

  • its listener takes another projection’s read model;

  • no single @AdHoc projection produces its type;

  • the projection looked up reads from the declaring one, or lies on a cycle of projections listening to each other.

What fails

A call fails when:

  • it names no unique projection, as described under Materializing;

  • a listener throws: the exception reaches the caller as thrown, with no retry and no error handler;

  • a message up to the instant cannot be read by the running code, such as a revision newer than it or a retired event type, whether or not the projection listens to its type;

  • a read-model key is longer than 255 characters;

  • it is made from a listener and closes a cycle, as described above;

  • for the present, an upstream projection is rebuilt: UpstreamRebuildingException;

  • with an instant that has passed, projections it reads from listen to each other, so none can be rebuilt before the others;

  • on every engine but PostgreSQL, captured messages are missing that retention did not delete: MissingHistoryException;

  • the projection derives more read models than its memory bound allows: InMemoryReadModelLimitExceededException.

When history has been deleted

Retention can delete old captured messages; see Retention. Deleted history never fails a call. The read models come from what remains, incompleteHistory is set on the result, and a warning names the projection and the topic. The warning appears once per projection and topic, then at debug level, while incompleteHistory stays set on every call.

val summaries = adHoc.materialize<AccountSummary>()
if (summaries.incompleteHistory) {
    // retention deleted deposits these balances would include
}

Keep retention longer than the history you materialize over. The events of event-sourced aggregates are never deleted.

A message that is missing without retention having deleted it means the history is broken. On MySQL, MariaDB, SQL Server, H2 and HSQLDB the call then fails with MissingHistoryException, which names the source, since an answer built on it would be wrong and nobody would notice. With until, a gap after the instant changes nothing; the call fails only when the first message after the gap occurred at or before the instant, or none follows it. On PostgreSQL such a message cannot be told from a position that a transaction which rolled back never used, so the call answers from the messages it finds; see Choosing a database.

Cost

Every call replays from scratch: nothing is cached, so its cost grows with the history it reads. Bound it with a key: materializeOne on a projection without @ReadModelKey reads only that key’s messages. An @AdHocUpstream parameter materializes again on every invocation of its listener.

For a lookup you make often, keep a persistent projection instead and query its read models.

Memory bound

A materialization keeps all the read models it derives in memory, and is bounded like an in-memory projection.

Out of the box nothing is bounded, so one call can take as much memory as the projection has read models. Set a bound before you materialize from a web request, so a growing history cannot use up the application’s memory.

One setting, spring.ddd.cqrs.projection.in-memory.max-read-models, covers every in-memory projection and materialization, and a projection can have its own; see Bounding the read models.

A call that derives more read models than the bound fails with InMemoryReadModelLimitExceededException and returns nothing. The settings are listed under CQRS in-memory projections.