Upgrading to 2.2.0

This page covers what you need to know when upgrading to Quine Enterprise 2.2.0.

Back up your data store before upgrading

2.2.0 upgrades the persistence format in place, and stamps the store with the new format version the first time it starts. A 2.1.1 process will not start against a store carrying that stamp, so restoring a backup is the route back to 2.1.1. See The upgrade is one-way.

What's New

• A node's history is queryable from Cypher — six new procedures read a node's journal directly: history.propertyChanges, history.nodeChanges, history.edgeChanges, history.edgeChangesBetween, history.nodeAt, and history.queryAt. Together they answer both halves of a historical question: what changed and when, and what a node or a whole query returned at any past moment. Because the moment is an ordinary Cypher argument, a single query can find the moment that matters and then read the graph at it. See Historical Queries.

• Standing query results are independent of node sleep — DistinctId standing queries report the same results whatever the sleep and wake pattern of the nodes involved. See Behavior Changes.

• Tunable snapshot frequency — the new quine.persistence.snapshot-after-events setting lets a journaling deployment snapshot a node only once it has journaled enough to be worth it, cutting the write volume reaching your data store. See New configuration.
• Hot node metrics — new metrics rank individual nodes by mailbox backlog, message rate, and estimated wait. When a low-cardinality node on the critical path of a high-volume ingest is holding up the stream, these metrics name it by ID so you can rework the ingest query. See Metrics.
• Type-distinct result values in the UI — the results card renders values as syntax-highlighted JSON, so the number 42 and the string "42" are distinguishable at a glance, as are a null value and the word "null".
• Higher outbound HTTP connection limits — webhook and HTTP destinations reach the throughput their parallelism setting asks for.
• Streaming journal reads — a node's journal is now read as a stream rather than collected first, so walking a long history no longer costs memory proportional to its length.

• Nodes keep processing under clock skew — a node processes its messages whatever the offset between its own clock and its peers'. See max-catch-up-sleep is removed.

• A cleaner cluster-wide ingest API — every response carrying a cluster ingest returns the configuration it is running, with secrets redacted, and status is a typed model with published field and enum values. See Breaking Changes.

• Cluster-wide ingest counts survive relocation — an ingest's record and byte counts come from each member's persisted ledger, so a partition moving between hosts loses none of the work already counted and the total reads the same from any host.

For the full list of changes, see the Release Notes.

Upgrade procedure

1. Stop Quine Enterprise. Stop every cluster member — see A full cluster restart is required.
2. Back up the data store. This is your route back to 2.1.1.
3. Remove max-catch-up-sleep from your configuration if it is set. 2.2.0 does not recognize the key and refuses to start while it is present.
4. Leave snapshot-singleton as it is. If you want to change it, do that in a separate restart once the upgrade has settled. See Change snapshot-singleton in its own step.
5. Start 2.2.0. Start every member. The store's format version is stamped at startup, before any node wakes.
6. Update any client of the cluster-wide ingest v2 API for the slice → partition rename.

There is no bulk migration step and no deadline by which the store has to be converted. A node reads its 2.1.1 records when it wakes and writes the new format the next time it sleeps with something to save. A node that never wakes keeps its 2.1.1 records and reads them correctly every time.

Breaking Changes

The upgrade is one-way

Quine Enterprise stamps its data store with the persistence format version it writes, and checks that stamp at startup before any node wakes. 2.2.0 raises the stamp:

• A 2.2.0 process accepts a store written by 2.1.1 and stamps it with the new version.
• A 2.1.1 process started against a store carrying that stamp stops immediately, reporting an incompatible version.

The startup check is what makes the upgrade safe to perform: an older build is stopped at the door rather than allowed to read a format it does not understand. It also means the upgrade is one-way. Take a backup before you start 2.2.0, because restoring it is the route back to 2.1.1.

Under the default time-keyed snapshots your 2.1.1 snapshot rows remain in the store, though the version stamp still prevents 2.1.1 from starting. Under snapshot-singleton = true, each node overwrites its single snapshot row the first time it sleeps, so the backup is the only copy of the 2.1.1 bytes.

A full cluster restart is required

Every member has to be on the same persistence format, and a member checks the stamp when it starts. Stop every member, then start every member on 2.2.0. A rolling, member-by-member upgrade is not supported for this transition, in the same way it was not supported when moving to 2.1.1.

If you deploy with Helm, scale the cluster to 0 members and back up, following the Helm chart quickstart.

What the upgrade does not change

No table, column family, keyspace, or column is added, renamed, or dropped by any persistor. Cassandra, Keyspaces, MapDB, RocksDB, ClickHouse, and in-memory all keep the schema they had, so there is no DDL to run before upgrading, unlike the move to 2.1.1 which added two Cassandra tables.

max-catch-up-sleep is removed

Nodes now process their messages whatever the offset between their own clock and their peers', so no setting is needed to govern how long a node waits for a clock to catch up. quine.max-catch-up-sleep is removed.

2.2.0 does not recognize the key, and stops at startup with a configuration error naming it, as it does for any other unrecognized setting. Remove it from your configuration before upgrading.

Keep your clocks synchronized. A historical query assembles its answer from timestamps each member wrote, so run NTP on every member. See Cluster Resilience.

subscribers and subscriptions change their columns

These two procedures now report which standing queries each subscription is held for, and their columns change. Update any saved query that reads them.

Procedure	2.1.1 columns	2.2.0 columns
subscribers(node)	queryId, queryDepth, receiverId, lastResult	dgnId, subscriber, forQueries, lastResult
subscriptions(node)	queryId, queryDepth, receiverId, lastResult	dgnId, peer, forQueries, answer

Cluster-wide ingest API: slice becomes partition

The cluster-wide ingest v2 endpoints change shape. The unit of work is named partition throughout, which is the standard term for dividing work across a cluster, and status is now a typed model with published field names and enum values. No stored record or in-flight request changes shape, so there is nothing to migrate, but a client reading these responses needs updating.

On the ingest record:

2.1.1	2.2.0	Note
slices	partitions	Each entry's sliceIndex is now partitionIndex
sliceStatus	status	A typed status model in place of a free-form object
generation	removed	Orchestration bookkeeping
—	settings	New. The configuration this ingest is running, with secrets redacted, in the shape a per-member ingest reports and the body a PATCH replays. Present on every response carrying a record

On the dry-run plan (clusterIngests:plan): sliceCount becomes partitionCount and slices becomes partitions.

Status now carries the fields a caller can depend on. The definition fingerprint, the superseded flag, the answering host's actor-system uid, its member index and tier, and the engine name were internal to orchestration and are no longer returned. goal and the two state fields are enums with published values, spelled in SCREAMING_SNAKE_CASE like the rest of the v2 API, in place of free-form strings, so goal is now RUNNING or PAUSED where 2.1.1 returned lowercase.

Two changes to what status reports:

• An ingest's ingestedCount and bytesIngested come from every member's persisted ingest ledger. A partition that moves between hosts carries its history with it, and the count reads the same from whichever host answers. Members write that ledger about once a minute, so the value reads zero until the first write after the ingest starts and advances a minute at a time thereafter. The per-partition counts underneath measure one host's share of one partition, metered live, so read each for what it measures rather than summing the second into the first. See What the counts mean.
• A status read that no host answers returns 503. The condition is transient and clears on its own, and the distinct status code tells a client to retry rather than treating the answer as the ingest's current state. A single unreachable host is still reported in partitionsNotReporting, as before.

See Cluster-Wide Ingest Streams for the full status shape.

Cluster ingest host addresses and metrics are permissioned

Two things now sit behind permissions that govern the same information elsewhere:

• Host addresses and pins in a cluster ingest's record and status require ClusterStatusRead. IngestRead remains the floor for the endpoint: a caller holding only IngestRead sees the ingest's counts and state, with runningOn and pinnedTo blanked.
• Cluster ingest metrics require IngestRead, the same permission that governs the ingests themselves.

A client running under a role narrower than before may find these fields blank. See OIDC and RBAC Setup for the role and capability tables.

New configuration: snapshot-after-events

One persistence key is added, along with a quine.metrics.hot-nodes block for the new metrics (see Configuration), and none is renamed. A configuration written for 2.1.1 is accepted unchanged unless it sets max-catch-up-sleep.

quine.persistence {
  # only snapshot a sleeping node once it has journaled at least this many
  # events since its last snapshot. 0, the default, snapshots on every sleep
  snapshot-after-events = 0
}

With journaling enabled, a snapshot bounds how much journal has to be replayed when a node next wakes. A node that journaled a handful of events replays quickly with or without one, so on a deployment with many lightly-touched nodes, snapshotting every one of them at every sleep is most of the write volume reaching your data store. Raising snapshot-after-events spends a little more replay time at wake to buy that write volume back.

The default is inert: at snapshot-after-events = 0, 2.2.0 snapshots on every sleep exactly as 2.1.1 did.

A non-zero value applies to sleep-time snapshots on a journaling deployment, and is accepted in the combinations where it decides something:

snapshot-after-events	journal-enabled	snapshot-singleton	snapshot-schedule	Outcome
0 (default)	any	any	any	Accepted — every 2.1.1 configuration lands here
negative	any	any	any	Refused
positive	false	any	any	Refused: without a journal, the snapshot is the durable record
positive	true	true	any	Refused: the single snapshot row is meant to hold the latest state
positive	true	false	not on-node-sleep	Refused: it decides only sleep-time snapshots
positive	true	false	on-node-sleep	Accepted

Each refusal names its reason at startup. See Persistors for the surrounding settings.

Behavior Changes

DistinctId standing query results are independent of node sleep

A DistinctId standing query now reports the same results whatever the sleep and wake pattern of the nodes involved. A node rebuilt from disk comes back holding what it held while awake, because replay records history rather than re-deciding it, and because each subscription and each cached answer now records which standing queries it is held for.

Two things follow for an upgrade:

• A match already delivered is not delivered again. What a node last reported carries forward from its 2.1.1 snapshot, or is re-derived when replaying a 2.1.1 journal.
• Your 2.1.1 state is read where it stands. Nothing is rewritten in bulk. Where a 2.1.1 record does not carry something 2.2.0 tracks, the value is derived from what the record does carry, erring toward keeping a subscription a little longer than needed. Each node narrows that to the exact set as it wakes.

Standing query state is saved on sleep independently of snapshots

MultipleValues standing query state is now written when a node sleeps, whether or not that sleep also writes a snapshot. Deployments running snapshot-schedule = never or on-node-update therefore keep this state across a node's sleep and wake, and no configuration change is needed to get it.

State written before journaling was turned off is read back

A node reads its journal at every wake. If journaling was enabled at any point in a deployment's life, the state written then comes back when the node next wakes, even if journal-enabled is false now.

For a deployment that has always run with journal-enabled = false, this is two additional data store reads per node wake, against the journals and domain_index_events tables, each returning nothing. On Cassandra that is two extra round trips per wake. If you run journaling off with a high node wake rate, measure the rate before and after the upgrade.

Change snapshot-singleton in its own step

Wake-time snapshot key resolution does nothing when the stored key shape already matches the setting, which is the case for any upgrade that leaves snapshot-singleton alone. Changing the setting moves snapshot rows to the other key shape: the bytes move verbatim rather than being re-encoded, and each write completes before its delete, so an interruption mid-move leaves two rows rather than none.

Change one thing at a time. Upgrade to 2.2.0 first, let it settle, then change snapshot-singleton in a later restart.

Server-Sent Events ingests run alongside other HTTP traffic

An SSE ingest holds its connection for the life of the stream. SSE ingests now use a connection path of their own, outside the shared outbound HTTP pool, so a long-lived SSE stream and the rest of a deployment's HTTP traffic to the same host no longer compete for the same connections.

Partition moves log at information level

A cluster relocating an ingest partition is routine, and a worker superseded by the placement that replaced it now says so at information level. Errors in the log are reserved for conditions that want attention.

Startup failures exit with the reason stated

A refused data store configuration, or a member that could not start, exits reporting which condition stopped it.