Skip to content

Cluster-Wide Ingest Placement

Cluster-wide ingest streams split one ingest definition into slices and run each slice on a host. This page describes how the cluster chooses those hosts, what it does when the cluster changes shape, and what the resulting data guarantees are.

Two words are used precisely here. A host is any machine in the cluster, and it is what a slice is assigned to. A member is a host that holds a graph position and serves part of the graph. Every member is a host; not every host is a member, and a slice can run on either.

New in Quine Enterprise 2.1.1

Cluster-wide ingest streams are available from Quine Enterprise 2.1.1 onward.

Placement is decided live, not stored

The cluster manages slice placement on your behalf, continuously and for the life of the ingest. It is not a decision made once when the ingest is created, and it is not persisted: the cluster holds the ingest's definition, not a record of where its slices went.

This has a practical consequence when you read status. There is no stored assignment to report, so the runningOn field in slice status tells you where the slice is running now, which is the only answer available. A slice nothing answered for appears in slicesNotReporting rather than being reported at a location it has already left.

The same cluster shape and the same load produce the same answer, and load is smoothed over time rather than sampled raw, so a momentary spike on one host does not move slices around.

The tiers a slice can land on

Hosts fall into three tiers for the purpose of ingest placement.

Positioned members serve the graph. Each holds a position from 0 until the cluster's target size, and owns a range of shards. They run slices too, but they are filled last. See Cluster Resilience for how positions are assigned.

Ingest executors are hot spares that have been given ingest work. They serve no shards, and appear in slice status with "tier": "ingestExecutor" and a null memberIdx. A host cannot hold a graph position and be an ingest executor at the same time; taking a position ends executor duty.

Idle hot spares are spares that have not been given any ingest work. They remain available for promotion into a vacated graph position.

Hot spares run ingest work

A hot spare normally serves no graph APIs, but that limitation does not extend to cluster-wide ingest slices. A spare running slices stays in the spare pool and is still available for promotion into a vacated position.

How a host is chosen

Slices go to hosts that are not serving the graph before they go to hosts that are. The order is:

  1. hosts already running slices
  2. idle hot spares
  3. members serving the graph

A slice that is already running somewhere stays there while that host can carry it, so ordinary operation does not shuffle slices between hosts.

The practical consequence is that hosts beyond your target cluster size absorb ingest work. A cluster with spares runs its ingest on them and leaves members free to serve queries. A cluster sized exactly to its target size has nowhere else to put the work, so ingest and query serving compete for the same hosts. If ingest is loading your graph members, adding hosts is the lever. See Cluster Sizing for sizing the cluster overall.

How capacity is measured

Each host reports its own shape and its own utilization on an ongoing basis, so both figures are measured rather than assumed. Capacity is expressed in whole work units, and a host's free capacity is measured against a target utilization rather than against the machine.

Per-slice demand is not measured today; every slice costs one work unit. A host that has not reported has no capacity to spend, so no slice is placed on it.

Placement waits for a first report

The cluster will not place an unclaimed slice until every live host has reported its capacity at least once, so that no slice is placed on the basis of a partial view. Expect a brief delay before slices settle after a restart. If it lasts, the logs name the hosts still being waited for.

What happens when the cluster changes

A host fails. Its slices stop being reported, become unclaimed, and are placed on the remaining hosts. How much data this costs depends entirely on the source's resume state. See Relocation and data guarantees below.

A spare is promoted into a vacated position. Taking a graph position ends ingest-executor duty, so any slices it was running are released and re-placed.

A host joins. It reports, gains capacity, and becomes eligible. Existing slices are not moved to it merely because it is empty, since incumbency keeps them where they are, but new and unclaimed slices will prefer it if it is in the dedicated tier.

A host stops taking assignments while staying alive. Its slices stay with it rather than moving. A deliberate hand-off is serialised, so the cluster does not give a slice to a new owner while the current one still reports holding it. The logs name a host in this state and the slices it still claims.

Two hosts can nonetheless hold one slice, which is what slicesReportedByMultipleHosts reports. It happens by design while an unresumable source relocates, and it can happen when the cluster gives up on a host that is in fact still running and still writing. This is safe rather than merely tolerated: a slice's progress is recorded as a count of records processed from that slice's own starting point, so two owners can disagree about how far the work has got, never about what has been done.

Relocation and data guarantees

Cluster-wide ingest preserves at-least-once delivery. What relocation costs depends on where the source's resume state lives.

Resume state held by Sources On relocation
The source system Kafka (committed offsets), Kinesis KCL (lease table), SQS (visibility timeout) The new owner resumes where the old one committed
Quine Bounded number iterator, file The new owner resumes from the persisted record offset
Nothing Server-sent events, WebSocket, Reactive Stream Owners overlap: both run briefly, producing bounded duplicates

The last row needs care. A live feed with no resume state cannot be stopped and restarted without losing whatever arrives in between, and that loss is silent. For those three sources the cluster overlaps owners instead, accepting a bounded window of duplicate records in place of an undetectable gap. The overlap is reported as a warning when the ingest is planned and when it is created.

This is why idempotent ingest queries matter more for cluster-wide ingest than for a single-member ingest: relocation is routine here, not exceptional.

During an overlap, both hosts answer for the same slice, and the slice index appears in slicesReportedByMultipleHosts. Briefly, that is expected. Persistently, it is worth investigating.

Pinned slices

A slice pinned with pinnedTo is exempt from all of the above. It runs on its named host or nowhere: it is never relocated, and it stops until that host returns. This is what makes it possible to run a local file or standard input as a cluster-wide ingest. See Pinning an ingest to one host.

Because a lawful wait and a typo'd address look identical from the outside, a pin naming a host the cluster does not currently know is reported in warnings when the ingest is created, and listed in pinnedAwaitingHost in status for as long as it stays true.

Monitoring placement

The slice status response is the operational view. Three fields answer the questions placement raises:

  • slicesNotReporting: placed slices no host answered for. During normal operation this is empty; briefly non-empty during a relocation.
  • slicesReportedByMultipleHosts: slices two hosts both claim. Expected during an overlapping relocation, and expected to clear.
  • pinnedAwaitingHost: pinned slices whose host is absent. Lawful, but worth confirming the address is right.

Each slice's runningOn.tier tells you which tier absorbed it. A cluster where every slice reports positioned is one where ingest is competing with query serving, which is the signal to add spares.

Next steps