Delivery Guarantees¶
Quine Enterprise provides different delivery guarantees for different parts of the data pipeline. Understanding these guarantees helps you design systems with appropriate reliability characteristics.
| Component | Guarantee |
|---|---|
| Ingest | At-Least-Once |
| Standing Query Outputs | Best Effort |
Ingest: At-Least-Once¶
Ingest streams provide at-least-once delivery guarantees when properly configured. This means:
- Every ingested record will be processed at least once
- In failure scenarios (crash, restart), some records may be reprocessed
- No records will be silently dropped
How It Works¶
Quine Enterprise achieves at-least-once semantics by committing source offsets after successfully writing data to the graph:
Source → Deserialize → Write to Graph → Commit Offset
If Quine Enterprise crashes after writing but before committing, the record will be reprocessed on restart. This is the standard trade-off for at-least-once delivery.
To achieve at-least-once guarantees, you need a source that supports offset tracking (such as Kafka, SQS, or Kinesis with KCL) configured to commit offsets only after successful processing.
Idempotent Ingest Queries¶
Because at-least-once delivery may result in duplicate processing, your ingest queries should be idempotent. Processing the same record twice should produce the same result. The idFrom() function helps achieve this by generating deterministic node IDs from your data.
Standing Query Outputs: Best Effort¶
Standing query outputs are delivered on a best-effort basis. While Quine Enterprise includes backpressure mechanisms to prevent result loss during normal operation, results can be dropped under certain conditions.
Backpressure Mechanism¶
When standing query results are produced faster than outputs can consume them, Quine Enterprise applies backpressure by pausing ingest. This keeps the result queue from growing unbounded and allows outputs to catch up.
However, the result queue has a maximum size. If the queue fills completely, new results are dropped rather than buffered indefinitely. This can happen in pathological cases such as patterns that trigger cascading matches.
When Results May Be Lost¶
Standing query results can be lost in these scenarios:
- Queue overflow: New results are dropped when the result queue is full
- Output failure: If an output destination fails, the standing query is cancelled
- Process restart: In-flight results in the queue can be lost on restart
Designing for Best-Effort Delivery¶
When building systems that consume standing query outputs:
- Treat outputs as notifications rather than authoritative records
- Design for duplicate delivery: The same result may be emitted more than once in edge cases
- Consider downstream buffering: If guaranteed delivery is required, route outputs through a message queue that provides stronger guarantees