Supported Connectors and Persistors¶
Persistors¶
The persistor is where Novelty saves the model. RocksDB is the default and needs no configuration, so most deployments only revisit this choice when one of the following applies:
- RocksDB has no build for your architecture. It ships as a native binary used through JNI, and startup fails on an unsupported platform. MapDB is the portable fallback, at the cost of a 2 GB practical limit per memory-mapped file and off-heap memory use.
- You need replication and failover. Cassandra provides them, and
reaches Cassandra-compatible services too. Amazon Keyspaces is the same
trade with AWS operating it, at the cost of accepting only the
ONE,LOCAL_ONE, andLOCAL_QUORUMread consistency levels. - Nothing needs to survive the run. In-memory simulates a persistor without touching disk, and Empty makes every write a no-op, leaving only the node cache. Both are for tests and demos.
Set the choice with
thatdot.novelty.store.type.
Persistors covers each in depth, and backup is
delegated to whichever store you pick.
| Persistor | Configuration value | Description |
|---|---|---|
| RocksDB | rocks-db |
An embedded log-structured merge tree on the local filesystem. The default, and the fastest choice for a single host. |
| MapDB | map-db |
An embedded Java store on the local filesystem. The fallback where RocksDB has no native build for the host architecture. |
| Apache Cassandra | cassandra |
A distributed database giving high throughput, replication, and failover. |
| ScyllaDB | cassandra |
A Cassandra-compatible database, connected through the Cassandra persistor. |
| Astra DB | cassandra |
DataStax's serverless Cassandra-compatible service. Needs an application token and a secure connect bundle. |
| Amazon Keyspaces | keyspaces |
AWS's managed Cassandra-compatible service, for a distributed store without operating Cassandra yourself. |
| In-memory | in-memory |
Holds everything in memory and writes nothing to disk. Useful for tests and short experiments; all data is lost on shutdown. |
| Empty | empty |
Discards every write and returns nothing on read. No history, and nothing survives a restart. |
ScyllaDB and Astra DB are Cassandra-compatible, so they share the cassandra
configuration value and differ only in how you point it at them.
Ingest sources¶
An ingest stream pairs one source with an observation transformation that shapes
each record. The source decides only where the bytes
come from, so choose it by where your data already lives: a broker such as Kafka
or Kinesis for a continuous feed, a file or S3 object for data already at rest,
or Standard input and Number iterator to try something without setting up any
infrastructure.
Set it in the source block of
Create Ingest Stream: POST /api/v2/model/{modelName}/ingests.
| Name | Configuration value | Formats | Compression (ZLIB, GZIP, BASE64) | Description |
|---|---|---|---|---|
| File Ingest | File |
AvroContainer, CSV, Json, JsonL, Line, Parquet | ✓ | An active stream of data being ingested from a file on this Quine host. |
| Kafka Ingest Stream | Kafka |
Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kafka. |
| Kinesis Data Stream | Kinesis |
Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kinesis. |
| Kinesis Data Stream Using Kcl lib | KinesisKCL |
Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kinesis |
| Number Iterator Ingest | NumberIterator |
— | An infinite ingest stream which requires no data source and just produces new sequential numbers every time the stream is (re)started. The numbers are Java Longs` and will wrap at their max value. |
|
| Reactive Stream Ingest | ReactiveStream |
Avro, Drop, Json, Protobuf, Raw | A stream of data being ingested from a reactive stream. | |
| S3 Ingest | S3 |
AvroContainer, CSV, Json, JsonL, Line, Parquet | ✓ | An ingest stream from a file in S3, newline delimited. This ingest source is experimental and its behavior is subject to change. It's best suited to continuously active streams; durability is not guaranteed once a stream has been inactive for 1 minute or more. |
| Server Sent Events Stream | ServerSentEvent |
Avro, Drop, Json, Protobuf, Raw | ✓ | A server-issued event stream, as might be handled by the EventSource JavaScript API. Only consumes the data portion of an event. |
| Simple Queue Service Queue | SQS |
Avro, Drop, Json, Protobuf, Raw | ✓ | An active stream of data being ingested from AWS SQS. |
| Standard Input Ingest | StdInput |
AvroContainer, CSV, Json, JsonL, Line, Parquet | An active stream of data being ingested from standard input to this Quine process. | |
| WebSocket File Upload | WebSocketFileUpload |
AvroContainer, CSV, Json, JsonL, Line, Parquet | Streamed file upload via WebSocket protocol. | |
| Websockets Ingest Stream (Simple Startup) | WebsocketClient |
Avro, Drop, Json, Protobuf, Raw | A websocket stream started after a sequence of text messages. |
Observation result destinations¶
Each observation result passes an optional filter and then reaches one or more destinations, routed in parallel. Every destination sends results down stream, so pick by the system that should receive them: a broker such as Kafka or Kinesis to feed another pipeline, or an HTTP endpoint to notify a service directly. Drop and Log JSON to Console are what you use while building, before pointing anything at a real system.
Destinations are set in the
outputWorkflow of an ingest stream rather than on a resource of their own, through
Create Ingest Stream: POST /api/v2/model/{modelName}/ingests.
| Name | Configuration value | Formats | Description |
|---|---|---|---|
| Broadcast to Reactive Stream | ReactiveStream |
JSON, Protobuf | Broadcasts data to a created Reactive Stream. Other thatDot products can subscribe to Reactive Streams. Reactive Stream outputs are only supported in standalone (single-host) deployments. |
| Drop | Drop |
— | Effectively no destination at all, this does nothing but forget the data sent to it. |
| Log JSON to Console | StandardOut |
JSON | Prints each result as a single-line JSON object to stdout on the application server. |
| POST to HTTP[S] Webhook | HttpEndpoint |
JSON | Makes an HTTP[S] POST for each result. For the format of the result, see "Standing Query Result Output". |
| Publish to Kafka Topic | Kafka |
JSON, Protobuf | Publishes provided data to the specified Apache Kafka topic. |
| Publish to Kinesis Data Stream | Kinesis |
JSON, Protobuf | Publishes provided data to the specified Amazon Kinesis stream. |
| Publish to SNS Topic | SNS |
JSON, Protobuf | Publishes an AWS SNS record to the provided topic. To guarantee delivery, writes that fail are retried indefinitely, so confirm the credentials and topic ARN before starting this output. An unfixable error (e.g., an invalid topic ARN or missing credentials) will retry forever without emitting results, which may stop the Standing Query this output is attached to. |
| Write JSON to File | File |
JSON | Writes each result as a single-line JSON record. For the format of the result, see "Standing Query Result Output". |