Kafka¶
With services.kafka: local, quinectl runs a single-node Kafka broker, creates the topics you list, and
can load each topic once from a file of sample records. It then prints the exact request that creates
an ingest reading from each topic.
quine:
edition: enterprise
licenseKey: ${QE_LICENSE_KEY}
customerId: ${QE_CUSTOMER_ID}
services:
persistor: cassandra
auth: keycloak
proxy: none
metrics: none
kafka: local
Create the seed file the example loads into events, in your project directory:
mkdir -p seed
cat > seed/events.jsonl <<'EOF'
{"id":1,"kind":"demo"}
{"id":2,"kind":"demo"}
{"id":3,"kind":"demo"}
{"id":4,"kind":"demo"}
{"id":5,"kind":"demo"}
{"id":6,"kind":"demo"}
EOF
Then add the topics to quinectl.yaml:
quine:
edition: enterprise
licenseKey: ${QE_LICENSE_KEY}
customerId: ${QE_CUSTOMER_ID}
services:
persistor: cassandra
auth: keycloak
proxy: none
metrics: none
kafka:
local:
namespace: my-project
topics:
- name: events
partitions: 3
seedFile: ./seed/events.jsonl
- name: users
Every project on the machine shares the one broker container, quinectl-kafka. It has two addresses:
| From | Address |
|---|---|
| Your machine, or a product jar | 127.0.0.1:9092 |
| A product running from its image | host.docker.internal:9094 |
The second is the broker's port plus 2, and works on macOS and Linux alike. If you set
services.kafka.local.port, both addresses move with it.
Use the short form when your own code produces to Kafka
kafka: local on its own starts an empty
broker: no
topics, no seeding, and no printed ingest. The broker doesn't create topics on first use, so create
each one before producing to it:
docker exec quinectl-kafka kafka-topics --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1
quinectl up and quinectl kafka-reset leave topics you created yourself alone, but quinectl gc
offers to delete any topic no project declares, so read quinectl gc --dry-run before you confirm.
Topics¶
Each topic has a name, a partitions count (default 1), and an optional seedFile.
A topic's partition count can't change after it is created. To change it, run quinectl kafka-reset,
which deletes this project's topics so the next up creates them again.
Namespaces¶
Because the broker is shared, topic names are shared too. Without a namespace, quinectl creates each
topic under the name in the file. Two projects that both declare events then use one events topic,
and it holds one seed file at a time. The second project's first up deletes the topic and loads its
own file in its place, and a seed-file change in either project does the same for both. up warns when
this happens, naming the other project.
namespace gives this project its own topics. With namespace: my-project, the topic events is
created on the broker as my-project.events:
✓ topic events created as my-project.events (3 partitions)
Inside quinectl, the topic is still events: in quinectl.yaml, in quinectl status, and as the
name of the printed ingest. Anything that talks to the broker directly needs the full name: the
printed ingest's topics: already has it, and so must your own producers, consumers and
kafka-topics commands. quinectl kafka-reset deletes only this project's topics, under its
namespace. Without a namespace, it deletes a topic shared with another project for both.
A namespace may contain letters, digits, _ and -, and no dots; quinectl refuses any other
value before starting anything. Changing the namespace creates the topics again under the new name,
and leaves the old ones on the broker; run quinectl kafka-reset before the change to remove them.
Leave the namespace empty when only one project uses Kafka on the machine and your clients expect the plain topic names. To run several projects side by side, see Multi-tenancy.
Seed files¶
A seedFile is loaded into its topic once, on the up that creates the topic. Supported formats, by
extension:
| Extension | Contents |
|---|---|
.jsonl or .ndjson |
One JSON object per line |
.json |
A JSON array of objects, or one object |
.csv |
A header line, then one record per line; every value is kept as a string |
Relative paths are relative to quinectl.yaml, and the file must exist: if it doesn't, up stops
before starting anything, with an error naming the file. Among up's other lines, the end of the load
looks like this:
→ kafka: seeding events 100% [0s]
✓ topic events seeded (6 records from seed/events.jsonl)
quinectl status shows how many records each topic holds:
kafka topics (127.0.0.1:9092):
TOPIC STATE PARTITIONS RECORDS PER PARTITION
events present 3 6 p0=2 p1=2 p2=2
users present 1 0 p0=0
When other projects have topics on the broker, status adds a line under the table that counts them
and names the directories that declare them.
An unchanged seed file is never loaded again. If you edit it, the next up deletes the topic, creates
it again, and loads the new contents, so the topic matches the file again.
Editing a seed file replays the topic
An ingest already reading the topic doesn't know it was recreated. An ingest on one member reads the whole topic again from the start: keep its query idempotent, as the printed one is, or stop it before editing a seed file. A cluster-wide ingest may read only some partitions again, so stop it first and create it again afterwards.
Ingest a topic¶
quinectl doesn't create ingests itself. Once Quine Enterprise is running, up prints a ready-to-run
request for each topic. Copy it exactly as printed, including the final EOF line, and
change the query to fit your data:
curl -s -X POST http://localhost:8080/api/v2/graph/quine/ingests \
-H 'Content-Type: application/yaml' \
-H "Authorization: Bearer $(quinectl token)" \
--data-binary @- <<'EOF'
name: events
source:
type: Kafka
format: {type: Json}
topics: [my-project.events]
bootstrapServers: host.docker.internal:9094
groupId: quinectl-63354509e1be-events
autoOffsetReset: EARLIEST
query: |
MATCH (n)
WHERE id(n) = idFrom($that)
SET n = $that
parameter: that
EOF
- The
Authorizationline appears only when auth is on. bootstrapServersis the address the product uses to reach the broker from inside its container. It differs from the127.0.0.1:9092you use from your machine. Use the value quinectl prints.groupIdis unique to this project, so two projects reading one topic each get every record.- The same request is in
quinectl status --json, underkafka.topics[].ingest.
The ingest API is described in the REST API reference.
Novelty gets no printed request. Create its ingest yourself under /api/v2/model/{modelName}/ingests,
with the broker's address for a product image, host.docker.internal:9094, and the topic's full name;
see the Novelty REST API reference.
Cluster-wide ingest¶
On a Quine Enterprise cluster with a size of 2 or more, hot spares not counted, quinectl
prints a cluster-wide ingest instead. It posts to
/api/v2/graph/quine/clusterIngests
and places the topic's partitions across the cluster.
- Placement: hot spares take partitions first, so the shares aren't always even. With spares, a small topic can run entirely on one spare; see Cluster-Wide Ingest Placement.
- Preview: to see how many partitions it will have before creating it, post the same body to
/api/v2/graph/quine/clusterIngests:plan. The plan doesn't say which member runs each partition; once the ingest is running, its status shows that inrunningOn. - Which kind:
quinectl status --jsonreports which kind of ingest a topic gets inkafka.topics[].ingest.scope, asmemberorcluster.
For what a cluster-wide ingest is and how to manage one, see Cluster-Wide Ingest Streams.
Reset and clean up¶
To start this project's topics over:
quinectl kafka-reset
This asks first; in a script, add --yes. It deletes this project's own topics and forgets that their seed
files were loaded, so the next up creates and loads them again. Other projects' topics and the broker are
not touched, except a topic shared with another project because neither uses a namespace or both use the same one.
Removing the broker is covered in Multi-tenancy. Run
kafka-reset first when you're finished with a project, or its topics stay on the broker until
quinectl gc removes them.
Next steps¶
- To change the printed ingest beyond its
query, see the Kafka ingest source in Apache Kafka. - To ingest a topic across several members, Run a Local Cluster.