Skip to content

Kafka

With services.kafka: local, quinectl runs a single-node Kafka broker, creates the topics you list, and can load each topic once from a file of sample records. It then prints the exact request that creates an ingest reading from each topic.

quinectl.yaml
quine:
  edition: enterprise
  licenseKey: ${QE_LICENSE_KEY}
  customerId: ${QE_CUSTOMER_ID}
services:
  persistor: cassandra
  auth: keycloak
  proxy: none
  metrics: none
  kafka: local

Create the seed file the example loads into events, in your project directory:

mkdir -p seed
cat > seed/events.jsonl <<'EOF'
{"id":1,"kind":"demo"}
{"id":2,"kind":"demo"}
{"id":3,"kind":"demo"}
{"id":4,"kind":"demo"}
{"id":5,"kind":"demo"}
{"id":6,"kind":"demo"}
EOF

Then add the topics to quinectl.yaml:

quinectl.yaml
quine:
  edition: enterprise
  licenseKey: ${QE_LICENSE_KEY}
  customerId: ${QE_CUSTOMER_ID}
services:
  persistor: cassandra
  auth: keycloak
  proxy: none
  metrics: none
  kafka:
    local:
      namespace: my-project
      topics:
        - name: events
          partitions: 3
          seedFile: ./seed/events.jsonl
        - name: users

Every project on the machine shares the one broker container, quinectl-kafka. It has two addresses:

From Address
Your machine, or a product jar 127.0.0.1:9092
A product running from its image host.docker.internal:9094

The second is the broker's port plus 2, and works on macOS and Linux alike. If you set services.kafka.local.port, both addresses move with it.

Use the short form when your own code produces to Kafka

kafka: local on its own starts an empty broker: no topics, no seeding, and no printed ingest. The broker doesn't create topics on first use, so create each one before producing to it:

docker exec quinectl-kafka kafka-topics --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1

quinectl up and quinectl kafka-reset leave topics you created yourself alone, but quinectl gc offers to delete any topic no project declares, so read quinectl gc --dry-run before you confirm.

Topics

Each topic has a name, a partitions count (default 1), and an optional seedFile.

A topic's partition count can't change after it is created. To change it, run quinectl kafka-reset, which deletes this project's topics so the next up creates them again.

Namespaces

Because the broker is shared, topic names are shared too. Without a namespace, quinectl creates each topic under the name in the file. Two projects that both declare events then use one events topic, and it holds one seed file at a time. The second project's first up deletes the topic and loads its own file in its place, and a seed-file change in either project does the same for both. up warns when this happens, naming the other project.

namespace gives this project its own topics. With namespace: my-project, the topic events is created on the broker as my-project.events:

✓ topic events created as my-project.events (3 partitions)

Inside quinectl, the topic is still events: in quinectl.yaml, in quinectl status, and as the name of the printed ingest. Anything that talks to the broker directly needs the full name: the printed ingest's topics: already has it, and so must your own producers, consumers and kafka-topics commands. quinectl kafka-reset deletes only this project's topics, under its namespace. Without a namespace, it deletes a topic shared with another project for both.

A namespace may contain letters, digits, _ and -, and no dots; quinectl refuses any other value before starting anything. Changing the namespace creates the topics again under the new name, and leaves the old ones on the broker; run quinectl kafka-reset before the change to remove them.

Leave the namespace empty when only one project uses Kafka on the machine and your clients expect the plain topic names. To run several projects side by side, see Multi-tenancy.

Seed files

A seedFile is loaded into its topic once, on the up that creates the topic. Supported formats, by extension:

Extension Contents
.jsonl or .ndjson One JSON object per line
.json A JSON array of objects, or one object
.csv A header line, then one record per line; every value is kept as a string

Relative paths are relative to quinectl.yaml, and the file must exist: if it doesn't, up stops before starting anything, with an error naming the file. Among up's other lines, the end of the load looks like this:

→ kafka: seeding events 100% [0s]
✓ topic events seeded (6 records from seed/events.jsonl)

quinectl status shows how many records each topic holds:

kafka topics (127.0.0.1:9092):
  TOPIC   STATE    PARTITIONS  RECORDS  PER PARTITION
  events  present  3           6        p0=2 p1=2 p2=2
  users   present  1           0        p0=0

When other projects have topics on the broker, status adds a line under the table that counts them and names the directories that declare them.

An unchanged seed file is never loaded again. If you edit it, the next up deletes the topic, creates it again, and loads the new contents, so the topic matches the file again.

Editing a seed file replays the topic

An ingest already reading the topic doesn't know it was recreated. An ingest on one member reads the whole topic again from the start: keep its query idempotent, as the printed one is, or stop it before editing a seed file. A cluster-wide ingest may read only some partitions again, so stop it first and create it again afterwards.

Ingest a topic

quinectl doesn't create ingests itself. Once Quine Enterprise is running, up prints a ready-to-run request for each topic. Copy it exactly as printed, including the final EOF line, and change the query to fit your data:

curl -s -X POST http://localhost:8080/api/v2/graph/quine/ingests \
  -H 'Content-Type: application/yaml' \
  -H "Authorization: Bearer $(quinectl token)" \
  --data-binary @- <<'EOF'
name: events
source:
  type: Kafka
  format: {type: Json}
  topics: [my-project.events]
  bootstrapServers: host.docker.internal:9094
  groupId: quinectl-63354509e1be-events
  autoOffsetReset: EARLIEST
query: |
  MATCH (n)
  WHERE id(n) = idFrom($that)
  SET n = $that
parameter: that
EOF
  • The Authorization line appears only when auth is on.
  • bootstrapServers is the address the product uses to reach the broker from inside its container. It differs from the 127.0.0.1:9092 you use from your machine. Use the value quinectl prints.
  • groupId is unique to this project, so two projects reading one topic each get every record.
  • The same request is in quinectl status --json, under kafka.topics[].ingest.

The ingest API is described in the REST API reference.

Novelty gets no printed request. Create its ingest yourself under /api/v2/model/{modelName}/ingests, with the broker's address for a product image, host.docker.internal:9094, and the topic's full name; see the Novelty REST API reference.

Cluster-wide ingest

On a Quine Enterprise cluster with a size of 2 or more, hot spares not counted, quinectl prints a cluster-wide ingest instead. It posts to /api/v2/graph/quine/clusterIngests and places the topic's partitions across the cluster.

  • Placement: hot spares take partitions first, so the shares aren't always even. With spares, a small topic can run entirely on one spare; see Cluster-Wide Ingest Placement.
  • Preview: to see how many partitions it will have before creating it, post the same body to /api/v2/graph/quine/clusterIngests:plan. The plan doesn't say which member runs each partition; once the ingest is running, its status shows that in runningOn.
  • Which kind: quinectl status --json reports which kind of ingest a topic gets in kafka.topics[].ingest.scope, as member or cluster.

For what a cluster-wide ingest is and how to manage one, see Cluster-Wide Ingest Streams.

Reset and clean up

To start this project's topics over:

quinectl kafka-reset

This asks first; in a script, add --yes. It deletes this project's own topics and forgets that their seed files were loaded, so the next up creates and loads them again. Other projects' topics and the broker are not touched, except a topic shared with another project because neither uses a namespace or both use the same one.

Removing the broker is covered in Multi-tenancy. Run kafka-reset first when you're finished with a project, or its topics stay on the broker until quinectl gc removes them.

Next steps

  • To change the printed ingest beyond its query, see the Kafka ingest source in Apache Kafka.
  • To ingest a topic across several members, Run a Local Cluster.