Multi-tenancy¶
Each directory with a quinectl.yaml is its own project, and several projects can run on one machine at
once. They share one Cassandra, Keycloak, Kafka, Prometheus and Grafana, and each keeps its own graph,
users, topics, dashboards and product members. Use it to keep a demo, a test suite and your own
experiments apart without running a second copy of each service.
What is shared and what isn't¶
| Service | Shared by every project? | What keeps projects apart |
|---|---|---|
| Cassandra | Yes, quinectl-cassandra |
A different quine.keyspace (and novelty.keyspace) |
| Keycloak | Yes, quinectl-keycloak |
A different quine.realm (and novelty.realm) |
| Kafka | Yes, quinectl-kafka |
A different services.kafka.local.namespace |
| Product members | No | A different quine.port and novelty.port |
| Prometheus | Yes, quinectl-prometheus |
Nothing to set: quinectl labels each project's members and picks their ports |
| Grafana | Yes, quinectl-grafana |
Nothing to set: each project gets its own data source and dashboards |
| Reverse proxy | No, one per project | A different services.proxy.nginx.port |
The shared ports, CQL 9042, Keycloak 8443, Kafka 9092, Prometheus 9090 and Grafana 3000, stay
the same for every project. Don't change them to separate projects: separation comes from the keyspace,
realm and namespace.
Example: two projects on one machine¶
Two directories, project-a and project-b, each with its own file. The highlighted lines are what
keeps them apart. project-a keeps the default port; project-b moves its web port out of the way.
Both turn metrics on, and neither needs a metrics setting:
quine:
edition: enterprise
licenseKey: ${QE_LICENSE_KEY}
customerId: ${QE_CUSTOMER_ID}
keyspace: project_a
realm: project-a
services:
persistor: cassandra
auth: keycloak
proxy: none
metrics: prometheus-grafana
kafka:
local:
namespace: project-a
topics:
- name: events
quine:
edition: enterprise
port: 8081
licenseKey: ${QE_LICENSE_KEY}
customerId: ${QE_CUSTOMER_ID}
keyspace: project_b
realm: project-b
services:
persistor: cassandra
auth: keycloak
proxy: none
metrics: prometheus-grafana
kafka:
local:
namespace: project-b
topics:
- name: events
Run quinectl up in each directory. The first up starts the shared services, and the second reuses
them. Among its other output, the second up prints these lines about project-b's own topic, realm,
dashboards and web UI:
✓ topic events created as project-b.events (1 partition)
✓ keycloak realm project-b converged
✓ metrics converged: grafana http://localhost:3000/d/quinectl-quine-afc995e30d9f (no login needed) — prometheus http://localhost:9090, scraping 1/1 members
✓ quine is up: http://localhost:8081
The dashboard id in the Grafana link is derived from the directory's path, so yours differs.
The two projects now have:
| What | project-a | project-b | Shared container |
|---|---|---|---|
| Quine Enterprise | http://localhost:8080 | http://localhost:8081 | None, one each |
| Graph | Keyspace project_a |
Keyspace project_b |
quinectl-cassandra |
| Users and tokens | Realm project-a |
Realm project-b |
quinectl-keycloak |
events topic |
project-a.events |
project-b.events |
quinectl-kafka |
| Metrics | Member port 11091, if started first |
Member port 11092, if started second |
quinectl-prometheus |
| Dashboards | Its own, showing only its data | Its own, showing only its data | quinectl-grafana |
Each project has its own sample users, so quinectl token gives a token for that project only. A
token from project-a is refused by project-b with 401. Both projects write the events topic
in their files, but on the broker they are two topics, and quinectl kafka-reset in project-b
deletes only project-b.events.
quinectl status in either directory lists the members of every project, with each one's directory.
What keeps projects apart¶
Graph: keyspace¶
quine.keyspace (and novelty.keyspace) names the Cassandra keyspace that holds the graph. The
default is quine for Quine Enterprise and novelty for Novelty, so two projects that both keep the default
share one graph. quinectl cass-reset deletes only this project's keyspaces, and refuses while another
project declares the same keyspace. RocksDB and in-memory storage are already private to a project; see
Graph Storage.
Users: realm¶
quine.realm (and novelty.realm) names the Keycloak realm with the project's users and roles. The
default is quine-enterprise for Quine Enterprise. Users you list under users: are created in this project's
realm only. See Auth and RBAC with Keycloak.
Kafka topics: namespace¶
services.kafka.local.namespace puts this project's topics under a prefix on the shared broker: with
namespace: project-a, the topic events is project-a.events on the broker. Without a namespace,
topics keep the names in the file, so two projects that declare events share one topic. See
Kafka namespaces.
Ports¶
Each project needs its own ports for these, because two processes can't share a port, and up refuses
two projects whose ports overlap:
quine.portis the web UI and REST API, default8080, and one more for each extra member.novelty.portis Novelty's,8090in a generated file.quine.cluster.portBasecan be shared by projects running from the image. Withruntime: jar, give each project its own range.services.proxy.nginx.portis the reverse proxy's. It runs one per project, so give each its own.
Metrics: nothing to set¶
Prometheus and Grafana need no settings to keep projects apart. quinectl picks each project's member metrics ports, and gives it its own data source and dashboards, which show only that project's data. A few settings, such as Grafana's sign-in, apply to every project. See Sharing with other projects.
When two projects collide¶
If two projects name the same keyspace, realm or Kafka topic, up warns, naming the other directory,
and carries on:
warning: kafka topic "project-a.events" is also declared by the config in /Users/me/project-a (enterprise, 2.2.1) — both reconcile and seed the same topic on the shared broker; a seed-file change in either directory recreates it
Sharing is allowed when you want it, for example two projects reading one topic. But two projects running different product versions on one keyspace can stop a product from starting, and a shared topic follows whichever project last changed its seed file.
Let quinectl pick the values¶
To keep a project clear of the defaults without choosing values yourself, write its file with
quinectl init --agent. It picks the web and cluster ports, keyspace and realm from the directory's
path, and suggests a proxy port and Kafka namespace in comments:
| Value | Default | With --agent |
|---|---|---|
quine.port |
8080 |
Between 18080 and 18979 |
quine.cluster.portBase |
25520 |
35520 and up, spaced 16 apart |
quine.keyspace |
quine |
quine_<directory>_<hash> |
quine.realm |
quine-enterprise |
quine-<directory>-<hash> |
Kafka namespace |
none | Suggested in a comment: quine-<directory>-<hash> |
| Proxy port | 80 |
Suggested in a comment, in the 10000s |
The same directory always gets the same values. Two directories can occasionally get the same port;
up then refuses, so set quine.port to a free one. Novelty, with --with-novelty, gets values of its
own beside Quine Enterprise's. The shared services' ports stay as they are.
The proxy port and Kafka namespace are comments because neither service is on yet. When you turn one
on, use the long form and copy the suggested value. The short form proxy: nginx uses port 80, and
Kafka topics without the namespace line are shared with every project on the machine.
Stopping and removing¶
To stop this project and keep everything for the next up, run quinectl down. To free memory or
start clean, use --rm or --purge, below.
| Command | Stops | Removes |
|---|---|---|
quinectl down |
This project's members | Their containers, and no data |
quinectl down --rm |
Every quinectl service, if no other project uses them | The service containers and the graph in Cassandra, if no other project still uses them. Kafka topics and Keycloak users are kept |
quinectl down --purge |
Every quinectl service, if no other project uses them | Containers and their data, if no other project uses them |
quinectl down --all |
Every project's members, every quinectl service | Member containers, and no data |
quinectl down --all --rm |
Everything quinectl started on the machine | Every quinectl container and the graph in Cassandra |
quinectl down --all --purge |
Everything quinectl started on the machine | Every quinectl container and its data |
- Plain
downis always safe. It stops only this directory's members, and leaves every service running for the nextup. -
--rmand--purgewithout--allneed this project's members stopped first. While any may still be running, they refuse and change nothing:error: refusing --rm: this directory still has members that may be running: container quinectl-quine-766ac9d7fada-0 — nothing was touched; run 'quinectl down' first (it stops them and reports any pid record it cannot read or verify), then re-run 'quinectl down --rm'Run plain
quinectl down, thenquinectl down --rm. ---rmand--purgewithout--allrefuse to remove a shared container another project still uses, and name each project that uses each one:error: refusing --rm: quinectl-cassandra/quinectl-keycloak are still in use by other tenants quinectl-cassandra: /Users/me/project-b (keyspace project_b) quinectl-keycloak: /Users/me/project-b (realm project-b) note: this directory is already unregistered and its own data is untouched by this refusal; if you are tearing down only this directory (after 'quinectl down', 'quinectl cass-reset', 'quinectl kafka-reset'), you are done — the shared containers stay for the users listed above fix: to remove the shared containers too, each listed directory's owner runs 'quinectl cass-reset' / 'quinectl kafka-reset' (or drops the realm) then 'quinectl down --rm' there — or deletes that directory and runs 'quinectl gc'; once no users remain, this directory's --rm/--purge proceedsThe first line names each shared container still in use. With Kafka and metrics on, as in the example projects above, it also names
quinectl-kafkaand the metrics stack, and lists their users.The refusal still changes this project. If it uses metrics, its dashboards are removed from the shared Grafana. It also drops off quinectl's records until its next
up. Until then,quinectl gccounts its keyspace, realm and topics as unused and offers to delete them, so runquinectl uphere before you rungc.To remove the shared containers, use
quinectl down --all --rm, or delete the other projects' directories and runquinectl gc. ---all --rmand--all --purgelist the other projects they would affect and ask before removing anything. If no other project uses the services, they remove everything without asking.--yesanswers for a script. Preview first;--dry-runchanges nothing:quinectl down --all --purge --dry-run -
quinectl downnever stops or removes anything quinectl didn't start. A Quine process it finds running without having started it is reported, and is stopped only with--orphans.
To delete only the data, and keep the containers running, use quinectl cass-reset for the graph and
quinectl kafka-reset for the topics. cass-reset refuses while another project declares the same
keyspace. When two projects share a Kafka topic, because neither uses a namespace or both use the same
one, kafka-reset deletes it for both.
Cleaning up after deleted projects¶
Deleting a project directory doesn't remove its keyspace, realm or topics. quinectl gc finds what
deleted projects left behind on this machine. That includes quinectl's own records, their Grafana
dashboards, and keyspaces, realms, Kafka topics and per-project containers that no remaining project
uses. It offers to remove them:
quinectl gc --dry-run
quinectl gc
It asks before deleting anything, and lists, but never deletes, anything in a service quinectl didn't
start. It counts anything no project declares as left behind, including Kafka topics you created yourself,
so read the --dry-run list before you confirm.
Next steps¶
- Keep topics apart on the shared broker: Kafka namespaces.
- Give each project its own users and roles: Auth and RBAC with Keycloak.