Skip to content

Multi-tenancy

Each directory with a quinectl.yaml is its own project, and several projects can run on one machine at once. They share one Cassandra, Keycloak, Kafka, Prometheus and Grafana, and each keeps its own graph, users, topics, dashboards and product members. Use it to keep a demo, a test suite and your own experiments apart without running a second copy of each service.

What is shared and what isn't

Service Shared by every project? What keeps projects apart
Cassandra Yes, quinectl-cassandra A different quine.keyspace (and novelty.keyspace)
Keycloak Yes, quinectl-keycloak A different quine.realm (and novelty.realm)
Kafka Yes, quinectl-kafka A different services.kafka.local.namespace
Product members No A different quine.port and novelty.port
Prometheus Yes, quinectl-prometheus Nothing to set: quinectl labels each project's members and picks their ports
Grafana Yes, quinectl-grafana Nothing to set: each project gets its own data source and dashboards
Reverse proxy No, one per project A different services.proxy.nginx.port

The shared ports, CQL 9042, Keycloak 8443, Kafka 9092, Prometheus 9090 and Grafana 3000, stay the same for every project. Don't change them to separate projects: separation comes from the keyspace, realm and namespace.

Example: two projects on one machine

Two directories, project-a and project-b, each with its own file. The highlighted lines are what keeps them apart. project-a keeps the default port; project-b moves its web port out of the way. Both turn metrics on, and neither needs a metrics setting:

quinectl.yaml
quine:
  edition: enterprise
  licenseKey: ${QE_LICENSE_KEY}
  customerId: ${QE_CUSTOMER_ID}
  keyspace: project_a
  realm: project-a
services:
  persistor: cassandra
  auth: keycloak
  proxy: none
  metrics: prometheus-grafana
  kafka:
    local:
      namespace: project-a
      topics:
        - name: events
quinectl.yaml
quine:
  edition: enterprise
  port: 8081
  licenseKey: ${QE_LICENSE_KEY}
  customerId: ${QE_CUSTOMER_ID}
  keyspace: project_b
  realm: project-b
services:
  persistor: cassandra
  auth: keycloak
  proxy: none
  metrics: prometheus-grafana
  kafka:
    local:
      namespace: project-b
      topics:
        - name: events

Run quinectl up in each directory. The first up starts the shared services, and the second reuses them. Among its other output, the second up prints these lines about project-b's own topic, realm, dashboards and web UI:

✓ topic events created as project-b.events (1 partition)
✓ keycloak realm project-b converged
✓ metrics converged: grafana http://localhost:3000/d/quinectl-quine-afc995e30d9f (no login needed) — prometheus http://localhost:9090, scraping 1/1 members
✓ quine is up: http://localhost:8081

The dashboard id in the Grafana link is derived from the directory's path, so yours differs.

The two projects now have:

What project-a project-b Shared container
Quine Enterprise http://localhost:8080 http://localhost:8081 None, one each
Graph Keyspace project_a Keyspace project_b quinectl-cassandra
Users and tokens Realm project-a Realm project-b quinectl-keycloak
events topic project-a.events project-b.events quinectl-kafka
Metrics Member port 11091, if started first Member port 11092, if started second quinectl-prometheus
Dashboards Its own, showing only its data Its own, showing only its data quinectl-grafana

Each project has its own sample users, so quinectl token gives a token for that project only. A token from project-a is refused by project-b with 401. Both projects write the events topic in their files, but on the broker they are two topics, and quinectl kafka-reset in project-b deletes only project-b.events.

quinectl status in either directory lists the members of every project, with each one's directory.

What keeps projects apart

Graph: keyspace

quine.keyspace (and novelty.keyspace) names the Cassandra keyspace that holds the graph. The default is quine for Quine Enterprise and novelty for Novelty, so two projects that both keep the default share one graph. quinectl cass-reset deletes only this project's keyspaces, and refuses while another project declares the same keyspace. RocksDB and in-memory storage are already private to a project; see Graph Storage.

Users: realm

quine.realm (and novelty.realm) names the Keycloak realm with the project's users and roles. The default is quine-enterprise for Quine Enterprise. Users you list under users: are created in this project's realm only. See Auth and RBAC with Keycloak.

Kafka topics: namespace

services.kafka.local.namespace puts this project's topics under a prefix on the shared broker: with namespace: project-a, the topic events is project-a.events on the broker. Without a namespace, topics keep the names in the file, so two projects that declare events share one topic. See Kafka namespaces.

Ports

Each project needs its own ports for these, because two processes can't share a port, and up refuses two projects whose ports overlap:

  • quine.port is the web UI and REST API, default 8080, and one more for each extra member.
  • novelty.port is Novelty's, 8090 in a generated file.
  • quine.cluster.portBase can be shared by projects running from the image. With runtime: jar, give each project its own range.
  • services.proxy.nginx.port is the reverse proxy's. It runs one per project, so give each its own.

Metrics: nothing to set

Prometheus and Grafana need no settings to keep projects apart. quinectl picks each project's member metrics ports, and gives it its own data source and dashboards, which show only that project's data. A few settings, such as Grafana's sign-in, apply to every project. See Sharing with other projects.

When two projects collide

If two projects name the same keyspace, realm or Kafka topic, up warns, naming the other directory, and carries on:

warning: kafka topic "project-a.events" is also declared by the config in /Users/me/project-a (enterprise, 2.2.1) — both reconcile and seed the same topic on the shared broker; a seed-file change in either directory recreates it

Sharing is allowed when you want it, for example two projects reading one topic. But two projects running different product versions on one keyspace can stop a product from starting, and a shared topic follows whichever project last changed its seed file.

Let quinectl pick the values

To keep a project clear of the defaults without choosing values yourself, write its file with quinectl init --agent. It picks the web and cluster ports, keyspace and realm from the directory's path, and suggests a proxy port and Kafka namespace in comments:

Value Default With --agent
quine.port 8080 Between 18080 and 18979
quine.cluster.portBase 25520 35520 and up, spaced 16 apart
quine.keyspace quine quine_<directory>_<hash>
quine.realm quine-enterprise quine-<directory>-<hash>
Kafka namespace none Suggested in a comment: quine-<directory>-<hash>
Proxy port 80 Suggested in a comment, in the 10000s

The same directory always gets the same values. Two directories can occasionally get the same port; up then refuses, so set quine.port to a free one. Novelty, with --with-novelty, gets values of its own beside Quine Enterprise's. The shared services' ports stay as they are.

The proxy port and Kafka namespace are comments because neither service is on yet. When you turn one on, use the long form and copy the suggested value. The short form proxy: nginx uses port 80, and Kafka topics without the namespace line are shared with every project on the machine.

Stopping and removing

To stop this project and keep everything for the next up, run quinectl down. To free memory or start clean, use --rm or --purge, below.

Command Stops Removes
quinectl down This project's members Their containers, and no data
quinectl down --rm Every quinectl service, if no other project uses them The service containers and the graph in Cassandra, if no other project still uses them. Kafka topics and Keycloak users are kept
quinectl down --purge Every quinectl service, if no other project uses them Containers and their data, if no other project uses them
quinectl down --all Every project's members, every quinectl service Member containers, and no data
quinectl down --all --rm Everything quinectl started on the machine Every quinectl container and the graph in Cassandra
quinectl down --all --purge Everything quinectl started on the machine Every quinectl container and its data
  • Plain down is always safe. It stops only this directory's members, and leaves every service running for the next up.
  • --rm and --purge without --all need this project's members stopped first. While any may still be running, they refuse and change nothing:

    error: refusing --rm: this directory still has members that may be running: container quinectl-quine-766ac9d7fada-0 — nothing was touched; run 'quinectl down' first (it stops them and reports any pid record it cannot read or verify), then re-run 'quinectl down --rm'
    

    Run plain quinectl down, then quinectl down --rm. - --rm and --purge without --all refuse to remove a shared container another project still uses, and name each project that uses each one:

    error: refusing --rm: quinectl-cassandra/quinectl-keycloak are still in use by other tenants
      quinectl-cassandra:
        /Users/me/project-b (keyspace project_b)
      quinectl-keycloak:
        /Users/me/project-b (realm project-b)
    note: this directory is already unregistered and its own data is untouched by this refusal; if you are tearing down only this directory (after 'quinectl down', 'quinectl cass-reset', 'quinectl kafka-reset'), you are done — the shared containers stay for the users listed above
    fix: to remove the shared containers too, each listed directory's owner runs 'quinectl cass-reset' / 'quinectl kafka-reset' (or drops the realm) then 'quinectl down --rm' there — or deletes that directory and runs 'quinectl gc'; once no users remain, this directory's --rm/--purge proceeds
    

    The first line names each shared container still in use. With Kafka and metrics on, as in the example projects above, it also names quinectl-kafka and the metrics stack, and lists their users.

    The refusal still changes this project. If it uses metrics, its dashboards are removed from the shared Grafana. It also drops off quinectl's records until its next up. Until then, quinectl gc counts its keyspace, realm and topics as unused and offers to delete them, so run quinectl up here before you run gc.

    To remove the shared containers, use quinectl down --all --rm, or delete the other projects' directories and run quinectl gc. - --all --rm and --all --purge list the other projects they would affect and ask before removing anything. If no other project uses the services, they remove everything without asking. --yes answers for a script. Preview first; --dry-run changes nothing:

    quinectl down --all --purge --dry-run
    
  • quinectl down never stops or removes anything quinectl didn't start. A Quine process it finds running without having started it is reported, and is stopped only with --orphans.

To delete only the data, and keep the containers running, use quinectl cass-reset for the graph and quinectl kafka-reset for the topics. cass-reset refuses while another project declares the same keyspace. When two projects share a Kafka topic, because neither uses a namespace or both use the same one, kafka-reset deletes it for both.

Cleaning up after deleted projects

Deleting a project directory doesn't remove its keyspace, realm or topics. quinectl gc finds what deleted projects left behind on this machine. That includes quinectl's own records, their Grafana dashboards, and keyspaces, realms, Kafka topics and per-project containers that no remaining project uses. It offers to remove them:

quinectl gc --dry-run
quinectl gc

It asks before deleting anything, and lists, but never deletes, anything in a service quinectl didn't start. It counts anything no project declares as left behind, including Kafka topics you created yourself, so read the --dry-run list before you confirm.

Next steps