# Quine Enterprise on Kubernetes: the deployment tour, as text This is the complete content of the interactive Kubernetes Deployment Tour at https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/, derived from the same data the tour renders. It is written for readers that do not run JavaScript, including LLM agents. What is here: every card of the tour in order with its teaching, and the complete configuration it walks through. The configuration is plain Kubernetes organized by kustomize: one shared base, then one overlay per capability, each a strict superset of the one before, so applying an overlay applies everything up to it. Every file below is quoted in full and is also served as a real file at the raw URL beside it, so the tree can be reconstructed on disk exactly as shown. The deployment placeholders are yours to replace: hostnames under `example.com`, the license key, the keystore and its password, and your identity provider realms. ## Section 1: intro ### Configuring Quine Enterprise on Kubernetes Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#welcome/1 wired into Cassandra, Keycloak, and Kafka, with Prometheus scraping metrics into a Grafana dashboard, one capability at a time Interactive: guided look around the finished deployment. Buttons frame each part of the end state in turn (the QE cluster, Cassandra, Kafka, Keycloak, metrics, the license service), with a caption naming what it does. Detail: - What you're looking at is a mental model, not a live view: the scene shows how the pieces of the deployment relate, but it is not connected to the real cluster and doesn't reflect its actual state. You can explore freely: orbit the scene, page around, try things in the controls panel. - Quine Enterprise is **pluggable by design**: its external dependencies are configuration choices, and this tour wires each one to a concrete service. - The plug points, precisely: a [persistor](https://docs.thatdot.com/quine-enterprise/learn/persistors/) is always present (the default is pod-local RocksDB; in-memory is an option). Cassandra plugs in when the graph must outlive any single pod. An [identity provider](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/) like Keycloak is only required once auth is enabled. Ingest can come from [many source types](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/#data-sources); Kafka here. And QE always [emits metrics](https://docs.thatdot.com/quine-enterprise/learn/metrics/); Prometheus + Grafana are the optional collection and rendering on top. - What Quine Enterprise adds over open-source Quine: **clustering** (members, target-size, hot spares) and **authentication** spanning the whole deployment (OIDC login + RBAC on QE itself, JWT auth to Cassandra, private-key JWT-assertion OAuth to Kafka). The boundary, precisely: open-source Quine can already persist to Cassandra and ingest from Kafka, TLS and standard SASL (PLAIN, SCRAM, OAuth client-secret) included, and ships the same metrics reporters; what Enterprise gates is running as more than one process, and that authentication story: exactly what this tour configures. - The arc of the tour: the platform and services are already provisioned when it opens; the work is configuring Quine Enterprise against them, capability by capability, until a streaming workload runs through the finished deployment. - Each configuration is a strict superset of the one before, which is why this scene accretes as you page forward and strips back down as you page backwards. - The license flow, precisely: QE reaches `https://license.thatdot.com` (outbound HTTPS 443, certificate-pinned TLS): `POST /entitlements` validates the license on startup, `POST /usage-report` sends periodic check-ins; if the server is briefly unreachable, QE keeps running through a grace period. Details: [license management](https://docs.thatdot.com/quine-enterprise/core-concepts/license-management/) in the docs. ## Section 2: baseline ### Anatomy of the deployment Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#baseline/1 plain Kubernetes manifests, organized by kustomize: one shared base holds the invariant objects, and each section of this tour applies one small overlay on top Configuration in focus: `overlays/baseline/kustomization.yaml` what the baseline deploys, by kind ``` # from the shared base (invariant across the tour): Deployment the QE pod: image, license, probes Service ClusterIP the load balancer targets Ingress https://quine.example.com ExternalSecret materializes the license Secret # from the overlay (one per section of this tour): ConfigMap this overlay's conf, generated # ...plus the overlay's patch on the Deployment (replicas, strategy) ``` **how a conf change becomes a running pod**: QE reads its conf once, at boot, and Kubernetes does not restart pods when a ConfigMap changes in place. `configMapGenerator` closes that gap: it names the ConfigMap after a hash of its contents and rewrites every reference to match. A conf edit renames the ConfigMap, the pod template changes, and the Deployment rolls out fresh pods: stale config cannot survive an apply. **portable on purpose**: everything here is plain Kubernetes. The one cloud-specific edge is the Ingress and its load-balancer annotations (an AWS ALB here); swap those for your cluster's ingress and the rest transfers unchanged. Even the license Secret arrives through External Secrets Operator, which keeps the backing store's identity out of the manifests entirely. Detail: - The base never changes across the tour: a Deployment (image, license wiring, probes, resources), a Service, an Ingress, and an ExternalSecret that materializes the license Secret. Each capability is one overlay on top: a complete `quine_enterprise.conf` (which becomes the generated ConfigMap), a small patch on the Deployment (replicas, rollout strategy), and the kustomization that binds them. - One base detail worth noticing early: BOTH pod probes point at the liveness endpoint, and the real readiness check belongs to the load balancer. That is deliberate, and the clustering section shows why (a hot spare fails readiness on purpose). - The overlays are strict supersets: each carries the previous one's entire config plus its own additions, so every overlay stands alone and the diff between neighbors is exactly the new capability. Paging forward in this tour mirrors applying the next overlay. ### The whole config, one small file Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#baseline/2 the entire runtime configuration of the baseline: everything not in it is a factory default Configuration in focus: `overlays/baseline/quine_enterprise.conf` **complete means complete**: everything not in this file is a QE factory default (webserver on 8080, uuid node ids, cluster target-size 1); what you see is everything QE is told. The one counter-intuitive line: `metrics-reporters = []` switches metrics OFF. QE's built-in default is a JMX reporter, which the Prometheus javaagent on these pods would pick up immediately; the empty list keeps the dashboard deliberately dark until the observability section turns it on. Detail: - The conf never holds a secret: it rides in a world-readable ConfigMap. Secret values arrive as environment variables instead: HOCON `${?ENV}` substitution for values the conf names explicitly, plus `CONFIG_FORCE_*` env overrides for keys injected later in the tour. The Kubernetes Secret behind them is materialized by External Secrets Operator from the cluster's secret store; the store itself stays opaque to these manifests. - Everything not in this file is a QE factory default; the docs publish [the full reference config, defaults and all](https://docs.thatdot.com/quine-enterprise/reference/config/configuration/#reference-documentation), so every default this conf leans on is inspectable. - [RocksDB](https://rocksdb.org) is an embedded key-value store (a log-structured merge tree): it runs as a library inside the QE process, so the baseline has no separate database server. - The storage default, precisely (verified in the QE source): with no `quine.store` block at all, QE runs a local RocksDB and resolves its path as conf `filepath`, else the `QUINE_DATA` env var, else `quine.db` in the working directory. The QE image ships `QUINE_DATA=/var/quine` baked in, and the Deployment mounts an [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) there: the untouched default lands the database on node-local scratch storage whose lifetime is exactly the pod's. The docs page: [where RocksDB stores data](https://docs.thatdot.com/quine-enterprise/learn/persistors/#where-rocksdb-stores-data). ### One key, two jobs Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#baseline/3 the license key thatDot issues is both the registry password that pulls the image and the runtime license the process heartbeats Configuration in focus: `base/deployment.yaml` the Secret behind both *(the shape a client creates; values redacted)* ``` kind: Secret metadata: { name: quine-license } type: kubernetes.io/dockerconfigjson # a docker login, stored as data stringData: .dockerconfigjson: '{"auths": {"registry.license.thatdot.com": {"username": "", "password": ""}}}' license-key: ``` **the two names, mapped**: the `password` pulls the image from `registry.license.thatdot.com`; `license-key` is env-fed to the process, which heartbeats it to the license server. Detail: - Licensing as you will live it: thatDot issues one license key, and it does two jobs. It is the registry password that authorizes pulling the `quine-enterprise` image from `registry.license.thatdot.com` (via `imagePullSecrets` and a [kubernetes.io/dockerconfigjson Secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/), the standard private-registry credential shape), and it is the runtime license: env-fed into the conf's `${?QUINE_LICENSE_KEY}` substitution, then heartbeated to the license server while QE runs. The docs state it the same way: [your key is also your registry credential](https://docs.thatdot.com/quine-enterprise/core-concepts/license-management/#your-key-is-also-your-registry-credential). - The Secret on the card is the shape a client creates once, by hand or from a secret store. In the deployment this tour models, it is materialized by External Secrets Operator from an external store; `kubectl create secret` from a vault of your choice works the same, because everything downstream (the image pull, the env var) only sees a normal Kubernetes Secret. ### Baseline: one pod, nothing to lose Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#baseline/4 a single member · in-pod RocksDB · no auth · the smallest thing that is Quine Enterprise Interactive: kill the pod. Deletes the single baseline pod. The pod-local RocksDB dies with it, so the replacement pod comes back with an empty graph. Detail: - Storage is an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir)-backed RocksDB: delete the pod, lose the graph. That is not a bug here; it is the teaching property the persistence section fixes. - Everything lived in that emptyDir: every node the graph builds, the ingest that built them, standing queries, UI styling. A fresh pod is a fresh QE. Fine when a clean slate on every restart is acceptable; disqualifying for production: the docs say the same under [storage in production](https://docs.thatdot.com/quine-enterprise/core-concepts/operational-considerations/#storage). - The dim shapes around QE (Cassandra, Keycloak, Kafka, Prometheus + Grafana) are already provisioned and sitting idle; each coming section wires Quine Enterprise to one of them. ### Configuration this section adds ```sh kubectl apply -k overlays/baseline/ ``` #### `base/kustomization.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/base/kustomization.yaml ```yaml # Base for Quine Enterprise: invariant resources shared by every overlay. # # ⚠️ This base is DELIBERATELY NOT DEPLOYABLE ALONE: the Deployment mounts # ConfigMap `quine-config`, which only an overlay's configMapGenerator creates. # Do not "fix" this by adding a ConfigMap here; pick an overlay: # kubectl apply -k ../overlays/baseline apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: quine labels: - pairs: app.kubernetes.io/name: quine-enterprise app.kubernetes.io/part-of: quine-deployment includeSelectors: true resources: - externalsecret.yaml - deployment.yaml - services.yaml - ingress.yaml ``` #### `base/externalsecret.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/base/externalsecret.yaml ```yaml # Pulls the thatDot license material from the cluster's external secret # store (secret quine-enterprise/license) into k8s Secret `quine-license`: # - .dockerconfigjson : registry credentials for registry.license.thatdot.com # (username = customerId, password = the license key) # - license-key : the SAME license key, fed to QE as env QUINE_LICENSE_KEY # One key, two jobs: your production license key both pulls the image and # runs the product. apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: quine-license spec: refreshInterval: 1h secretStoreRef: kind: ClusterSecretStore name: secret-store target: name: quine-license template: type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: | {"auths":{"registry.license.thatdot.com":{"username":"{{ .customerId }}","password":"{{ .licenseKey }}","auth":"{{ printf "%s:%s" .customerId .licenseKey | b64enc }}"}}} license-key: "{{ .licenseKey }}" data: - secretKey: customerId remoteRef: key: quine-enterprise/license property: customerId - secretKey: licenseKey remoteRef: key: quine-enterprise/license property: licenseKey ``` #### `base/deployment.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/base/deployment.yaml ```yaml # Quine Enterprise Deployment: the invariant shape. Per-overlay knobs (replicas, # rollout strategy) are overlay patches; ALL runtime configuration lives in the # overlay's quine_enterprise.conf (ConfigMap quine-config), except secrets, # which arrive as CONFIG_FORCE_* env vars (HOCON env overrides). apiVersion: apps/v1 kind: Deployment metadata: name: quine-enterprise spec: selector: matchLabels: {} # filled by kustomize labels(includeSelectors) template: metadata: annotations: # (No prometheus.io/* scrape annotations here: the metrics overlay # patches them in together with the conf's jmx reporter, so emission # and collection stay one kit.) # Keep Karpenter consolidation from evicting QE pods underneath you (on # a live deployment of this exact configuration, consolidation # replaced whole pod sets twice). kubectl delete and rollouts are # unaffected; this only blocks autonomous node-consolidation evictions. karpenter.sh/do-not-disrupt: "true" spec: imagePullSecrets: - name: quine-license # registry credentials (ESO-managed) terminationGracePeriodSeconds: 30 containers: - name: quine-enterprise image: registry.license.thatdot.com/thatdot/quine-enterprise:2.1.0 imagePullPolicy: IfNotPresent env: # (Nothing cluster-related here: QUINE_SEED_DNS arrives with the # cluster overlay patch, alongside the seed Service it names.) # Consumed by the `quine.license-key = ${?QUINE_LICENSE_KEY}` line # every conf carries: the mechanism is visible everywhere, the # value nowhere. - name: QUINE_LICENSE_KEY valueFrom: secretKeyRef: name: quine-license key: license-key # (No CONFIG_FORCE_* secrets here either: each overlay's patch # adds its own. The base is deliberately overlay-agnostic: image, # license, config mount, probes, resources, nothing more.) - name: JDK_JAVA_OPTIONS value: >- -Dconfig.file=/etc/quine/quine_enterprise.conf -Dconfig.override_with_env_vars=true -Dthatdot.loglevel=WARN -XX:MaxRAMPercentage=75 -javaagent:jmx_prometheus_javaagent.jar=9090:/exporter.yaml # containerPort lists are documentation, not gates: these are the # ports the process binds in EVERY overlay (8080 webserver; 9090 the # always-running javaagent). The cluster port 25520 is only bound # once a cluster-join config exists, so it is declared by the # cluster overlay patches (verified live: a single member does not bind it). ports: - name: http containerPort: 8080 - name: metrics containerPort: 9090 # Both probes use the LIVENESS path on purpose: a hot spare answers 503 # on /api/v2/system/readiness until promoted; a readiness-path pod probe # would leave spares permanently unready and wedge every rolling update # from the cluster overlay on. Traffic gating still works: the ALB health # check (on the Service) uses the real readiness endpoint, keeping # spares out of the target group until promoted. livenessProbe: httpGet: path: /api/v2/system/liveness port: 8080 initialDelaySeconds: 5 timeoutSeconds: 10 readinessProbe: httpGet: path: /api/v2/system/liveness port: 8080 initialDelaySeconds: 5 timeoutSeconds: 10 resources: # requests == limits → Guaranteed QoS; node requests: # autoscalers size nodes off requests. 8 CPU / 16Gi is the cpu: "8" # docs.thatdot.com operating-env floor (8-32 memory: 16Gi # cores, 16-20GB); MaxRAMPercentage=75 above limits: # derives a ~12GiB heap from the 16Gi limit. cpu: "8" memory: 16Gi volumeMounts: - name: quine-config mountPath: /etc/quine readOnly: true - name: quine-data mountPath: /var/quine volumes: - name: quine-config configMap: name: quine-config # generated (hash-suffixed) by the overlay - name: quine-data emptyDir: {} # dies with the pod: the baseline teaching property ``` #### `base/services.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/base/services.yaml ```yaml # Front Service (ALB target). The headless seed Service is NOT here; it is # introduced by the cluster overlay together with the quine.cluster # config block, so everything that forms a cluster lives in one place. apiVersion: v1 kind: Service metadata: name: quine-enterprise annotations: # ALB health checks (these belong on the backing Service, not the Ingress). # Unlike the pod probes, the ALB uses the REAL readiness endpoint: a hot # spare's 503 keeps it out of the target group until it is promoted. alb.ingress.kubernetes.io/healthcheck-path: /api/v2/system/readiness alb.ingress.kubernetes.io/success-codes: "204" spec: type: ClusterIP ports: - name: http port: 80 targetPort: 8080 protocol: TCP ``` #### `base/ingress.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/base/ingress.yaml ```yaml # Public entry: https://quine.example.com → quine-enterprise:80. # IngressClass `alb` is the AWS Load Balancer Controller's stock class. # TLS terminates at the ALB with an ACM certificate; no certificate-arn # annotation here: this manifest is applied by plain `kubectl apply -k`, # so we rely on the controller's ACM auto-discovery by rule host (a # certificate covering quine.example.com matches). # # ⚠️ Until the auth overlay is applied, QE is publicly reachable with NO # auth. Acceptable only for a short-lived evaluation on non-sensitive # data; never leave a real deployment in this state. apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: quine-enterprise annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]' alb.ingress.kubernetes.io/ssl-redirect: "443" external-dns.alpha.kubernetes.io/aws-evaluate-target-health: "false" spec: ingressClassName: alb rules: - host: quine.example.com http: paths: - path: / pathType: Prefix backend: service: name: quine-enterprise port: number: 80 ``` #### `overlays/baseline/kustomization.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/baseline/kustomization.yaml ```yaml # Baseline: 1 member, ephemeral RocksDB. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # Namespace + labels are REPEATED in every overlay on purpose: resources created # by this overlay's configMapGenerator do not inherit the base's transformers; # without the repeat the ConfigMap lands in the kubectl context's default # namespace and label-based cleanup never finds it. namespace: quine labels: - pairs: app.kubernetes.io/name: quine-enterprise app.kubernetes.io/part-of: quine-deployment includeSelectors: true resources: - ../../base # The hash suffix on the generated ConfigMap is the rollout mechanism: switching # overlays rewrites the ConfigMap name in the pod template → clean rollout. configMapGenerator: - name: quine-config files: - quine_enterprise.conf patches: - path: deployment-patch.yaml ``` #### `overlays/baseline/quine_enterprise.conf` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/baseline/quine_enterprise.conf ```hocon # ── Baseline ──────────────────────────────────────────────────────────────── # A single Quine Enterprise member, pod-local RocksDB storage (emptyDir). # Delete the pod and the database dies with it: that is the point of the baseline. # Requires: nothing beyond the platform (no Cassandra, no Keycloak, no metrics). # # Anything not set here is a QE factory default (webserver on 8080, uuid ids, # cluster target-size 1, ...): these files carry only what differs and why. # Licensing: the key's VALUE never appears here or anywhere in the repo. HOCON # ${?ENV} substitution reads it from the pod environment (fed from a k8s Secret). # This line appears in every overlay. quine.license-key = ${?QUINE_LICENSE_KEY} # Storage: deliberately unset. QE's factory default is exactly the baseline's # store: a local RocksDB, at the path named by the QUINE_DATA env var (the QE # image bakes in QUINE_DATA=/var/quine and the Deployment mounts an emptyDir # there), or ./quine.db if that env var is unset too. Two ways to override # where the data lives: set QUINE_DATA on the container, or uncomment this # block (shown holding the resolved defaults) and edit: # # quine.store = { # type = rocks-db # filepath = "/var/quine" # } # Explicitly NO metrics reporting yet: QE's built-in default is [ { type = jmx } ] # (the Prometheus javaagent on these pods would pick that up immediately), so the # empty list keeps the dashboard dark until the metrics overlay deliberately turns it on. quine.metrics-reporters = [] ``` #### `overlays/baseline/deployment-patch.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/baseline/deployment-patch.yaml ```yaml # Single member on a pod-local store: Recreate, never RollingUpdate; two # single members must not run at once. The explicit `rollingUpdate: null` is # required or the API rejects any RollingUpdate→Recreate transition # (downgrading from a cluster overlay back to a single-member overlay). apiVersion: apps/v1 kind: Deployment metadata: name: quine-enterprise spec: replicas: 1 strategy: type: Recreate rollingUpdate: null ``` ## Section 3: persistence ### Real off-machine persistence with Cassandra Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#persistence/1 the persistence overlay's whole conf: against the baseline, the commented-out RocksDB default becomes a real `quine.store` block with its oauth companion, and nothing else moves Configuration in focus: `overlays/persistence/quine_enterprise.conf` Detail: - The overlays are strict supersets, and this is the first time that pays off: the baseline conf you already know rides along byte-identical (the license line, the metrics-off line), and one capability arrives as two new blocks. Every section repeats this shape. - `should-create-keyspace` and `should-create-tables` are called out as a development convenience by the docs: in production the keyspace and tables are created deliberately, with your replication settings, not by the first process to boot. [Cassandra persistor setup](https://docs.thatdot.com/quine-enterprise/learn/persistors/cassandra-setup/) carries the full config reference. - The consistency lines matter later: `LOCAL_QUORUM` against this deployment's `replication-factor = 1` is a quorum of one, but the same two lines are already correct for a replicated production ring. The conf is written for where it is going, not just where it is. - The one secret in the story (the keystore password) is not in this file: it arrives as a `CONFIG_FORCE_*` environment variable from a Kubernetes Secret. The conf stays world-readable. ### Why Cassandra Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#persistence/2 the persistor is a configuration choice: local stores end with the pod, and the cluster on the horizon needs a store every member can share the persistor menu *(quine.store.type)* ``` # local: an embedded store inside the pod; the data lives and dies with it rocks-db the default: an embedded LSM tree, no server (the baseline) map-db pure JVM, the most portable; memory-mapped files in-memory RAM only, nothing on disk: gone when the process exits # remote: a database the pod connects to; the data outlives every pod cassandra distributed, replicated, high-throughput; also speaks ScyllaDB and Astra DB ``` **why Cassandra, here**: two sections from now this deployment grows to four members, and the docs are unambiguous: "the cluster's persistence backend must be shared by all cluster members". None of the three local options can be: on its disk or in its RAM, a store inside the pod cannot back a cluster. Of the two supported cluster backends (Cassandra and ClickHouse), Cassandra is the recommended one. Choosing it now means the storage story never has to change again. [persistors](https://docs.thatdot.com/quine-enterprise/learn/persistors/) · [cluster resilience](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/) Detail: - How a persistor stores the graph, in one breath: a node's changes ("deltas" to its properties and edges) are appended to a per-node log, event-sourcing style, with periodic snapshots so a node can be restored without replaying everything. The persistor is the agent that reads and writes that data; the [persistors page](https://docs.thatdot.com/quine-enterprise/learn/persistors/) is the reference. - MapDB exists mostly as the zero-native-dependencies fallback: RocksDB ships as a platform-specific native library, and on an unsupported platform QE will suggest MapDB instead. Its memory-mapped files slow down past 2GB; it is not a production choice. - Cassandra compatibility travels: the same persistor type speaks to ScyllaDB and Astra DB, and Amazon Keyspaces has its own `type = keyspaces` with a few settings fixed by AWS (replication factor 3, LOCAL_QUORUM writes): the docs carry [Keyspaces](https://docs.thatdot.com/quine-enterprise/learn/persistors/cassandra-setup/#amazon-keyspaces-configuration) and [Astra DB](https://docs.thatdot.com/quine-enterprise/learn/persistors/cassandra-setup/#astradb-configuration) configuration sections. - ClickHouse is the other cluster-capable backend in Quine Enterprise, but the docs recommend Cassandra: a proven expansion model, and native TTL support for bounding data growth (the next cards use exactly that). ### What QE stores Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#persistence/3 one keyspace, eight tables: the graph as an append-only log of changes, plus everything the app needs to come back keyspace quine *(what actually lands in Cassandra)* ``` journals every change to a node, append-only: the event-sourced truth of the graph snapshots a node's assembled state, saved as it sleeps standing_queries the registered standing queries themselves standing_query_states per-node partial matches in progress domain_graph_nodes compiled DistinctId standing-query patterns domain_index_events per-node events against those patterns edges supernode support: an enormous node's edges, offloaded to the store (Enterprise addition) meta_data app state: ingest definitions, sample queries, UI styling; everything QE needs to come back ``` **everything the running system accumulates is in here**: node history lands in `journals` and `snapshots`, and an ingest added in the UI lands in `meta_data`. The full DDL (primary keys, compaction strategies) is one page: [Cassandra schema](https://docs.thatdot.com/quine-enterprise/learn/persistors/cassandra-setup/#cassandra-schema). Detail: - The write path, roughly: as events touch a node, its deltas are appended to `journals`; when a node goes quiet ("sleeps"), its assembled state is written to `snapshots` so waking it later does not require replaying its whole history. Both are keyed by node id, which is what makes the persistor fast for streaming writes. - When to save is itself configuration: the `quine.persistence` block controls whether journals are kept at all (`journal-enabled`), when snapshots happen (`snapshot-schedule`, default on-node-sleep), and when standing-query state is saved. The defaults are the durable choices; relaxing them trades safety for speed. The docs show the block with its defaults: [persistence event configuration](https://docs.thatdot.com/quine-enterprise/learn/persistors/#quine-enterprise-persistence-event-configuration). - The `edges` table is a Quine Enterprise addition for supernode support: past a configured threshold, a node with an enormous edge set keeps its edges directly in the store instead of in its snapshots. It sits empty until that feature is switched on. - `meta_data` is why the app comes back whole, not just the graph: ingest definitions, sample queries, node appearances, standing-query registrations all live there as small records. It is also the one table the TTL card will tell you never to expire. - The exact DDL (partition keys, clustering order, compaction strategies) is in the [schema section](https://docs.thatdot.com/quine-enterprise/learn/persistors/cassandra-setup/#cassandra-schema) of the Cassandra setup page. ### TTL and tuning at scale Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#persistence/4 bound the keyspace's growth with table TTLs, and size the Cassandra ring alongside the QE cluster: the docs on this card carry the depth bounding growth: set TTLs on the data tables *(cqlsh)* ``` ALTER TABLE quine.journals WITH default_time_to_live = 604800; # one week # TTL-able (per the docs): journals, snapshots, standing_queries, # standing_query_states, domain_graph_nodes, domain_index_events. # Never TTL meta_data: it is the app's memory, not graph history. ``` sizing Cassandra alongside QE *(rules of thumb; the sizing guide carries the method)* ``` 1 Cassandra node per 4 QE members # the common starting ratio never fewer than 3 Cassandra nodes # the minimum production ring scale Cassandra out, not up # more, smaller nodes beat fewer big ones: I/O, compaction, and failures spread wider keep QE and Cassandra close # every node sleep and wake is a round trip: network latency lands straight on graph throughput replication-factor = 3, LOCAL_QUORUM # so reads and writes survive a node loss ``` **this deployment's ring is deliberately tiny**: one Cassandra node, so the conf pins `replication-factor = 1` and LOCAL_QUORUM degenerates to that one node. And tuning is never a live knob: changing Cassandra means redeploying Cassandra. These two pages carry that depth when you need it: [data expiration: which tables to TTL](https://docs.thatdot.com/quine-enterprise/learn/persistors/cassandra-setup/#data-expiration) · [cluster sizing: Cassandra hosts](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-sizing/) Detail: - TTL is the docs' first operational recommendation for Cassandra-backed deployments: expire what the use case no longer needs, per table, with plain `ALTER TABLE`. The `journals` and `domain_index_events` tables are created with TimeWindowCompactionStrategy, which is what makes TTL-expired data cheap for Cassandra to drop. - The 4:1 member ratio and the 3-node floor are the docs' common starting point, not a law: write-heavy workloads (many properties or edges updated per event) pull the ratio down, and the sizing guide's real advice is to measure persistor latency and pending compactions under peak load, then resize. [Cluster sizing](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-sizing/) walks the whole method, QE hosts included. - The scale-out lean comes from deployments; the docs frame node size as a trade-off (fewer, larger nodes minimize operational overhead) but list the wins on the more-smaller side: reads spread across more machines with lower tail latencies, compaction work spread wider, and less throughput lost when one node fails. - Latency budgets both layers: Cassandra hosts want fast local NVMe storage (Cassandra is I/O-intensive, and network-attached storage adds latency that lands directly on QE's throughput), and the network between QE and Cassandra deserves the same care: place them close (same cluster or same zone), because every node sleep, wake, and journal write is a round trip. - For everything else that changes at scale (consistency levels against a real replication factor, compaction backlog, backup strategy), the persistor is standard Cassandra: your existing Cassandra operational practice applies unchanged. ### Private-key JWT to Cassandra Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#persistence/5 even the persistor speaks OIDC: no database password anywhere, just a keystore, a signed assertion, and a token round trip Configuration in focus: `overlays/persistence/quine_enterprise.conf` the oauth block, key by key *(what each is and where it comes from)* ``` client-id the OAuth client your IdP admin registers for QE: confidential, grant type jwt-bearer; the name is whatever you registered discovery-url your IdP's discovery document, from the realm or tenant that owns the client; QE reads the token endpoint out of it resource-uri sent as the token request's resource parameter: you pick the value, your IdP mints it into the token's audience (aud) claim (Keycloak: an audience mapper on the client), and Cassandra's validator must accept the same value cert-file a PKCS12 or JKS keystore holding QE's X.509 cert and private key, mounted into the pod; the IdP trusts its public key (registered on the client, e.g. as inline JWKS) cert password the one secret: env-fed from a Kubernetes Secret (the CONFIG_FORCE_* line), never in the conf ``` **the flow, in one sentence**: QE signs a JWT with the keystore's private key, trades it at the IdP for an access token, and presents that token to Cassandra, whose JWT-aware authenticator validates it: no database password exists to leak or rotate. Keycloak is only this deployment's IdP; the block maps onto any OIDC provider. The full contract (keystore formats, IdP requirements, TLS fine print): [Cassandra OAuth authentication](https://docs.thatdot.com/quine-enterprise/learn/persistors/cassandra-setup/#cassandra-oauth-authentication) Detail: - Every value on this card comes from YOUR identity provider, not from thatDot: the flow is standard OIDC (a private_key_jwt client assertion traded for an access token), supported by Keycloak, Okta, Microsoft Entra ID, Auth0, and any provider implementing the JWT bearer grant. This deployment runs Keycloak; nothing in the block knows or cares. The page to hand your identity team: [identity provider requirements](https://docs.thatdot.com/quine-enterprise/learn/persistors/cassandra-setup/#identity-provider-requirements). - The keystore must be PKCS12 or JKS (PEM files are not supported). In PKCS12 the certificate and key share an alias, so the optional `cert-alias`/`key-alias` settings can usually be omitted. - Switching on `quine.store.oauth` also forces the Cassandra connection onto TLS, unconditionally: bearer tokens never travel plaintext. And QE validates that TLS against the JVM's DEFAULT truststore only, so a self-signed or internal CA must be merged into a copy of the stock `cacerts` (never replace the file: the license server and IdP connections trust the same store). In this deployment the container entrypoint does that merge at startup, from a CA cert mounted off a Kubernetes Secret. - The other side of the handshake is Cassandra's: it needs a JWT-aware authenticator that validates tokens against the IdP's JWKS endpoint and accepts the agreed audience. This deployment's Cassandra runs one; the docs describe the requirement. - Foreshadowing the finale: the same keystore identity later authenticates QE's Kafka ingest and output. One machine identity, registered once with the IdP, for data in, data out, and the store. - The canonical reference, proven by this very deployment: [Cassandra OAuth authentication](https://docs.thatdot.com/quine-enterprise/learn/persistors/cassandra-setup/#cassandra-oauth-authentication). ### The same kill, the other ending Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#persistence/6 still one member, but the graph lives in Cassandra now: delete the pod and the data does not even notice Interactive: kill the pod, with Cassandra behind it. The same pod deletion after the persistence overlay is applied. Journals and snapshots are already in Cassandra, so the replacement pod reads the same graph back. Detail: - Same button as the baseline's closer, opposite ending: the pod dies, and everything of consequence is already in the keyspace. The fresh pod reads its world back from Cassandra: the graph, the ingest definition, sample queries, UI styling, standing queries. - What still dies with the pod: only the in-memory working set, the nodes currently awake. The fresh pod rebuilds that lazily as queries and ingest touch nodes again; nothing is lost, some first touches are just a read slower. How big that working set is allowed to get is itself configuration: [memory configuration, the node cache limits](https://docs.thatdot.com/quine-enterprise/reference/config/configuration/#memory-configuration). - This is the property everything after this section builds on: a pod that owns no data is replaceable. Next section, four pods share this same store, and replaceable becomes promotable. The docs state it as the operational rule: [Cassandra is the recovery target](https://docs.thatdot.com/quine-enterprise/learn/orchestration/disaster-recovery/#cassandra-is-the-recovery-target). ### Configuration this section adds ```sh kubectl apply -k overlays/persistence/ ``` #### `overlays/persistence/kustomization.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/persistence/kustomization.yaml ```yaml # Cassandra: still 1 member, storage moves to Cassandra. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # Namespace + labels are REPEATED in every overlay on purpose: resources created # by this overlay's configMapGenerator do not inherit the base's transformers; # without the repeat the ConfigMap lands in the kubectl context's default # namespace and label-based cleanup never finds it. namespace: quine labels: - pairs: app.kubernetes.io/name: quine-enterprise app.kubernetes.io/part-of: quine-deployment includeSelectors: true resources: - ../../base configMapGenerator: - name: quine-config files: - quine_enterprise.conf patches: - path: deployment-patch.yaml ``` #### `overlays/persistence/quine_enterprise.conf` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/persistence/quine_enterprise.conf ```hocon # ── Cassandra ──────────────────────────────────────────────────────────────── # This overlay is ALL about the persistor: the pod-local RocksDB quine.store is # swapped for Cassandra, and nothing else changes. Still a single member (QE's # default cluster target-size is 1), so the diff against the baseline is purely # the storage story. Delete the pod now and the data survives, because the data # no longer lives in the pod. # Licensing: the key's VALUE never appears here or anywhere in the repo. HOCON # ${?ENV} substitution reads it from the pod environment (fed from a k8s Secret). # This line appears in every overlay. quine.license-key = ${?QUINE_LICENSE_KEY} quine.store = { type = cassandra endpoints = ["cassandra-dc1-service.cassandra.svc:9042"] keyspace = quine should-create-keyspace = true # dev convenience; never in prod should-create-tables = true replication-factor = 1 # matches the single-node dc1 read-consistency = LOCAL_QUORUM write-consistency = LOCAL_QUORUM local-datacenter = "dc1" read-timeout = "10s" write-timeout = "10s" } # Even the persistor speaks OIDC: QE authenticates to Cassandra with # an OIDC private_key_jwt client assertion: it signs a JWT with the key in # cert-file, trades it at Keycloak for an access token, and presents THAT to # Cassandra (whose JwtAuthenticator validates it). No password anywhere. quine.store.oauth { client-id = "quine-cassandra-client" discovery-url = "https://keycloak.example.com/realms/cassandra/.well-known/openid-configuration" resource-uri = "cassandra://example" # audience Cassandra accepts cert-file = "/opt/certs/keystore.p12" # cert password arrives via CONFIG_FORCE_quine_store_oauth_cert__file__password } # Explicitly NO metrics reporting yet: QE's built-in default is [ { type = jmx } ] # (the Prometheus javaagent on these pods would pick that up immediately), so the # empty list keeps the dashboard dark until the metrics overlay deliberately turns it on. quine.metrics-reporters = [] ``` #### `overlays/persistence/deployment-patch.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/persistence/deployment-patch.yaml ```yaml # Single member (the persistor moved, the topology didn't): Recreate, never # RollingUpdate; two single members must not share a persistor. The explicit # `rollingUpdate: null` is required or the API rejects any RollingUpdate→Recreate # transition (downgrading from a cluster overlay back to a single-member overlay). # # The Cassandra JWT-auth kit joins HERE, with the persistor that needs it # (nothing of this in the base): # - keystore.p12 (+ its password via CONFIG_FORCE_*): the private_key_jwt # assertion identity the conf's quine.store.oauth block points at; # - cassandra-ca.crt + USE_SYSTEM_CA_CERTS + the command override: QE's # store-oauth mode forces the datastax driver into TLS, and it trusts via # the DEFAULT JVM truststore only. The image ships the Temurin # __cacert_entrypoint.sh but does not run it; chaining it here merges our # Cassandra cert into a COPY of the stock cacerts; every public CA stays, # so the license-server TLS check is untouched. apiVersion: apps/v1 kind: Deployment metadata: name: quine-enterprise spec: replicas: 1 strategy: type: Recreate rollingUpdate: null template: spec: containers: - name: quine-enterprise command: ["/__cacert_entrypoint.sh", "/init-quine.sh"] env: - name: USE_SYSTEM_CA_CERTS value: "1" - name: CONFIG_FORCE_quine_store_oauth_cert__file__password valueFrom: secretKeyRef: name: quine-cassandra-cert key: password volumeMounts: - name: quine-cassandra-cert mountPath: /opt/certs/keystore.p12 subPath: keystore.p12 readOnly: true - name: quine-cassandra-cert mountPath: /certificates/cassandra-ca.crt subPath: cassandra-ca.crt readOnly: true volumes: - name: quine-cassandra-cert secret: secretName: quine-cassandra-cert ``` ## Section 4: clustering ### The cluster, declared in two lines Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#clustering/1 the cluster overlay's whole conf: one new block declares how big the cluster is and how members find each other, and everything else is carried forward byte-identical Configuration in focus: `overlays/cluster/quine_enterprise.conf` Detail: - The superset discipline pays off a second time: the persistence conf you already know rides along byte-identical (the license line, the whole `quine.store` block with its oauth, the metrics-off line), and one capability arrives as one new block. A side-by-side diff between neighbors shows exactly clustering, nothing else. - `target-size` is the cluster's fixed operational size, set in config at startup: the cluster assigns positions 0, 1, 2 and starts operating when every position is filled by a live member. A position is a property of the cluster, not of any pod: that distinction is the whole promotion story two cards from now. - Why 3 and not 2: a cluster below 3 members cannot reliably resolve a network partition, because deciding which side survives takes a majority. The mechanism is documented under [cluster partition detection](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/#cluster-partition-detection). The tuning card returns to this floor. - Changing `target-size` later is a real operation, not an edit: the docs have all members restart with the new size (persisted data and standing queries survive; ingest streams are defined per member and must be recreated). Size the cluster deliberately, not incrementally. The procedure: [validating and resizing](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-sizing/#validating-and-resizing). ### The cluster kit Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#clustering/2 the Kubernetes half: a seed Service appears, and the patch reshapes the Deployment: replicas 4, RollingUpdate returns, the cluster port opens Configuration in focus: `overlays/cluster/seed-service.yaml` **the one section where a new Kubernetes object appears**: every other capability arrives as a conf change plus a patch tweak; clustering also needs a Service that exists purely so members can find each other. Read the kit as three answers: how many members (`replicas: 4` over `target-size 3`: the fourth is the spare), how they meet (the headless Service the conf names), and how they talk (port 25520, which the process only binds once a cluster-join config exists). And one flip back: a single member had to die before its replacement started (Recreate); a cluster can roll members one at a time, so RollingUpdate returns. The other half of the kit is in the tree: `overlays/cluster/deployment-patch.yaml`, whose diff against the persistence overlay shows exactly the three answers arriving. Detail: - The seed Service is headless (`clusterIP: None`): asking Kubernetes DNS for its name returns the member pods' own addresses, not a proxy in front of them. That is exactly what discovery needs: a list of peers, not a load balancer. - `publishNotReadyAddresses: true` is the bootstrap enabler: a booting member is not ready yet, but its peers must still be able to find it. A normal Service hides unready pods; this one advertises them on purpose. - Port 25520 is the cluster protocol port, and the process only binds it once a cluster-join config exists (verified on this deployment: a single member never opens it). The base Deployment leaves it undeclared; the patch adds it together with the config that makes it real. - Every member binds the SAME port, on purpose: a member's identity is its address:port pair (`quine.cluster.this-member`, defaulting to the pod's own IP and 25520), and on Kubernetes one-IP-per-pod makes the pair unique without touching the port (verified live: each pod listens on 25520 bound to its own pod IP). Ports only need to differ when members share a host, which is why the config also accepts port 0: auto-assign. The docs' reference for the pair: [member identity, address, port, and position](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/#member-identity). - `QUINE_SEED_DNS` is belt and suspenders: the conf already names the seed Service, and the image also honors this environment variable as a fallback. Either alone would do. - Why RollingUpdate comes back: the single-member overlays used Recreate, because one member has no one to hand off to: the old pod must die before its replacement starts. A cluster rolls members one at a time while the rest keep the graph, so the strategy flips back. ### Discovery: static seeds or DNS Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#clustering/3 two ways a member finds its cluster: name the machines, or name one name that resolves to them; a scheduler-managed environment wants the second Configuration in focus: `overlays/cluster/quine_enterprise.conf` the OTHER cluster-join type *(fixed machines: VMs, bare metal; not this deployment's file)* ``` cluster-join = { type = static-seed-addresses seed-addresses.0.address = 172.31.1.100 seed-addresses.0.port = 25520 seed-addresses.1.address = 172.31.1.101 seed-addresses.1.port = 25520 } a fixed list of member host:ports. Any reachable seed that is part of the cluster gets a joining member in; list several, so joining never depends on one machine being up. A member may name ITSELF as a seed: if no other seed answers, it forms a fresh cluster that the rest then join ``` **why DNS on Kubernetes instead** (the `dns-entry` block emphasized in the file): a seed list wants stable addresses, and pods have none: their IPs are unknowable in advance and change on every reschedule. The headless Service inverts the problem: one stable name, resolved at join time to whatever pods exist right now. The docs recommend exactly this for scheduler-managed environments. [cluster seeds](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/#cluster-seeds) · [DNS discovery](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/#dns-discovery) Detail: - Discovery is only the introduction: once joined, each member advertises its own address to the others (`quine.cluster.this-member`, defaulting to the IP it finds at runtime), and the cluster gossips membership from there. The join config just answers "who do I call first?". - Static seeds fit machines with addresses you know and keep: VMs, bare metal. The docs recommend listing several seeds so joining never depends on one machine, and a member may list itself: if no other seed answers, it bootstraps a fresh cluster for the rest to join. - On Kubernetes the addresses are the problem: pod IPs are assigned at schedule time and change on every reschedule, so a fixed seed list is stale before it is written. `dns-entry` flips the dependency: the conf names one stable DNS name, and Kubernetes keeps that name resolving to the current pods. - This is the docs' own recommendation for scheduler-managed environments, and the seed Service card you just saw is its Kubernetes incarnation: headless, publishing every pod, ready or not. - Under the hood (skippable depth): a DNS answer carries only addresses, no ports. So dns-entry joining is a two-step bootstrap: members probe each discovered address on a dedicated management port (7626) to agree on who forms or joins the cluster, and the 25520 cluster transport takes over from there. Both ports are verifiably bound on every live pod of this deployment. ### Hot spares Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#clustering/4 replicas exceeding target-size, nothing more: a booted, joined member holding no position, waiting to turn a member death from minutes of pause into seconds when an active member dies ``` the cluster pauses input: a position is empty, and processing stays paused until every position is filled again without a spare a replacement must boot, then join: the pause is minutes with a spare the spare is already booted and already joined; it takes the empty position immediately: the pause is seconds ``` what the spare is *(and is not)* ``` booted + joined it gossips with the cluster and knows the graph holds no position no memberIdx; it does no graph work until promoted graph APIs say no ingest, queries, standing queries: positioned members only; admin routes like status still answer ``` **what the docs promise, exactly**: at least one hot spare is recommended "to minimize downtime and maximize overall throughput". Not zero downtime: a member death pauses cluster input either way; the spare decides whether that pause is seconds or minutes. Two honest notes for Kubernetes: a scheduler restarts failed pods on its own, so "you may not need a hot spare at all depending on your downtime tolerance"; and with spares, keep hosts homogeneous: any spare must be able to assume the full workload of any member. [hot spares](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/#hot-spares) · [cluster performance](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-performance/) Detail: - The honest failure model first: when a positioned member is lost, the cluster leaves its operating state and input processing pauses until every position is filled again. Nothing about a spare changes that; the spare changes how long the pause lasts. The named states behind this claim: [the cluster lifecycle](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/#cluster-lifecycle), Operating to Unavailable and back. - Without a spare, refilling the position means a replacement process must boot and join: minutes, dominated by startup. A spare has already done both, so it is swapped in immediately: the docs call it the best way to minimize downtime. - A spare is a full member of the conversation but not of the graph: it gossips, it knows the topology, and its administrative routes (like cluster status) answer, but ingest, queries, and standing queries are only served by positioned members. - How many spares: at least one for any production deployment, per the docs, and one per physical rack in a datacenter. The Kubernetes nuance is stated just as plainly: a scheduler already restarts failed pods, so "you may not need a hot spare at all depending on your downtime tolerance". This deployment runs one because the promotion is worth watching. - The fine print that comes with spares: keep the hosts homogeneous. Any spare must be able to assume the full workload of any member it replaces, so a cluster of mixed machine sizes undermines the guarantee. ### Two probes, one endpoint, on purpose Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#clustering/5 the seed planted in the baseline section pays off: the spare answers 503 on readiness by design, so the pod probes must not ask that question Configuration in focus: `base/deployment.yaml` **why liveness for BOTH pod probes**: a hot spare answers 503 on the readiness endpoint until it is promoted; a readiness-path pod probe would mark spares permanently unready and wedge every rolling update. Traffic gating is the load balancer's job instead: `base/services.yaml` (in the tree) annotates the load balancer's own health check onto the Service, pointing it at the real readiness endpoint (`/api/v2/system/readiness`, a ready member answers `204`), keeping unready members out of the target group. The kubelet asks "alive?"; the load balancer asks "ready for traffic?". Detail: - The anomaly from the baseline section, now observable: this overlay finally has a spare, and the spare answers 503 on `/api/v2/system/readiness` while Kubernetes reports its pod Ready. Both statements are correct, because two different questions are being asked. - The kubelet asks "is this process alive?": both pod probes point at `/api/v2/system/liveness`, which a healthy spare passes. The load balancer asks "should this member receive traffic?": its health check (annotated on the Service) hits the real readiness endpoint, so the spare stays out of the target group until promoted. - What the naive wiring would cost, precisely: a readiness-path pod probe would mark the spare permanently unready, and this overlay just switched to RollingUpdate: a rollout that waits for 4 ready pods out of a set that can only ever show 3 wedges forever. - One caution the docs and this deployment agree on: do not use readiness codes to identify the spare. During cluster formation "not ready yet" and "spare" both answer 503; the status API is the authoritative source of roles, and the next card leans on it. The docs name that surface under [inspecting the live cluster](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/#inspecting-the-live-cluster). ### Tuning at scale Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#clustering/6 the base Deployment already encodes the documented operating floor; the sizing method takes over from there Configuration in focus: `base/deployment.yaml` sizing the cluster *(the method, from the sizing guide)* ``` never fewer than 3 members # resolving a network partition takes a majority: the split-brain floor 2,000-10,000 events/s per host # the observed range; most production workloads land at 2-5k. divide required throughput by a conservative per-host rate at least 1 hot spare # for any production deployment plan Kafka partitions with the member count # partitions are Kafka's unit of parallelism: a member with no partition sits idle on that stream. the streaming section shows this live ``` **the Deployment already encodes the floor**: the resources block (emphasized in the file) pins requests equal to limits at exactly the documented 8 CPU / 16Gi floor, and its comments cite the [operating environment](https://docs.thatdot.com/quine-enterprise/reference/operating-env/) page because that is where the numbers came from. The method and the arithmetic: [cluster sizing](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-sizing/) · [aligning with Kafka partitions](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-performance/#aligning-with-kafka-partitions) Detail: - The resources block is deliberate teaching: requests equal limits (Guaranteed QoS, no throttling surprises), at exactly the documented floor of 8 CPU / 16Gi, and the JVM flag derives a ~12Gi heap from the container limit. The docs' own [heap guidance](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-sizing/#quine-enterprise-hosts): fixed 12 to 16GB, never more (larger heaps risk long garbage collection pauses). - The sizing arithmetic, worked: need 15,000 events/s and expect a conservative 4,000 events/s per host? Start at 4 members plus a spare. Most production workloads land between 2,000 and 5,000 events/s per host; trivial ones reach higher, connected graphs pull lower. [The worked example in the sizing guide](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-sizing/#quine-enterprise-hosts_1) is this arithmetic. - Sizing is then a measurement loop, not a formula: watch persistor latency (is Cassandra the bottleneck?), standing-query backpressure, and per-host CPU during PEAK traffic, and resize toward roughly 80% utilization. The sizing guide's [evaluation method](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-sizing/#evaluating-your-cluster-size) walks the whole loop, including when the right move is not resizing QE at all. - The Kafka forward-mention, precisely: this deployment's ingest runs the members as one consumer group, so partitions divide across members and a member with no partition sits idle on that stream (partitions are Kafka's unit of parallelism). The docs additionally [align the two counts for data locality](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-performance/#aligning-with-kafka-partitions): members as an even divisor or multiple of the partition count keeps each partition's data on one member. The streaming section makes both visible. - The AWS flag, as always the incarnation not the lesson: [fixed-performance instance families](https://docs.thatdot.com/quine-enterprise/reference/operating-env/) (m7a here) over burstable t-series, whose CPU-credit throttling shows up as mysterious latency spikes in a backpressured system (the operating environment page carries this as its EC2 instance-type guidance). ### Kill a member, watch the promotion Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#clustering/7 the spare exists for this moment: an active member dies, the pause is seconds, and the restarted pod becomes the new spare Interactive: kill an active member. Removes a positioned cluster member. The hot spare takes the empty position immediately, and the restarted pod rejoins as the new spare. A readout tracks each member position and role, mirroring what GET /api/v2/system/status reports. Detail: - The readout mirrors what the real cluster reports: `GET /api/v2/system/status` maps member addresses to their `memberIdx` and lists hot spares separately; a spare's own entry has NO `memberIdx`. That absence is the authoritative role test, not readiness codes. - The full promotion story, as the docs tell it: the failure detector ejects the unresponsive member, the cluster pauses input on the empty position, the spare is swapped in immediately, and when the dead member's pod restarts it finds its old position taken and joins as the new spare. Positions belong to the cluster; pods just fill them. How failure is decided: [the member failure detector](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/#member-failure-detector). - The operational tip that falls out: when scripting against a cluster, preflight with a status call so you are talking to a positioned member: graph APIs (ingest, queries) answer only there, and a spare will refuse them. - Everything the promoted member needs is already in Cassandra: the shared store is what makes ANY member able to take ANY position. The persistence section's "replaceable" just became "promotable". [cluster resilience](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-resilience/) carries the whole model. ### Configuration this section adds ```sh kubectl apply -k overlays/cluster/ ``` #### `overlays/cluster/kustomization.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/cluster/kustomization.yaml ```yaml # Cluster: 3 members + 1 hot spare (replicas 4 > target-size 3). apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # Namespace + labels are REPEATED in every overlay on purpose: resources created # by this overlay's configMapGenerator do not inherit the base's transformers; # without the repeat the ConfigMap lands in the kubectl context's default # namespace and label-based cleanup never finds it. namespace: quine labels: - pairs: app.kubernetes.io/name: quine-enterprise app.kubernetes.io/part-of: quine-deployment includeSelectors: true resources: - ../../base - seed-service.yaml # the cluster-discovery mechanism, introduced with clustering configMapGenerator: - name: quine-config files: - quine_enterprise.conf patches: - path: deployment-patch.yaml ``` #### `overlays/cluster/quine_enterprise.conf` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/cluster/quine_enterprise.conf ```hocon # ── Cluster ────────────────────────────────────────────────────────────────── # Clustering appears here: the quine.cluster block raises target-size from its # default of 1 to 3 and names the headless seed Service for member discovery; # everything before this overlay was a single member. The overlay also raises # replicas to 4: one MORE than target-size, which makes the fourth member a hot # spare. There is no HOCON key for spares: spare-ness is purely replicas # exceeding target-size (cluster membership is config; spare-ness is # scheduling). The spare answers 503 on /api/v2/system/readiness until an # active member dies and it is promoted. # Licensing: the key's VALUE never appears here or anywhere in the repo. HOCON # ${?ENV} substitution reads it from the pod environment (fed from a k8s Secret). # This line appears in every overlay. quine.license-key = ${?QUINE_LICENSE_KEY} quine.cluster.target-size = 3 # default is 1; three active members from here on quine.cluster.cluster-join = { type = dns-entry name = "quine-enterprise-seed" # headless seed Service; members discover each # other through its DNS entry } quine.store = { type = cassandra endpoints = ["cassandra-dc1-service.cassandra.svc:9042"] keyspace = quine should-create-keyspace = true # dev convenience; never in prod should-create-tables = true replication-factor = 1 # matches the single-node dc1 read-consistency = LOCAL_QUORUM write-consistency = LOCAL_QUORUM local-datacenter = "dc1" read-timeout = "10s" write-timeout = "10s" } # Even the persistor speaks OIDC: QE authenticates to Cassandra with # an OIDC private_key_jwt client assertion: it signs a JWT with the key in # cert-file, trades it at Keycloak for an access token, and presents THAT to # Cassandra (whose JwtAuthenticator validates it). No password anywhere. quine.store.oauth { client-id = "quine-cassandra-client" discovery-url = "https://keycloak.example.com/realms/cassandra/.well-known/openid-configuration" resource-uri = "cassandra://example" # audience Cassandra accepts cert-file = "/opt/certs/keystore.p12" # cert password arrives via CONFIG_FORCE_quine_store_oauth_cert__file__password } # Explicitly NO metrics reporting yet: QE's built-in default is [ { type = jmx } ] # (the Prometheus javaagent on these pods would pick that up immediately), so the # empty list keeps the dashboard dark until the metrics overlay deliberately turns it on. quine.metrics-reporters = [] ``` #### `overlays/cluster/deployment-patch.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/cluster/deployment-patch.yaml ```yaml # 4 replicas = target-size 3 + 1 hot spare. Spare-ness is PURELY replicas # exceeding target-size; there is no HOCON key for it (cluster membership is # config, spare-ness is scheduling). A clustered QE can roll safely, so the # strategy flips back to RollingUpdate. # # The Cassandra JWT-auth kit is carried forward from the persistence overlay (see its patch # for the full story): assertion keystore + cert password env, plus the # Temurin-entrypoint chaining that adds Cassandra's TLS cert to JVM trust. apiVersion: apps/v1 kind: Deployment metadata: name: quine-enterprise spec: replicas: 4 strategy: type: RollingUpdate template: spec: containers: - name: quine-enterprise command: ["/__cacert_entrypoint.sh", "/init-quine.sh"] ports: - name: cluster # bound only when a cluster-join config containerPort: 25520 # exists: part of the cluster kit env: - name: USE_SYSTEM_CA_CERTS value: "1" - name: CONFIG_FORCE_quine_store_oauth_cert__file__password valueFrom: secretKeyRef: name: quine-cassandra-cert key: password # Belt-and-suspenders for cluster-join.type=dns-entry: the conf # names the seed Service explicitly; the image also honors this. - name: QUINE_SEED_DNS value: quine-enterprise-seed volumeMounts: - name: quine-cassandra-cert mountPath: /opt/certs/keystore.p12 subPath: keystore.p12 readOnly: true - name: quine-cassandra-cert mountPath: /certificates/cassandra-ca.crt subPath: cassandra-ca.crt readOnly: true volumes: - name: quine-cassandra-cert secret: secretName: quine-cassandra-cert ``` #### `overlays/cluster/seed-service.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/cluster/seed-service.yaml ```yaml # The dns-entry discovery mechanism, introduced WITH clustering, not before: # a headless Service whose DNS entry resolves to every member pod. The conf's # cluster-join.name points at exactly this name. publishNotReadyAddresses # matters: booting members must be discoverable before they are "ready". apiVersion: v1 kind: Service metadata: name: quine-enterprise-seed spec: clusterIP: None publishNotReadyAddresses: true ports: - name: cluster port: 25520 targetPort: 25520 protocol: TCP ``` ## Section 5: authentication ### Auth on: one block and one line Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#authentication/1 the auth overlay's whole conf: a `quine.auth` block and one webserver-advertise line appear, everything else is carried forward byte-identical, and the overlay's patch adds exactly two env vars: the client secret and the session secret Configuration in focus: `overlays/auth/quine_enterprise.conf` Detail: - The superset discipline, third time: the cluster conf you already know rides along byte-identical (the license line, the cluster block, the whole `quine.store` with its oauth, the metrics-off line), and a side-by-side diff between neighbors shows exactly one capability arriving: authentication. - The two new secrets arrive exactly like the keystore password did: `CONFIG_FORCE_quine_auth_oidc_full_client_secret` and `CONFIG_FORCE_quine_auth_session_secret` env vars, each fed from a Kubernetes Secret. The mechanism is HOCON's env override (`config.override_with_env_vars`, enabled in the base Deployment's JVM options): at load time the prefix is stripped and the underscores become the key path, completing the world-readable conf with its secrets. - Strict validation, precisely: the auth block is parsed as a whole, and a partial block is a fatal boot error, not a warning. The live-proven failure mode: add any `CONFIG_FORCE_quine_auth_*` env var to a pre-auth overlay and its mere presence force-activates the block, crash-looping the pod with `Key not found: provider/id/expiration-seconds`. Wiring auth is all-or-nothing, which is why the shared base carries none of it. - The parenthetical in the file's header comment is a real constraint, not a musing: with auth on, the default API version must not be v1; the factory default (v2) is already correct, so the conf deliberately sets nothing. The docs state it flatly: [when auth is on, only v2 is mounted](https://docs.thatdot.com/quine-enterprise/reference/upgrade/migrating-from-api-v1/#api-version-overview). - This section's cards track the docs' own setup guide almost one to one: [Integrating Quine Enterprise with Your Identity Provider](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/). ### Any IdP, one contract Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#authentication/2 this deployment runs Keycloak, but QE speaks standard OIDC: every value in this block comes out of YOUR identity provider's discovery document and client registration Configuration in focus: `overlays/auth/quine_enterprise.conf` what QE needs from an identity provider ``` endpoint URLs where logins go and codes become tokens: published in the discovery document every OIDC provider serves at /.well-known/openid-configuration a confidential client an id + secret registered for QE a roles claim the six role names, at the top level of the access token (the next two cards) ``` the provider + client blocks, key by key *(where each value comes from)* ``` location-url your IdP's issuer URL, the discovery document's issuer field (a realm here; a tenant, org, or directory elsewhere) authorization-url login-path together, your discovery document's authorization_endpoint: base plus final path segment (auth here, authorize on many IdPs) token-url your discovery document's token_endpoint, verbatim access-token-audience the audience your IdP mints into this deployment's access tokens (aud claim); QE rejects bearer tokens that do not carry it. where it comes from varies: an audience mapper here, an API identifier, an authorization-server audience, an app id client.id the confidential client your IdP admin registered for QE; its secret is env-fed from a Kubernetes Secret, never in the conf ``` **any OIDC identity provider works**: ADFS, Okta, Microsoft Entra ID, Auth0, or a self-hosted provider like this deployment's Keycloak. Every value comes out of YOUR provider's discovery document and client registration; nothing in QE knows or cares which vendor answered. The checklist form of this card: [required configuration](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#required-configuration) Detail: - The discovery document is the universal contract: every OIDC-compliant provider serves `/.well-known/openid-configuration`, and it publishes the issuer, authorization endpoint, and token endpoint this block needs. Fill the block from your provider's document, not from memory of anyone's vendor docs. - The client to register: confidential (it holds a secret), standard authorization-code flow enabled, QE's callback as a redirect URI (the behind-the-load-balancer card has the fine print on that). Your IdP admin registers it once and hands over the id and secret; the secret goes straight into a Kubernetes Secret. - `access-token-audience` does double duty (verified in the QE source): it names the audience every bearer token must carry in `aud`, and setting it is what enables bearer-token authentication at all: unset, QE rejects bearers outright, because an unbound token from any of the IdP's other clients would otherwise pass. - Where the audience value comes from, per provider: an audience mapper on the client (Keycloak), an API identifier (Auth0), the authorization server's audience setting (Okta), an application or client id (Microsoft Entra ID), a relying-party trust identifier (ADFS). Same field, different admin console. - The docs' checklist form of this card: [required configuration](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#required-configuration). ### The six roles Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#authentication/3 six role names your identity provider must carry: four on a containment ladder, two specialists outside it the six roles *(who each is for)* ``` SuperAdmin platform owners; nothing restricted: use sparingly Architect senior engineers designing the pipeline: everything DataEngineer can, plus namespaces, shard limits, user-defined functions, application metrics DataEngineer builds and operates the pipelines: ingests, queries and writes, standing queries, algorithms Analyst explores the data: read-only queries and the Exploration UI; creates, modifies, deletes nothing SRE keeps it healthy: full operational visibility plus lifecycle control, but no reading or writing graph data Billing finance and procurement: license usage, nothing else ``` the containment chain ``` Analyst ⊂ DataEngineer ⊂ Architect ⊂ SuperAdmin each role includes every capability of the one before it SRE · Billing specialized sets OUTSIDE the chain: operations and finance are jobs, not steps on the product ladder ``` **this deployment's realm mirrors this exactly**: six client roles on the QE client, these names, nothing else, one seeded demo user per role. The names come from the docs, not from this deployment: your IdP must carry the same six. [required roles](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#required-roles) · [role hierarchy](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#role-hierarchy) Detail: - SRE, precisely: full operational visibility (metrics, application state, cluster status, ingest statuses, standing-query configurations, roles, license usage) plus lifecycle control (shutdown, system control, node sleep, pausing and resuming ingests, cancelling queries), and NO graph access of any kind: no reading or writing data, no creating or deleting resources. - Billing is deliberately tiny: the license usage view and its backing endpoint, nothing else. It exists so procurement can self-serve a number without a standing request to engineering. - Assignment rules: a user can hold several roles (the union applies), and must hold at least one of the six: a user with none of them cannot access QE at all. - The full matrix behind this card crosses every role with every capability, thirty-plus operations from `GraphRead` to `Shutdown`: [capabilities](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#capabilities). The closing card of this section plays a slice of it. - The realm mirrors the docs exactly: it defines exactly these six as client roles on the QE client, with one seeded demo user each. If you meet an `Admin` role in older thatDot material: it no longer exists; `SRE` replaced it. ### The token QE accepts Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#authentication/4 authorization rides in the access token: a top-level `roles` claim, spelled exactly right, or it does not exist a token QE accepts *(the docs' example; other claims trimmed)* ``` { "iss": "https://idp.example.com/...", "aud": "your-client-id", "sub": "a76a8d54-50d4-4ff2-a38b-135d426af310", "roles": ["SuperAdmin"] } ``` the two non-negotiables ``` a top-level roles claim QE reads roles from the JWT root only: nested homes like resource_access..roles or realm_access.roles are not consulted. most IdPs default to nested and need a claim mapper exact PascalCase values SuperAdmin, SRE, Architect, DataEngineer, Analyst, Billing: case-sensitive, no normalization; sre, ARCHITECT, data-engineer are silently discarded ``` **your IdP needs one rule for each**: providers do not emit this shape by default; whatever yours calls it (claim mapper, claim transformation, token customization rule), it must emit the top-level `roles` claim and the audience QE expects. This deployment's IdP carries both rules; its logins only succeed because of them. The provider-agnostic walkthrough: [setting up roles in your identity provider](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#setting-up-roles-in-your-identity-provider) Detail: - Why the docs shout about nesting: most providers emit roles nested by default (`resource_access..roles` here, `realm_access.roles` or schema-URI claim names elsewhere), and every one of them needs a mapper, transformation, or rule to ALSO emit the top-level claim. - Wrong-shape symptoms, from the docs' diagnosis table: roles missing from the JWT root = the browser loops between login and QE, and `/api/v2/auth/me` answers 401 with `CouldNotDecodeClaim`; roles present but misspelled = login succeeds and every action is denied, with `/api/v2/auth/me` showing empty `roles: []`. Two different symptoms, one config mistake each. - To see what your IdP actually emits, decode the access token's payload (the middle base64 segment, between the dots) and look for `roles` at the top level. - Three roles also accept their long display names as alternates: `Super Administrator`, `Data Engineer`, `Site Reliability Engineer`. Everything else is the exact PascalCase or silence. - The provider-agnostic walkthrough (create the roles, assign users, configure the claim): [setting up roles in your identity provider](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#setting-up-roles-in-your-identity-provider). ### Humans and machines Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#authentication/5 browsers get a session cookie after the login round trip; scripts and services present a bearer token: either way, the roles in the token decide Configuration in focus: `overlays/auth/quine_enterprise.conf` the session block, key by key ``` session secret signs the session cookie (env-fed, the CONFIG_FORCE_* line). every member must hold the SAME secret: the load balancer sends your next request to any member, and each must verify cookies the others minted. keep it stable: rotating it logs every user out at once expiration-seconds session lifetime: the one setting here with no default; strict validation refuses to boot without it secure-cookies the cookie only ever travels over https ``` machines skip the browser *(client_credentials bearers)* ``` POST # from the discovery document grant_type=client_credentials client_id + client_secret # a service account, not a human → access_token # carries the service account's roles Authorization: Bearer # on every API call ``` **two doors, one role check**: humans get a session cookie after the login round trip; scripts and services present a bearer token from the client_credentials grant. Either way, the roles in the token decide what is allowed. This deployment's service account carries `SRE`: status and metrics reads, no graph access. [authentication methods](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#authentication-methods) · [API access: bearer tokens](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#api-access-bearer-tokens) Detail: - The shared-secret rule is mechanical, not stylistic (verified in the QE source): the session cookie is itself a signed token, HMAC'd with the session secret, and every member validates cookies with the secret it was configured with. Behind a load balancer, consecutive requests land on different members, so the members must agree. Rotate the secret and every session fails validation at once: a mass logout, not an error. The three session keys' reference: [session management properties](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#session-management-properties). - Sessions also replicate (verified in source): members share the session store over the cluster's own replication layer, so the login you completed through one member is honored by all of them. - A denied request has one exact shape: a JSON error whose message reads `Missing Permission: `, naming precisely the capability the role lacked. The closing card's captions quote it verbatim. [how to interpret auth errors](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#how-to-interpret-auth-errors). - Audit logging is one system property away (`-Dthatdot.audit.loglevel=INFO`): every login step and every API call logged with correlatable ids (a per-login attempt id, a per-request action id, hashed token identifiers, never the tokens themselves): the trail auditors ask for. [audit logging](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#audit-logging) · [the control mapping auditors ask for](https://docs.thatdot.com/quine-enterprise/reference/compliance-overview/#audit-and-accountability-au). - Bearer hygiene: access tokens expire on the IdP's schedule, often within minutes; a service simply re-runs the same client_credentials call on expiry. [API access: bearer tokens](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#api-access-bearer-tokens). ### Behind the load balancer Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#authentication/6 QE binds a plain pod port but lives at an https front door: one conf line teaches it its public name, and logins depend on it Configuration in focus: `overlays/auth/quine_enterprise.conf` two addresses are true at once ``` what QE binds 0.0.0.0:8080 inside the pod, plain http: TLS ended at the load balancer what users reach https://quine.example.com (port 443) whenever QE writes a URL that points at itself (the OIDC login callback above all), it must name the front door: this line is how it knows the name ``` and register what the callback really renders *(at the IdP)* ``` https://quine.example.com/api/v2/auth/callback https://quine.example.com:443/api/v2/auth/callback BOTH: the advertised callback renders its port, and Keycloak does not treat :443 as equal to the bare https form (proven live here: the bare registration alone gets logins rejected) ``` **the failure you would see**: QE cannot see the front door on its own; from inside the pod every request arrives as plain http. Without this line's `use-tls` the callback renders `http://…:443`, the IdP finds no such registered redirect, and every login dies at the IdP before a password is typed. The reference, including the general reverse-proxy story: [webserver configuration](https://docs.thatdot.com/quine-enterprise/reference/config/quine-webserver-advertise/) Detail: - What QE does without the line (verified in source): it infers its address from the incoming request, and from inside the pod that request is plain http, because TLS ended at the load balancer. Any URL QE writes about itself comes out wrong the moment a proxy fronts it; the advertise line states the truth the proxy hides. - `use-tls` is the load-bearing field for OIDC, and it overrides the bind side's scheme (source-verified): the pod keeps serving plain http, but every self-referential URL, the login callback above all, now says https. Without it the callback renders `http://…:443`, and the IdP compares that against its registered redirect list and refuses. - The registration fine print, proven live on this deployment: the advertised callback renders its port explicitly, and Keycloak does not treat `:443` as equal to the bare https form, so the realm registers both spellings (of the callback and of the app root). Whatever your IdP: register what the callback actually renders, character for character. - Nothing here is specific to this deployment's load balancer: nginx, Envoy, HAProxy, any TLS-terminating front door creates the same two-truths situation. The reference page carries the general reverse-proxy story: [serving Quine Enterprise behind a reverse proxy](https://docs.thatdot.com/quine-enterprise/reference/config/quine-webserver-advertise/#serving-quine-enterprise-behind-a-reverse-proxy). ### Who may do what: try it Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#authentication/7 six users, one per role, four actions, one capability check each: the same grid proven live on a real deployment of this exact configuration Interactive: who may do what. Pick one of the six role accounts, then try an action (run a query, create an ingest, read metrics, read license usage). The request travels through the load balancer to a member and is allowed or denied by the capability the role carries. Detail: - Every cell of this grid is the docs' capabilities matrix, quoted: run a query is `GraphRead`, create an ingest is `IngestWrite`, read metrics is `ApplicationMetricsRead`, read license usage is `LicenseRead`. The deny caption is the real error's message field, verbatim. [the full matrix](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#capabilities). - The one asterisk: Analyst may query (`GraphRead`) but not scan: a query that may touch every node additionally requires `AllNodeScan` (verified in the QE source: the query endpoints check it on top of GraphRead), and Analyst lacks it. An analyst's first exploratory look-at-everything query is the classic place to meet that denial. - The Keycloak round trip happens once per user: after it, the session cookie answers for them, which is why repeat actions skip the violet hop. Switching users is what sends the simulator (and a real deployment) back through the IdP. - Notice what never happens: the IdP is not consulted per action. Keycloak's job ended when the token was minted; every allow and deny after that is QE reading roles out of the token it already holds. The docs' three-line version of this division of labor: [how OIDC authentication works](https://docs.thatdot.com/quine-enterprise/learn/oidc-setup/#how-oidc-authentication-works). ### Configuration this section adds ```sh kubectl apply -k overlays/auth/ ``` #### `overlays/auth/kustomization.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/auth/kustomization.yaml ```yaml # Cassandra + Keycloak: the login wall appears. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # Namespace + labels are REPEATED in every overlay on purpose: resources created # by this overlay's configMapGenerator do not inherit the base's transformers; # without the repeat the ConfigMap lands in the kubectl context's default # namespace and label-based cleanup never finds it. namespace: quine labels: - pairs: app.kubernetes.io/name: quine-enterprise app.kubernetes.io/part-of: quine-deployment includeSelectors: true resources: - ../../base - seed-service.yaml # the cluster-discovery mechanism, introduced with clustering configMapGenerator: - name: quine-config files: - quine_enterprise.conf patches: - path: deployment-patch.yaml ``` #### `overlays/auth/quine_enterprise.conf` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/auth/quine_enterprise.conf ```hocon # ── Cassandra + Keycloak ───────────────────────────────────────────────────── # Adds authentication: the presence of the quine.auth block is the on-switch. # Users now hit a Keycloak login wall; roles come from the client-role claim. # (Deliberately NOT setting quine.default-api-version: it must not be v1 under # auth, and the default v2 is correct.) # Licensing: the key's VALUE never appears here or anywhere in the repo. HOCON # ${?ENV} substitution reads it from the pod environment (fed from a k8s Secret). # This line appears in every overlay. quine.license-key = ${?QUINE_LICENSE_KEY} quine.cluster.target-size = 3 # default is 1; three active members from here on quine.cluster.cluster-join = { type = dns-entry name = "quine-enterprise-seed" # headless seed Service; members discover each # other through its DNS entry } quine.store = { type = cassandra endpoints = ["cassandra-dc1-service.cassandra.svc:9042"] keyspace = quine should-create-keyspace = true # dev convenience; never in prod should-create-tables = true replication-factor = 1 # matches the single-node dc1 read-consistency = LOCAL_QUORUM write-consistency = LOCAL_QUORUM local-datacenter = "dc1" read-timeout = "10s" write-timeout = "10s" } # Even the persistor speaks OIDC: QE authenticates to Cassandra with # an OIDC private_key_jwt client assertion: it signs a JWT with the key in # cert-file, trades it at Keycloak for an access token, and presents THAT to # Cassandra (whose JwtAuthenticator validates it). No password anywhere. quine.store.oauth { client-id = "quine-cassandra-client" discovery-url = "https://keycloak.example.com/realms/cassandra/.well-known/openid-configuration" resource-uri = "cassandra://example" # audience Cassandra accepts cert-file = "/opt/certs/keystore.p12" # cert password arrives via CONFIG_FORCE_quine_store_oauth_cert__file__password } quine.auth { session { # secret arrives via CONFIG_FORCE_quine_auth_session_secret expiration-seconds = 3600 # MANDATORY: QE crashes if absent secure-cookies = true } oidc.full { provider { location-url = "https://keycloak.example.com/realms/quine-enterprise" authorization-url = "https://keycloak.example.com/realms/quine-enterprise/protocol/openid-connect" login-path = "auth" token-url = "https://keycloak.example.com/realms/quine-enterprise/protocol/openid-connect/token" access-token-audience = "quine-enterprise-client" } client { id = "quine-enterprise-client" # secret arrives via CONFIG_FORCE_quine_auth_oidc_full_client_secret } } } # QE sits behind a TLS-terminating load balancer: advertise the public https front door so # the OIDC callback is built as https://… (without use-tls the callback renders # http://…:443 and Keycloak rejects the redirect). quine.webserver-advertise = { address = "quine.example.com", port = 443, use-tls = true } # Explicitly NO metrics reporting yet: QE's built-in default is [ { type = jmx } ] # (the Prometheus javaagent on these pods would pick that up immediately), so the # empty list keeps the dashboard dark until the metrics overlay deliberately turns it on. quine.metrics-reporters = [] ``` #### `overlays/auth/deployment-patch.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/auth/deployment-patch.yaml ```yaml # Carried forward from the cluster overlay: 4 replicas = target-size 3 + 1 hot spare, # RollingUpdate (a clustered QE can roll safely), and the Cassandra JWT-auth # kit (see the persistence overlay's patch for the full story). # # The auth secrets join HERE, not in the base: CONFIG_FORCE_quine_auth_* env # vars inject quine.auth.* config keys, and the mere presence of any of them # force-activates QE's strictly-validated auth block; a pre-auth overlay carrying # them crash-loops with "Key not found: provider/id/expiration-seconds". apiVersion: apps/v1 kind: Deployment metadata: name: quine-enterprise spec: replicas: 4 strategy: type: RollingUpdate template: spec: containers: - name: quine-enterprise command: ["/__cacert_entrypoint.sh", "/init-quine.sh"] ports: - name: cluster # bound only when a cluster-join config containerPort: 25520 # exists: part of the cluster kit env: - name: USE_SYSTEM_CA_CERTS value: "1" - name: CONFIG_FORCE_quine_store_oauth_cert__file__password valueFrom: secretKeyRef: name: quine-cassandra-cert key: password # Belt-and-suspenders for cluster-join.type=dns-entry (see the cluster overlay). - name: QUINE_SEED_DNS value: quine-enterprise-seed - name: CONFIG_FORCE_quine_auth_oidc_full_client_secret valueFrom: secretKeyRef: name: quine-oidc key: client-secret - name: CONFIG_FORCE_quine_auth_session_secret valueFrom: secretKeyRef: name: quine-session key: session-secret volumeMounts: - name: quine-cassandra-cert mountPath: /opt/certs/keystore.p12 subPath: keystore.p12 readOnly: true - name: quine-cassandra-cert mountPath: /certificates/cassandra-ca.crt subPath: cassandra-ca.crt readOnly: true volumes: - name: quine-cassandra-cert secret: secretName: quine-cassandra-cert ``` #### `overlays/auth/seed-service.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/auth/seed-service.yaml ```yaml # The dns-entry discovery mechanism, introduced WITH clustering, not before: # a headless Service whose DNS entry resolves to every member pod. The conf's # cluster-join.name points at exactly this name. publishNotReadyAddresses # matters: booting members must be discoverable before they are "ready". apiVersion: v1 kind: Service metadata: name: quine-enterprise-seed spec: clusterIP: None publishNotReadyAddresses: true ports: - name: cluster port: 25520 targetPort: 25520 protocol: TCP ``` ## Section 6: observability ### Metrics on: one changed line Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#observability/1 the metrics overlay's whole conf: one line flips from an empty list to the JMX reporter, and everything else is carried forward byte-identical: the smallest diff of the tour Configuration in focus: `overlays/metrics/quine_enterprise.conf` Detail: - The superset discipline, one last time: the auth conf you already know rides along byte-identical, and the capability arrives as one changed line. A side-by-side diff between neighbors is a header and a list literal. - Why the line reads as a SWAP and not an addition: QE's factory default already is `[ { type = jmx } ]`. The baseline section explained the suppression (the empty list kept the always-present javaagent serving nothing); this overlay simply stops suppressing. The counter-intuitive line from the baseline conf card resolves here. - JMX is one entry on a menu: the reporters list also speaks CSV files, InfluxDB, and SLF4J logging, and entries can be combined. [metrics output formats](https://docs.thatdot.com/quine-enterprise/learn/metrics/quick-start/#metrics-output-formats) shows the shapes; this deployment uses JMX because the image's javaagent turns JMX into a Prometheus endpoint for free. - The auth section reaches into this one (verified on the docs' metrics pages): reading metrics through the REST API requires the `ApplicationMetricsRead` capability, held by `SRE` and `Architect`. The Prometheus scrape path on :9090 is separate plumbing inside the pod network; the role check guards the public API. The claim's source, first block on [the metrics docs](https://docs.thatdot.com/quine-enterprise/learn/metrics/). - No-tooling verification while the kit settles: `GET /api/v2/system/metrics` returns the same numbers as JSON, straight from a member. [metrics quick start](https://docs.thatdot.com/quine-enterprise/learn/metrics/quick-start/). ### The other half rides in the patch Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#observability/2 the conf made QE emit; three pod annotations let Prometheus collect: either half alone leaves Grafana dark Configuration in focus: `overlays/metrics/deployment-patch.yaml` deployment.yaml *(base, one line that was always there)* ``` -javaagent:jmx_prometheus_javaagent.jar=9090:/exporter.yaml in the container's JVM options at EVERY overlay: the agent has served :9090 since the baseline; the conf's empty list just kept it empty of QE metrics ``` **either half alone leaves Grafana dark**: the conf makes QE emit (JMX, re-served on :9090 by the javaagent); the patch's three annotations make Prometheus collect (its discovery reads them off every pod). Pods from the earlier overlays carry no annotations and are entirely unknown to Prometheus: the before/after on the dashboard is real, not staged. What the payload holds: [collected metrics](https://docs.thatdot.com/quine-enterprise/learn/metrics/metrics/#available-metrics) Detail: - The annotations are a discovery convention, not a Kubernetes standard: THIS Prometheus (the community chart, annotation-based discovery) reads `prometheus.io/scrape|port|path` off pods to build its target list. A cluster running the Prometheus operator would declare a PodMonitor object instead. The portable statement: tell your collector which pods to scrape, on which port and path; these annotations are this cluster's spelling of it. The docs' chart-driven spelling of the same contract: [Prometheus and Grafana on Kubernetes](https://docs.thatdot.com/quine-enterprise/learn/orchestration/prometheus-grafana-k8s/#prometheus-helm-values-prometheus-valuesyaml). - The annotations live on the pod TEMPLATE, so applying the overlay rolls the Deployment, and the fresh pods are born discoverable. Earlier overlays' pods carried no annotations and were entirely unknown to Prometheus: the dashboard's before/after is real, not staged. - The javaagent line is the quiet third member of the kit, and it was in the base all along: it turns the JVM's JMX beans into a Prometheus-format /metrics page on :9090 at every overlay. What was missing until now was anything QE-shaped in those beans (the conf half) and anyone asking (this half). - One small tease: the same patch carries one more addition that has nothing to do with metrics. It gets its own card in the finale section. ### From JMX bean to dashboard panel Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#observability/3 QE reports to JMX, the in-pod javaagent serves :9090, Prometheus pulls on its schedule, Grafana queries Prometheus: four hops, one direction the pipeline, end to end ``` QE metrics registry counters, meters, histograms: node counts, ingest rates, persistor latency, standing-query lag ↓ reports via JMX the conf's one flipped line javaagent on :9090 baked into the image, running at every overlay: re-serves the JMX beans as a Prometheus-format /metrics page ↓ Prometheus PULLS on its own schedule, from every pod its discovery found (the patch's annotations) Prometheus stores the samples as time series ↓ Grafana queries Grafana draws the dashboard the finale will light up ``` **pull, not push**: QE never sends metrics anywhere. It exposes them, and Prometheus initiates every collection (a scrape is an HTTP GET of that /metrics page). That is why the scene's scrape traffic flows INTO Prometheus and nothing flows back out: losing Prometheus loses history, never data, and never touches QE. No-tooling checks of the same numbers: [metrics quick start](https://docs.thatdot.com/quine-enterprise/learn/metrics/quick-start/) Detail: - What actually rides the pipe (the [collected metrics reference](https://docs.thatdot.com/quine-enterprise/learn/metrics/metrics/#available-metrics) lists all of it): node and edge counters, per-stream ingest meters, persistor operation latency histograms, standing-query result meters, plus the JVM's own memory and GC numbers. - The scene animates the model faithfully: scrape balls flow INTO Prometheus only (it pulls; nothing flows back out), one from every active member per wave, and every second wave Grafana sends a query ball to Prometheus and gets a response ball back. - Losing the collector is survivable by design: if Prometheus is down, QE neither notices nor slows; the scrapes simply stop and that window of history is never collected. The metrics are re-exposed continuously, not queued. - The dashboard was provisioned with the rest of the services and has been sitting dark since the baseline card first pointed at the dim shapes. This section lights the plumbing; the data arrives next section. ### Reading the dashboard Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#observability/4 ingest rate, persistor latency, standing-query lag, per-member CPU: the sizing loop's inputs, now live instead of theoretical four signals to read first *(the sizing loop's inputs)* ``` ingest rate records/s per member, per stream: the throughput you are actually getting persistor latency Cassandra round trips as QE experiences them: where a storage bottleneck shows first standing-query lag matches queuing faster than outputs drain them; sustained backpressure pauses ingest per-member CPU the engine's own ceiling: the wall you hit when storage keeps up ``` and one graph-health panel *(the supernode detector)* ``` Nodes by Edge Count how many awake nodes fall into each edge-count bucket. The upper buckets answer "do supernodes exist?", and the per-bucket counts show their topology: a few huge ones and many merely-large ones are different problems ``` **this is the clustering section's sizing loop, live**: measure these four under peak traffic, resize toward roughly 80% utilization, repeat ([the evaluation method](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-sizing/#evaluating-your-cluster-size)). The docs also turn each signal into a production alert rule with thresholds: [ingest rate](https://docs.thatdot.com/quine-enterprise/learn/metrics/recommended-alerts/#ingest-rate-per-stream) · [persistor latency](https://docs.thatdot.com/quine-enterprise/learn/metrics/recommended-alerts/#persistor-latency) · [standing-query backpressure](https://docs.thatdot.com/quine-enterprise/learn/metrics/recommended-alerts/#standing-query-backpressure) Detail: - The clustering section's tuning card promised a measurement loop; these are its instruments. Watch the four signals under PEAK traffic, decide which resource is actually the bottleneck, resize toward roughly 80% utilization, measure again: [evaluating your cluster size](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-sizing/#evaluating-your-cluster-size). The tour's own cross-reference: [the tuning card](#clustering/6). The dashboard itself is [published in the docs](https://docs.thatdot.com/quine-enterprise/learn/orchestration/prometheus-grafana-k8s/#quine-enterprise-grafana-dashboard), importable on your own cluster. - How to read the pair that matters most: ingest rate falling while persistor latency climbs says storage is the wall (Cassandra needs attention); ingest rate flat while member CPU is pegged says the engine is the wall (more or bigger members). The panel-to-metric-name mapping: [key metrics to monitor](https://docs.thatdot.com/quine-enterprise/learn/metrics/quick-start/#key-metrics-to-monitor). - Standing-query lag deserves respect in a streaming system: sustained backpressure from an output that cannot keep up (a slow webhook, a throttled topic) propagates upstream and pauses ingest. The [recommended alerts page](https://docs.thatdot.com/quine-enterprise/learn/metrics/recommended-alerts/#standing-query-backpressure) gives it an alert rule with thresholds, alongside rules for the other three signals. - The supernode detector, precisely (the docs' [supernode edge counts](https://docs.thatdot.com/quine-enterprise/learn/metrics/recommended-alerts/#supernode-edge-counts) alert): the panel draws the `node.edge-counts` histogram, and the buckets carry the story. The `2048-16383` bucket staying populated is the warning (nodes with thousands of edges accumulating); `16384-infinity` going non-zero is critical (a live supernode). Supernodes are expensive everywhere at once: slow to wake, sleep, and snapshot, heavier on the persistor, and they serialize traversals through one hot node. - Two honest caveats from the same docs section: the histogram only counts nodes currently awake in memory, so pair it with the log signal (QE logs `Node has: edges` every 10,000 edges on a node, awake or not) for durable detection. And the storage tie-in from the persistence section: the enterprise `edges` table exists exactly for supernode support ([what QE stores](#persistence/3)). - The dashboard right now shows a healthy IDLE cluster: near-zero rates, flat latency, quiet CPU. That is deliberate: nothing is flowing yet. The next section's streams are what make every panel climb, and watching them climb is that section's payoff. ### Configuration this section adds ```sh kubectl apply -k overlays/metrics/ ``` #### `overlays/metrics/kustomization.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/metrics/kustomization.yaml ```yaml # Cassandra + Keycloak + metrics: the Grafana dashboard lights up. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # Namespace + labels are REPEATED in every overlay on purpose: resources created # by this overlay's configMapGenerator do not inherit the base's transformers; # without the repeat the ConfigMap lands in the kubectl context's default # namespace and label-based cleanup never finds it. namespace: quine labels: - pairs: app.kubernetes.io/name: quine-enterprise app.kubernetes.io/part-of: quine-deployment includeSelectors: true resources: - ../../base - seed-service.yaml # the cluster-discovery mechanism, introduced with clustering configMapGenerator: - name: quine-config files: - quine_enterprise.conf patches: - path: deployment-patch.yaml ``` #### `overlays/metrics/quine_enterprise.conf` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/metrics/quine_enterprise.conf ```hocon # ── Cassandra + Keycloak + metrics ─────────────────────────────────────────── # The metrics kit, in two halves. App half (HERE): metrics-reporters flips from # [] to the JMX reporter: QE starts emitting, and the always-present Prometheus # javaagent serves the metrics on :9090. Infra half (this overlay's deployment # patch): the prometheus.io/* scrape annotations that let Prometheus discover # the pods. Either half alone leaves Grafana dark. Everything else matches the auth overlay. # Licensing: the key's VALUE never appears here or anywhere in the repo. HOCON # ${?ENV} substitution reads it from the pod environment (fed from a k8s Secret). # This line appears in every overlay. quine.license-key = ${?QUINE_LICENSE_KEY} quine.cluster.target-size = 3 # default is 1; three active members from here on quine.cluster.cluster-join = { type = dns-entry name = "quine-enterprise-seed" # headless seed Service; members discover each # other through its DNS entry } quine.store = { type = cassandra endpoints = ["cassandra-dc1-service.cassandra.svc:9042"] keyspace = quine should-create-keyspace = true # dev convenience; never in prod should-create-tables = true replication-factor = 1 # matches the single-node dc1 read-consistency = LOCAL_QUORUM write-consistency = LOCAL_QUORUM local-datacenter = "dc1" read-timeout = "10s" write-timeout = "10s" } # Even the persistor speaks OIDC: QE authenticates to Cassandra with # an OIDC private_key_jwt client assertion: it signs a JWT with the key in # cert-file, trades it at Keycloak for an access token, and presents THAT to # Cassandra (whose JwtAuthenticator validates it). No password anywhere. quine.store.oauth { client-id = "quine-cassandra-client" discovery-url = "https://keycloak.example.com/realms/cassandra/.well-known/openid-configuration" resource-uri = "cassandra://example" # audience Cassandra accepts cert-file = "/opt/certs/keystore.p12" # cert password arrives via CONFIG_FORCE_quine_store_oauth_cert__file__password } quine.auth { session { # secret arrives via CONFIG_FORCE_quine_auth_session_secret expiration-seconds = 3600 # MANDATORY: QE crashes if absent secure-cookies = true } oidc.full { provider { location-url = "https://keycloak.example.com/realms/quine-enterprise" authorization-url = "https://keycloak.example.com/realms/quine-enterprise/protocol/openid-connect" login-path = "auth" token-url = "https://keycloak.example.com/realms/quine-enterprise/protocol/openid-connect/token" access-token-audience = "quine-enterprise-client" } client { id = "quine-enterprise-client" # secret arrives via CONFIG_FORCE_quine_auth_oidc_full_client_secret } } } # QE sits behind a TLS-terminating load balancer: advertise the public https front door so # the OIDC callback is built as https://… (without use-tls the callback renders # http://…:443 and Keycloak rejects the redirect). quine.webserver-advertise = { address = "quine.example.com", port = 443, use-tls = true } # The app half of the metrics kit: report metrics via JMX, where the # (always-present) Prometheus javaagent serves them on :9090. The infra half, # the scrape annotations Prometheus discovers pods by, is in this overlay's # deployment patch. Together they light up Grafana. quine.metrics-reporters = [ { type = jmx } ] ``` #### `overlays/metrics/deployment-patch.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/metrics/deployment-patch.yaml ```yaml # Carried forward from the cluster overlay: 4 replicas = target-size 3 + 1 hot spare, # RollingUpdate (a clustered QE can roll safely), and the Cassandra JWT-auth # kit (see the persistence overlay's patch for the full story). # # The auth secrets join at the auth overlay: injecting # quine.auth.* keys via env force-activates QE's strictly-validated auth block, # so earlier overlays must not carry these env vars. apiVersion: apps/v1 kind: Deployment metadata: name: quine-enterprise spec: replicas: 4 strategy: type: RollingUpdate template: metadata: annotations: # The infra half of the metrics kit: these annotations are how the # (annotation-discovery) Prometheus finds QE pods. The conf's # metrics-reporters swap is the app half; emission without collection, # or collection without emission, both leave Grafana dark. prometheus.io/scrape: "true" prometheus.io/port: "9090" prometheus.io/path: "/metrics" spec: containers: - name: quine-enterprise command: ["/__cacert_entrypoint.sh", "/init-quine.sh"] ports: - name: cluster # bound only when a cluster-join config containerPort: 25520 # exists: part of the cluster kit env: - name: USE_SYSTEM_CA_CERTS value: "1" - name: CONFIG_FORCE_quine_store_oauth_cert__file__password valueFrom: secretKeyRef: name: quine-cassandra-cert key: password # Belt-and-suspenders for cluster-join.type=dns-entry (see the cluster overlay). - name: QUINE_SEED_DNS value: quine-enterprise-seed - name: CONFIG_FORCE_quine_auth_oidc_full_client_secret valueFrom: secretKeyRef: name: quine-oidc key: client-secret - name: CONFIG_FORCE_quine_auth_session_secret valueFrom: secretKeyRef: name: quine-session key: session-secret volumeMounts: - name: quine-cassandra-cert mountPath: /opt/certs/keystore.p12 subPath: keystore.p12 readOnly: true - name: quine-cassandra-cert mountPath: /certificates/cassandra-ca.crt subPath: cassandra-ca.crt readOnly: true # Kafka OAuth: the Strimzi cluster CA, for the Kafka ingest's # client-scoped ssl.truststore.* (PEM); NOT part of JVM trust. # Mounted ONLY in this overlay: the streaming section's Kafka # ingests are its sole consumers, and they are installed only # here. Note an installed ingest persists in persistor metadata: # switching to an earlier overlay afterwards restores it as # FAILED (this mount path is absent there). - name: kafka-ca mountPath: /opt/kafka-ca/ca.crt subPath: ca.crt readOnly: true volumes: - name: quine-cassandra-cert secret: secretName: quine-cassandra-cert - name: kafka-ca secret: secretName: kafka-ca ``` #### `overlays/metrics/seed-service.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/overlays/metrics/seed-service.yaml ```yaml # The dns-entry discovery mechanism, introduced WITH clustering, not before: # a headless Service whose DNS entry resolves to every member pod. The conf's # cluster-join.name points at exactly this name. publishNotReadyAddresses # matters: booting members must be discoverable before they are "ready". apiVersion: v1 kind: Service metadata: name: quine-enterprise-seed spec: clusterIP: None publishNotReadyAddresses: true ports: - name: cluster port: 25520 targetPort: 25520 protocol: TCP ``` ## Section 7: streaming ### Nothing left to configure Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#streaming/1 the finale arrives with no overlay to apply: configuration made Quine Enterprise ready, and the workload itself is runtime API assets, created through the authenticated API what the installer POSTs *(in dependency order)* ``` 1 · query-UI styling PUT /api/v2/queryUi/sampleQueries · quickQueries · nodeAppearances replace semantics: idempotent, cluster-wide 2 · standing queries POST /api/v2/graph/quine/standingQueries once each: cluster-wide, and registered BEFORE the ingests, so every record is matched as it arrives (matching is prospective) 3 · the Kafka ingests POST /api/v2/graph/quine/ingests four streams, six POSTs: an ingest lives on ONE member (the Quine-Member-Idx header says which); events goes to all three members, the other three pin to members 0, 1, and 2 ``` **the assets are the contract; the installer is a detail**: here a Kubernetes Job makes these calls with a bearer token, but a CI pipeline, a script, or a colleague with curl would install the identical assets. And all of it persists in the graph's metadata: the installer runs once, not at every boot. [ingest streams](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/) · [standing queries](https://docs.thatdot.com/quine-enterprise/learn/standing-queries/standing-queries/) Detail: - The line this section closes: every section so far was configuration: a persistor, a cluster, an identity provider, a metrics kit. The workload itself never needed a conf file, because ingests and standing queries are runtime API objects. - The calls ride the auth section's machinery: the installer authenticates like any other API client (a bearer token from the identity provider) as a user whose roles grant creating ingests and standing queries. The capability names on the auth section's matrix are exactly what gets checked. The API surface itself: [the REST API](https://docs.thatdot.com/quine-enterprise/core-concepts/rest-api/#authentication), and every running server hosts [interactive API docs](https://docs.thatdot.com/quine-enterprise/core-concepts/rest-api/#interactive-docs) with the full ingest and standing-query schemas. - The assets persist in the graph's metadata, in the persistor (the persistence section's meta_data table, doing its job): they survive pod restarts and rolling updates, and a keyspace reset is what removes them. Install once, not at every boot. - Step 1 is cosmetic and stays a footnote: sample queries, quick queries, and node appearances style the exploration UI's right-click flows. Useful, not architectural. - Why the ordering matters: standing queries match prospectively, as data arrives. Registering them before the ingest means every record is matched on arrival; registering them after would mean the earlier records pass unmatched. ### The ingest stream, verbatim Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#streaming/2 one YAML asset per stream defines the whole intake: topic, consumer group, offset discipline, rate limit, record format, and the query that lands each record in the graph Configuration in focus: `runtime assets/ingest-events.yaml` Detail: - The offset discipline is load-bearing (proven on this cluster): with `ExplicitCommit`, the group's progress survives member disruption and the stream resumes where it left off. Without it, any disruption restarts from EARLIEST: the counts reset and the whole topic replays. The product-level rule behind it: [ingest is at least once](https://docs.thatdot.com/quine-enterprise/core-concepts/delivery-guarantees/#ingest-at-least-once). - `autoOffsetReset: EARLIEST` answers a different question: where a BRAND NEW consumer group begins. First run: the topic's beginning. The committed offsets take over from there. - Operating gotcha worth knowing before it bites: a per-name GET or DELETE of an ingest answers only on the member hosting that stream; through the load balancer, other members return 404, which reads as "the ingest vanished". The LIST endpoint tells the truth from any member; target one member with the `Quine-Member-Idx` header. The endpoint list itself: [inspecting ingest streams via the API](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/#inspecting-ingest-streams-via-the-api). - The Cypher at the bottom is deliberately minimal: each record lands as one Event node carrying its fields. Real ingest queries model a domain, and writing them is [a discipline of its own](https://docs.thatdot.com/quine-enterprise/learn/cypher/), beyond this tour's scope. - `maxPerSecond` is the deliberate pacing of the feed: it throttles THIS stream instance, so three members at 75 records/s each hold an intended 225/s total, and the feed flows like production traffic instead of racing to the end. The other three streams carry their own rates. - What this payload deliberately omits: an error policy. When a record fails to parse or its query errors, the stream's configured handling decides between retrying, logging, and shipping the record to a dead letter queue; the docs carry the menu: [record and stream error handling](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/#error-handling) · [dead letter queue](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/#dead-letter-queue). - The full source reference, including every field on this card: [Kafka ingest](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/kafka/) · [ingest streams](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/). ### One identity: store, data in, data out Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#streaming/3 a standing query matches as data arrives and emits to the matches topic; its Kafka destination carries the same private-key JWT assertion the ingest source does: one keystore identity for the store, the data in, and the data out Configuration in focus: `runtime assets/standing-query.yaml` the assertion block, key by key *(what each is and where it comes from)* ``` type OAuthBearerAssertionLogin: the Enterprise assertion flow; no client secret exists. QE signs a JWT with the keystore's private key and trades it at the IdP for the bearer token the broker sees clientId the OAuth client your IdP admin registers for Kafka access: confidential, client authenticator "signed JWT" (the IdP trusts the certificate's public key) certFile the SAME keystore the persistence section mounted: one machine identity, registered once per realm certFilePassword the one secret: substituted at POST time from a Kubernetes Secret, redacted in every API response resourceUri the audience the broker validates; your IdP mints it into the token's aud claim discoveryUrl the IdP realm that owns the Kafka client; QE reads the token endpoint out of its discovery document sasl.mechanism explicit in kafkaProperties: NOT derived from saslJaasConfig ssl.truststore.* trust for the broker's TLS listener, scoped to this Kafka client only; the mount that provides the file has its own card ``` **consuming and producing, symmetrical**: this file's Kafka destination carries the identical block the ingest source does, so ONE registered identity covers the store, the data in, and the data out. One asymmetry to know: destinations have no `securityProtocol` field, so `security.protocol` rides in `kafkaProperties` there. The full contract, both directions: [OAuth JWT assertion authentication](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/kafka/#oauth-jwt-assertion-authentication-private_key_jwt) · [identity provider requirements](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/kafka/#identity-provider-requirements) · [TLS trust](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/kafka/#tls-trust-configuration) Detail: - The flow in one sentence, same shape as the persistor's: QE signs a JWT with the keystore's private key, trades it at the identity provider for an access token, and presents that token to the broker over SASL_SSL/OAUTHBEARER. No shared secret exists anywhere on the path, which is precisely the point for security models that disallow shared-secret clients. - A standing query is the streaming half of the product: instead of running a query when someone asks, the graph continuously matches the pattern as data arrives and fires the outputs on each new match. Naming the concept is enough here; writing patterns is a craft of its own. [structure](https://docs.thatdot.com/quine-enterprise/learn/standing-queries/standing-queries/#standing-query-structure) · [the destination menu](https://docs.thatdot.com/quine-enterprise/learn/standing-queries/standing-queries/#output-destinations). - Unlike the per-member ingests, ONE POST registers the standing query on every member (the response carries per-member stats). And the redaction rule applies here too: a GET shows the keystore password field redacted, exactly like the ingest; the registration persists in the graph's metadata and survives restarts. - Any OIDC identity provider plays the IdP role (Keycloak is this deployment's incarnation), and the broker side is any Kafka whose listener validates the IdP's tokens: here a Strimzi-operated broker validating against the realm's JWKS. The docs' [identity provider requirements](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/kafka/#identity-provider-requirements) spell the registration contract. - A convenience the client-secret flow does not get (verified in the docs and in this deployment): when the type is `OAuthBearerAssertionLogin`, QE injects its own login callback handler automatically; only `sasl.mechanism` still needs setting by hand. - If your broker team does allow client secrets, the simpler [OAuthBearerLogin](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/kafka/#oauthbearerlogin) type does the same dance with an id and secret; the surrounding configuration is identical. - The enterprise boundary from the opening card, now concrete: open-source Quine parses this configuration shape and speaks standard SASL, but the assertion login handler ships only in Quine Enterprise. - Why the output goes to a topic: a topic survives pods, fans in from every member, and is browsable live: the green balls in the scene are the emitter at work. It runs one partition, a deliberate contrast with events' three: matches fan IN, and at this volume ordering is worth more than parallel writes. Downstream is ordinary Kafka: whatever consumes `matches` (an alerting pipeline, a downstream service, a notebook) needs no knowledge of Quine at all; one contract to know before building on it: [standing query outputs are best effort](https://docs.thatdot.com/quine-enterprise/core-concepts/delivery-guarantees/#standing-query-outputs-best-effort); buffer and de-duplicate downstream. - This page section is deployment-proven: the configuration on these cards is the one running behind the scene's amber and green balls. [OAuth JWT assertion authentication](https://docs.thatdot.com/quine-enterprise/learn/ingest-sources/kafka/#oauth-jwt-assertion-authentication-private_key_jwt). ### The prerequisite that rode ahead Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#streaming/4 one mount in the metrics overlay's patch was never about metrics: the broker's CA certificate, waiting for this section's ingest to need it Configuration in focus: `overlays/metrics/deployment-patch.yaml` Detail: - The scoping is the lesson: this CA authenticates one TLS endpoint (the broker's OAuth listener), and only the Kafka consumer and producer ever contact it, so the trust rides as Kafka-client configuration (`ssl.truststore.*` in the payload) instead of joining the JVM's global trust store. Compare the Cassandra CA, which DOES merge into JVM trust, because the persistor's TLS is validated against the JVM default: two CAs, two correct scopes. - The certificate is operator-managed and renews, so the platform provisioning refreshes the in-namespace copy each time it runs. The portable need: distribute your broker's CA to its clients, however your platform does that. - The gotcha the quoted comment warns about, live-proven: the ingest persists in the graph's metadata, so stepping the deployment back to an earlier overlay without a reset restores the ingest on boot, and it comes back FAILED: this mount does not exist there. This tour only ever moves forward, and a keyspace reset clears the assets too. - Why no seventh overlay: the overlays are strict supersets of configuration, and the finale changes no configuration. Its one Kubernetes prerequisite ships with the last overlay that exists, and this card is the tease from the metrics kit card, paid off. ### Partitions × members Deep link: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/#streaming/5 partitions are Kafka's unit of parallelism: drag the count and watch the consumer group divide them, idle members included; the three pinned 1-partition streams stay put Interactive: partitions x members. A slider changes the partition count on the featured topic and shows the consumer group dividing partitions across members, idle members included. The three pinned single-partition streams stay put alongside it. Detail: - The rule the slider demonstrates is standard Kafka consumer-group behavior: a partition is consumed by exactly one member of a group, so partitions cap the group's parallelism. Fewer partitions than members leaves members idle on that stream; more partitions than members is fine but uneven counts skew the load. - The steady beams are the second placement pattern, for topics that cannot split: entities, entity-updates, and sessions carry one partition each, and one partition means one consumer, so no consumer group can spread them. The deployment balances them the other way instead, pinning one stream per member through the API's member targeting: entities on member 0, entity-updates on member 1, sessions on member 2. And on either pattern the hot spare consumes nothing: ingest runs on positioned members only, the clustering section's spare doing exactly its job. - The docs add a second, subtler reason to align the counts ([aligning with Kafka partitions](https://docs.thatdot.com/quine-enterprise/learn/clustering/cluster-performance/#aligning-with-kafka-partitions)): with the member count an even divisor or multiple of the partition count, upstream partitioning can keep each partition's data on one member for data locality (the docs name `locIdFrom` for that pattern). - Partition count is decided on the Kafka side and changing it is a real operation: plan it together with the member count at design time, alongside the sizing method on [the clustering tuning card](#clustering/6). - The other lever is `parallelism`, per member and per stream (default 16): how many consumed records cause their effects concurrently. This deployment leaves it alone: the streams sit far below capacity by design (`maxPerSecond` caps each instance), which keeps the three levers separate and visible: partitions decide who consumes, parallelism decides in-member concurrency, and the rate limit decides the feed's tempo. The docs' guidance: [parallelism configuration](https://docs.thatdot.com/quine-enterprise/learn/troubleshooting/ingest/#1-parallelism-configuration), and before reaching for it, [identify which bottleneck you actually have](https://docs.thatdot.com/quine-enterprise/learn/troubleshooting/diagnosing-bottlenecks/#identifying-the-bottleneck). - How to read the real cluster's version of this picture: the consumer-group description shows the partition assignments (one per member here), and Grafana's per-member ingest rates are equal by construction when the counts align (this deployment's paced run draws three equal ~75 records/s lines). ### Configuration this section adds No new overlay: these are API payloads POSTed to the running deployment, not manifests. #### `runtime assets/ingest-events.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/runtime-assets/ingest-events.yaml ```yaml # THE per-member ingest: the same POST goes to every member # (Quine-Member-Idx 0/1/2), and the shared consumer group (groupId below) # splits the topic's 3 partitions one per member - partitions are the unit # of parallelism. # # Auth: the broker's 9093 listener is SASL_SSL/OAUTHBEARER. QE authenticates # to Keycloak with a private_key_jwt client assertion, signed with the SAME # keystore as the persistor identity (/opt/certs/keystore.p12), and presents # the resulting bearer token to the broker. No client secret on this path. # __CERT_FILE_PASSWORD__ is substituted at POST time from a k8s Secret; no # secret value sits in this file, and the API redacts the field in every # response. File paths are pod-side paths on the QE member hosting the stream. name: events # Records per second for THIS stream instance. Three members each run one # instance, so 75 per member is an intended 225 records/second stream total: # the feed's tempo is configuration, not accident. maxPerSecond: 75 source: type: Kafka topics: - events bootstrapServers: kafka-kafka-bootstrap.kafka.svc:9093 groupId: events securityProtocol: SASL_SSL autoOffsetReset: EARLIEST offsetCommitting: type: ExplicitCommit maxInterval: 10s saslJaasConfig: type: OAuthBearerAssertionLogin clientId: quine-kafka-client certFile: /opt/certs/keystore.p12 certFilePassword: __CERT_FILE_PASSWORD__ certFileType: PKCS12 resourceUri: kafka://example discoveryUrl: https://keycloak.example.com/realms/kafka/.well-known/openid-configuration kafkaProperties: # NOT derived from saslJaasConfig; must be explicit: sasl.mechanism: OAUTHBEARER # Trust for the broker's TLS listener (Strimzi cluster CA), Kafka-client # scoped; JVM-global trust is untouched: ssl.truststore.type: PEM ssl.truststore.location: /opt/kafka-ca/ca.crt format: type: Json # A deliberately minimal landing query: each record becomes one Event node # carrying all its fields. Real ingest queries model a domain; this tour # teaches the configuration around them, not the Cypher. query: | MATCH (e) WHERE id(e) = idFrom('event', $that.id) SET e = $that, e: Event ``` Post it with: ```sh # $TOKEN is an access token from the realm the auth overlay names, for a # client whose roles claim carries the capability this call needs # (IngestWrite) and whose audience matches access-token-audience. Before the # auth overlay is applied there is no auth: drop the Authorization header. TOKEN=$(curl -s -X POST \ https://keycloak.example.com/realms/quine-enterprise/protocol/openid-connect/token \ -d grant_type=client_credentials \ -d client_id= -d client_secret=$CLIENT_SECRET \ | jq -r .access_token) # the keystore password is the one value the file does not carry: read it # from the same Secret the pods mount, and substitute it as the body is sent PW=$(kubectl get secret quine-cassandra-cert -n quine \ -o jsonpath='{.data.password}' | base64 -d) # the v2 API accepts this yaml body as-is sed "s|__CERT_FILE_PASSWORD__|$PW|" ingest-events.yaml \ | curl -X POST http://localhost:8080/api/v2/graph/quine/ingests \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/yaml" \ --data-binary @- # per-member streams: repeat with -H "Quine-Member-Idx: 0" (then 1, 2) ``` #### `runtime assets/standing-query.yaml` Raw file: https://docs.thatdot.com/quine-enterprise/learn/orchestration/kubernetes-deployment-tour/config/runtime-assets/standing-query.yaml ```yaml # The standing query: unlike the per-member ingests, ONE POST registers the # rule cluster-wide (the response carries per-member stats). It matches # prospectively, as data arrives, so it is registered BEFORE the ingests. # # The match goes to the matches Kafka topic, and the producer authenticates # exactly like the ingests' consumers: same OAuth broker listener (:9093, # SASL_SSL/OAUTHBEARER), same private_key_jwt client assertion, same # keystore identity (the persistor's). One machine identity authenticates # the data in AND the data out. One asymmetry to know: the destination has # no typed securityProtocol field (unlike the ingest source), so the # protocol rides in kafkaProperties. # __CERT_FILE_PASSWORD__ is substituted at POST time from a k8s Secret; the # API redacts the field in every response. File paths are pod-side paths # on the QE members, not the posting container. name: emit-matches # A deliberately minimal pattern: watch for Event nodes whose flagged field # is true. Real patterns are a craft of their own; the configuration around # them is what this file teaches. pattern: type: Cypher mode: MULTIPLE_VALUES query: >- MATCH (e:Event) WHERE e.flagged = true RETURN id(e) AS eventId outputs: - name: publish destinations: - type: Kafka topic: matches bootstrapServers: kafka-kafka-bootstrap.kafka.svc:9093 saslJaasConfig: type: OAuthBearerAssertionLogin clientId: quine-kafka-client certFile: /opt/certs/keystore.p12 certFilePassword: __CERT_FILE_PASSWORD__ certFileType: PKCS12 resourceUri: kafka://example discoveryUrl: https://keycloak.example.com/realms/kafka/.well-known/openid-configuration kafkaProperties: # No typed securityProtocol on the destination side - the protocol # AND mechanism ride as raw properties: security.protocol: SASL_SSL sasl.mechanism: OAUTHBEARER # Trust for the broker's TLS listener (Strimzi cluster CA), Kafka- # client scoped; JVM-global trust is untouched: ssl.truststore.type: PEM ssl.truststore.location: /opt/kafka-ca/ca.crt ``` Post it with: ```sh # $TOKEN as in the ingest tip (this call needs StandingQueryWrite), and the # same substitution: the password comes from the Secret, never from the file. # Register this BEFORE the ingests: standing queries match prospectively, as # data arrives. PW=$(kubectl get secret quine-cassandra-cert -n quine \ -o jsonpath='{.data.password}' | base64 -d) # the v2 API accepts this yaml body as-is sed "s|__CERT_FILE_PASSWORD__|$PW|" standing-query.yaml \ | curl -X POST http://localhost:8080/api/v2/graph/quine/standingQueries \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/yaml" \ --data-binary @- ```