Skip to content

Metrics

With services.metrics: prometheus-grafana, quinectl runs Prometheus and Grafana on your machine. Prometheus collects metrics from every member, hot spares included. Grafana opens with thatDot's dashboard for each product you run. Every project on the machine shares one Prometheus and one Grafana, and each project gets dashboards of its own; see Sharing with other projects.

quinectl.yaml
quine:
  edition: enterprise
  licenseKey: ${QE_LICENSE_KEY}
  customerId: ${QE_CUSTOMER_ID}
services:
  persistor: cassandra
  auth: keycloak
  proxy: none
  metrics: prometheus-grafana
  kafka: none
quinectl.yaml
quine:
  edition: enterprise
  licenseKey: ${QE_LICENSE_KEY}
  customerId: ${QE_CUSTOMER_ID}
services:
  persistor: cassandra
  auth: keycloak
  proxy: none
  metrics:
    prometheusGrafana:
      prometheusPort: 9090
      grafanaPort: 3000
      scrapeInterval: 15s
      scrapeTimeout: 12s
      grafanaAnonymous: true
      grafanaAnonymousRole: Editor
      prometheusImage: prom/prometheus:v3.5.0
      grafanaImage: grafana/grafana:11.5.2
  kafka: none

The long form shows the defaults. Use it to change ports, how often metrics are collected, or Grafana sign-in; each field is in the quinectl.yaml Reference.

After up, the metrics line looks like this:

✓ metrics converged: grafana http://localhost:3000/d/quinectl-quine-cb2508d240e1 (no login needed) — prometheus http://localhost:9090, scraping 1/1 members
What Where
Grafana http://localhost:3000
Prometheus http://localhost:9090

The Grafana link goes straight to this project's dashboard; the id at its end differs for each project directory. Each project has its own dashboard for each product it runs, a Quine Prometheus Dashboard and a Novelty Prometheus Dashboard, titled with the directory name, such as Quine Prometheus Dashboard — my-project (cb2508d240e1). up checks that every member is being collected and the dashboards are in place, and fails if they aren't.

The first up downloads the dashboards and the Prometheus JMX exporter agent that publishes each member's metrics, and keeps them for later runs. On a machine without internet access, point dashboard, noveltyDashboard and agentJar at local files instead.

Grafana sign-in

By default, Grafana opens without a sign-in. Choose who can view dashboards and who can change them. These settings apply to every project, because Grafana is shared; see Sharing with other projects.

Anyone who opens Grafana can view dashboards and create their own. thatDot's dashboards can't be overwritten: to change one, save a copy under a new name. This is what metrics: prometheus-grafana alone gives you, written out:

quinectl.yaml
quine:
  edition: enterprise
  licenseKey: ${QE_LICENSE_KEY}
  customerId: ${QE_CUSTOMER_ID}
services:
  persistor: cassandra
  auth: keycloak
  proxy: none
  metrics:
    prometheusGrafana:
      grafanaAnonymous: true
      grafanaAnonymousRole: Editor
  kafka: none

Anyone can view dashboards, but saving a new one needs a sign-in as admin / admin. Set the anonymous role to Viewer:

quinectl.yaml
quine:
  edition: enterprise
  licenseKey: ${QE_LICENSE_KEY}
  customerId: ${QE_CUSTOMER_ID}
services:
  persistor: cassandra
  auth: keycloak
  proxy: none
  metrics:
    prometheusGrafana:
      grafanaAnonymousRole: Viewer
  kafka: none

Grafana asks everyone to sign in. Sign in as admin / admin. Turn anonymous access off:

quinectl.yaml
quine:
  edition: enterprise
  licenseKey: ${QE_LICENSE_KEY}
  customerId: ${QE_CUSTOMER_ID}
services:
  persistor: cassandra
  auth: keycloak
  proxy: none
  metrics:
    prometheusGrafana:
      grafanaAnonymous: false
  kafka: none

up shows which applies: (admin/admin) after the Grafana address, instead of (no login needed).

grafanaAnonymousRole also accepts Admin, which lets anonymous visitors change Grafana's own settings too.

Ports and timing

Setting Default What it is
prometheusPort 9090 Prometheus on your machine
grafanaPort 3000 Grafana on your machine
memberBasePort Picked for you, from 11091 Where member N publishes metrics: memberBasePort + N
scrapeInterval 15s How often Prometheus collects
scrapeTimeout 12s How long one collection may take; must be less than the interval

The default timeout is close to the interval because collecting a product's metrics can take several seconds. If members keep dropping off the dashboard, raise both values.

Sharing with other projects

Every project with metrics on uses the same three containers: quinectl-prometheus, quinectl-grafana, and quinectl-prom-label-proxy, which limits each project's dashboards to that project's data. A second project needs no metrics settings: quinectl picks free member ports for it and adds its dashboards to the same Grafana.

Applies to Settings
Every project prometheusPort, grafanaPort, prometheusImage, grafanaImage, grafanaAnonymous, grafanaAnonymousRole, grafanaPublicUrl
This project memberBasePort, scrapeInterval, scrapeTimeout, dashboard, noveltyDashboard, agentJar

Keep the settings that apply to every project the same in each file:

  • A prometheusPort or grafanaPort that differs from the running container's is refused before anything starts:

    error: grafana port 3001 differs from the shared quinectl-grafana's port 3000 (used by /Users/me/project-a) — set services.metrics.prometheusGrafana.grafanaPort: 3000 in this file
    
  • A different image, sign-in setting or grafanaPublicUrl replaces the shared container for every project, and up warns, naming the others. Replacing Grafana removes dashboards you saved yourself, and replacing Prometheus discards the metrics collected so far, for every project; thatDot's dashboards come back on their own. The last up wins, so two files that disagree switch Grafana back and forth:

    warning: replacing shared quinectl-grafana (config drift) — other directories using it: /Users/me/project-a
    

    A different image shows (image drift) in place of (config drift).

A stopped project's dashboards stay in Grafana after quinectl down, ready for its next up. quinectl down --rm and quinectl down --purge remove them, and quinectl gc removes those of a deleted project. See Multi-tenancy.

Next steps