Skip to content

Monitor Novelty with Prometheus and Grafana

Novelty can be monitored with Prometheus and Grafana. The container image that Novelty ships with can be configured to enable the Prometheus JMX exporter within to emit metrics to be scraped by a Prometheus instance. Grafana can then use Prometheus as a datasource and visualize the provided metrics. This guide will show you how to set this up in a Kubernetes cluster using Helm.

Prerequisites

Installing Novelty alongside Prometheus and Grafana, using Helm

We will be using Helm charts for Novelty, Prometheus, and Grafana to stand up a Novelty instance, with Prometheus scraping metrics emitted from Novelty, and Grafana using Prometheus as a datasource to visualize metrics using the Novelty Grafana Dashboard.

Provided below will be 4 assets to help you install these resources:

  • Novelty Helm values
  • Prometheus Helm values
  • Grafana Helm values
  • Bash script to install all 3 services

Novelty Helm values (novelty-values.yaml)

These Helm values will launch an instance of the Novelty trial, with the Prometheus JMX Exporter enabled, and Prometheus pod annotations applied to the Novelty pod for service discovery within Kubernetes.

trial:
  email: <EMAIL>
  apiKey: <API_KEY>

metrics:
  prometheus:
    enabled: true

Prometheus Helm values (prometheus-values.yaml)

The Prometheus Helm values has defaults which will automatically perform Kubernetes service discovery. The Novelty Helm chart above, by setting metrics.prometheus.enabled to true, will add the annotations the Prometheus deployment will expect.

server:
  global:
    scrape_interval: 15s
    scrape_timeout: 10s
configmapReload:
  prometheus:
    enabled: false
alertmanager:
  enabled: false
kube-state-metrics:
  enabled: false
prometheus-node-exporter:
  enabled: false
prometheus-pushgateway:
  enabled: false

Grafana Helm values (grafana-values.yaml)

The Grafana Helm values below configures the above Prometheus deployment as a datasource, so it is made available right away.

adminUser: admin
adminPassword: admin
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        access: proxy
        url: http://prometheus-server
        isDefault: true

Bash script to install everything (install-cluster.sh)

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add thatdot https://helm.thatdot.com
helm repo update

helm \
  upgrade --install \
  prometheus prometheus-community/prometheus \
  -f prometheus-values.yaml

helm \
  upgrade --install \
  grafana grafana/grafana \
  -f grafana-values.yaml

helm upgrade --install \
  novelty thatdot/novelty \
  -f novelty-values.yaml

Info

Requires thatdot/novelty Helm chart version 0.4.4 or later.

After running this script, port-forward and navigate to the Grafana pod, login using admin/admin, and then import the Novelty Dashboard.

Novelty Grafana Dashboard

Novelty Grafana Dashboard

The Novelty Grafana Dashboard has the following controls:

  • Ingest Rate per stream
  • Bucketed Node Edge Counts
  • Persistor Latency
  • Heap Memory Usage