Monitor Novelty with Prometheus and Grafana¶
Novelty can be monitored with Prometheus and Grafana. The container image that Novelty ships with can be configured to enable the Prometheus JMX exporter within to emit metrics to be scraped by a Prometheus instance. Grafana can then use Prometheus as a datasource and visualize the provided metrics. This guide will show you how to set this up in a Kubernetes cluster using Helm.
Prerequisites¶
kubectl
configured for the target cluster- Helm installed
- Access to the thatDot Helm repository
- Enrolled in the Novelty trial for a trial API Key
Installing Novelty alongside Prometheus and Grafana, using Helm¶
We will be using Helm charts for Novelty, Prometheus, and Grafana to stand up a Novelty instance, with Prometheus scraping metrics emitted from Novelty, and Grafana using Prometheus as a datasource to visualize metrics using the Novelty Grafana Dashboard.
Provided below will be 4 assets to help you install these resources:
- Novelty Helm values
- Prometheus Helm values
- Grafana Helm values
- Bash script to install all 3 services
Novelty Helm values (novelty-values.yaml
)¶
These Helm values will launch an instance of the Novelty trial, with the Prometheus JMX Exporter enabled, and Prometheus pod annotations applied to the Novelty pod for service discovery within Kubernetes.
trial:
email: <EMAIL>
apiKey: <API_KEY>
metrics:
prometheus:
enabled: true
Prometheus Helm values (prometheus-values.yaml
)¶
The Prometheus Helm values has defaults which will automatically perform Kubernetes service discovery. The Novelty Helm chart above, by setting metrics.prometheus.enabled
to true
, will add the annotations the Prometheus deployment will expect.
server:
global:
scrape_interval: 15s
scrape_timeout: 10s
configmapReload:
prometheus:
enabled: false
alertmanager:
enabled: false
kube-state-metrics:
enabled: false
prometheus-node-exporter:
enabled: false
prometheus-pushgateway:
enabled: false
Grafana Helm values (grafana-values.yaml
)¶
The Grafana Helm values below configures the above Prometheus deployment as a datasource, so it is made available right away.
adminUser: admin
adminPassword: admin
datasources:
datasources.yaml:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus-server
isDefault: true
Bash script to install everything (install-cluster.sh
)¶
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add thatdot https://helm.thatdot.com
helm repo update
helm \
upgrade --install \
prometheus prometheus-community/prometheus \
-f prometheus-values.yaml
helm \
upgrade --install \
grafana grafana/grafana \
-f grafana-values.yaml
helm upgrade --install \
novelty thatdot/novelty \
-f novelty-values.yaml
Info
Requires thatdot/novelty Helm chart version 0.4.4 or later.
After running this script, port-forward and navigate to the Grafana pod, login using admin
/admin
, and then import the Novelty Dashboard.
Novelty Grafana Dashboard¶
The Novelty Grafana Dashboard has the following controls:
- Ingest Rate per stream
- Bucketed Node Edge Counts
- Persistor Latency
- Heap Memory Usage