Quine Enterprise Helm Chart Quickstart

Full Helm chart available at: https://github.com/thatdot/helm-charts/tree/main/charts/quine-enterprise

Prerequisites

Kubernetes

A Kubernetes cluster is required to install Quine Enterprise with Helm.

Helm

Helm must be installed to use the charts. Please refer to Helm’s documentation to get started.

Once Helm has been set up correctly, add the repo as follows:
helm repo add thatdot https://helm.thatdot.com

If you have previously added this repo, run helm repo update to retrieve the latest versions of the packages. You can then run helm search repo thatdot to see the charts.

Cassandra Compatible Database

Apache Cassandra or a Cassandra-compatible database such as ScyllaDB is required to use Quine Enterprise.

The database used with Quine Enterprise may be installed in the same Kubernetes cluster as quine, but this is not required. However, it is recommended that it be physically located as close to the Quine Enterprise installation as possible to reduce network latency, such as ensuring they are both located in the same AWS region or availability zone.

There are a variety of ways to install Apache Cassandra or ScyllaDB into Kubernetes. Both databases have published Helm charts to make installation and administration easier.

See helm-charts/extras/persistor for an example to quickly bootstrap a thatDot environment with an Apache Cassandra cluster.

Apache Cassandra Helm chart: https://artifacthub.io/packages/helm/bitnami/cassandra
ScyllaDB Helm chart: https://operator.docs.scylladb.com/stable/installation/helm.html

License

Quine Enterprise is proprietary software and a license is required to use it.

A free trial evaluation license is available from thatdot.com. When using this helm chart with a trial license, Values.trial.enabled must be set to true and the Values.trial.email and Values.trial.apiKey fields must be set. The API key will be provided via email after registering for a trial license.

To schedule a demo or purchase a full license, please reach out at https://www.thatdot.com/request-a-demo/, email sales@thatdot.com, or contact thatDot directly.

Installation

Helm Values

First create a values file called quine-enterprise-values.yaml with required helm configuration values (see values.yaml for a reference of available configuration and defaults):

# 3 Quine Enterprise hosts is a good minimum size for a cluster. Feel free to use more, up to the amount of available CPU and memory in your cluster.
hostCount: 3

# For simple installations, ClusterIP is a good choice. LoadBalancer can also be used to create an externally available endpoint for your cluster.
service:
  type: ClusterIP

# Settings for using Cassandra cluster as the persistor. REQUIRED.
cassandra:
  enabled: true
  endpoints: "<CASSANDRA_ADDRESS_1>,<CASSANDRA_ADDRESS_2>"
  plaintextAuth:
    enabled: true
    username: <CASSANDRA_USERNAME>
    password: <CASSANDRA_PASSWORD>
  shouldCreateKeyspace: true
  shouldCreateTables: true

# Evaluation resource settings, should increase to 3-32cpu and at least 15Gi memory for production
resources:
  requests:
    cpu: 2
    memory: 7Gi
  limits:
    cpu: 4
    memory: 7Gi

If a trial license is being used, be sure to include a section with the trial license credentials.

trial:
  enabled: true
  email: "<EMAIL_USED_TO_REGISTER>"
  apiKey: "<API_KEY_RECEIVED_IN_EMAIL>"

If you are not using a trial license, be sure to set the trial to disabled. And you will be required to specify an alternative docker image and tag pointing to a hosted version of the software you have been provided (see below).

trial:
  enabled: false

(Optional) specify a docker image and tag. This is required if you are not using the trial license.

# Override image repository and tag to change the docker image used to run thatDot Quine Enterprise
image:
  repository: public.ecr.aws/thatdot/streaming-graph-trial
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: ""

Install the cluster with Helm using the command (Don’t forget to first helm repo add the thatdot Helm repo in the prerequisites section):

helm install --values=quine-enterprise-values.yaml \
    my-quine-enterprise thatdot/quine-enterprise

Uninstall the cluster:
helm uninstall my-quine-enterprise

Cluster Scaling

In order to change the number of active cluster members of a Quine Enterprise, it must be scaled down to zero hosts and then back up. This is because Quine uses a static sharding scheme to determine which nodes each member will manage. However, all persisted data and standing queries can remain between different sized clusters, so it is possible to effectively pause a workload, scale up or down, and then resume the workload.

Recommended

Delete all ingests before starting to scale down. Use the delete ingest API endpoint.

Note

Quine ingests are defined per-member, so it is important to consider the topology of which member is ingesting from which stream. Deleting and recreating ingests is preferable when each host should be ingesting from a partitioned Kafka cluster or similar source.

Use Helm to scale the cluster down to 0 active hosts:

helm upgrade --wait --reuse-values \
    --set hostCount=0 \
    my-quine-enterprise

Once the operation has completed, update the quine-enterprise-values.yaml file with the desired cluster size and use Helm to deploy the changes:

helm upgrade --wait --values=quine-enterprise-values.yaml \
    my-quine-enterprise

Recommended

Re-create all required ingests. Use the create ingest API endpoint.

Upgrade/Downgrade Version

If upgrading from or downgrading to a version of Quine Enterprise prior to 1.8.3, the entire cluster must be scaled up and down in a similar procedure to the Cluster Scaling section:

helm upgrade --wait --reuse-values \
    --set hostCount=0 \
    my-quine-enterprise

If you are upgrading or downgrading between versions 1.8.3 or newer, you can simply run helm upgrade as below without scaling the cluster down to 0 members. Note that all work will be saved, but ingests, standing query outputs, and ad-hoc queries will be paused during the transition period.

Update the quine-enterprise-values.yaml file with the desired Quine Enterprise docker image (example: downgrading to trial version 1.8.3):

# Override image repository and tag to change the docker image used to run thatDot Quine Enterprise
image:
  repository: public.ecr.aws/thatdot/streaming-graph-trial
  # Overrides the image tag whose default is the chart appVersion.
  tag: "1.8.3"

If you do not want to specify the version explicitly, and a new version is available, you can simply run helm repo update to get the latest versions of all helm charts.

Then deploy the update with Helm:

helm upgrade --wait --values=quine-enterprise-values.yaml \
    my-quine-enterprise

You may also specify a chart version explicitly with Helm:

helm upgrade --wait --values=quine-enterprise-values.yaml \
	--version 0.2.0
    my-quine-enterprise

It is important to always use the latest helm chart version available for the particular appVersion of Quine Enterprise you want to deploy.

Metrics and Monitoring

Quine Enterprise exposes a large number of JVM and application metrics via the Dropwizard Metrics Library.

The recommended monitoring stack consists of influxDBv1 to record metrics and grafana to graph and visualize them.

See helm-charts/extras/monitoring-stack for an example to quickly bootstrap a thatDot environment with an InfluxDb and a Grafana instance.

# Metrics Configuration:
metrics:
  influx:
    enabled: true
    database: metrics
    scheme: http
    # Set "host" and "port" to the reachable address of the influx server
    host: "<INFLUXDB_HOST_ADDRESS>"
    port: 8086

1.8.1