Skip to content

Novelty Helm Chart Quickstart

Overview

Novelty shines as a stream processor. It excels in scenarios where you can stream observations into Novelty, and can receive flowing observations results from Novelty.

In this quickstart, you will:

  1. Spin up Novelty and a minimal Kafka cluster in Kubernetes via Helm.
  2. Create a test transformation and bidirectional Kafka stream with Novelty’s Interactive API.
  3. Produce sample observations and watch scored results flow into a Kafka topic.

By the end you will have a working Kafka → Novelty → Kafka pipeline you can adapt to your own data.

Prerequisites

  • kubectl configured for the target cluster
  • Helm must be installed to use the charts. Please refer to Helm's documentation to get started.
  • A Kubernetes cluster will be needed to deploy Kafka and Novelty in this quickstart.
  • Signed up for a Novelty trial for an API Key.

Installing Novelty and Kafka using Helm

We will be using Helm charts to install Novelty and Kafka.

Novelty Helm Values (novelty-values.yaml)

Use this as your Novelty helm values. Use your email and API Key.

trial:
  email: <EMAIL>
  apiKey: <API_KEY>

Kafka Helm Values (kafka-values.yaml)

We will create as simple of a Kafka deployment as possible.

listeners:
  client:
    protocol: PLAINTEXT
  controller:
    protocol: PLAINTEXT
  interbroker:
    protocol: PLAINTEXT
  external:
    protocol: PLAINTEXT

Bash script to install helm charts

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add thatdot https://helm.thatdot.com
helm repo update

helm upgrade --install \
  kafka bitnami/kafka \
  -f kafka-values.yaml

helm upgrade --install \
  novelty thatdot/novelty \
  -f novelty-values.yaml

Running this script will install the Kafka and Novelty Helm charts.

Ensuring Deployments are Ready

Let's use kubectl to ensure that both Novelty and Kafka are ready to serve our requests.

Use kubectl to list all the pods deployed into our current namespace. We should see Novelty and Kafka

kubectl get pods

NAME                       READY   STATUS    RESTARTS   AGE
kafka-controller-0         1/1     Running   0          31s
kafka-controller-1         1/1     Running   0          31s
kafka-controller-2         1/1     Running   0          31s
novelty-5fd657fccc-4kzs4   1/1     Running   0          30s

If they aren't all in a ready state, wait a minute or two and try again.

Port-forward to Novelty and Establish Kafka stream with Interactive API

We will now port-forward to Novelty, and then use its Interactive API to create a simple transformation, along with creating a new stream that will consume data from Kafka, transform it with our transformation, score the observation results, and then stream them back to Kafka.

Kafka -> Novelty -> Kafka

To open up a port-forward to Novelty, use the following command in a spare terminal. Keep this terminal up as we explore with Novelty's Interactive API:

kubectl port-forward deployments/novelty 8080

After that, navigate to Novelty at localhost:8080

Novelty UI

After loading up Novelty in your browser, click on the Interactive API button on the left navigation. We will be using two of the API's to create our Kafka stream:

Interactive API

Create a named transformation

Click on the create a named transformation function link, give your transformation a name (for example, test-transformation), set the body type to application/yaml, and then paste in this sample transformation:

type: JavaScript
function: that => [ that[0], that[1], that[2], that[3] ]

After that, click the blue "Send API Request" button to create that transformation.

Create Transformation

Create a new Novelty stream

Once you've created a transformation, we can now create a new Novelty stream. Click on the create a new novelty stream link, give your stream a name (for example, test-stream), set the body type to application/yaml, and then paste in this stream configuration.

noveltyContext: test-context
transformation: test-transformation
failureMode: LogAndKeepConsuming
inputStream:
  type: KafkaInput
  topics:
    - observations
  bootstrapServers: kafka:9092
  groupId: novelty-group
outputStream:
  type: KafkaOutput
  topic: observation-results
  bootstrapServers: kafka:9092

After that, click the blue "Send API Request" button to create the new Novelty Kafka stream.

Stream Create

Produce Observations to Kafka, Consume Observations Results from Kafka

Now that we've established a Kafka stream in Novelty, we can stop our port-forward since we will be interactive with Kafka now. We will SSH twice into Kafka, one to create a producer and produce some sample observations, and a consumer to see those observations scored in real time.

Create 2 terminals, and use this command to SSH them into one of those Kafka instances:

kubectl exec -it kafka-controller-0 -- bash

In one of those terminals, consume from the observation-results with this command:

kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic observation-results

In the other terminal, prepare to produce some sample observations with this command:

kafka-console-producer.sh --bootstrap-server kafka:9092 --topic observations

After that, the shell will wait for you to enter newline delimited messages to Kafka. Our transformation is expecting an array of 4 strings, so we will send some extremely basic observations, to assert a working Novelty stream.

Send the following messages to Kafka, and you should observe scored observation results flowing out of the observation-results topic we are consuming from

  • ["a", "b", "c", "d"]
  • ["a", "b", "c", "e"]
  • ["a", "b", "c", "f"]
  • ["a", "q", "c", "d"]

Flowing observation results

Summary

This quickstart showed how to provision Novelty and a simple Kafka deployment, to test out streaming data in and out of Novelty. Once you've established this steel-thread, you can expand upon it, and feed data for your use-case into Kafka, and mold your transformation so that it creates observations from your data that help answer your questions as you look for anomolous behavior in your data.