Apache Kafka

Kafka Configuration

Novelty supports Apache Kafka for both input streams (consuming observations) and output streams (publishing scored results). Kafka configuration is available via the REST API.

In addition to the API-mapped Kafka options, arbitrary Kafka configuration can be provided via the kafkaProperties field.

In order to avoid confusion and mitigate certain vulnerabilities in the Kafka client libraries, certain configuration keys are disabled by a validation step. The bootstrap.servers property is disallowed because it duplicates the bootstrapServers field. Similarly, certain values for the sasl.jaas.config property are known to introduce vulnerabilities, so those property values are forbidden.

Because of the extreme variety of possible configuration combinations, we cannot provide a comprehensive guide on configuring Kafka. However, we recommend using securityProtocol: "SSL" wherever possible to encrypt requests between Novelty and the Kafka broker.

Secure Kafka Configuration

Novelty supports typed "secret" parameters for Kafka security configuration. These parameters are redacted in API responses and logs (replaced with Secret(****), usually, or **** if a small part of a larger value).

SSL/TLS Passwords

Parameter	Type	Description
sslKeystorePassword	string (secret)	Password for the SSL keystore file
sslTruststorePassword	string (secret)	Password for the SSL truststore file
sslKeyPassword	string (secret)	Password for the private key in the keystore

SASL Authentication

The saslJaasConfig parameter accepts one of three authentication types:

PlainLogin

For SASL/PLAIN authentication:

{
  "saslJaasConfig": {
    "type": "PlainLogin",
    "username": "my-username",
    "password": "my-password"
  }
}

ScramLogin

For SASL/SCRAM-SHA-256 or SCRAM-SHA-512 authentication:

{
  "saslJaasConfig": {
    "type": "ScramLogin",
    "username": "my-username",
    "password": "my-password"
  }
}

OAuthBearerLogin

For SASL/OAUTHBEARER authentication:

{
  "saslJaasConfig": {
    "type": "OAuthBearerLogin",
    "clientId": "my-client-id",
    "clientSecret": "my-client-secret",
    "scope": "optional-scope",
    "tokenEndpointUrl": "https://auth.example.com/oauth/token"
  }
}

Field	Required	Description
clientId	Yes	OAuth client identifier
clientSecret	Yes	OAuth client secret (redacted in responses)
scope	No	OAuth scope for the token request
tokenEndpointUrl	No	Token endpoint URL

Migrating from kafkaProperties

The typed Secret parameters take precedence over corresponding entries in kafkaProperties. When both are configured, a warning is logged. For example:

WARN - Kafka property 'ssl.keystore.password' in kafkaProperties will be overridden
by typed Secret parameter. Remove 'ssl.keystore.password' from kafkaProperties to
suppress this warning.

Affected properties:

kafkaProperties key	Typed parameter
ssl.keystore.password	sslKeystorePassword
ssl.truststore.password	sslTruststorePassword
ssl.key.password	sslKeyPassword
sasl.jaas.config	saslJaasConfig

Before (unprotected):

{
  "type": "Kafka",
  "topic": "events",
  "bootstrapServers": "kafka:9093",
  "kafkaProperties": {
    "security.protocol": "SASL_SSL",
    "ssl.keystore.password": "keystore-secret",
    "sasl.jaas.config": "org.apache.kafka...PlainLoginModule required username=\"user\" password=\"pass\";"
  }
}

After (protected):

{
  "type": "Kafka",
  "topic": "events",
  "bootstrapServers": "kafka:9093",
  "sslKeystorePassword": "keystore-secret",
  "saslJaasConfig": {
    "type": "PlainLogin",
    "username": "user",
    "password": "pass"
  },
  "kafkaProperties": {
    "security.protocol": "SASL_SSL"
  }
}

Credential Redaction

The typed Secret parameters are automatically redacted in API responses, displaying as Secret(****). Values in kafkaProperties are not redacted—migrate sensitive values to the typed parameters for protection.