Observation Outputs

API v2 is now the default

The v2 API is the default for all observation output operations. API v1 remains available but is planned for deprecation and will be removed in a future release. See Migrating from API v1 for migration guidance.

When Novelty detects anomalies or patterns in your data, observation outputs route those detection results to external systems. Each observation can have any number of output destinations, which are processed in parallel. Outputs support various destinations including Kafka, Kinesis, SNS, HTTP webhooks, and local files.

Output Workflows

Novelty provides workflow-based output processing with filtering and parallel destination routing. Each observation result passes through optional filtering before reaching one or more destination outputs.

Results can be routed to any of the following destinations, set with the type field of an output.

Name	Configuration value	Formats	Description
Broadcast to Reactive Stream	ReactiveStream	JSON, Protobuf	Broadcasts data to a created Reactive Stream. Other thatDot products can subscribe to Reactive Streams. Reactive Stream outputs are only supported in standalone (single-host) deployments.
Drop	Drop	—	Effectively no destination at all, this does nothing but forget the data sent to it.
Log JSON to Console	StandardOut	JSON	Prints each result as a single-line JSON object to stdout on the application server.
POST to HTTP[S] Webhook	HttpEndpoint	JSON	Makes an HTTP[S] POST for each result. For the format of the result, see "Standing Query Result Output".
Publish to Kafka Topic	Kafka	JSON, Protobuf	Publishes provided data to the specified Apache Kafka topic.
Publish to Kinesis Data Stream	Kinesis	JSON, Protobuf	Publishes provided data to the specified Amazon Kinesis stream.
Publish to SNS Topic	SNS	JSON, Protobuf	Publishes an AWS SNS record to the provided topic. To guarantee delivery, writes that fail are retried indefinitely, so confirm the credentials and topic ARN before starting this output. An unfixable error (e.g., an invalid topic ARN or missing credentials) will retry forever without emitting results, which may stop the Standing Query this output is attached to.
Write JSON to File	File	JSON	Writes each result as a single-line JSON record. For the format of the result, see "Standing Query Result Output".

Filtering

Filter results before processing with predicates. Use OnlyPositiveMatch to route only positive matches (when the pattern first matches), filtering out cancellations:

{
  "filter": {
    "type": "OnlyPositiveMatch"
  }
}

Observation results include an isPositiveMatch metadata flag:

• true - Pattern newly matched (positive match)
• false - Pattern no longer matches (cancellation)

Output Formats

Destinations that support formats can serialize as JSON or Protobuf.

JSON (Default):

{
  "format": { "type": "JSON" }
}

Protobuf:

{
  "format": {
    "type": "Protobuf",
    "schemaUrl": "http://schema-registry:8081/schemas/ids/1",
    "typeName": "com.example.ResultMessage"
  }
}

Output Destinations

Drop

Drop the current result output and end processing the destination.

POST to Webhook

Makes an HTTP[S] POST for each result.

Log JSON to Standard Out

Prints each result as a single-line JSON object to standard output on the Novelty server.

Log JSON to a File

Write each result as a single-line JSON object to a file on the local filesystem.

Publish to Kafka Topic

Publishes a record for each result to the provided Apache Kafka topic. Records can be serialized as JSON or Protocol Buffers before being published to Kafka.

Setting	Description	Default
topic	Kafka topic name	—
bootstrapServers	Kafka bootstrap servers	—
format	Output format (JSON or Protobuf)	JSON
kafkaProperties	Additional Kafka producer configuration	None
sslKeystorePassword	Password for SSL keystore (secret)	None
sslTruststorePassword	Password for SSL truststore (secret)	None
sslKeyPassword	Password for private key in keystore (secret)	None
saslJaasConfig	SASL authentication configuration (secret)	None

Credential Redaction

For security, sslKeystorePassword, sslTruststorePassword, sslKeyPassword, and saslJaasConfig credentials are automatically redacted in API responses, displaying as Secret(****). See Kafka Configuration for details on SASL authentication types.

Publish to Kinesis Stream

Publishes a record for each result to the provided Kinesis stream. Records can be serialized as JSON or Protocol Buffers before being published to Kinesis.

Setting	Description	Default
streamName	Kinesis stream name	—
credentials	AWS credentials (optional)	None
region	AWS region (optional)	None
format	Output format (JSON or Protobuf)	JSON
kinesisParallelism	Concurrent publish operations	None
kinesisMaxBatchSize	Maximum records per batch	None
kinesisMaxRecordsPerSecond	Rate limit (records/second)	None
kinesisMaxBytesPerSecond	Rate limit (bytes/second)	None

Credential Redaction

For security, accessKeyId and secretAccessKey values are automatically redacted in API responses, displaying as Secret(****).

Publish to SNS Topic

Publishes an AWS SNS record to the provided topic containing JSON for each result.

Setting	Description	Default
topic	SNS topic ARN	—
credentials	AWS credentials (optional)	None
region	AWS region (optional)	None
format	Output format (JSON or Protobuf)	JSON

Credential Redaction

For security, accessKeyId and secretAccessKey values are automatically redacted in API responses, displaying as Secret(****).

Credential Validation

Ensure your credentials and topic ARN are correct. If writing to SNS fails, the write will be retried indefinitely. If the error is not fixable (e.g., the topic or credentials cannot be found), the outputs will never be emitted and the output could stop running.

Publish to Reactive Stream

Broadcasts results to a TCP-based reactive stream endpoint. Clients can connect to receive a continuous stream of results.

Setting	Description	Default
address	Address to bind the stream server	localhost
port	Port to bind the stream server	—
format	Output format (JSON or Protobuf)	JSON

Cluster Limitation

Reactive Stream outputs do not function correctly when running in a cluster. Use Kafka or Kinesis for clustered deployments.