Skip to content

Configuration

Configuration is supported by Typesafe Config, enabling multiple ways to pass in options. Most commonly, configuration is provided via either Java system properties (passed as command-line options) or via a HOCON config file. HOCON is a JSON-like format that is very flexible and human-readable. The reference config below is in HOCON format.

# Example of setting configuration via configuration file
java \
  -Dconfig.file=quine.conf \
  -jar quine-1.10.0.jar

# Example of overriding configuration via system properties
java \
  -Dquine.webserver.port=9000 \
  -Dquine.id.type=uuid-3 \
  -jar quine-1.10.0.jar

# Example of overriding configuration via environment variables
CONFIG_FORCE_quine_webserver_port=9000 \
CONFIG_FORCE_quine_id_type=uuid-5 \
java \
  -Dconfig.override_with_env_vars=true \
  -jar quine-1.10.0.jar

Differences from Quine

In general, the configuration for Quine Enterprise is very similar to the configuration for Quine Open Source. Quine Enterprise's complete documented configuration reference is located here. Quine Enterprise exposes a configuration surface that is nearly a superset of Quine's.

Quine Enterprise also includes some additional configuration sections that are not present in Quine:

  • cluster: This section is used to configure the cluster membership and communication settings. Its full breadth is documented in the config reference, but special attention should be given to the cluster joining configuration.

The quine.cluster.cluster-join configuration block specifies how the cluster members will be discovered. The type is either dns-entry or static-seed-addresses. If using dns-entry, a name should also be provided that will resolve a set of A records, one for each cluster member. If using static-seed-addresses, a seed-addresses list should be provided.

At minimum, a typical Kubernetes-based deployment will use dns-entry cluster joining with the following configuration:

quine.cluster {
  target-size = 4 # the number of members who must occupy positions in the cluster for the graph to be considered available
  cluster-join {
    type = "dns-entry" # Use the A records for a DNS name to find cluster members
    name = "streaminggraph" # The DNS name to use
  }
}

A typical configuration for an EC2-based deployment using static seed addresses would look like this:

quine.cluster {
  target-size = 4 # the number of members who must occupy positions in the cluster for the graph to be considered available
  cluster-join {
    type = "static-seed-addresses" # Use the specified, statically-known seed addresses to find cluster members
    seed-addresses = [ # note that not all cluster members need to be specified as seeds
      {
        address = "10.0.5.1" # hostname or IP at which a seed member can be found
        port = 17750
      },
      {
        address = "10.0.1.12"
        port = 17750
      }
    ]
  }
}

In lieu of a full cluster-join config block, environment variables may be used: QUINE_SEED_DNS can be set instead of cluster-join.name when using dns-entry cluster joining, and QUINE_SEED_ADDRESSES can be set instead of cluster-join.seed-addresses when using static seed addresses (formatted as a comma-separated list of address:port pairs like 10.0.5.1:17750,10.0.1.12:17750).

Memory Configuration

Quine Enterprise caches nodes in memory as necessary. The setting for in-memory-soft-node-limit and in-memory-hard-node-limit along with the shard-count configuration determines how many nodes can be cached at a time, and therefore how much heap space the JVM needs available for processing. The shard count defaults to 4 and does not typically need to be changed. Each shard in a cluster member retains its own cache of in memory nodes. The soft limit (defaulting to 10000) determines the minimum size of this cache, and the hard limit (defaulting to 75000) determines the maximum size of the cache. The cache uses the flexible range between these values for nodes which are being put to sleep while other nodes are being rehydrated into memory (i.e. the difference between these values determines how many nodes can be going to sleep and waking up at the same time). The default settings would enable between 40,000 and 300,000 nodes in memory at a time. Depending on the use case, and the average memory footprint of a node, These values can be adjusted up or down to maximize usage of the memory on the machine and the JVM heap.

Note

The heap space requirement is primarily a function of the soft and hard node limit settings. Beyond that, leaving overhead for the operating system to manage direct memory requests will ensure the performance of the host server.

To avoid Garbage Collection (GC) pauses in the JVM heap, it is recommended that you set the memory allocated to Quine Enterprise to a fixed size if possible. You can do this by setting Xms and Xmx to the same value, discouraging the JVM from dynamically resizing the heap. Resizing the heap triggers a full GC, which can mean a lengthy pause (meaning everything in the app is absolutely locked), depending on the size of the heap. Keep in mind that large heap spaces take longer in garbage collection so simply adding as much as possible could negatively affect performance. For this reason, it is not recommended to set the heap size larger than 16GB, and 12GB tends to be a good starting point for large graph implementations with high ingest requirements.

Network traffic can also use memory. This is typically a small amount and is well-managed by the system, but in some extreme cases (like high-volume data without proper data locality), this memory use can add up to something non-trivial—especially if the heap is using all available memory on the system or other processes are running and competing for memory. Off-heap memory is typically used and in the worst case, not leaving some memory available for off-heap usage can lead to the OS killing the process. It is recommended that the physical memory on a host be 25% to 33% larger than the Xms and Xmx heap setting. This will guarantee there is enough off-heap memory available for these other operations.

Data Pipeline Security Recommendations

Data pipelines in Quine Enterprise can be very flexible to almost any use-case, but with that flexibility comes potential for misuse. In order to keep Quine Enterprise, its data, and its environment as secure as possible, we recommend you follow these best practices:

  1. Use the latest version of Quine Enterprise. We regularly release updates with security fixes.
  2. Deploy Quine Enterprise behind a reverse proxy that performs TLS termination, user authentication, and HTTP request logging. This will allow you to control access to your Quine Enterprise instance and gives you more information about how and when your graph is being accessed.

  3. Configure your data source according to that data source's best practices. For example, if using a Kafka data source, keep your Kafka servers up to date and use the kafkaProperties field in the Quine Enterprise ingest configuration to enforce TLSv1.3 encryption between Quine Enterprise and your Kafka cluster.

  4. Do not pass unsanitized data to any Cypher procedures (functionality invoked with the CALL syntax). This can lead to query injection attacks (via procedures like cypher.doIt and cypher.do.case) or server-side request forgery (via procedures like loadJsonLines).

  5. Configure file ingest security controls to restrict file access to specific directories. Use the file-ingest.allowed-directories setting to whitelist permitted directories and set file-ingest.resolution-mode to "static" in production environments to only allow files present at startup. See the Files and Named Pipes documentation for details.

Loglevel Configuration

Quine Enterprise uses the Logback framework for logging. Loglevel is configurable for several loggers. The root.loglevel Java system property will set the log level for all loggers including dependencies. thatdot.loglevel will set the log level only for Quine Enterprise logs directly.

Available log levels are ERROR, WARN, INFO, DEBUG, TRACE. See documentation for more details: https://logback.qos.ch/manual/architecture.html#effectiveLevel

Example:

thatdot {
    loglevel = INFO
}

Reference Documentation

Uncommented values are the defaults, unless otherwise noted. Unexpected configuration keys or values in the quine block will report an error at startup.

A single underscore _ is used to indicate a required property with no default value. There are none of these in the default configuration.