Operational Considerations¶
Deploying: Basic Resource Planning¶
Novelty is extremely flexible in its deployment model. It is robust and powerful enough to run on a server with hundreds of CPU cores and terabytes of RAM, but also small and lightweight enough to run in a container or on a Raspberry Pi. Custom deployment configurations (e.g. on-premise deployment, non-containerized deployments, alternate storage configurations, etc.) are available upon request.
Novelty is a backpressured system; it will deliberately slow down when resource-constrained. This is a highly desirable quality because the only alternative is to crash the system. Making Novelty run faster is essentially a matter of allocating resources to the machine/environment so that the slowest component is not a bottleneck.
CPU¶
As a distributed system, Novelty is designed from the ground up to take advantage of multiple CPU cores. Novelty will automatically configure increased parallelism where there are more CPU cores available.
High CPU Utilization¶
By design, we expect Novelty to utilize as many resources as it is allotted. High CPU usage alone, without additional symptoms, is desirable and reflects good utilization of the system.
Low CPU Utilization¶
Novelty is designed to use backpressure to limit incoming data from producers. If the data storage layer is slow, Novelty will backpressure incoming data to ensure all work can complete. This mechanism provides data safety by not dropping incoming records, and system resilience by not sending more data than it can handle.
Performance is primarily measured as throughput of the number of observations per second. A system the size/scale of a modern laptop (with SSD) will often witness throughput around 20,000 observations per second. Increasing CPU and RAM will allow for higher throughput, though the total throughput is limited by several other considerations, including the parallelism query parameter in the Record and Score Observation: POST /api/v2/contexts/{context}/observations and Record and Score Observations Batch: POST /api/v2/contexts/{context}/observations/bulk API endpoints.
RAM¶
System memory (RAM) is used by Novelty primarily to keep nodes in the graph warm in the cache. When a node required for a query or operation is needed, if it is live in the cache, then calls to disk can often be entirely avoided. This significantly increases performance and throughput of the system overall. So in general, the more nodes live in the cache, the faster the system will run.
Note
Novelty uses a novel caching strategy that we call "semantic caching". The net result is that there is a decreasing benefit to having additional nodes kept in memory. There is only a performance improvement if those nodes are needed.
Since Novelty runs in the JVM, you can have Novelty use additional system RAM in the standard Java fashion by increasing the maximum heap size with the -Xmx flag when starting the application. E.g.: java -Xmx6g -jar novelty-0.15.0.jar. In order for Novelty to receive a benefit from increased RAM, you will probably also need to increase the limit of nodes kept in memory per shard by changing the in-memory-soft-node-limit, and possibly the in-memory-hard-node-limit sizes. See the Configuration Reference for details. The ideal setting for these limits depends on the use case and will vary. We suggest monitoring usage with the tools described below to determine the best settings for your application. In-memory node limits can be changed dynamically from the built-in REST API with the shard-sizes endpoint.
Warning
Keep in mind that setting the maximum heap size is not the same as limiting the total memory used by the application. Some amount of additional off-heap memory will be used when the system is operating normally. Off-heap memory usage is usually trivial, but can become significant in some cases, e.g.: Lots of network traffic, or using memory mapped files. The latter case can becomes very significant when using the MapDB persistor, which defaults to using memory mapped files for local data storage.
Out of Memory Error¶
Novelty can be configured to use any amount of RAM. The settings for in-memory-soft-node-limit and in-memory-hard-node-limit should be adjusted according to the workload in proportion to the amount of memory available on the system. If you encounter an Out-of-Memory error, we suggest lowering the soft limit. This setting can be adjusted while the system is running via an API call.
Storage¶
Novelty stores data using one or more Persistor. Persistors are either local or remote. A local persistor will store data on the same machine where Novelty is running, at a location in the filesystem determined by its configuration. A remote persistor will store data on a separate machine. By default, Novelty will use the local RocksDB persistor.
For production deployments, we recommend using a remote persistor like Cassandra to provide data redundancy and high-availability. Additionally, old data can be expired from disk using Cassandra's time-to-live setting in order to limit the total size of data stored.
Monitoring¶
It is often helpful to watch an operating system to confirm it is behaving as anticipated. Novelty includes multiple mechanisms for monitoring a running system.
Web Browser¶
A basic operational dashboard is built into the pages served from the Novelty web browser.
This page includes live updating reports on memory usage, nodes in the cache (per shard), basic histograms of the the number of properties and edges per node, and latency measurements of various important operations.
REST API¶
Novelty serves a REST API from its built-in web server. That web server provides separate Process Readiness: GET /api/v2/admin/readiness and Process Liveness: GET /api/v2/admin/liveness API endpoints for monitoring the status of the Novelty system. Detailed metrics can be retrieved from the Metrics: GET /api/v2/admin/metrics endpoint.
JMX¶
Novelty can be monitored and controlled live using JMX. There is a wide array of tools available for monitoring JVM applications through JMX. If you don't already have a preference, you might start with VisualVM.
Saving Data¶
The application will save data produced from the observations passed in. Upon restart, the system resumes with all data available from a prior session. So the underlying model created from past observations will be preserved across system restarts, if the underlying file system is restored.
For alternative options for data storage, high availability, clustered operation, and alternate data storage formats, please contact thatDot's sales team.
Shutting Down¶
A set of administrative API endpoints are included in the built-in documentation. To shut down the system cleanly and ensure that all data are properly persisted, use the Graceful Shutdown: POST /api/v2/admin/shutdown endpoint. Please be sure to call this API and await a successful response before shutting down the application or container. Failure to properly shut down the system can result in loss of data.