Application/JVM

Process Killed

If the operating system kills the process, it is likely due to insufficient RAM available for the total workload managed by the OS. Like most JVM applications, thatDot Streaming Graph uses RAM for the heap, but also for off-heap allocations. We recommend setting a lower Java heap size to allow some RAM to be used by off-heap allocations. We also suggest that you do not run other processes on the same machines running Streaming Graph.

Application Won’t Start

If the application will not start, ensure that the resources you are allocating to the system are actually available. thatDot Streaming Graph operates in a JVM and can be tuned accordingly. If there are other services hosted on the same machine with Streaming Graph, ensure that all services can operate with the available system resources. We recommend running Streaming Graph without other intensive services on the same machine for most deployments.

Pekko Heartbeat Warning/Timeout

In clustered operation, sometimes log messages such as:

[quine-cluster-pekko.actor.default-dispatcher-19] org.apache.pekko.remote.PhiAccrualFailureDetector - heartbeat interval is growing too large for address pekko://quine-cluster@10.0.15.51:25520: 2585 millis

This can be caused by network latency, but often this can be due to CPU limits created by virtualization layers such as Docker or Kubernetes. If CPU limits are the cause, the recommended solution is to remove CPU limits which utilize CFS (Completely Fair Scheduler).

In Docker, this can be done by avoiding the use of the --cpus and --cpu-quotas options, opting instead for the --cpu-shares option instead if necessary.

In Kubernetes, this can be done by avoiding the use of resource limits for Streaming Graph pods or, if necessary, ensuring those limits are set to at least double the resource requests for each pod.

See this Akka documentation page for more information on deploying to Docker or Kubernetes.