Files and Named Pipes¶

Files are a stream of data. It's easy to think of a file as a singular chunk, but on disk it is a linear sequence of bits. Reading data from a file is the process of starting at the beginning, and reading each bit in sequence, until you reach the end. Programs usually do this under the hood and deliver a single result when finished, but that data can be handled by a program before the end of file is reached. Reading a file until a particular sequence is found (the "delimiter") will yield a sequence of bytes which can be handled as a single event. This is how data is streamed into Quine Enterprise from a file—making it a very natural and convenient way to load data into Quine Enterprise.

Security Controls¶

Quine Enterprise implements directory allow listing and file enumeration controls to protect against path traversal vulnerabilities. File ingests are restricted to specific directories configured at startup.

Configure file ingest security in your configuration file:

quine.file-ingest {
  # Allow list of directories
  allowed-directories = ["/path/to/data"]

  # File resolution mode
  resolution-mode = "static"  # or "dynamic"
}

The allowed-directories setting specifies an allow list of directories from which files can be ingested. Relative paths are resolved against the working directory at startup. An empty list means no file ingests are allowed except from recipes. By default, Quine Enterprise restricts ingestion to the file_ingests subdirectory in the working directory (["file_ingests"]) for more restrictive production security.

Symbolic links are not allowed

Symbolic links (symlinks) are not permitted within allowed directories. A symlink could be used by a bad actor to reference sensitive files outside the allowed directory — for example, linking to a password file — thereby tricking Quine Enterprise into ingesting secret information. Quine Enterprise will reject any file ingest that resolves to a symlink.

The resolution-mode setting controls which files within allowed directories can be ingested. Set to "static" to only allow files that were present at startup (more secure), or "dynamic" to allow any file in allowed directories, even files added after startup (more flexible). Quine Enterprise defaults to "static" for production security, ensuring only files present at startup can be ingested.

File Ingest Format¶

Each file has one special feature which distinguishes it from a normal stream of data: it has an end. When ingesting data from one of these file types, the stream will continue until the end-of-file signal is reached. When the end-of-file signal is reached, the stream will finish and be marked by Quine Enterprise as COMPLETED.

JSON[L]¶

"type" : "CypherJson"

JSON data is ubiquitous; it is a very natural source of a data for a streaming system. The typical approach is to save records as JSON objects to a file, separated by new-lines: \n. It is very common to call that format a .json file. That format is not technically valid JSON, and so sometimes it is referred to as "JSON Lines" and given the .jsonl file extension.

Quine Enterprise reads these .json or .jsonl files one line at a time, passing each parsed JSON object to the Cypher ingest query as $that to be used in the rest of the ingest query.

CSV¶

"type" : "CypherCsv"

Comma-Separated Values are a convenient way to store tabular data. Quine Enterprise reads CSV files natively, passing each parsed line into a Cypher query as: $that. Fields from each row are accessible by numeric index or by key, depending on whether headers are provided. The headers field can either be read from the first line of the CSV, provided manually as a list of strings, or disabled entirely. See the REST API reference for details.

Text¶

"type" : "CypherLine"

Any text file can be used as as stream of input to Quine Enterprise. Using the CypherLine ingest option will read in the specified file, passing each individual line as a string to the provided Cypher query, where it can be used directly as a string value, or parsed by hand into a more meaningful structure.

Named Pipe¶

A named pipe is an operating-system/file-system abstraction typically used for inter-process communication. A named pipe has a file-like representation in the filesystem—so it looks like a file to most programs—but it does not persist data. The operating system uses this object that looks like a file as a communication channel. One program can write to a named pipe, and if another program is actively reading from the same named pipe, then the second program receives the data written by the first program.

Quine Enterprise supports named pipes on Unix-like operating systems (MacOS, Linux, etc), but not on Windows. On most Unix-like operating systems, Quine Enterprise will automatically detect whether the provided path refers to a regular file or a named pipe. Automatic detection can be overridden by setting the fileIngestMode setting in the ingest API.

When Quine Enterprise has an active stream ingesting from a named pipe, it will consume data from that pipe until the stream is cancelled, or it encounters an error. The format of the incoming data from the named pipe can be any of the File Ingest Formats listed above. Data written to the named pipe will be separated by new-line characters (\n) and passed to the provided Cypher query exactly as described above.

Cluster-Wide Ingest¶

A file exists on one host's filesystem. Nothing about a path tells the cluster which members can read it, so a cluster-wide ingest of a file source is refused unless you pin it. A member chosen by the cluster might not have the file, and reading nothing while reporting healthy is worse than refusing.

Supply pinnedTo with the cluster address of the host that holds the file:

{
  "ingest": {
    "name": "audit-log",
    "source": {
      "type": "File",
      "path": "file_ingests/audit.log",
      "format": {
        "type": "Line"
      }
    },
    "query": "MATCH (n) WHERE id(n) = idFrom($that) SET n.line = $that"
  },
  "pinnedTo": "10.0.0.7:2551"
}

This runs the file ingest under cluster management: the definition lives in the cluster manager rather than on one member, it survives that host restarting, and it is created, paused, resumed, updated, and deleted through the same API as every other cluster-wide ingest stream. What it does not do is distribute, since a pinned ingest is one slice on one host.

A pinned ingest is never relocated. It runs on its host or nowhere, and it stops until that host returns. Because a lawful wait for an absent host and a mistyped address look identical, a pin naming a host the cluster does not currently know is reported in warnings at creation and listed in pinnedAwaitingHost in status.

The pinned host's file security controls still apply: allowed-directories and resolution-mode are evaluated on the host running the slice. Quine checks the pinned host's policy when the ingest is created where it can; if the host cannot be asked, creation proceeds and a denial surfaces as the ingest's failure cause instead.

Updating a pinned file ingest

pinnedTo is part of the definition, and an update that omits it would normally unpin the ingest. For a file source that would hand the slice to a host without the file, so it is refused instead. Include pinnedTo in every update to a pinned file ingest.

Named pipes behave the same way, and for the same reason.