Recipe Data Service¶
Most Quine recipes come with sample data files that you download and point the
recipe's ingest streams at. For the recipes below, thatDot also hosts that data at
data.thatdot.com as Server-Sent Events (SSE) streams, one event per line of the
file. Each of these recipe pages has a "Recipe v2 (data.thatdot.com)" tab with a
version of the recipe whose ingest streams read from the hosted stream instead of a
local file, so you can run it without downloading anything.
The stream accepts query parameters that control which lines are sent, how fast, and
whether the stream closes when the file ends. To change how a recipe loads, add them
to the url: in its ingest stream:
ingestStreams:
- name: email-ingest
source:
type: ServerSentEvent
url: https://data.thatdot.com/recipe/email.jsonl?rate=1
format:
type: Json
query: |-
MATCH (sender), (message)
WHERE id(sender) = idFrom('email', $that.from)
...
Every file is served from https://data.thatdot.com/recipe/. Click a file name to
watch the stream in your browser at ten lines per second.
Controlling the stream¶
Every control can be set as a query parameter on the URL or as an HTTP header of the same name.
| What it does | Query parameter | Header | Default |
|---|---|---|---|
| First line to send. | ?start=500 |
start: 500 |
1 |
| No lines after this one are sent. | ?end=510 |
end: 510 |
last line of the file |
| Maximum number of lines to send. | ?limit=100 |
limit: 100 |
no limit |
Maximum lines per second. Decimals are allowed, so 0.5 sends one line every two seconds. |
?rate=1 |
rate: 1 |
unlimited |
Seconds between heartbeats. 0 turns heartbeats off and closes the stream after the last line. |
?keepalive=0 |
keepalive: 0 |
30 |
| Reconnect delay the client is told to use, in milliseconds. | ?retry=10000 |
retry: 10000 |
3000 |
If the same control is sent both as a query parameter and as an HTTP header, the query parameter wins. A missing or invalid value falls back to the default value.
Event IDs
Lines are numbered starting from 1. Each event's id is its line number in the file, meaning ?start=500
gives ids 500, 501, etc. Since lines start at 1, ?start=0 is invalid, and will default to starting on
line 1.
A client that reconnects after a dropped connection, sends the last id it saw in the Last-Event-ID header, and
the stream resumes on the line after that. The Last-Event-ID header overrides ?start=, allowing a reconnected
stream to resume from the next line, instead of from the beginning.
Examples¶
The -N flag stops curl from buffering so lines print as they arrive, and the keepalive=0 is on every example so curl
exits at the end of the stream.
# the first three lines
curl -N "https://data.thatdot.com/recipe/apache.log?limit=3&keepalive=0"
# lines 500 through 510
curl -N "https://data.thatdot.com/recipe/apache.log?start=500&end=510&keepalive=0"
# one line per second
curl -N "https://data.thatdot.com/recipe/apache.log?rate=1&keepalive=0"
# the same limit, sent as a header instead of a query parameter
curl -N -H "limit: 3" "https://data.thatdot.com/recipe/apache.log?keepalive=0"
# resume after event 500
curl -N -H "Last-Event-ID: 500" "https://data.thatdot.com/recipe/apache.log?limit=3&keepalive=0"
Heartbeats
Heartbeats are blank lines sent every keepalive seconds for the life of the connection, keeping the connection open,
which is what an ingest stream wants. It's also why a curl without keepalive=0 prints every line and then hangs. When
keepalive=0 is set, the server closes the connection after the last line.