Capless
Docs Streams & cursors

View as Markdown

Streams & cursors

A stream is a durable, ordered, append-only collection of JSON records. Streams are immutable except for erasures.

H="authorization: Bearer $CAPLESS_API_KEY"

curl -s https://api.capless.sh/v1/streams -H "$H" -d '{"name":"events","validator":"entry.user_id != \"\""}'
curl -s https://api.capless.sh/v1/streams -H "$H"                       # list (by name)
curl -s https://api.capless.sh/v1/streams/events -H "$H"                # metadata: cursor (head), entryCount, …
curl -s -X PATCH https://api.capless.sh/v1/streams/events -H "$H" -d '{"validator":null}'
curl -s -X DELETE https://api.capless.sh/v1/streams/events -H "$H"      # asynchronous
  • Names match ^[A-Za-z0-9][A-Za-z0-9_.\-]{0,127}$. Use the name or the ID (str_…) in paths.
  • A validator is a CEL expression over entry that must return true for every inserted record. A change (PATCH) applies to batches committed after it and increments validatorVersion.
  • Deleting releases the name immediately; reads and writes return 410 stream_deleted; storage is removed in the background (deleting → deleted), and rows are deleted from every materialization the stream fed.

Inserting

curl -s https://api.capless.sh/v1/streams/events/entries -H "$H" \
  -d '{"idempotencyKey":"req-1","entries":[{"user_id":"u1","amount":100},{"user_id":"u2","amount":200}]}'
{
  "batchId": "019a…",
  "committedAt": "2026-10-30T12:00:00.123Z",
  "inserted": 2,
  "duplicate": false,
  "startCursor": "cur_…",
  "committedCursor": "cur_…",
  "entries": [{ "id": "ent_…", "cursor": "cur_…" }, { "id": "ent_…", "cursor": "cur_…" }]
}
  • Atomic: validation runs over all entries first; one failure rejects everything (422 validation_failed, failing indexes in details.failures).
  • Durable: success is returned only after the batch is in object storage (R2) and appended to the stream's history. Acknowledged data is never lost.
  • Batches hold 1–100 000 entries; each entry ≤ 1 MiB and ≤ 32 levels deep; request bodies ≤ 8 MiB (413 request_too_large). Split larger loads into several batches.
  • "returnEntryIds": false omits the per-entry array from large responses.

Idempotency

idempotencyKey (per stream, per operation type, ≤ 256 chars) makes retries safe:

  • same key, identical body → the original result with duplicate: true;
  • same key, different body → 409 idempotency_key_reused;
  • concurrent retries resolve to one commit. Keys are retained for 7 days.

Batches and positions

A stream's history is a sequence of batches in commit order. A batch is either an entries batch (1..100 000 records) or an erasure marker (one position). Every item has a position (batchId, offset).

  • Batch IDs are UUIDv7 strings assigned at commit, never at upload. Within a stream they are strictly increasing, and lexicographic order equals commit order. The first 48 bits are the commit millisecond; clock regressions and bursts reuse the previous millisecond and increment a sequence, so the embedded time is approximate but monotonic.
  • There is no per-entry sequence number; ordering batches is sufficient.

Cursors

Cursors (cur_…, constant size) are boundaries between items, not items.

origin │ B1[0] │ B1[1] │ B2[0] │ B2[1] │ head
       ↑       ↑               ↑
  cur(B1,0) cur(B1,1)       cur(B2,1)

In this two-batch stream, afterCursor: null reads from the origin, cur(B1,1) sits between B1[0] and B1[1] (it is B1[0]'s cursor), and B2[1]'s cursor is the head.

  • cur(B, k) sits immediately before item k of batch B. An entry's cursor is the boundary right after it.
  • The origin cursor is the boundary before everything; afterCursor: null means origin.
  • The head (stream.cursor) is the boundary after the last committed item; null for an empty stream.
  • Cursors and entry IDs are stable forever: compaction and physical purges rewrite storage objects, never offsets.

Visibility: no gaps

Each stream has exactly one sequencer (a Durable Object). The head moves in the same storage transaction that appends a batch, and batch IDs are assigned in that transaction, so a reader can never observe batch N+1 while batch N could still appear: there are no uncommitted gaps. A consumer that persists its cursor never misses a record committed before it.

Throughput

One stream handles on the order of a thousand commits per second. Batching amortizes this (100 000 entries per commit). Shard a very hot logical stream across several streams.