Skip to content

Streaming recorder

This page mirrors implementation notes maintained in CLAUDE.md. Update both when changing this subsystem.

When Enable Streaming is on, Cue runs a small video capture pipeline in the background and feeds it to a digest model every few seconds. The recorder is an external CLI (vendored under vendor/ocap-{macos,windows}) launched as a subprocess at ~10 fps. ocap uses GStreamer + hardware H.265 encoding (near-zero CPU on Apple Silicon / NVIDIA) and writes:

  • <streaming root>/stream/chunk_<ts>.mkv — video.
  • <streaming root>/stream/chunk_<ts>.mcap — structured keyboard / mouse / window / screen events.

Pipeline

sequenceDiagram
    participant Cue as Cue (parent)
    participant Rotator
    participant Ocap as ocap subprocess
    participant Pruner as pruner worker
    participant Evictor
    participant Digest as digest worker

    Cue->>Rotator: start()
    loop every chunk_secs (default 30 s)
        Rotator->>Ocap: spawn (ocap-macos / ocap-windows CLI)
        Ocap-->>Rotator: chunk_<ts>.mkv + chunk_<ts>.mcap
        Rotator->>Pruner: spawn for closed chunk
        Pruner->>Pruner: dhash + event-aware keyframe extraction
        Pruner-->>Cue: keyframe_<ts>.jpg files
    end
    loop every 60 s
        Evictor->>Evictor: drop chunks/keyframes outside window_secs (default 15 min)
    end
    loop every digest tick (5 s)
        Digest->>Digest: read recent MCAP events + keyframes
        Digest->>Digest: select up to 10 frames + scrub events
        Digest->>Digest: call cue.llm.summarize_digest_with_policy
        Digest->>Cue: write digest.md + insert digests row
    end

Cue's threads on top of ocap

Thread What it does
rotator Stops the current ocap and starts a fresh chunk every streaming.chunk_secs (30 s default), or when ocap dies unexpectedly. After each rotation it fires a per-chunk pruner subprocess.
pruner worker Reads the closed MKV via owa.gstreamer.gst.mkv_reader, dhashes each frame, keeps only frames that either differ significantly from the last keyframe or follow a real input event (keyboard / click / window change). Writes small JPEGs to the keyframes dir. Spawns its own subprocess so its GStreamer env doesn't leak into the parent.
evictor Every 60 s, deletes chunk files whose end time is older than streaming.window_secs (15 min default). Same cutoff applies to keyframe JPEGs.
digest Reads recent MCAP messages, PII-scrubs text, sends to the configured backend, writes digest.md. See Digest pipeline.

Why ocap is a subprocess

GStreamer + hardware H.265 encoding can't easily live in-process with Cue's own Python — PyGObject pulls in GLib, GStreamer.framework dylibs that can clash with the rest of Cue's process state. ocap as a subprocess sidesteps this and gives a clean process boundary for the privacy-pause SIGKILL.

The subprocess is launched via the ocap CLI (owa.ocap_macos.cli or owa.ocap_windows.cli); Cue itself never imports the GStreamer recorder modules.

Snapshot context

Hotkey path reads digest.md + the most recent MCAP events + up to 3 pruner keyframes via recorder.snapshot_context(recent_secs, max_events). Opus gets "accumulated knowledge" on the hotkey path with no on-hotkey capture latency.

Streaming root path quirks (macOS)

Stream + keyframe dirs live under ~/Library/Caches/Cue/ rather than ~/Library/Application Support/Cue/ because GStreamer's filesink location=... doesn't tolerate spaces. Windows uses the usual %LOCALAPPDATA%\Cue.

Vendored ocap

The recorder lives in two repos as git submodules under vendor/:

  • vendor/ocap-macos — provides owa.ocap_macos.recorder. Pyproject pinned to owa-env-desktop==0.6.5.
  • vendor/ocap-windows — provides owa.ocap_windows.recorder.

Both rely on vendor/open-world-agents-private for owa-core, owa-msgs, owa-env-desktop, mcap-owa-support, owa-cli. After cloning Cue, run:

git submodule update --init --recursive

before the first install. See Submodules & vendoring.

Privacy interaction

Privacy pause kills ocap immediately (SIGKILL — see Privacy pause for why graceful shutdown isn't used). Pruner / evictor / digest threads keep running but find no new chunks, so the pipeline naturally drains. On resume a fresh ocap session is spawned.

See also