Skip to content

cue.llm

Digest backend abstraction.

DigestBackend is the protocol the digest pipeline calls into to produce a summary string from (prompt_header, prev_summary, images, timeline). CloudVisionBackend is the existing Haiku-with-images path lifted out of cue.digest._build_digest. LocalVisionBackend (commit 6) will land behind the same protocol so the call site doesn't change again.

Output scrub stays at the call site (cue.digest._build_digest) so a single defense-in-depth layer covers every backend's output.

DigestBackend

Bases: Protocol

Producer of a digest summary from a prompt + images + timeline.

Returns the raw model output. The caller is responsible for the output-side PII scrub — keeping the scrub there means each backend doesn't need to remember to call it.

CloudVisionBackend

Anthropic Haiku via the Messages API.

Builds a temporally interleaved content array (text + image blocks) so the model anchors each event burst to the keyframe it follows. Window-event narration is scrubbed inline before it reaches the prompt — the API never sees raw app names / titles.

LocalVisionBackend

Local Gemma 4 multimodal digest via the bundled llama-server subprocess. Posts to localhost/v1/chat/completions with the same text + image_url interleave the cloud path uses, except images go in as data:image/jpeg;base64,... URLs (downscaled + EXIF-stripped by cue.image_preprocess.prepare_for_digest).

Failure modes always raise LocalUnavailable / LocalTimeout — NEVER fall back to cloud silently. The policy gate in summarize_digest_with_policy decides whether allow_cloud_fallback is opt-in.

get_digest_backend 

get_digest_backend() -> DigestBackend

Return the configured digest backend. digest_backend="local" → LocalVisionBackend; default "cloud" → CloudVisionBackend.

The policy gate (skip-on-local-unavailable vs. opt-in cloud fallback) lives in summarize_digest_with_policy — _build_digest is its only caller, so the policy stays at the orchestration layer and individual backends only have to raise typed errors.

summarize_digest_with_policy 

summarize_digest_with_policy(prompt_header: str, prev_summary: str, images: list[Path], timeline: list[dict]) -> str | None

Policy gate around the configured digest backend.

digest_backend == "local" AND the local backend raises: - allow_cloud_fallback=True → re-route to CloudVisionBackend (raw prompt + images leave the device — this is the user's explicit opt-in). - allow_cloud_fallback=False → return None. The caller skips the digest cycle. NO cloud call made. digest_backend == "cloud" → call CloudVisionBackend directly.