mcp: project-plan — OB1 JSONL Watcher — Ambient Capture from Claude Code Sessions
This commit is contained in:
Lovebug MCP
parent
b168f1cc07
commit
9bff04eebf
@ -0,0 +1,159 @@
|
|||||||
|
---
|
||||||
|
created: '2026-05-14'
|
||||||
|
path: Sources/Homelab
|
||||||
|
project: ob1-jsonl-watcher
|
||||||
|
status: active
|
||||||
|
tags:
|
||||||
|
- ob1
|
||||||
|
- claude-code
|
||||||
|
- automation
|
||||||
|
- python
|
||||||
|
- docker
|
||||||
|
- pgvector
|
||||||
|
type: project-plan
|
||||||
|
updated: '2026-05-14'
|
||||||
|
---
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Ambient capture of Claude Code and Cowork session conversations into OB1 (Open Brain), so Lovebug accumulates a searchable corpus of past work without explicit `add_thought` calls. Captures should happen passively, "save everything and see what happens," with filtering tuned later from actual usage data rather than up-front guessing.
|
||||||
|
|
||||||
|
OB1 becomes the raw-corpus safety net under Trellis. Trellis stays as Lovebug-curated working memory (high signal, deliberate writes). OB1 becomes the broader semantic search across everything that's been said in sessions (wider, noisier, eventually filtered). Two-tier memory model, not unified.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
**Capture surface: JSONL watcher only.**
|
||||||
|
The proxy/egress capture paths (LiteLLM, network mirror via Traefik, mitmproxy egress) were explored and ruled out:
|
||||||
|
- LiteLLM only captures traffic that opts into pointing at it. Claude Code, Cowork, and Claude Desktop use vendor endpoints directly and can't be retargeted without breaking auth. The capture surface LiteLLM does cover (scripts, n8n) is lower-signal.
|
||||||
|
- Traefik traffic mirroring is real and well-supported, but only mirrors traffic that flows through Traefik. Outbound vendor API traffic from herbydev doesn't traverse Traefik (it's outbound, not ingress).
|
||||||
|
- mitmproxy egress with TLS interception is mechanically possible but introduces a CA whose compromise would MITM any client trusting it, sits in a gray-zone TOS posture, and depends on Claude Code not pinning certificates. Not worth the operational burden for the same capture surface JSONL provides.
|
||||||
|
|
||||||
|
JSONL captures exactly what we care about (Claude Code + Cowork sessions on herbydev) by reading files those clients write for their own purposes. No certificates, no interception, no trust changes.
|
||||||
|
|
||||||
|
**Save-everything-first, filter-later.**
|
||||||
|
Initial deployment captures every meaningful turn pair. Structural filtering only (drop tool-call events, drop empty turns, drop pure-acknowledgment turns). No "is this insightful" scoring at capture time. Run for 2–3 weeks, observe what surfaces in searches, then add filter rules based on what's actually noise.
|
||||||
|
|
||||||
|
**Synchronous pipeline first, async split deferred.**
|
||||||
|
OB1's existing `capture_thought` MCP tool does embedding + metadata extraction + insert atomically. Daemon calls it synchronously per turn pair. Async split (separate workers for embedding and metadata) is deferred until the daemon shows meaningful lag during heavy session activity. Measure first, optimize after.
|
||||||
|
|
||||||
|
**Metadata extraction must be local before launch.**
|
||||||
|
OB1's upstream `extractMetadata` function calls OpenRouter (`gpt-4o-mini`) per capture. At ambient-capture volume this would silently bill OpenRouter for every turn. Must be swapped to a local Ollama chat model before the watcher goes live. `nomic-embed-text` (already deployed) handles embeddings; a separate chat model (`qwen2.5:3b` or `llama3.2:3b`) handles metadata.
|
||||||
|
|
||||||
|
**Turn-pair as the unit of capture.**
|
||||||
|
Each captured thought is one user turn + one assistant turn. The assistant turn is the `content` (semantically searchable). The user turn lives in metadata for context. Reasoning: a user question alone is rarely useful to retrieve; an assistant answer alone is missing the prompt that produced it. The pair is the unit of meaning.
|
||||||
|
|
||||||
|
**Capture daemon is dumb-as-rocks.**
|
||||||
|
Watcher tails files, parses JSONL, pairs turns, posts to OB1's `capture_thought` via MCP HTTP. No filtering logic beyond skipping structural noise (tool events, empty turns). No deduplication (OB1's content-hash idempotency handles it). No queueing (synchronous fire-and-forget).
|
||||||
|
|
||||||
|
**Tailscale-only.**
|
||||||
|
No public exposure. The daemon runs on herbydev, calls ob1-mcp over the Tailscale interface. Same posture as the rest of the OB1 deployment.
|
||||||
|
|
||||||
|
## Open Items
|
||||||
|
|
||||||
|
- [ ] Confirm exact Claude Code JSONL schema in the version currently deployed on herbydev (event types, field names, parentUuid chaining behavior, how Cowork's JSONL differs if at all)
|
||||||
|
- [ ] Decide on chat model for metadata extraction (`qwen2.5:3b` vs `llama3.2:3b`) — pull both, run a small test corpus through each, pick by JSON-validity rate and tag quality
|
||||||
|
- [ ] Confirm how Cowork session files differ from Claude Code's (path, format, naming) — assume same pattern from past memories but verify
|
||||||
|
- [ ] Decide observability story: structured JSON logs to stdout (captured by Docker) plus a small `/healthz` endpoint, or something more ambitious like a Streamlit-scrape pattern
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase 0 — Prerequisites (Lovebug owns)
|
||||||
|
|
||||||
|
Tasks:
|
||||||
|
- [ ] Pull chat model into Ollama on the box hosting OB1's Ollama instance (`qwen2.5:3b` and/or `llama3.2:3b`)
|
||||||
|
- [ ] Modify OB1's `server/index.ts`: replace OpenRouter `extractMetadata` call with Ollama equivalent. Same JSON-mode pattern (`format: "json"` parameter in Ollama). Keep the same JSON schema in the system prompt
|
||||||
|
- [ ] Restart ob1-mcp. Test manual `capture_thought` via curl, verify metadata extraction returns valid JSON with `type`, `topics`, `people`, `action_items`, `dates_mentioned` populated
|
||||||
|
- [ ] Measure: time per capture end-to-end (embedding + metadata + insert). Document the number so we know what "synchronous lag" looks like before adding ambient load
|
||||||
|
|
||||||
|
Stop condition: confirmed working capture via the now-fully-local pipeline. No OpenRouter calls.
|
||||||
|
|
||||||
|
### Phase 1 — Schema reconnaissance
|
||||||
|
|
||||||
|
Tasks:
|
||||||
|
- [ ] Locate Claude Code session JSONL files on herbydev (likely `~/.config/Claude/local-agent-mode-sessions/` based on past Dispatch debugging notes)
|
||||||
|
- [ ] Run a Claude Code session, then `tail -f` the JSONL while it's in progress. Catalog the event types observed: `user`, `assistant`, `tool_use`, `tool_result`, system messages
|
||||||
|
- [ ] Map the structure: how is a user turn distinguished from a tool_result event (both have `type: "user"` in some schemas)? Where does the actual content live? What's the timestamp format? How does `parentUuid` chain across turns?
|
||||||
|
- [ ] Identify Cowork session storage — same format/path or different?
|
||||||
|
- [ ] Document the pairing algorithm: "skip tool events, buffer most recent real user turn, emit pair on next real assistant turn"
|
||||||
|
|
||||||
|
Deliverable: a written schema reference (in this project's notes or a sibling reference page) that the daemon's code can be built against.
|
||||||
|
|
||||||
|
Stop condition: confident understanding of the file format. Don't write daemon code until this is solid — schema assumptions baked into untested code are the most expensive thing to unwind later.
|
||||||
|
|
||||||
|
### Phase 2 — Daemon implementation
|
||||||
|
|
||||||
|
Tasks:
|
||||||
|
- [ ] Scaffold `ob1-jsonl-watcher` Python project under `/opt/projects/docker/ob1-jsonl-watcher/` using `uv init`
|
||||||
|
- [ ] Dependencies: `watchdog` (file events), `httpx` (HTTP to OB1), `pyyaml` (config file), `structlog` (JSON logging)
|
||||||
|
- [ ] `compose.yml` with bind mount to `~/.config/Claude/` (read-only), Tailscale network access, restart policy
|
||||||
|
- [ ] Daemon entry point: walk the sessions directory, set up watchdog observers for new files and modifications, maintain per-session state (last user turn buffer, last processed line offset for resume-after-restart)
|
||||||
|
- [ ] Turn pairing logic: as new lines arrive, parse JSONL, skip tool events and empty turns, buffer user turns, emit pair-as-thought when assistant turn arrives
|
||||||
|
- [ ] OB1 client: POST to `ob1-mcp` `capture_thought` tool via MCP HTTP transport. Auth via `x-brain-key` header. `content` field gets the assistant turn; user turn and session metadata go in OB1's `metadata` jsonb via the call (note: OB1's `capture_thought` currently only accepts `content`; this may need an extension to accept supplementary metadata, or we accept that user-turn context lives only in the daemon's logs for now)
|
||||||
|
- [ ] Idempotency check: rely on OB1's content-hash dedup. If a daemon restart re-reads lines that were already captured, OB1 rejects the duplicate insert. Track per-file line offsets in a small state file to minimize re-reads
|
||||||
|
- [ ] Config file (`watcher.yml`): JSONL search paths, daemon poll interval, daemon enable/disable kill switch, structural filter rules (which event types to skip), OB1 endpoint URL and auth key
|
||||||
|
- [ ] Kill switch: env var `OB1_CAPTURE_ENABLED=false` makes the daemon log "skipped" for every event but write nothing
|
||||||
|
|
||||||
|
Stop condition: daemon runs against a known test session JSONL file, produces the expected captures in OB1, search returns them. Verified end-to-end before deploying as systemd-managed service.
|
||||||
|
|
||||||
|
### Phase 3 — Deploy and observe
|
||||||
|
|
||||||
|
Tasks:
|
||||||
|
- [ ] Bring up the daemon as a long-running service (systemd unit or docker compose with `restart: unless-stopped`)
|
||||||
|
- [ ] Verify it survives daemon restart, machine reboot, Ollama restart (with graceful retry on capture failures)
|
||||||
|
- [ ] Run for 2 weeks of normal Claude Code / Cowork usage
|
||||||
|
- [ ] Daily: spot-check OB1 via search and `list_thoughts` to see what's being captured. Note any obvious noise patterns
|
||||||
|
- [ ] Watch for daemon lag during heavy sessions: does the watcher fall behind realtime? Does Ollama queue up captures?
|
||||||
|
- [ ] Log volume: how many captures per day? How does that compare to expectations?
|
||||||
|
|
||||||
|
Stop condition: 2 weeks of data, plus a written summary of what surfaced (signal vs noise) and any pain points.
|
||||||
|
|
||||||
|
### Phase 4 — Filter tuning
|
||||||
|
|
||||||
|
Based on Phase 3 observations:
|
||||||
|
- [ ] Identify noise categories (rapid clarification chatter, code-block-only turns, ack-only turns, low-content turns)
|
||||||
|
- [ ] Add filter rules to `watcher.yml`. Rules should be inclusion-by-default — only rule out things observed to be clearly noisy
|
||||||
|
- [ ] Optional: introduce length thresholds, marker-based promotion (e.g., `[remember]` tag in turn elevates score), or first-and-last-turn-of-session bookending
|
||||||
|
- [ ] Re-deploy with new filter config, run another 2 weeks, iterate
|
||||||
|
|
||||||
|
Stop condition: subjective sense that search results are useful more often than not. Quantitative target is "I find what I'm looking for via OB1 semantic search at least once a week without trying."
|
||||||
|
|
||||||
|
### Phase 5 — Async pipeline split (conditional)
|
||||||
|
|
||||||
|
Only execute if Phase 3 observations show daemon lag is a real problem.
|
||||||
|
|
||||||
|
Tasks:
|
||||||
|
- [ ] Schema change: `ALTER TABLE thoughts ALTER COLUMN embedding DROP NOT NULL` (and same for metadata fields if NOT NULL)
|
||||||
|
- [ ] Daemon path becomes: POST raw `content` → OB1 inserts row with NULL embedding and `metadata: {pending: true}` → return immediately
|
||||||
|
- [ ] New worker: `ob1-enricher`. Polls for rows with `embedding IS NULL`, embeds via Ollama, updates row. Polls for rows with `metadata->>'pending' = 'true'`, extracts metadata via local chat model, updates row. Sleeps when no work
|
||||||
|
- [ ] Run enricher continuously or scheduled-during-idle, depending on observed load
|
||||||
|
|
||||||
|
Stop condition: daemon no longer lags under heavy session load.
|
||||||
|
|
||||||
|
### Phase 6 — Future capture sources (deferred)
|
||||||
|
|
||||||
|
Sources to add later, all feeding the same daemon-to-OB1 contract:
|
||||||
|
- [ ] Dispatch pre-compact hook (Phase 7 of original OB1 plan)
|
||||||
|
- [ ] LiteLLM proxy callback (if/when script/n8n traffic becomes worth capturing)
|
||||||
|
- [ ] n8n email-summary intake (parallel write to OB1 alongside the existing wiki write, per OB1 addendum Phase 5)
|
||||||
|
|
||||||
|
Out of scope for this project — just noting they slot into the same architecture.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
**Architecture summary.** Sources (JSONL watcher today, others later) post turn-pairs to OB1 via the existing `capture_thought` MCP tool. OB1 owns embedding, metadata extraction, and insert atomically. MCP server stays the single write path; daemon doesn't bypass it. This keeps OB1 as sole owner of its data shape and avoids re-implementing the capture pipeline in the watcher.
|
||||||
|
|
||||||
|
**Why the metadata extraction model swap is gating.** With OpenRouter still wired in upstream, every captured turn would cost real money and add network latency to a path that should be local-only. Confirming this swap is Phase 0's job, not something to discover mid-deployment.
|
||||||
|
|
||||||
|
**The "save everything" philosophy is load-bearing.** Filtering at capture time before we know retrieval patterns is guessing. The dataset is local, storage is cheap, embeddings are free on Ollama, and OB1's content-hash dedup means noise doesn't compound. The discipline is to resist designing the filter until Phase 3's data is in hand.
|
||||||
|
|
||||||
|
**Trellis stays unaffected.** Trellis is curated working memory, Lovebug decides what goes in. This watcher feeds OB1, the raw-corpus layer. Two parallel systems, neither replacing the other. Same framing as the wiki/OB1 split in the OB1 addendum.
|
||||||
|
|
||||||
|
**No deduplication in the daemon.** OB1's `thoughts` table enforces `UNIQUE(md5(content), user_id)` (per the original ob1-deployment plan). A daemon restart that re-reads already-captured lines results in rejected inserts at the database, not data corruption. Per-file offset tracking is for efficiency, not correctness.
|
||||||
|
|
||||||
|
**The 30-minute mitmproxy viability test.** Mentioned during planning. Not part of this project plan but worth doing once for closure: pull mitmproxy, run in transparent mode, see if Claude Code respects `HTTPS_PROXY` and `NODE_EXTRA_CA_CERTS`. If it works cleanly, the egress path remains technically viable for a future revisit (not as the primary capture, but as a complementary surface for non-JSONL traffic). If Claude Code pins certs, the door is closed and we never wonder again.
|
||||||
|
|
||||||
|
**References.**
|
||||||
|
- `ob1-main-deployment` (original OB1 project plan, phases 1-6)
|
||||||
|
- `ob1-deployment` (addendum, phase 4 deferral, phase 7 dispatch hook addition)
|
||||||
|
- `trellis-mcp` (sibling working-memory system, curated layer)
|
||||||
|
- OB1 upstream: `github.com/NateBJones-Projects/OB1` (server/index.ts is the canonical reference for the capture pipeline)
|
||||||
Loading…
Reference in New Issue
Block a user