Compare commits
No commits in common. "9bff04eebf2b9c5e3afb90e7538ab59c72cc7d03" and "7eb8d756d1228402eaf0eee302a865b4a07c1364" have entirely different histories.
9bff04eebf
...
7eb8d756d1
@ -1,46 +0,0 @@
|
|||||||
---
|
|
||||||
created: '2026-05-10'
|
|
||||||
path: Sources/Dev
|
|
||||||
project: dispatch-replacement
|
|
||||||
status: active
|
|
||||||
tags:
|
|
||||||
- dispatch
|
|
||||||
- claude-code
|
|
||||||
- mcp
|
|
||||||
- automation
|
|
||||||
type: session-notes
|
|
||||||
updated: '2026-05-10'
|
|
||||||
---
|
|
||||||
|
|
||||||
## Outcome
|
|
||||||
|
|
||||||
Worked through what a Dispatch replacement actually requires given the April 2026 Anthropic policy change blocking subscription auth in third-party tools. Landed on a two-layer architecture that splits dispatch (API-billed) from agent execution (CLI subprocess on subscription). Pi evaluated and set aside as a possible engine, not the harness frame. No code written this session — pure design.
|
|
||||||
|
|
||||||
## Topics Covered
|
|
||||||
|
|
||||||
- Pi as a candidate harness; what it actually does (terminal coding agent with session tree, extension-based context engineering, multi-provider) vs. what it doesn't (mobile interface, subscription auth post-April 2026)
|
|
||||||
- Anthropic's April 4, 2026 policy: subscription OAuth tokens restricted to first-party products only; Claude Code and Claude.ai exempt
|
|
||||||
- Why the policy exists: prompt-cache efficiency in first-party tools, telemetry, and protecting the customer relationship from harnesses that abstract Anthropic into swappable infrastructure
|
|
||||||
- Auth mechanism distinctions: OAuth tokens (in-process, restricted) vs. CLI subprocess (process boundary, harness never touches credentials) vs. Agent SDK (in-process, API-key-only by Anthropic policy)
|
|
||||||
- Prompt caching mechanics: automatic vs. explicit breakpoints, 0.1x cache reads, 1.25x cache writes, 5-minute default TTL, prefix-based hashing
|
|
||||||
- Caching gotcha for subprocess pattern: per-spawn dynamic headers in `claude -p` reduce cache hit rates vs. interactive sessions; long-lived `--input-format stream-json` recovers most of it
|
|
||||||
- Cost modeling: heavy daily API usage with good caching ≈ $300/month, light usage ≈ $60/month; subscription pricing assumes human-paced consumption that automation defeats
|
|
||||||
- Final architecture: dispatch layer (harness, API + caching) handles phone-driven conversation and long-term context; agent layer (CLI subprocess on subscription) handles task execution
|
|
||||||
|
|
||||||
## Key Learnings
|
|
||||||
|
|
||||||
- **Subscription pricing is structurally a different product from API pricing.** Subscriptions price for human typing speed; harnesses defeat that assumption regardless of caching efficiency. No clever workaround makes API billing match subscription cost at heavy use.
|
|
||||||
- **The subprocess vs. SDK distinction is meaningful, not just technical.** Subprocess keeps auth inside Claude Code's process boundary; the harness never possesses subscription credentials. SDK is in-process and explicitly API-key-only by Anthropic policy. The boundary maps to what Anthropic actually cares about.
|
|
||||||
- **Pi's value is in the engine layer, not the harness frame.** Strong session model, real context-injection hooks via extensions, MCP support — but no mobile interface and the OAuth path it advertises is now per-token billed for Anthropic. It's a candidate component, not the answer.
|
|
||||||
- **Two-layer split is naturally good engineering, not just a billing hack.** Dispatch wants long-lived state, harness-controlled context, structured tool use over a small token volume. Agent execution wants fresh per-task context, codebase awareness, heavy tool use within bursts. Different layers want different model interfaces; the auth/billing situation happens to align with that split.
|
|
||||||
- **Cache structure for the dispatch layer is textbook-friendly.** Stable system prompt → cached tool definitions → cached long-term context (wiki/OpenBrain) → conversation tail. High-90s hit rates achievable for personal-scale usage.
|
|
||||||
- **`list_vault_projects` MCP tool has a sort bug** — fails with `'<' not supported between instances of 'str' and 'datetime.date'` when frontmatter dates aren't consistently typed. Lovebug task.
|
|
||||||
|
|
||||||
## Follow-ons
|
|
||||||
|
|
||||||
- [ ] File Lovebug ticket for `list_vault_projects` date-sort bug
|
|
||||||
- [ ] Decide on mobile interface approach for dispatch layer (Telegram/Discord bot vs. simple web UI on `herbydev` over Tailscale vs. ntfy-style push)
|
|
||||||
- [ ] Sketch dispatch ↔ agent task-brief contract (what context gets passed when dispatch spawns Claude Code)
|
|
||||||
- [ ] Design the dispatch layer's prompt structure with explicit cache breakpoints (system → tools → long-term context → conversation tail)
|
|
||||||
- [ ] Decide how the dispatch layer reads from wiki/OpenBrain — direct MCP, fetched at turn time, or pre-assembled
|
|
||||||
- [ ] Revisit Pi-as-engine question once dispatch layer shape is concrete; may end up using its RPC mode rather than building from scratch
|
|
||||||
@ -1,94 +0,0 @@
|
|||||||
---
|
|
||||||
created: '2026-05-11'
|
|
||||||
path: Sources/Homelab
|
|
||||||
project: traefik-deployment
|
|
||||||
status: active
|
|
||||||
tags:
|
|
||||||
- homelab
|
|
||||||
- traefik
|
|
||||||
- tls
|
|
||||||
- cloudflare
|
|
||||||
- dns
|
|
||||||
- docker
|
|
||||||
type: session-notes
|
|
||||||
updated: '2026-05-11'
|
|
||||||
---
|
|
||||||
|
|
||||||
# Traefik — Phase 1 close-out + Phase 2 canary complete
|
|
||||||
|
|
||||||
## Outcome
|
|
||||||
|
|
||||||
Closed Phase 1.6 (wildcard cert) and shipped all of Phase 2 (canary, both
|
|
||||||
surfaces, dashboard lockdown) in one session. The full Traefik
|
|
||||||
architecture is now running end-to-end. Every service from here forward
|
|
||||||
inherits this pattern.
|
|
||||||
|
|
||||||
What's live:
|
|
||||||
|
|
||||||
- Traefik 3.7 on VM 102 (`10.0.21.181`), Docker + file + ACME providers
|
|
||||||
- LE prod wildcard `*.herbylab.dev` issued via Cloudflare DNS-01, in
|
|
||||||
`acme.json`, ~90 day expiry
|
|
||||||
- `whoami.herbylab.dev` reachable on both surfaces with real certs
|
|
||||||
- `traefik.herbylab.dev` — dashboard routed through Traefik itself
|
|
||||||
(TLS + basic auth), no insecure `:8080` exposed
|
|
||||||
- New `cloudflared` tunnel `traefik-vm` on the Traefik VM, parallel to
|
|
||||||
the existing `cloudflared-mcp` on HerbyDev (no migration yet)
|
|
||||||
|
|
||||||
## Topics Covered
|
|
||||||
|
|
||||||
- Traefik VM vs LXC clarification (it's VM 102, not an LXC — Knot is the
|
|
||||||
LXC at 103)
|
|
||||||
- Docker compose layout under `/opt/traefik/`, `/opt/whoami/`,
|
|
||||||
`/opt/cloudflared/` — Traefik infra + services as separate compose
|
|
||||||
projects sharing the external `traefik` Docker network
|
|
||||||
- Static + dynamic config split (`traefik.yml` + `dynamic/` file
|
|
||||||
provider directory)
|
|
||||||
- ACME staging → prod flip (wipe `acme.json`, remove `caServer`, recreate)
|
|
||||||
- Cross-host routing model: same-host services use Docker labels;
|
|
||||||
HerbyDev services will use file provider when migrated
|
|
||||||
- cloudflared token method, second tunnel running parallel to the
|
|
||||||
existing one
|
|
||||||
- Dashboard exposure tiers (private+basicauth vs Authentik SSO vs
|
|
||||||
disabled) — picked basicauth for now, Authentik comes with MCP migration
|
|
||||||
|
|
||||||
## Key Learnings
|
|
||||||
|
|
||||||
- **`.env` is read at compose command time, not container runtime.** An
|
|
||||||
empty `CF_DNS_API_TOKEN` produced a Cloudflare `6111: Invalid format
|
|
||||||
for Authorization header` error — the auth header was `Bearer ` with
|
|
||||||
nothing after it. Diagnostic: `docker compose exec traefik printenv
|
|
||||||
CF_DNS_API_TOKEN` shows what the container actually sees.
|
|
||||||
- **CF edge cert ≠ origin cert.** Public clients always see Cloudflare's
|
|
||||||
edge cert (Google Trust Services). The LE wildcard lives on Traefik
|
|
||||||
behind cloudflared and is never seen by public traffic. This is
|
|
||||||
expected and correct; tripped on it briefly when reading curl output.
|
|
||||||
- **cloudflared → `traefik:80` causes a redirect loop.** Traefik's `web`
|
|
||||||
entrypoint redirects HTTP→HTTPS, cloudflared forwards the 301 back to
|
|
||||||
the client, client loops. Fix: point cloudflared at `traefik:443` and
|
|
||||||
enable "No TLS Verify" on the CF tunnel public hostname (the cert is
|
|
||||||
for `whoami.herbylab.dev`, not `traefik`).
|
|
||||||
- **Tunnel != tunnel credentials.** A cloudflared instance owns one
|
|
||||||
tunnel. Cloning the existing setup means cloning the compose pattern,
|
|
||||||
not the credentials. New tunnel `traefik-vm` was created separately;
|
|
||||||
old `cloudflared-mcp` on HerbyDev continues running untouched.
|
|
||||||
- **Compose label escaping for bcrypt.** `$` is variable expansion in
|
|
||||||
compose. Every `$` in a bcrypt hash must be doubled to `$$` when used
|
|
||||||
inline in a label. Caught before deploy.
|
|
||||||
- **Capability-based naming pays off immediately.** `traefik.herbylab.dev`
|
|
||||||
for the dashboard, `whoami.herbylab.dev` for the canary — no host
|
|
||||||
infix, the names mean what they do.
|
|
||||||
|
|
||||||
## Follow-ons
|
|
||||||
|
|
||||||
- [ ] Snapshot VM 102 as `phase2-complete`
|
|
||||||
- [ ] Phase 3 — MCP migration (file provider, parallel-path cutover, new
|
|
||||||
tunnel hosts `mcp.herbylab.dev`)
|
|
||||||
- [ ] Decommission `cloudflared-mcp` on HerbyDev after MCP cutover verified
|
|
||||||
- [ ] Decide whoami fate (keep as permanent diagnostic vs tear down
|
|
||||||
after Phase 3)
|
|
||||||
- [ ] Migrate dashboard auth from basicauth to Authentik forward-auth
|
|
||||||
when Authentik is wired into Traefik (likely as part of MCP work)
|
|
||||||
- [ ] Knot zone file into git
|
|
||||||
- [ ] Ansible role for Knot primary
|
|
||||||
- [ ] Knot secondary on NAS
|
|
||||||
- [ ] Move Traefik VM to Lab VLAN (cosmetic, not blocking)
|
|
||||||
@ -1,174 +0,0 @@
|
|||||||
---
|
|
||||||
created: '2026-05-11'
|
|
||||||
path: Sources/Homelab
|
|
||||||
project: herbylab-project-management
|
|
||||||
status: active
|
|
||||||
tags:
|
|
||||||
- mcp
|
|
||||||
- automation
|
|
||||||
- n8n
|
|
||||||
- obsidian
|
|
||||||
type: project-plan
|
|
||||||
updated: '2026-05-11'
|
|
||||||
---
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Extend `herbylab` MCP to manage projects and notes as first-class artifacts backed by Postgres, with Trello as the visual projection. Vault remains the immutable content store in Phase 1; Postgres becomes sole source in a later phase.
|
|
||||||
|
|
||||||
Solves the pain of status hygiene and project visibility — 44+ vault entries, almost all flagged `active`, no aggregate view, no real query surface, frontmatter status edits feel wrong.
|
|
||||||
|
|
||||||
## Locked Decisions
|
|
||||||
|
|
||||||
**Three-layer model.** Mirrors Trellis:
|
|
||||||
- Vault md files — content, narrative, immutable. Feeds wiki-vault.
|
|
||||||
- Postgres (`herbylab` schema) — state, metadata, queryable index.
|
|
||||||
- Trello — visual projection of project state. Read-mostly.
|
|
||||||
|
|
||||||
**Single MCP, not split.** Project tools live alongside vault tools in `herbylab`. One connector, one tool surface.
|
|
||||||
|
|
||||||
**Polymorphic tool shape.** `create_artifact(kind, ...)` and `update_artifact(kind, slug, ...)` dispatch on `kind`. Adds future artifact types without growing the tool surface.
|
|
||||||
|
|
||||||
**Postgres co-located with Trellis.** Same instance, separate schema (`herbylab.*`). Trellis's schema-namespace pattern leaves room for this by design.
|
|
||||||
|
|
||||||
**Phase 1: dual-write, immutable entries.**
|
|
||||||
- `create_artifact` writes Postgres row + vault file atomically.
|
|
||||||
- No content updates via MCP — entries immutable once created.
|
|
||||||
- `update_artifact` only mutates metadata (status, tags, title, summary, domain).
|
|
||||||
- Vault stays the read surface for Obsidian and wiki-vault.
|
|
||||||
- New phases/addenda get new files, matching existing pattern (`ob1-deployment` + `ob1-main-deployment`).
|
|
||||||
|
|
||||||
**Phase N: vault retires, Postgres becomes sole source.**
|
|
||||||
- Viewer TBD — wiki-vault adaptation or new web UI served from Postgres.
|
|
||||||
- Out of scope for this plan.
|
|
||||||
|
|
||||||
**Source of truth for status: Postgres.** Status changes mid-conversation via `update_artifact` — never a file edit.
|
|
||||||
|
|
||||||
**Status propagates via NOTIFY → n8n → Trello.** Postgres trigger fires `pg_notify('project_changed', ...)` on insert/update. n8n LISTENs, projects to Trello.
|
|
||||||
|
|
||||||
**Frontmatter scope simplifies.** Vault frontmatter holds identity (`project`, `type`, `path`, `tags`, `created`). Status and `updated` leave frontmatter — Postgres concerns.
|
|
||||||
|
|
||||||
## Schema (Phase 1)
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE SCHEMA herbylab;
|
|
||||||
|
|
||||||
CREATE TYPE herbylab.artifact_kind AS ENUM ('note', 'project');
|
|
||||||
CREATE TYPE herbylab.project_status AS ENUM ('active', 'paused', 'blocked', 'completed', 'archived');
|
|
||||||
CREATE TYPE herbylab.project_domain AS ENUM ('homelab', 'dev', 'venture', 'reference');
|
|
||||||
|
|
||||||
CREATE TABLE herbylab.artifacts (
|
|
||||||
slug TEXT NOT NULL,
|
|
||||||
kind herbylab.artifact_kind NOT NULL,
|
|
||||||
title TEXT NOT NULL,
|
|
||||||
domain herbylab.project_domain,
|
|
||||||
tags TEXT[] DEFAULT '{}',
|
|
||||||
vault_path TEXT NOT NULL,
|
|
||||||
body TEXT NOT NULL,
|
|
||||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
||||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
||||||
PRIMARY KEY (kind, slug)
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE herbylab.projects (
|
|
||||||
slug TEXT PRIMARY KEY,
|
|
||||||
status herbylab.project_status NOT NULL DEFAULT 'active',
|
|
||||||
summary TEXT,
|
|
||||||
trello_card_id TEXT,
|
|
||||||
FOREIGN KEY (kind, slug) REFERENCES herbylab.artifacts (kind, slug)
|
|
||||||
DEFERRABLE INITIALLY DEFERRED
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE INDEX idx_artifacts_kind ON herbylab.artifacts(kind);
|
|
||||||
CREATE INDEX idx_artifacts_tags ON herbylab.artifacts USING GIN(tags);
|
|
||||||
CREATE INDEX idx_projects_status ON herbylab.projects(status);
|
|
||||||
```
|
|
||||||
|
|
||||||
`body` stores the markdown blob. Vault file holds the same content. Dual-write keeps them in lockstep; immutability eliminates drift risk.
|
|
||||||
|
|
||||||
## NOTIFY trigger
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE OR REPLACE FUNCTION herbylab.notify_project_change() RETURNS TRIGGER AS $$
|
|
||||||
DECLARE payload JSON;
|
|
||||||
BEGIN
|
|
||||||
payload := json_build_object(
|
|
||||||
'op', TG_OP,
|
|
||||||
'slug', COALESCE(NEW.slug, OLD.slug),
|
|
||||||
'status', COALESCE(NEW.status, OLD.status)
|
|
||||||
);
|
|
||||||
PERFORM pg_notify('project_changed', payload::text);
|
|
||||||
RETURN NEW;
|
|
||||||
END;
|
|
||||||
$$ LANGUAGE plpgsql;
|
|
||||||
|
|
||||||
CREATE TRIGGER projects_notify
|
|
||||||
AFTER INSERT OR UPDATE OR DELETE ON herbylab.projects
|
|
||||||
FOR EACH ROW EXECUTE FUNCTION herbylab.notify_project_change();
|
|
||||||
```
|
|
||||||
|
|
||||||
## Tool surface
|
|
||||||
|
|
||||||
**Vault tools (existing, unchanged):**
|
|
||||||
- `save_vault_note`
|
|
||||||
- `get_vault_project`
|
|
||||||
- `check_vault_file`
|
|
||||||
- `list_vault_projects`
|
|
||||||
- `get_vault_schema`
|
|
||||||
|
|
||||||
**Artifact tools (new):**
|
|
||||||
- `create_artifact(kind, slug, title, vault_path, content, domain?, tags?, summary?)` — writes Postgres row + vault file. For `kind="project"`, also inserts `projects` row with default `status='active'`.
|
|
||||||
- `update_artifact(kind, slug, **fields)` — mutates metadata only (status, title, tags, summary, domain). Never touches body or vault file.
|
|
||||||
- `get_artifact(kind, slug)` — returns row + linked file path.
|
|
||||||
- `list_artifacts(kind?, status?, domain?, tag?)` — filtered query.
|
|
||||||
|
|
||||||
`list_vault_projects` deprecates in favor of `list_artifacts` once cutover completes.
|
|
||||||
|
|
||||||
## Open Items
|
|
||||||
|
|
||||||
- [ ] Postgres deployment shape — depends on Trellis decision (LXC vs container)
|
|
||||||
- [ ] Migration tooling — match whatever Trellis lands on
|
|
||||||
- [ ] Trello board layout — single board with status lists vs board per domain
|
|
||||||
- [ ] Duplicate cleanup before backfill: `ob1-deployment` + `ob1-main-deployment`, `editor-stack-upgrade` (Homelab + Venture), `content-hub-phase5-architecture` + `-planning`
|
|
||||||
- [ ] Naming pin-down: memory has both `herbylab` and `herby-dev` — settle on one
|
|
||||||
|
|
||||||
## Phases
|
|
||||||
|
|
||||||
### Phase 1 — Infrastructure
|
|
||||||
- [ ] Postgres instance available with `herbylab` schema on the Trellis-shared instance
|
|
||||||
- [ ] Migration tooling chosen (match Trellis decision)
|
|
||||||
- [ ] Apply schema, types, indexes, NOTIFY trigger
|
|
||||||
|
|
||||||
### Phase 2 — MCP tools
|
|
||||||
- [ ] `create_artifact` (dual-write, transactional)
|
|
||||||
- [ ] `update_artifact` (metadata only)
|
|
||||||
- [ ] `get_artifact`, `list_artifacts` (read paths)
|
|
||||||
- [ ] Tag normalization mirroring Trellis pattern
|
|
||||||
|
|
||||||
### Phase 3 — Trello projection
|
|
||||||
- [ ] Trello board structure: lists for `active`, `paused`, `blocked`, `completed`. Labels for domain.
|
|
||||||
- [ ] n8n workflow: Postgres LISTEN node on `project_changed`
|
|
||||||
- [ ] Branch on op: INSERT → create card, UPDATE → move/edit card, DELETE → archive
|
|
||||||
- [ ] Store `trello_card_id` back to `projects` row on creation
|
|
||||||
|
|
||||||
### Phase 4 — Backfill and cutover
|
|
||||||
- [ ] Script walks existing vault project-plan files, populates `artifacts` + `projects` rows
|
|
||||||
- [ ] Status defaults to current frontmatter value (42 of 44 are `active`)
|
|
||||||
- [ ] Manual triage pass in Trello — drag to honest columns
|
|
||||||
- [ ] Strip `status` / `updated` from vault frontmatter going forward
|
|
||||||
- [ ] Backfill of existing frontmatter optional
|
|
||||||
|
|
||||||
### Phase N — Vault retirement
|
|
||||||
- Deferred. Revisit once Phase 1–4 are operational and read patterns are clearer.
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
**Why dual-write in Phase 1.** Keeps two readers (Obsidian, wiki-vault) functional during transition. Immutability of entries kills the main drift risk — entries land once, neither side updates content. Update tool only touches metadata, which lives only in Postgres.
|
|
||||||
|
|
||||||
**Why immutable entries.** Matches existing behavior — new phases get new files (`-phase2`, `-addendum`) rather than edits. Codifying this as a constraint removes a class of bugs and makes Phase N (Postgres-only) trivially achievable later.
|
|
||||||
|
|
||||||
**Why Postgres co-located with Trellis.** Trellis's plan explicitly chose schema-namespacing to leave room for consolidation. Same instance, separate schemas, no new infrastructure. Backup/recovery story is shared.
|
|
||||||
|
|
||||||
**Why not Trellis itself for project tracking.** Trellis is for conversation threads — coarse session-level checkpoints, append-only events, paused/resumed work. Projects are different: durable, named, status-bearing, visualized externally. Different lifecycle, different reader, different surface.
|
|
||||||
|
|
||||||
**Why the polymorphic tool shape.** Avoids tool sprawl. Adding a future artifact type (task, decision log, etc.) is a `kind` value, not a new MCP tool. Tradeoff is a fatter schema in the tool description.
|
|
||||||
@ -1,159 +0,0 @@
|
|||||||
---
|
|
||||||
created: '2026-05-14'
|
|
||||||
path: Sources/Homelab
|
|
||||||
project: ob1-jsonl-watcher
|
|
||||||
status: active
|
|
||||||
tags:
|
|
||||||
- ob1
|
|
||||||
- claude-code
|
|
||||||
- automation
|
|
||||||
- python
|
|
||||||
- docker
|
|
||||||
- pgvector
|
|
||||||
type: project-plan
|
|
||||||
updated: '2026-05-14'
|
|
||||||
---
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Ambient capture of Claude Code and Cowork session conversations into OB1 (Open Brain), so Lovebug accumulates a searchable corpus of past work without explicit `add_thought` calls. Captures should happen passively, "save everything and see what happens," with filtering tuned later from actual usage data rather than up-front guessing.
|
|
||||||
|
|
||||||
OB1 becomes the raw-corpus safety net under Trellis. Trellis stays as Lovebug-curated working memory (high signal, deliberate writes). OB1 becomes the broader semantic search across everything that's been said in sessions (wider, noisier, eventually filtered). Two-tier memory model, not unified.
|
|
||||||
|
|
||||||
## Locked Decisions
|
|
||||||
|
|
||||||
**Capture surface: JSONL watcher only.**
|
|
||||||
The proxy/egress capture paths (LiteLLM, network mirror via Traefik, mitmproxy egress) were explored and ruled out:
|
|
||||||
- LiteLLM only captures traffic that opts into pointing at it. Claude Code, Cowork, and Claude Desktop use vendor endpoints directly and can't be retargeted without breaking auth. The capture surface LiteLLM does cover (scripts, n8n) is lower-signal.
|
|
||||||
- Traefik traffic mirroring is real and well-supported, but only mirrors traffic that flows through Traefik. Outbound vendor API traffic from herbydev doesn't traverse Traefik (it's outbound, not ingress).
|
|
||||||
- mitmproxy egress with TLS interception is mechanically possible but introduces a CA whose compromise would MITM any client trusting it, sits in a gray-zone TOS posture, and depends on Claude Code not pinning certificates. Not worth the operational burden for the same capture surface JSONL provides.
|
|
||||||
|
|
||||||
JSONL captures exactly what we care about (Claude Code + Cowork sessions on herbydev) by reading files those clients write for their own purposes. No certificates, no interception, no trust changes.
|
|
||||||
|
|
||||||
**Save-everything-first, filter-later.**
|
|
||||||
Initial deployment captures every meaningful turn pair. Structural filtering only (drop tool-call events, drop empty turns, drop pure-acknowledgment turns). No "is this insightful" scoring at capture time. Run for 2–3 weeks, observe what surfaces in searches, then add filter rules based on what's actually noise.
|
|
||||||
|
|
||||||
**Synchronous pipeline first, async split deferred.**
|
|
||||||
OB1's existing `capture_thought` MCP tool does embedding + metadata extraction + insert atomically. Daemon calls it synchronously per turn pair. Async split (separate workers for embedding and metadata) is deferred until the daemon shows meaningful lag during heavy session activity. Measure first, optimize after.
|
|
||||||
|
|
||||||
**Metadata extraction must be local before launch.**
|
|
||||||
OB1's upstream `extractMetadata` function calls OpenRouter (`gpt-4o-mini`) per capture. At ambient-capture volume this would silently bill OpenRouter for every turn. Must be swapped to a local Ollama chat model before the watcher goes live. `nomic-embed-text` (already deployed) handles embeddings; a separate chat model (`qwen2.5:3b` or `llama3.2:3b`) handles metadata.
|
|
||||||
|
|
||||||
**Turn-pair as the unit of capture.**
|
|
||||||
Each captured thought is one user turn + one assistant turn. The assistant turn is the `content` (semantically searchable). The user turn lives in metadata for context. Reasoning: a user question alone is rarely useful to retrieve; an assistant answer alone is missing the prompt that produced it. The pair is the unit of meaning.
|
|
||||||
|
|
||||||
**Capture daemon is dumb-as-rocks.**
|
|
||||||
Watcher tails files, parses JSONL, pairs turns, posts to OB1's `capture_thought` via MCP HTTP. No filtering logic beyond skipping structural noise (tool events, empty turns). No deduplication (OB1's content-hash idempotency handles it). No queueing (synchronous fire-and-forget).
|
|
||||||
|
|
||||||
**Tailscale-only.**
|
|
||||||
No public exposure. The daemon runs on herbydev, calls ob1-mcp over the Tailscale interface. Same posture as the rest of the OB1 deployment.
|
|
||||||
|
|
||||||
## Open Items
|
|
||||||
|
|
||||||
- [ ] Confirm exact Claude Code JSONL schema in the version currently deployed on herbydev (event types, field names, parentUuid chaining behavior, how Cowork's JSONL differs if at all)
|
|
||||||
- [ ] Decide on chat model for metadata extraction (`qwen2.5:3b` vs `llama3.2:3b`) — pull both, run a small test corpus through each, pick by JSON-validity rate and tag quality
|
|
||||||
- [ ] Confirm how Cowork session files differ from Claude Code's (path, format, naming) — assume same pattern from past memories but verify
|
|
||||||
- [ ] Decide observability story: structured JSON logs to stdout (captured by Docker) plus a small `/healthz` endpoint, or something more ambitious like a Streamlit-scrape pattern
|
|
||||||
|
|
||||||
## Phases
|
|
||||||
|
|
||||||
### Phase 0 — Prerequisites (Lovebug owns)
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
- [ ] Pull chat model into Ollama on the box hosting OB1's Ollama instance (`qwen2.5:3b` and/or `llama3.2:3b`)
|
|
||||||
- [ ] Modify OB1's `server/index.ts`: replace OpenRouter `extractMetadata` call with Ollama equivalent. Same JSON-mode pattern (`format: "json"` parameter in Ollama). Keep the same JSON schema in the system prompt
|
|
||||||
- [ ] Restart ob1-mcp. Test manual `capture_thought` via curl, verify metadata extraction returns valid JSON with `type`, `topics`, `people`, `action_items`, `dates_mentioned` populated
|
|
||||||
- [ ] Measure: time per capture end-to-end (embedding + metadata + insert). Document the number so we know what "synchronous lag" looks like before adding ambient load
|
|
||||||
|
|
||||||
Stop condition: confirmed working capture via the now-fully-local pipeline. No OpenRouter calls.
|
|
||||||
|
|
||||||
### Phase 1 — Schema reconnaissance
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
- [ ] Locate Claude Code session JSONL files on herbydev (likely `~/.config/Claude/local-agent-mode-sessions/` based on past Dispatch debugging notes)
|
|
||||||
- [ ] Run a Claude Code session, then `tail -f` the JSONL while it's in progress. Catalog the event types observed: `user`, `assistant`, `tool_use`, `tool_result`, system messages
|
|
||||||
- [ ] Map the structure: how is a user turn distinguished from a tool_result event (both have `type: "user"` in some schemas)? Where does the actual content live? What's the timestamp format? How does `parentUuid` chain across turns?
|
|
||||||
- [ ] Identify Cowork session storage — same format/path or different?
|
|
||||||
- [ ] Document the pairing algorithm: "skip tool events, buffer most recent real user turn, emit pair on next real assistant turn"
|
|
||||||
|
|
||||||
Deliverable: a written schema reference (in this project's notes or a sibling reference page) that the daemon's code can be built against.
|
|
||||||
|
|
||||||
Stop condition: confident understanding of the file format. Don't write daemon code until this is solid — schema assumptions baked into untested code are the most expensive thing to unwind later.
|
|
||||||
|
|
||||||
### Phase 2 — Daemon implementation
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
- [ ] Scaffold `ob1-jsonl-watcher` Python project under `/opt/projects/docker/ob1-jsonl-watcher/` using `uv init`
|
|
||||||
- [ ] Dependencies: `watchdog` (file events), `httpx` (HTTP to OB1), `pyyaml` (config file), `structlog` (JSON logging)
|
|
||||||
- [ ] `compose.yml` with bind mount to `~/.config/Claude/` (read-only), Tailscale network access, restart policy
|
|
||||||
- [ ] Daemon entry point: walk the sessions directory, set up watchdog observers for new files and modifications, maintain per-session state (last user turn buffer, last processed line offset for resume-after-restart)
|
|
||||||
- [ ] Turn pairing logic: as new lines arrive, parse JSONL, skip tool events and empty turns, buffer user turns, emit pair-as-thought when assistant turn arrives
|
|
||||||
- [ ] OB1 client: POST to `ob1-mcp` `capture_thought` tool via MCP HTTP transport. Auth via `x-brain-key` header. `content` field gets the assistant turn; user turn and session metadata go in OB1's `metadata` jsonb via the call (note: OB1's `capture_thought` currently only accepts `content`; this may need an extension to accept supplementary metadata, or we accept that user-turn context lives only in the daemon's logs for now)
|
|
||||||
- [ ] Idempotency check: rely on OB1's content-hash dedup. If a daemon restart re-reads lines that were already captured, OB1 rejects the duplicate insert. Track per-file line offsets in a small state file to minimize re-reads
|
|
||||||
- [ ] Config file (`watcher.yml`): JSONL search paths, daemon poll interval, daemon enable/disable kill switch, structural filter rules (which event types to skip), OB1 endpoint URL and auth key
|
|
||||||
- [ ] Kill switch: env var `OB1_CAPTURE_ENABLED=false` makes the daemon log "skipped" for every event but write nothing
|
|
||||||
|
|
||||||
Stop condition: daemon runs against a known test session JSONL file, produces the expected captures in OB1, search returns them. Verified end-to-end before deploying as systemd-managed service.
|
|
||||||
|
|
||||||
### Phase 3 — Deploy and observe
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
- [ ] Bring up the daemon as a long-running service (systemd unit or docker compose with `restart: unless-stopped`)
|
|
||||||
- [ ] Verify it survives daemon restart, machine reboot, Ollama restart (with graceful retry on capture failures)
|
|
||||||
- [ ] Run for 2 weeks of normal Claude Code / Cowork usage
|
|
||||||
- [ ] Daily: spot-check OB1 via search and `list_thoughts` to see what's being captured. Note any obvious noise patterns
|
|
||||||
- [ ] Watch for daemon lag during heavy sessions: does the watcher fall behind realtime? Does Ollama queue up captures?
|
|
||||||
- [ ] Log volume: how many captures per day? How does that compare to expectations?
|
|
||||||
|
|
||||||
Stop condition: 2 weeks of data, plus a written summary of what surfaced (signal vs noise) and any pain points.
|
|
||||||
|
|
||||||
### Phase 4 — Filter tuning
|
|
||||||
|
|
||||||
Based on Phase 3 observations:
|
|
||||||
- [ ] Identify noise categories (rapid clarification chatter, code-block-only turns, ack-only turns, low-content turns)
|
|
||||||
- [ ] Add filter rules to `watcher.yml`. Rules should be inclusion-by-default — only rule out things observed to be clearly noisy
|
|
||||||
- [ ] Optional: introduce length thresholds, marker-based promotion (e.g., `[remember]` tag in turn elevates score), or first-and-last-turn-of-session bookending
|
|
||||||
- [ ] Re-deploy with new filter config, run another 2 weeks, iterate
|
|
||||||
|
|
||||||
Stop condition: subjective sense that search results are useful more often than not. Quantitative target is "I find what I'm looking for via OB1 semantic search at least once a week without trying."
|
|
||||||
|
|
||||||
### Phase 5 — Async pipeline split (conditional)
|
|
||||||
|
|
||||||
Only execute if Phase 3 observations show daemon lag is a real problem.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
- [ ] Schema change: `ALTER TABLE thoughts ALTER COLUMN embedding DROP NOT NULL` (and same for metadata fields if NOT NULL)
|
|
||||||
- [ ] Daemon path becomes: POST raw `content` → OB1 inserts row with NULL embedding and `metadata: {pending: true}` → return immediately
|
|
||||||
- [ ] New worker: `ob1-enricher`. Polls for rows with `embedding IS NULL`, embeds via Ollama, updates row. Polls for rows with `metadata->>'pending' = 'true'`, extracts metadata via local chat model, updates row. Sleeps when no work
|
|
||||||
- [ ] Run enricher continuously or scheduled-during-idle, depending on observed load
|
|
||||||
|
|
||||||
Stop condition: daemon no longer lags under heavy session load.
|
|
||||||
|
|
||||||
### Phase 6 — Future capture sources (deferred)
|
|
||||||
|
|
||||||
Sources to add later, all feeding the same daemon-to-OB1 contract:
|
|
||||||
- [ ] Dispatch pre-compact hook (Phase 7 of original OB1 plan)
|
|
||||||
- [ ] LiteLLM proxy callback (if/when script/n8n traffic becomes worth capturing)
|
|
||||||
- [ ] n8n email-summary intake (parallel write to OB1 alongside the existing wiki write, per OB1 addendum Phase 5)
|
|
||||||
|
|
||||||
Out of scope for this project — just noting they slot into the same architecture.
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
**Architecture summary.** Sources (JSONL watcher today, others later) post turn-pairs to OB1 via the existing `capture_thought` MCP tool. OB1 owns embedding, metadata extraction, and insert atomically. MCP server stays the single write path; daemon doesn't bypass it. This keeps OB1 as sole owner of its data shape and avoids re-implementing the capture pipeline in the watcher.
|
|
||||||
|
|
||||||
**Why the metadata extraction model swap is gating.** With OpenRouter still wired in upstream, every captured turn would cost real money and add network latency to a path that should be local-only. Confirming this swap is Phase 0's job, not something to discover mid-deployment.
|
|
||||||
|
|
||||||
**The "save everything" philosophy is load-bearing.** Filtering at capture time before we know retrieval patterns is guessing. The dataset is local, storage is cheap, embeddings are free on Ollama, and OB1's content-hash dedup means noise doesn't compound. The discipline is to resist designing the filter until Phase 3's data is in hand.
|
|
||||||
|
|
||||||
**Trellis stays unaffected.** Trellis is curated working memory, Lovebug decides what goes in. This watcher feeds OB1, the raw-corpus layer. Two parallel systems, neither replacing the other. Same framing as the wiki/OB1 split in the OB1 addendum.
|
|
||||||
|
|
||||||
**No deduplication in the daemon.** OB1's `thoughts` table enforces `UNIQUE(md5(content), user_id)` (per the original ob1-deployment plan). A daemon restart that re-reads already-captured lines results in rejected inserts at the database, not data corruption. Per-file offset tracking is for efficiency, not correctness.
|
|
||||||
|
|
||||||
**The 30-minute mitmproxy viability test.** Mentioned during planning. Not part of this project plan but worth doing once for closure: pull mitmproxy, run in transparent mode, see if Claude Code respects `HTTPS_PROXY` and `NODE_EXTRA_CA_CERTS`. If it works cleanly, the egress path remains technically viable for a future revisit (not as the primary capture, but as a complementary surface for non-JSONL traffic). If Claude Code pins certs, the door is closed and we never wonder again.
|
|
||||||
|
|
||||||
**References.**
|
|
||||||
- `ob1-main-deployment` (original OB1 project plan, phases 1-6)
|
|
||||||
- `ob1-deployment` (addendum, phase 4 deferral, phase 7 dispatch hook addition)
|
|
||||||
- `trellis-mcp` (sibling working-memory system, curated layer)
|
|
||||||
- OB1 upstream: `github.com/NateBJones-Projects/OB1` (server/index.ts is the canonical reference for the capture pipeline)
|
|
||||||
@ -1,115 +0,0 @@
|
|||||||
---
|
|
||||||
created: '2026-05-10'
|
|
||||||
path: Sources/Homelab
|
|
||||||
project: trellis-mcp
|
|
||||||
status: active
|
|
||||||
tags:
|
|
||||||
- mcp
|
|
||||||
- claude-code
|
|
||||||
- lovebug
|
|
||||||
type: project-plan
|
|
||||||
updated: '2026-05-10'
|
|
||||||
---
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Build an MCP server (`trellis`) that captures and resumes long-running threads of work across artificial conversation boundaries. The unit is a *thread* — a logical work effort that may span multiple chat sessions over days or weeks. Pause/resume verbs let you switch contexts without losing state.
|
|
||||||
|
|
||||||
Solves the "where exactly did we leave the Traefik debugging" problem. Memory is too lossy; chat history finds old conversations but doesn't restore working state. Trellis sits between them: synthesized state per thread, on demand.
|
|
||||||
|
|
||||||
## Locked Decisions
|
|
||||||
|
|
||||||
**Three-layer model** — keep these separate, none should swallow another's job:
|
|
||||||
- CLAUDE.md / LOVEBUG.md = instructions (how to behave)
|
|
||||||
- Trellis threads = record (where we are in the doing)
|
|
||||||
- Memory / vault = lessons and outcomes (what we learned, what we built)
|
|
||||||
|
|
||||||
**Naming** — `trellis`. Garden-themed, fits the brand world, metaphor maps cleanly (persistent structure things grow on, you keep returning to it).
|
|
||||||
|
|
||||||
**Deployment** — separate MCP server, not folded into herby-dev. Different scope: herby-dev is your knowledge management, trellis is cross-cutting infrastructure usable from any agent context. Same deployment pattern as herby-dev: Cloudflare Tunnel, FastMCP, Authentik in front. Hostname `trellis.herbylab.dev`.
|
|
||||||
|
|
||||||
**Storage** — Postgres on the existing instance. Database `trellis`, schema `trellis`. Schema namespace makes future consolidation into a single database trivial.
|
|
||||||
|
|
||||||
**Schema shape** — four tables, append-only events log:
|
|
||||||
- `threads` — one row per named thread (slug, topic, goal, status, parent_id for forks)
|
|
||||||
- `tags` — separate table with DB-level normalization (lowercase, dash-separated, regex-checked)
|
|
||||||
- `thread_tags` — join table
|
|
||||||
- `events` — append-only log (started, checkpoint, status_change, fork, metadata_change)
|
|
||||||
|
|
||||||
Checkpoint payload is freeform markdown. The agent synthesizes whatever shape fits the thread (current state, decisions, open questions, next steps) — structure not enforced at the DB level.
|
|
||||||
|
|
||||||
**Tool surface (v1)** — seven tools, all suffixed with `_thread`:
|
|
||||||
- `start_thread` — creates row, writes `started` event
|
|
||||||
- `pause_thread` — writes `checkpoint` event, requires state payload
|
|
||||||
- `resume_thread` — reads thread + latest checkpoint (no write)
|
|
||||||
- `get_thread` — read-only inspection, optional full event history
|
|
||||||
- `list_threads` — filtered summary view
|
|
||||||
- `fork_thread` — new thread row with parent_id, fork event on parent
|
|
||||||
- `update_thread` — mutates thread fields, writes metadata_change/status_change event
|
|
||||||
|
|
||||||
**Lifecycle is prompt-driven** — system does nothing autonomously. No background jobs, no auto-abandon, no reminders. Threads exist and persist until you explicitly act on them.
|
|
||||||
|
|
||||||
**Active context lives in the harness, not the DB** — pause is a signal to the harness ("save and switch"); the DB just stores the trail. One active thread at a time in any given chat session is the harness's concern; the DB doesn't care how many threads are in flight.
|
|
||||||
|
|
||||||
## Open Items
|
|
||||||
|
|
||||||
- [ ] Postgres deployment shape — LXC vs Docker, fresh instance vs reusing existing
|
|
||||||
- [ ] Migration tooling — Alembic, plain SQL files, or simpler
|
|
||||||
- [ ] Tunnel + Authentik setup for `trellis.herbylab.dev` (parallels herby-dev work)
|
|
||||||
- [ ] Server framework — presumably FastMCP + Python to match herby-dev, confirm at build time
|
|
||||||
- [ ] First-run question: when trellis is operational, the trellis build itself becomes the first thread; paste design state from this project plan as the initial checkpoint
|
|
||||||
|
|
||||||
## Phases
|
|
||||||
|
|
||||||
### Phase 1 — Infrastructure
|
|
||||||
|
|
||||||
- [ ] Postgres instance available with `trellis` database and `trellis` schema
|
|
||||||
- [ ] Cloudflare Tunnel hostname `trellis.herbylab.dev`
|
|
||||||
- [ ] Authentik application + provider for trellis
|
|
||||||
- [ ] Repo `trellis-mcp` scaffolded
|
|
||||||
|
|
||||||
### Phase 2 — Schema and migrations
|
|
||||||
|
|
||||||
- [ ] Apply v1 schema (threads, tags, thread_tags, events)
|
|
||||||
- [ ] Verify CHECK constraints (status enum, tag normalization regex)
|
|
||||||
- [ ] Verify GIN/btree indexes resolve the resume-by-slug hot path
|
|
||||||
|
|
||||||
### Phase 3 — Tool implementation
|
|
||||||
|
|
||||||
- [ ] `start_thread`, `pause_thread`, `resume_thread` (the core loop)
|
|
||||||
- [ ] `get_thread`, `list_threads` (read paths)
|
|
||||||
- [ ] `fork_thread`, `update_thread` (mutation paths)
|
|
||||||
- [ ] Tag normalization in tool layer with clear errors before DB constraint fires
|
|
||||||
|
|
||||||
### Phase 4 — Wire-up and first use
|
|
||||||
|
|
||||||
- [ ] Connect from desktop Claude, mobile Claude, Claude Code, Lovebug
|
|
||||||
- [ ] Bootstrap: file the trellis build itself as the first thread
|
|
||||||
- [ ] Iterate based on real use — payload shape, missing affordances, etc.
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
**Deferred to v2:**
|
|
||||||
|
|
||||||
- Multi-agent / non-Claude support
|
|
||||||
- Auto-thread for unnamed conversations
|
|
||||||
- Auto-status changes (abandon after N days, surfacing stale threads, etc.)
|
|
||||||
- Read patterns beyond resume-by-name: date ranges, full-text search, decision timelines, fork trees
|
|
||||||
- Fork merging
|
|
||||||
- Soft delete / archival
|
|
||||||
- Tag metadata (description, color)
|
|
||||||
- Transcript path as a real column (Claude Code transcripts at `~/.claude/projects/`)
|
|
||||||
- A `resumed` event kind if telemetry on resumes proves useful
|
|
||||||
|
|
||||||
**Design principles that drove the shape:**
|
|
||||||
|
|
||||||
- *Append-only in spirit* — historical information isn't adjusted, only added to. Even fields that look mutable (current state, next steps) are superseded by newer events rather than rewritten. The thread row reflects current state; events tell you how it got there.
|
|
||||||
- *Coarse event grain* — about session-level state ("we paused here, this is what we knew"), not per-interaction ("decision added"). The unit is the checkpoint, not the keystroke.
|
|
||||||
- *Pause = save-and-switch action, not a status* — every thread that isn't actively being worked on is implicitly paused. No separate "paused" status needed.
|
|
||||||
- *Trigger is naming* — short topical Q&A doesn't need a thread. Lifting to trellis happens when you explicitly say "start a thread called X."
|
|
||||||
|
|
||||||
**Schema-namespacing rationale** — putting trellis tables in a `trellis` schema (even inside its own database) costs nothing now and makes any future merge into another database a no-op. Connection sets `search_path` so queries don't qualify table names.
|
|
||||||
|
|
||||||
**Why two MCPs instead of one** — herby-dev is your knowledge management, scoped to your homelab. Trellis is cross-cutting infrastructure usable from any agent context. Folding them couples unrelated concerns; deployment overhead of a second MCP is low given herby-dev established the pattern.
|
|
||||||
|
|
||||||
**Why separate tags table** — preference for normalization upfront. Array column would work for v1 but you know you'll end up at a join table eventually, so start there.
|
|
||||||
Loading…
Reference in New Issue
Block a user