From 659b329199d6b3c0a701bad60e89d7bf8568fdfc Mon Sep 17 00:00:00 2001 From: Lovebug MCP Date: Fri, 15 May 2026 23:00:14 +0000 Subject: [PATCH] =?UTF-8?q?mcp:=20project-plan=20=E2=80=94=20Postgres=20Co?= =?UTF-8?q?nsolidation=20&=20OB1=20Reflection=20Layer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...gres-consolidation-ob1-reflection-layer.md | 381 ++++++++++++++++++ 1 file changed, 381 insertions(+) create mode 100644 Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md diff --git a/Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md b/Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md new file mode 100644 index 0000000..ac61a73 --- /dev/null +++ b/Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md @@ -0,0 +1,381 @@ +--- +created: '2026-05-15' +path: Sources/Homelab +project: postgres-consolidation-reflection-layer +status: active +tags: +- ob1 +- mcp +- pgvector +- embeddings +- automation +- claude-code +- dispatch +type: project-plan +updated: '2026-05-15' +--- + +# Postgres Consolidation & OB1 Reflection Layer + +## Goal + +Two coupled outcomes: + +1. **Consolidate** the three project databases (`wiki`, `ob1`, `trellis`) on + the shared Postgres instance into a single database with three schemas, + each owned by a dedicated MCP service user. +2. **Build the OB1 reflection layer**: a two-layer model in `ob1` (captures + + reflections) with a daily cron job that synthesizes generalizable + lessons from agent session content, plus vault indexing so semantic + search spans vault notes, captures, and reflections uniformly. + +These are paired because the reflection loop is fundamentally cross-schema +(citing vault notes from captures from reflections in one query). The +consolidation is the foundation that makes the reflection design clean. + +## Locked Decisions + +### Database consolidation + +- One Postgres database, three schemas: `wiki`, `ob1`, `trellis`. +- Migration direction: **OB1's existing database is the target.** Add + `wiki` and `trellis` schemas to it. Preserves OB1's existing pgvector + setup if installed. +- Target database name, schema-prefix-rewrite details, and final role + names: deferred to execution. Pick deliberate names at that stage. +- Nothing else lives on the Postgres instance — old databases can be + dropped cleanly after verification. +- Downtime during cutover is acceptable. No parallel-run period needed. + +### MCP service users (conservative v1) + +- `herby_mcp` — owns `wiki.*`. No reads on `ob1` or `trellis`. +- `ob1_mcp` — owns `ob1.*`. **Reads** on `wiki.*` and `trellis.*` + (needed for reflection generation). +- `trellis_mcp` — owns `trellis.*`. No reads on `wiki` or `ob1`. +- `lovebug` — read-write across all three schemas (preserves current + broad-access behavior; tightening deferred). +- Each MCP server connects with its own credentials and its own + `search_path` so unqualified table references resolve to its own schema. +- Permission model revisits after reflection ships and real cross-schema + read patterns are observed. + +### Embeddings table + +- Shared `public.embeddings` table with pointers back to source rows: + + ``` + public.embeddings ( + id bigserial PRIMARY KEY, + source_schema text NOT NULL, + source_table text NOT NULL, + source_id bigint NOT NULL, + embedding vector() NOT NULL, + embedding_model text NOT NULL, + embedded_at timestamptz NOT NULL DEFAULT now(), + UNIQUE (source_schema, source_table, source_id, embedding_model) + ) + ``` + +- One table for all embeddings across all content types. Cross-source + semantic search becomes one `ORDER BY embedding <-> query_vec` query. +- The unique constraint allows running multiple embedding models in + parallel for experimentation later without conflicting rows. +- Orphan rows possible (no FK because `source_table` varies); cleaned + up by a periodic job (defer; not Phase 1 work). + +### Two-layer OB1 schema + +- `ob1.captures` — raw layer. Holds Dispatch transcripts, Claude Code + session content, claude.ai chat captures. Written by the capture path; + never rewritten. +- `ob1.reflections` — compiled layer. Holds synthesized lessons generated + by the daily reflection job. Structured object: lesson text, citations + (jsonb array of source pointers), tags, confidence score, generation + metadata (model, date, source window). +- Mirrors the wiki's Sources/Wiki two-layer model. Same uniform pattern + across both systems: raw layer is the substrate, compiled layer is + regenerable. +- Both layers embedded the same way via `public.embeddings`. Retrieval + can scope to one or both via `source_table` filter. + +### Reflection job + +- **Trigger:** daily cron, 3am local. Off-hours, low contention. +- **Inputs:** all three streams: + - Dispatch session transcripts (Phase 7 of original ob1 plan; pre-compact + hook target — may need to coordinate timing) + - Claude Code session logs from `~/.claude/projects/` (ob1-jsonl-watcher + project is the existing capture mechanism) + - claude.ai chat captures (new path — needs design; see Open Items) +- **Output:** structured reflection records: + + ``` + ob1.reflections ( + id bigserial PRIMARY KEY, + lesson text NOT NULL, + citations jsonb NOT NULL, -- [{schema, table, id, relevance}, ...] + tags text[], + confidence float, -- model-reported or heuristic + source_window tstzrange, -- what time window this synthesized + generated_at timestamptz NOT NULL DEFAULT now(), + generated_by text NOT NULL, -- model name + version + job_run_id uuid -- groups reflections from one run + ) + ``` + +- **Run-as identity:** `ob1_mcp` user (which has the reads it needs on + `wiki` and `trellis` under v1 permissions). +- **Idle behavior:** if a day has no significant activity, the job writes + no reflections and logs a no-op. No "no significant activity" placeholder + rows. + +### Embedding model + +- Whatever OB1 currently uses for its own captures. Vault indexing, + capture embedding, and reflection embedding all use the same model + so vectors live in the same space. +- Phase 0 confirms which model OB1 is on. +- Switching the model later is a deliberate full re-embed operation, not + a config flip. The `embedding_model` column on `public.embeddings` + supports incremental migration when that day comes. + +### Vault indexing + +- In scope for this plan. Without it, reflections can't cite vault notes + semantically. +- Backfill: one-time pass over all existing `wiki.notes` rows, embed + each, insert into `public.embeddings`. +- Incremental: `save_vault_note` in the herby MCP server updated to also + embed new content and upsert into `public.embeddings`. + +## Open Items + +- [ ] Final names (target database, schemas, MCP users) — picked at + execution time. +- [ ] pgvector status — Phase 0 must check `\dx` on each existing + database. If installed in OB1's database, no new install needed; if + absent, install during consolidation. +- [ ] Embedding model + dimension — inherited from whatever OB1 is using. + Phase 0 confirms. +- [ ] claude.ai chat capture path — no existing mechanism captures these. + Out of scope for initial reflection job inputs unless a capture path + lands first. If it doesn't, the job runs on Dispatch + Claude Code + streams only and chat captures get added later. **Flag for execution: + confirm with Travis whether to attempt a capture path in this plan or + explicitly defer.** +- [ ] Dispatch pre-compact hook timing — Phase 7 of the original ob1 + deployment plan owns this. If it hasn't shipped, the reflection job's + Dispatch input is empty until it does. **Worth confirming current state + before starting Phase 4 of this plan.** +- [ ] Reflection prompt design — what does the synthesizer actually do? + Read N captures + relevant vault notes from the day → extract + generalizable lessons → produce structured output. The prompt itself + is real work; placeholder for now. +- [ ] Noise filtering for captures — what gets included in the daily + reflection window? All captures? Captures above some signal threshold? + Excludes routine command output? **Design step in Phase 4.** +- [ ] Reflection deduplication — re-running the job over overlapping + windows could produce near-duplicate lessons. Strategy: confidence- + weighted merge, manual review, or first-write-wins? **Design step in + Phase 4.** + +## Phases + +### Phase 0 — Inspection and prep + +- [ ] Confirm Postgres instance hosts only `wiki`, `ob1`, `trellis` + (nothing else). +- [ ] `\dx` on each database — record which has pgvector installed. +- [ ] Inspect each database's schema: confirm table list, row counts, + any vector columns and their dimensions. +- [ ] Identify OB1's current embedding model (env var, code reference, + or configuration file). +- [ ] Inventory each MCP server's connection configuration: where + credentials live, how to update them. +- [ ] Confirm backup destination on the NAS is reachable over Tailscale + with sufficient space. +- [ ] Document current state in a session note before any changes. + +### Phase 1 — Backups and rollback prep + +- [ ] `pg_dumpall` of the full instance to NAS (full safety net). +- [ ] Per-database `pg_dump` of `wiki`, `ob1`, `trellis` to NAS + (granular restore targets). +- [ ] Verify dumps are readable (test restore to a scratch database). +- [ ] Write rollback runbook: how to restore from dumps and revert MCP + server connection strings if cutover fails. + +### Phase 2 — Consolidation + +- [ ] Pick final names: target database, schema names, MCP user names. +- [ ] If OB1's database is the target and is named appropriately, use + in place. Otherwise, rename or create new and restore OB1's content. +- [ ] Ensure pgvector is installed on the target database + (`CREATE EXTENSION IF NOT EXISTS vector`). +- [ ] Create `wiki` and `trellis` schemas in the target database. +- [ ] Move OB1's existing tables into the `ob1` schema if not already + namespaced (`ALTER TABLE ... SET SCHEMA ob1`). +- [ ] Restore `wiki` dump into the `wiki` schema using `pg_dump`'s + `--schema=wiki` retargeting or a sed pass over the dump SQL. +- [ ] Restore `trellis` dump into the `trellis` schema the same way. +- [ ] Create the three MCP service users. +- [ ] Grant ownership and permissions per the locked decisions. +- [ ] Set per-user `search_path` defaults. +- [ ] Verify row counts in each schema match pre-migration counts. + +### Phase 3 — MCP server updates + +- [ ] Update herby MCP server: new connection string (target DB, + `herby_mcp` user), confirm `search_path` resolution works without + qualifying table names. +- [ ] Update OB1's MCP server: same shape, `ob1_mcp` user. +- [ ] Update trellis MCP server: same shape, `trellis_mcp` user. +- [ ] Restart each MCP server, verify health, run one representative + tool from each (`list_vault_projects`, OB1 search, `list_threads`). +- [ ] **Positive isolation test:** confirm each MCP user is *rejected* + when attempting to write outside its owned schema. (E.g., `herby_mcp` + attempting `INSERT INTO ob1.captures` fails with permission error.) +- [ ] Update Lovebug's connection string if needed. + +### Phase 4 — OB1 reflection layer (foundation) + +- [ ] Create `ob1.reflections` table per the locked schema. +- [ ] Create `public.embeddings` table per the locked schema. +- [ ] Add the unique constraint on `(source_schema, source_table, + source_id, embedding_model)`. +- [ ] Add HNSW or IVFFlat index on `public.embeddings.embedding` + (model choice depends on row count and recall/speed trade — research + current pgvector defaults at execution time). +- [ ] Decide noise filtering rules for capture inclusion. +- [ ] Design the reflection prompt: input shape (captures from window + + vault context), output shape (structured object matching + `ob1.reflections`). +- [ ] Decide dedup strategy. + +### Phase 5 — Vault indexing + +- [ ] Confirm embedding model + dimension from Phase 0. +- [ ] Write backfill script: walks all `wiki.notes`, embeds each + (body + title + tag context — decide chunking), inserts into + `public.embeddings`. +- [ ] Run backfill, verify row count matches `wiki.notes` count. +- [ ] Update herby MCP server's `save_vault_note`: embed new content + and upsert into `public.embeddings` on every write. +- [ ] Verify: write a new vault note via MCP, confirm embedding row + appears in `public.embeddings` referencing the new note. + +### Phase 6 — Reflection job + +- [ ] Implement the reflection job (Python script + cron entry, or + systemd timer). +- [ ] Job runs as `ob1_mcp` user; connects with its credentials. +- [ ] Inputs: + - Captures from the last 24 hours (or configured window) + - Relevant vault notes pulled via semantic search against the day's + captures +- [ ] Generates reflection objects via the chosen model. +- [ ] Writes to `ob1.reflections`, embeds each, upserts to + `public.embeddings`. +- [ ] Schedule for 3am local. +- [ ] First-run validation: run manually, inspect output, iterate on + prompt before letting cron take over. + +### Phase 7 — End-to-end verification + +- [ ] Semantic search query that hits all three content types in one + result set: vault notes, captures, reflections. Example query: "what + do we know about VLAN tagging" should return relevant rows from + multiple sources. +- [ ] Reflection citations resolve correctly — pick a recent reflection, + follow each citation back to its source row, confirm the join works. +- [ ] Permission model holds under stress — try writes from each MCP + user into other schemas; all should fail. +- [ ] Lovebug can do its job — confirm Lovebug's read patterns still + work under the new connection. + +### Phase 8 — Cleanup + +- [ ] Drop the old `wiki` and `trellis` databases (only after multi-day + verification period — keep dumps regardless). +- [ ] Archive pre-migration dumps to long-term backup location on NAS. +- [ ] Update connection-string documentation (KeePassXC, any internal + docs). +- [ ] Write a session note capturing the final shape, role model, and + operational commands (per-schema backup, restore, etc.). + +## Notes + +### Why consolidate at all + +The three databases grew separately, not by deliberate isolation choice. +The reflection layer wants to cross all three constantly (cite vault notes +from captures from reflections). Cross-database queries in Postgres are +possible via FDW but every cross-DB query is a small tax — connection +management, transaction scope, FDW config. Schemas in one database make +cross-cutting queries free. + +### Why isolation by user, not by database + +You already separate concerns by *identity* across the stack — `travadmin` +vs `herbyadmin` on hosts, Authentik in front of MCP servers, GitHub +deploy keys scoped per machine. The pattern is consistent: separate the +*who*, not the *where*. Three MCP service users with schema-scoped +permissions matches that pattern. Postgres rejects writes outside an +MCP's owned schema at the database level — stronger than relying on the +MCP's code to behave. + +### Why two layers in OB1 + +Same reason the wiki has two layers. Mixing raw captures and synthesized +lessons in one table pollutes semantic neighbors and degrades retrieval +quality as the reflection corpus grows. The wiki design explicitly +rejected this; OB1 deserves the same discipline. + +### Why a shared embeddings table + +Reflection retrieval is fundamentally cross-source. The shared table +makes "find similar to X across everything" the default query path. +Per-table vector columns would force `UNION ALL` queries across every +content type — workable but easy to forget a table when a new content +type is added later. Shared table is more extensible. + +### Why vault indexing belongs in this plan + +Without it, reflections can only cite captures and reflections — never +vault notes. The "pull anything on LLM memory projects" use case that +motivated this whole design needs vault content searchable. Building +reflection first and indexing later means the reflection layer is half- +useful for weeks. + +### Why permissions are conservative now + +Easier to grant access later than to revoke it after agents have come +to expect it. Starting with only OB1 having cross-schema reads (the +minimum needed for reflection to work) and revisiting after real usage +tells us what we actually need versus what we thought we needed. + +### Risks + +- **Cutover failure**: mitigated by full dumps and rehearsed rollback. +- **Embedding model mismatch**: if Phase 0 reveals OB1 is on a model + that's expensive to re-embed at scale, the choice of staying vs. + switching becomes load-bearing. Currently small corpus means low cost + either way; flag if discovered. +- **Reflection prompt quality**: bad prompts produce noisy reflections + that pollute retrieval. Phase 6's first-run validation gate is the + protection — don't let cron take over until manual runs produce good + output. +- **Permission boundary holes**: positive isolation tests in Phase 3 are + the protection. Verify rejection, not just success. + +### What this plan does not do + +- Does not redesign the trellis or wiki MCP servers beyond connection + changes. +- Does not implement claude.ai chat capture (open item). +- Does not address the Dispatch pre-compact hook (lives in the original + ob1 deployment plan, Phase 7). +- Does not tighten Lovebug's permissions (deferred; natural follow-on). +- Does not move Postgres off the current host or change its operational + footprint beyond schema reorganization. \ No newline at end of file