wiki-vault/Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md

created	path	project	status	tags	type	updated
2026-05-15	Sources/Homelab	postgres-consolidation-reflection-layer	active	ob1 mcp pgvector embeddings automation claude-code dispatch	project-plan	2026-05-15

Postgres Consolidation & OB1 Reflection Layer

Goal

Two coupled outcomes:

1. Consolidate the three project databases (wiki, ob1, trellis) on the shared Postgres instance into a single database with three schemas, each owned by a dedicated MCP service user.
2. Build the OB1 reflection layer: a two-layer model in ob1 (captures
   • reflections) with a daily cron job that synthesizes generalizable lessons from agent session content, plus vault indexing so semantic search spans vault notes, captures, and reflections uniformly.

These are paired because the reflection loop is fundamentally cross-schema (citing vault notes from captures from reflections in one query). The consolidation is the foundation that makes the reflection design clean.

Locked Decisions

Database consolidation

• One Postgres database, three schemas: wiki, ob1, trellis.
• Migration direction: OB1's existing database is the target. Add wiki and trellis schemas to it. Preserves OB1's existing pgvector setup if installed.
• Target database name, schema-prefix-rewrite details, and final role names: deferred to execution. Pick deliberate names at that stage.
• Nothing else lives on the Postgres instance — old databases can be dropped cleanly after verification.
• Downtime during cutover is acceptable. No parallel-run period needed.

MCP service users (conservative v1)

• herby_mcp — owns wiki.*. No reads on ob1 or trellis.
• ob1_mcp — owns ob1.*. Reads on wiki.* and trellis.* (needed for reflection generation).
• trellis_mcp — owns trellis.*. No reads on wiki or ob1.
• lovebug — read-write across all three schemas (preserves current broad-access behavior; tightening deferred).
• Each MCP server connects with its own credentials and its own search_path so unqualified table references resolve to its own schema.
• Permission model revisits after reflection ships and real cross-schema read patterns are observed.

Embeddings table

• Shared public.embeddings table with pointers back to source rows:

  public.embeddings (
    id bigserial PRIMARY KEY,
    source_schema text NOT NULL,
    source_table text NOT NULL,
    source_id bigint NOT NULL,
    embedding vector(<dim>) NOT NULL,
    embedding_model text NOT NULL,
    embedded_at timestamptz NOT NULL DEFAULT now(),
    UNIQUE (source_schema, source_table, source_id, embedding_model)
  )
  

• One table for all embeddings across all content types. Cross-source semantic search becomes one ORDER BY embedding <-> query_vec query.

• The unique constraint allows running multiple embedding models in parallel for experimentation later without conflicting rows.

• Orphan rows possible (no FK because source_table varies); cleaned up by a periodic job (defer; not Phase 1 work).

Two-layer OB1 schema

• ob1.captures — raw layer. Holds Dispatch transcripts, Claude Code session content, claude.ai chat captures. Written by the capture path; never rewritten.
• ob1.reflections — compiled layer. Holds synthesized lessons generated by the daily reflection job. Structured object: lesson text, citations (jsonb array of source pointers), tags, confidence score, generation metadata (model, date, source window).
• Mirrors the wiki's Sources/Wiki two-layer model. Same uniform pattern across both systems: raw layer is the substrate, compiled layer is regenerable.
• Both layers embedded the same way via public.embeddings. Retrieval can scope to one or both via source_table filter.

Reflection job

• Trigger: daily cron, 3am local. Off-hours, low contention.

• Inputs: all three streams:

  • Dispatch session transcripts (Phase 7 of original ob1 plan; pre-compact hook target — may need to coordinate timing)
  • Claude Code session logs from ~/.claude/projects/ (ob1-jsonl-watcher project is the existing capture mechanism)
  • claude.ai chat captures (new path — needs design; see Open Items)

• Output: structured reflection records:

  ob1.reflections (
    id bigserial PRIMARY KEY,
    lesson text NOT NULL,
    citations jsonb NOT NULL,        -- [{schema, table, id, relevance}, ...]
    tags text[],
    confidence float,                -- model-reported or heuristic
    source_window tstzrange,         -- what time window this synthesized
    generated_at timestamptz NOT NULL DEFAULT now(),
    generated_by text NOT NULL,      -- model name + version
    job_run_id uuid                  -- groups reflections from one run
  )
  

• Run-as identity: ob1_mcp user (which has the reads it needs on wiki and trellis under v1 permissions).

• Idle behavior: if a day has no significant activity, the job writes no reflections and logs a no-op. No "no significant activity" placeholder rows.

Embedding model

• Whatever OB1 currently uses for its own captures. Vault indexing, capture embedding, and reflection embedding all use the same model so vectors live in the same space.
• Phase 0 confirms which model OB1 is on.
• Switching the model later is a deliberate full re-embed operation, not a config flip. The embedding_model column on public.embeddings supports incremental migration when that day comes.

Vault indexing

• In scope for this plan. Without it, reflections can't cite vault notes semantically.
• Backfill: one-time pass over all existing wiki.notes rows, embed each, insert into public.embeddings.
• Incremental: save_vault_note in the herby MCP server updated to also embed new content and upsert into public.embeddings.

Open Items

• Final names (target database, schemas, MCP users) — picked at execution time.
• pgvector status — Phase 0 must check \dx on each existing database. If installed in OB1's database, no new install needed; if absent, install during consolidation.
• Embedding model + dimension — inherited from whatever OB1 is using. Phase 0 confirms.
• claude.ai chat capture path — no existing mechanism captures these. Out of scope for initial reflection job inputs unless a capture path lands first. If it doesn't, the job runs on Dispatch + Claude Code streams only and chat captures get added later. Flag for execution: confirm with Travis whether to attempt a capture path in this plan or explicitly defer.
• Dispatch pre-compact hook timing — Phase 7 of the original ob1 deployment plan owns this. If it hasn't shipped, the reflection job's Dispatch input is empty until it does. Worth confirming current state before starting Phase 4 of this plan.
• Reflection prompt design — what does the synthesizer actually do? Read N captures + relevant vault notes from the day → extract generalizable lessons → produce structured output. The prompt itself is real work; placeholder for now.
• Noise filtering for captures — what gets included in the daily reflection window? All captures? Captures above some signal threshold? Excludes routine command output? Design step in Phase 4.
• Reflection deduplication — re-running the job over overlapping windows could produce near-duplicate lessons. Strategy: confidence- weighted merge, manual review, or first-write-wins? Design step in Phase 4.

Phases

Phase 0 — Inspection and prep

• Confirm Postgres instance hosts only wiki, ob1, trellis (nothing else).
• \dx on each database — record which has pgvector installed.
• Inspect each database's schema: confirm table list, row counts, any vector columns and their dimensions.
• Identify OB1's current embedding model (env var, code reference, or configuration file).
• Inventory each MCP server's connection configuration: where credentials live, how to update them.
• Confirm backup destination on the NAS is reachable over Tailscale with sufficient space.
• Document current state in a session note before any changes.

Phase 1 — Backups and rollback prep

• pg_dumpall of the full instance to NAS (full safety net).
• Per-database pg_dump of wiki, ob1, trellis to NAS (granular restore targets).
• Verify dumps are readable (test restore to a scratch database).
• Write rollback runbook: how to restore from dumps and revert MCP server connection strings if cutover fails.

Phase 2 — Consolidation

• Pick final names: target database, schema names, MCP user names.
• If OB1's database is the target and is named appropriately, use in place. Otherwise, rename or create new and restore OB1's content.
• Ensure pgvector is installed on the target database (CREATE EXTENSION IF NOT EXISTS vector).
• Create wiki and trellis schemas in the target database.
• Move OB1's existing tables into the ob1 schema if not already namespaced (ALTER TABLE ... SET SCHEMA ob1).
• Restore wiki dump into the wiki schema using pg_dump's --schema=wiki retargeting or a sed pass over the dump SQL.
• Restore trellis dump into the trellis schema the same way.
• Create the three MCP service users.
• Grant ownership and permissions per the locked decisions.
• Set per-user search_path defaults.
• Verify row counts in each schema match pre-migration counts.

Phase 3 — MCP server updates

• Update herby MCP server: new connection string (target DB, herby_mcp user), confirm search_path resolution works without qualifying table names.
• Update OB1's MCP server: same shape, ob1_mcp user.
• Update trellis MCP server: same shape, trellis_mcp user.
• Restart each MCP server, verify health, run one representative tool from each (list_vault_projects, OB1 search, list_threads).
• Positive isolation test: confirm each MCP user is rejected when attempting to write outside its owned schema. (E.g., herby_mcp attempting INSERT INTO ob1.captures fails with permission error.)
• Update Lovebug's connection string if needed.

Phase 4 — OB1 reflection layer (foundation)

• Create ob1.reflections table per the locked schema.
• Create public.embeddings table per the locked schema.
• Add the unique constraint on (source_schema, source_table, source_id, embedding_model).
• Add HNSW or IVFFlat index on public.embeddings.embedding (model choice depends on row count and recall/speed trade — research current pgvector defaults at execution time).
• Decide noise filtering rules for capture inclusion.
• Design the reflection prompt: input shape (captures from window
  • vault context), output shape (structured object matching ob1.reflections).
• Decide dedup strategy.

Phase 5 — Vault indexing

• Confirm embedding model + dimension from Phase 0.
• Write backfill script: walks all wiki.notes, embeds each (body + title + tag context — decide chunking), inserts into public.embeddings.
• Run backfill, verify row count matches wiki.notes count.
• Update herby MCP server's save_vault_note: embed new content and upsert into public.embeddings on every write.
• Verify: write a new vault note via MCP, confirm embedding row appears in public.embeddings referencing the new note.

Phase 6 — Reflection job

• Implement the reflection job (Python script + cron entry, or systemd timer).
• Job runs as ob1_mcp user; connects with its credentials.
• Inputs:
  • Captures from the last 24 hours (or configured window)
  • Relevant vault notes pulled via semantic search against the day's captures
• Generates reflection objects via the chosen model.
• Writes to ob1.reflections, embeds each, upserts to public.embeddings.
• Schedule for 3am local.
• First-run validation: run manually, inspect output, iterate on prompt before letting cron take over.

Phase 7 — End-to-end verification

• Semantic search query that hits all three content types in one result set: vault notes, captures, reflections. Example query: "what do we know about VLAN tagging" should return relevant rows from multiple sources.
• Reflection citations resolve correctly — pick a recent reflection, follow each citation back to its source row, confirm the join works.
• Permission model holds under stress — try writes from each MCP user into other schemas; all should fail.
• Lovebug can do its job — confirm Lovebug's read patterns still work under the new connection.

Phase 8 — Cleanup

• Drop the old wiki and trellis databases (only after multi-day verification period — keep dumps regardless).
• Archive pre-migration dumps to long-term backup location on NAS.
• Update connection-string documentation (KeePassXC, any internal docs).
• Write a session note capturing the final shape, role model, and operational commands (per-schema backup, restore, etc.).

Notes

Why consolidate at all

The three databases grew separately, not by deliberate isolation choice. The reflection layer wants to cross all three constantly (cite vault notes from captures from reflections). Cross-database queries in Postgres are possible via FDW but every cross-DB query is a small tax — connection management, transaction scope, FDW config. Schemas in one database make cross-cutting queries free.

Why isolation by user, not by database

You already separate concerns by identity across the stack — travadmin vs herbyadmin on hosts, Authentik in front of MCP servers, GitHub deploy keys scoped per machine. The pattern is consistent: separate the who, not the where. Three MCP service users with schema-scoped permissions matches that pattern. Postgres rejects writes outside an MCP's owned schema at the database level — stronger than relying on the MCP's code to behave.

Why two layers in OB1

Same reason the wiki has two layers. Mixing raw captures and synthesized lessons in one table pollutes semantic neighbors and degrades retrieval quality as the reflection corpus grows. The wiki design explicitly rejected this; OB1 deserves the same discipline.

Why a shared embeddings table

Reflection retrieval is fundamentally cross-source. The shared table makes "find similar to X across everything" the default query path. Per-table vector columns would force UNION ALL queries across every content type — workable but easy to forget a table when a new content type is added later. Shared table is more extensible.

Why vault indexing belongs in this plan

Without it, reflections can only cite captures and reflections — never vault notes. The "pull anything on LLM memory projects" use case that motivated this whole design needs vault content searchable. Building reflection first and indexing later means the reflection layer is half- useful for weeks.

Why permissions are conservative now

Easier to grant access later than to revoke it after agents have come to expect it. Starting with only OB1 having cross-schema reads (the minimum needed for reflection to work) and revisiting after real usage tells us what we actually need versus what we thought we needed.

Risks

• Cutover failure: mitigated by full dumps and rehearsed rollback.
• Embedding model mismatch: if Phase 0 reveals OB1 is on a model that's expensive to re-embed at scale, the choice of staying vs. switching becomes load-bearing. Currently small corpus means low cost either way; flag if discovered.
• Reflection prompt quality: bad prompts produce noisy reflections that pollute retrieval. Phase 6's first-run validation gate is the protection — don't let cron take over until manual runs produce good output.
• Permission boundary holes: positive isolation tests in Phase 3 are the protection. Verify rejection, not just success.

What this plan does not do

• Does not redesign the trellis or wiki MCP servers beyond connection changes.
• Does not implement claude.ai chat capture (open item).
• Does not address the Dispatch pre-compact hook (lives in the original ob1 deployment plan, Phase 7).
• Does not tighten Lovebug's permissions (deferred; natural follow-on).
• Does not move Postgres off the current host or change its operational footprint beyond schema reorganization.