wiki-vault/Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md

---
created: '2026-05-15'
path: Sources/Homelab
project: postgres-consolidation-reflection-layer
status: active
tags:
- ob1
- mcp
- pgvector
- embeddings
- automation
- claude-code
- dispatch
type: project-plan
updated: '2026-05-15'
---

# Postgres Consolidation & OB1 Reflection Layer

## Goal

Two coupled outcomes:

1. **Consolidate** the three project databases (`wiki`, `ob1`, `trellis`) on
   the shared Postgres instance into a single database with three schemas,
   each owned by a dedicated MCP service user.
2. **Build the OB1 reflection layer**: a two-layer model in `ob1` (captures
   + reflections) with a daily cron job that synthesizes generalizable
   lessons from agent session content, plus vault indexing so semantic
   search spans vault notes, captures, and reflections uniformly.

These are paired because the reflection loop is fundamentally cross-schema
(citing vault notes from captures from reflections in one query). The
consolidation is the foundation that makes the reflection design clean.

## Locked Decisions

### Database consolidation

- One Postgres database, three schemas: `wiki`, `ob1`, `trellis`.
- Migration direction: **OB1's existing database is the target.** Add
  `wiki` and `trellis` schemas to it. Preserves OB1's existing pgvector
  setup if installed.
- Target database name, schema-prefix-rewrite details, and final role
  names: deferred to execution. Pick deliberate names at that stage.
- Nothing else lives on the Postgres instance — old databases can be
  dropped cleanly after verification.
- Downtime during cutover is acceptable. No parallel-run period needed.

### MCP service users (conservative v1)

- `herby_mcp` — owns `wiki.*`. No reads on `ob1` or `trellis`.
- `ob1_mcp` — owns `ob1.*`. **Reads** on `wiki.*` and `trellis.*`
  (needed for reflection generation).
- `trellis_mcp` — owns `trellis.*`. No reads on `wiki` or `ob1`.
- `lovebug` — read-write across all three schemas (preserves current
  broad-access behavior; tightening deferred).
- Each MCP server connects with its own credentials and its own
  `search_path` so unqualified table references resolve to its own schema.
- Permission model revisits after reflection ships and real cross-schema
  read patterns are observed.

### Embeddings table

- Shared `public.embeddings` table with pointers back to source rows:

  ```
  public.embeddings (
    id bigserial PRIMARY KEY,
    source_schema text NOT NULL,
    source_table text NOT NULL,
    source_id bigint NOT NULL,
    embedding vector(<dim>) NOT NULL,
    embedding_model text NOT NULL,
    embedded_at timestamptz NOT NULL DEFAULT now(),
    UNIQUE (source_schema, source_table, source_id, embedding_model)
  )
  ```

- One table for all embeddings across all content types. Cross-source
  semantic search becomes one `ORDER BY embedding <-> query_vec` query.
- The unique constraint allows running multiple embedding models in
  parallel for experimentation later without conflicting rows.
- Orphan rows possible (no FK because `source_table` varies); cleaned
  up by a periodic job (defer; not Phase 1 work).

### Two-layer OB1 schema

- `ob1.captures` — raw layer. Holds Dispatch transcripts, Claude Code
  session content, claude.ai chat captures. Written by the capture path;
  never rewritten.
- `ob1.reflections` — compiled layer. Holds synthesized lessons generated
  by the daily reflection job. Structured object: lesson text, citations
  (jsonb array of source pointers), tags, confidence score, generation
  metadata (model, date, source window).
- Mirrors the wiki's Sources/Wiki two-layer model. Same uniform pattern
  across both systems: raw layer is the substrate, compiled layer is
  regenerable.
- Both layers embedded the same way via `public.embeddings`. Retrieval
  can scope to one or both via `source_table` filter.

### Reflection job

- **Trigger:** daily cron, 3am local. Off-hours, low contention.
- **Inputs:** all three streams:
  - Dispatch session transcripts (Phase 7 of original ob1 plan; pre-compact
    hook target — may need to coordinate timing)
  - Claude Code session logs from `~/.claude/projects/` (ob1-jsonl-watcher
    project is the existing capture mechanism)
  - claude.ai chat captures (new path — needs design; see Open Items)
- **Output:** structured reflection records:

  ```
  ob1.reflections (
    id bigserial PRIMARY KEY,
    lesson text NOT NULL,
    citations jsonb NOT NULL,        -- [{schema, table, id, relevance}, ...]
    tags text[],
    confidence float,                -- model-reported or heuristic
    source_window tstzrange,         -- what time window this synthesized
    generated_at timestamptz NOT NULL DEFAULT now(),
    generated_by text NOT NULL,      -- model name + version
    job_run_id uuid                  -- groups reflections from one run
  )
  ```

- **Run-as identity:** `ob1_mcp` user (which has the reads it needs on
  `wiki` and `trellis` under v1 permissions).
- **Idle behavior:** if a day has no significant activity, the job writes
  no reflections and logs a no-op. No "no significant activity" placeholder
  rows.

### Embedding model

- Whatever OB1 currently uses for its own captures. Vault indexing,
  capture embedding, and reflection embedding all use the same model
  so vectors live in the same space.
- Phase 0 confirms which model OB1 is on.
- Switching the model later is a deliberate full re-embed operation, not
  a config flip. The `embedding_model` column on `public.embeddings`
  supports incremental migration when that day comes.

### Vault indexing

- In scope for this plan. Without it, reflections can't cite vault notes
  semantically.
- Backfill: one-time pass over all existing `wiki.notes` rows, embed
  each, insert into `public.embeddings`.
- Incremental: `save_vault_note` in the herby MCP server updated to also
  embed new content and upsert into `public.embeddings`.

## Open Items

- [ ] Final names (target database, schemas, MCP users) — picked at
  execution time.
- [ ] pgvector status — Phase 0 must check `\dx` on each existing
  database. If installed in OB1's database, no new install needed; if
  absent, install during consolidation.
- [ ] Embedding model + dimension — inherited from whatever OB1 is using.
  Phase 0 confirms.
- [ ] claude.ai chat capture path — no existing mechanism captures these.
  Out of scope for initial reflection job inputs unless a capture path
  lands first. If it doesn't, the job runs on Dispatch + Claude Code
  streams only and chat captures get added later. **Flag for execution:
  confirm with Travis whether to attempt a capture path in this plan or
  explicitly defer.**
- [ ] Dispatch pre-compact hook timing — Phase 7 of the original ob1
  deployment plan owns this. If it hasn't shipped, the reflection job's
  Dispatch input is empty until it does. **Worth confirming current state
  before starting Phase 4 of this plan.**
- [ ] Reflection prompt design — what does the synthesizer actually do?
  Read N captures + relevant vault notes from the day → extract
  generalizable lessons → produce structured output. The prompt itself
  is real work; placeholder for now.
- [ ] Noise filtering for captures — what gets included in the daily
  reflection window? All captures? Captures above some signal threshold?
  Excludes routine command output? **Design step in Phase 4.**
- [ ] Reflection deduplication — re-running the job over overlapping
  windows could produce near-duplicate lessons. Strategy: confidence-
  weighted merge, manual review, or first-write-wins? **Design step in
  Phase 4.**

## Phases

### Phase 0 — Inspection and prep

- [ ] Confirm Postgres instance hosts only `wiki`, `ob1`, `trellis`
  (nothing else).
- [ ] `\dx` on each database — record which has pgvector installed.
- [ ] Inspect each database's schema: confirm table list, row counts,
  any vector columns and their dimensions.
- [ ] Identify OB1's current embedding model (env var, code reference,
  or configuration file).
- [ ] Inventory each MCP server's connection configuration: where
  credentials live, how to update them.
- [ ] Confirm backup destination on the NAS is reachable over Tailscale
  with sufficient space.
- [ ] Document current state in a session note before any changes.

### Phase 1 — Backups and rollback prep

- [ ] `pg_dumpall` of the full instance to NAS (full safety net).
- [ ] Per-database `pg_dump` of `wiki`, `ob1`, `trellis` to NAS
  (granular restore targets).
- [ ] Verify dumps are readable (test restore to a scratch database).
- [ ] Write rollback runbook: how to restore from dumps and revert MCP
  server connection strings if cutover fails.

### Phase 2 — Consolidation

- [ ] Pick final names: target database, schema names, MCP user names.
- [ ] If OB1's database is the target and is named appropriately, use
  in place. Otherwise, rename or create new and restore OB1's content.
- [ ] Ensure pgvector is installed on the target database
  (`CREATE EXTENSION IF NOT EXISTS vector`).
- [ ] Create `wiki` and `trellis` schemas in the target database.
- [ ] Move OB1's existing tables into the `ob1` schema if not already
  namespaced (`ALTER TABLE ... SET SCHEMA ob1`).
- [ ] Restore `wiki` dump into the `wiki` schema using `pg_dump`'s
  `--schema=wiki` retargeting or a sed pass over the dump SQL.
- [ ] Restore `trellis` dump into the `trellis` schema the same way.
- [ ] Create the three MCP service users.
- [ ] Grant ownership and permissions per the locked decisions.
- [ ] Set per-user `search_path` defaults.
- [ ] Verify row counts in each schema match pre-migration counts.

### Phase 3 — MCP server updates

- [ ] Update herby MCP server: new connection string (target DB,
  `herby_mcp` user), confirm `search_path` resolution works without
  qualifying table names.
- [ ] Update OB1's MCP server: same shape, `ob1_mcp` user.
- [ ] Update trellis MCP server: same shape, `trellis_mcp` user.
- [ ] Restart each MCP server, verify health, run one representative
  tool from each (`list_vault_projects`, OB1 search, `list_threads`).
- [ ] **Positive isolation test:** confirm each MCP user is *rejected*
  when attempting to write outside its owned schema. (E.g., `herby_mcp`
  attempting `INSERT INTO ob1.captures` fails with permission error.)
- [ ] Update Lovebug's connection string if needed.

### Phase 4 — OB1 reflection layer (foundation)

- [ ] Create `ob1.reflections` table per the locked schema.
- [ ] Create `public.embeddings` table per the locked schema.
- [ ] Add the unique constraint on `(source_schema, source_table,
  source_id, embedding_model)`.
- [ ] Add HNSW or IVFFlat index on `public.embeddings.embedding`
  (model choice depends on row count and recall/speed trade — research
  current pgvector defaults at execution time).
- [ ] Decide noise filtering rules for capture inclusion.
- [ ] Design the reflection prompt: input shape (captures from window
  + vault context), output shape (structured object matching
  `ob1.reflections`).
- [ ] Decide dedup strategy.

### Phase 5 — Vault indexing

- [ ] Confirm embedding model + dimension from Phase 0.
- [ ] Write backfill script: walks all `wiki.notes`, embeds each
  (body + title + tag context — decide chunking), inserts into
  `public.embeddings`.
- [ ] Run backfill, verify row count matches `wiki.notes` count.
- [ ] Update herby MCP server's `save_vault_note`: embed new content
  and upsert into `public.embeddings` on every write.
- [ ] Verify: write a new vault note via MCP, confirm embedding row
  appears in `public.embeddings` referencing the new note.

### Phase 6 — Reflection job

- [ ] Implement the reflection job (Python script + cron entry, or
  systemd timer).
- [ ] Job runs as `ob1_mcp` user; connects with its credentials.
- [ ] Inputs:
  - Captures from the last 24 hours (or configured window)
  - Relevant vault notes pulled via semantic search against the day's
    captures
- [ ] Generates reflection objects via the chosen model.
- [ ] Writes to `ob1.reflections`, embeds each, upserts to
  `public.embeddings`.
- [ ] Schedule for 3am local.
- [ ] First-run validation: run manually, inspect output, iterate on
  prompt before letting cron take over.

### Phase 7 — End-to-end verification

- [ ] Semantic search query that hits all three content types in one
  result set: vault notes, captures, reflections. Example query: "what
  do we know about VLAN tagging" should return relevant rows from
  multiple sources.
- [ ] Reflection citations resolve correctly — pick a recent reflection,
  follow each citation back to its source row, confirm the join works.
- [ ] Permission model holds under stress — try writes from each MCP
  user into other schemas; all should fail.
- [ ] Lovebug can do its job — confirm Lovebug's read patterns still
  work under the new connection.

### Phase 8 — Cleanup

- [ ] Drop the old `wiki` and `trellis` databases (only after multi-day
  verification period — keep dumps regardless).
- [ ] Archive pre-migration dumps to long-term backup location on NAS.
- [ ] Update connection-string documentation (KeePassXC, any internal
  docs).
- [ ] Write a session note capturing the final shape, role model, and
  operational commands (per-schema backup, restore, etc.).

## Notes

### Why consolidate at all

The three databases grew separately, not by deliberate isolation choice.
The reflection layer wants to cross all three constantly (cite vault notes
from captures from reflections). Cross-database queries in Postgres are
possible via FDW but every cross-DB query is a small tax — connection
management, transaction scope, FDW config. Schemas in one database make
cross-cutting queries free.

### Why isolation by user, not by database

You already separate concerns by *identity* across the stack — `travadmin`
vs `herbyadmin` on hosts, Authentik in front of MCP servers, GitHub
deploy keys scoped per machine. The pattern is consistent: separate the
*who*, not the *where*. Three MCP service users with schema-scoped
permissions matches that pattern. Postgres rejects writes outside an
MCP's owned schema at the database level — stronger than relying on the
MCP's code to behave.

### Why two layers in OB1

Same reason the wiki has two layers. Mixing raw captures and synthesized
lessons in one table pollutes semantic neighbors and degrades retrieval
quality as the reflection corpus grows. The wiki design explicitly
rejected this; OB1 deserves the same discipline.

### Why a shared embeddings table

Reflection retrieval is fundamentally cross-source. The shared table
makes "find similar to X across everything" the default query path.
Per-table vector columns would force `UNION ALL` queries across every
content type — workable but easy to forget a table when a new content
type is added later. Shared table is more extensible.

### Why vault indexing belongs in this plan

Without it, reflections can only cite captures and reflections — never
vault notes. The "pull anything on LLM memory projects" use case that
motivated this whole design needs vault content searchable. Building
reflection first and indexing later means the reflection layer is half-
useful for weeks.

### Why permissions are conservative now

Easier to grant access later than to revoke it after agents have come
to expect it. Starting with only OB1 having cross-schema reads (the
minimum needed for reflection to work) and revisiting after real usage
tells us what we actually need versus what we thought we needed.

### Risks

- **Cutover failure**: mitigated by full dumps and rehearsed rollback.
- **Embedding model mismatch**: if Phase 0 reveals OB1 is on a model
  that's expensive to re-embed at scale, the choice of staying vs.
  switching becomes load-bearing. Currently small corpus means low cost
  either way; flag if discovered.
- **Reflection prompt quality**: bad prompts produce noisy reflections
  that pollute retrieval. Phase 6's first-run validation gate is the
  protection — don't let cron take over until manual runs produce good
  output.
- **Permission boundary holes**: positive isolation tests in Phase 3 are
  the protection. Verify rejection, not just success.

### What this plan does not do

- Does not redesign the trellis or wiki MCP servers beyond connection
  changes.
- Does not implement claude.ai chat capture (open item).
- Does not address the Dispatch pre-compact hook (lives in the original
  ob1 deployment plan, Phase 7).
- Does not tighten Lovebug's permissions (deferred; natural follow-on).
- Does not move Postgres off the current host or change its operational
  footprint beyond schema reorganization.