mcp: project-plan — Postgres Consolidation & OB1 Reflection Layer
This commit is contained in:
Lovebug MCP
parent
2db366f97c
commit
659b329199
381
Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md
Normal file
381
Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md
Normal file
@ -0,0 +1,381 @@
|
|||||||
|
---
|
||||||
|
created: '2026-05-15'
|
||||||
|
path: Sources/Homelab
|
||||||
|
project: postgres-consolidation-reflection-layer
|
||||||
|
status: active
|
||||||
|
tags:
|
||||||
|
- ob1
|
||||||
|
- mcp
|
||||||
|
- pgvector
|
||||||
|
- embeddings
|
||||||
|
- automation
|
||||||
|
- claude-code
|
||||||
|
- dispatch
|
||||||
|
type: project-plan
|
||||||
|
updated: '2026-05-15'
|
||||||
|
---
|
||||||
|
|
||||||
|
# Postgres Consolidation & OB1 Reflection Layer
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Two coupled outcomes:
|
||||||
|
|
||||||
|
1. **Consolidate** the three project databases (`wiki`, `ob1`, `trellis`) on
|
||||||
|
the shared Postgres instance into a single database with three schemas,
|
||||||
|
each owned by a dedicated MCP service user.
|
||||||
|
2. **Build the OB1 reflection layer**: a two-layer model in `ob1` (captures
|
||||||
|
+ reflections) with a daily cron job that synthesizes generalizable
|
||||||
|
lessons from agent session content, plus vault indexing so semantic
|
||||||
|
search spans vault notes, captures, and reflections uniformly.
|
||||||
|
|
||||||
|
These are paired because the reflection loop is fundamentally cross-schema
|
||||||
|
(citing vault notes from captures from reflections in one query). The
|
||||||
|
consolidation is the foundation that makes the reflection design clean.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
### Database consolidation
|
||||||
|
|
||||||
|
- One Postgres database, three schemas: `wiki`, `ob1`, `trellis`.
|
||||||
|
- Migration direction: **OB1's existing database is the target.** Add
|
||||||
|
`wiki` and `trellis` schemas to it. Preserves OB1's existing pgvector
|
||||||
|
setup if installed.
|
||||||
|
- Target database name, schema-prefix-rewrite details, and final role
|
||||||
|
names: deferred to execution. Pick deliberate names at that stage.
|
||||||
|
- Nothing else lives on the Postgres instance — old databases can be
|
||||||
|
dropped cleanly after verification.
|
||||||
|
- Downtime during cutover is acceptable. No parallel-run period needed.
|
||||||
|
|
||||||
|
### MCP service users (conservative v1)
|
||||||
|
|
||||||
|
- `herby_mcp` — owns `wiki.*`. No reads on `ob1` or `trellis`.
|
||||||
|
- `ob1_mcp` — owns `ob1.*`. **Reads** on `wiki.*` and `trellis.*`
|
||||||
|
(needed for reflection generation).
|
||||||
|
- `trellis_mcp` — owns `trellis.*`. No reads on `wiki` or `ob1`.
|
||||||
|
- `lovebug` — read-write across all three schemas (preserves current
|
||||||
|
broad-access behavior; tightening deferred).
|
||||||
|
- Each MCP server connects with its own credentials and its own
|
||||||
|
`search_path` so unqualified table references resolve to its own schema.
|
||||||
|
- Permission model revisits after reflection ships and real cross-schema
|
||||||
|
read patterns are observed.
|
||||||
|
|
||||||
|
### Embeddings table
|
||||||
|
|
||||||
|
- Shared `public.embeddings` table with pointers back to source rows:
|
||||||
|
|
||||||
|
```
|
||||||
|
public.embeddings (
|
||||||
|
id bigserial PRIMARY KEY,
|
||||||
|
source_schema text NOT NULL,
|
||||||
|
source_table text NOT NULL,
|
||||||
|
source_id bigint NOT NULL,
|
||||||
|
embedding vector(<dim>) NOT NULL,
|
||||||
|
embedding_model text NOT NULL,
|
||||||
|
embedded_at timestamptz NOT NULL DEFAULT now(),
|
||||||
|
UNIQUE (source_schema, source_table, source_id, embedding_model)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
- One table for all embeddings across all content types. Cross-source
|
||||||
|
semantic search becomes one `ORDER BY embedding <-> query_vec` query.
|
||||||
|
- The unique constraint allows running multiple embedding models in
|
||||||
|
parallel for experimentation later without conflicting rows.
|
||||||
|
- Orphan rows possible (no FK because `source_table` varies); cleaned
|
||||||
|
up by a periodic job (defer; not Phase 1 work).
|
||||||
|
|
||||||
|
### Two-layer OB1 schema
|
||||||
|
|
||||||
|
- `ob1.captures` — raw layer. Holds Dispatch transcripts, Claude Code
|
||||||
|
session content, claude.ai chat captures. Written by the capture path;
|
||||||
|
never rewritten.
|
||||||
|
- `ob1.reflections` — compiled layer. Holds synthesized lessons generated
|
||||||
|
by the daily reflection job. Structured object: lesson text, citations
|
||||||
|
(jsonb array of source pointers), tags, confidence score, generation
|
||||||
|
metadata (model, date, source window).
|
||||||
|
- Mirrors the wiki's Sources/Wiki two-layer model. Same uniform pattern
|
||||||
|
across both systems: raw layer is the substrate, compiled layer is
|
||||||
|
regenerable.
|
||||||
|
- Both layers embedded the same way via `public.embeddings`. Retrieval
|
||||||
|
can scope to one or both via `source_table` filter.
|
||||||
|
|
||||||
|
### Reflection job
|
||||||
|
|
||||||
|
- **Trigger:** daily cron, 3am local. Off-hours, low contention.
|
||||||
|
- **Inputs:** all three streams:
|
||||||
|
- Dispatch session transcripts (Phase 7 of original ob1 plan; pre-compact
|
||||||
|
hook target — may need to coordinate timing)
|
||||||
|
- Claude Code session logs from `~/.claude/projects/` (ob1-jsonl-watcher
|
||||||
|
project is the existing capture mechanism)
|
||||||
|
- claude.ai chat captures (new path — needs design; see Open Items)
|
||||||
|
- **Output:** structured reflection records:
|
||||||
|
|
||||||
|
```
|
||||||
|
ob1.reflections (
|
||||||
|
id bigserial PRIMARY KEY,
|
||||||
|
lesson text NOT NULL,
|
||||||
|
citations jsonb NOT NULL, -- [{schema, table, id, relevance}, ...]
|
||||||
|
tags text[],
|
||||||
|
confidence float, -- model-reported or heuristic
|
||||||
|
source_window tstzrange, -- what time window this synthesized
|
||||||
|
generated_at timestamptz NOT NULL DEFAULT now(),
|
||||||
|
generated_by text NOT NULL, -- model name + version
|
||||||
|
job_run_id uuid -- groups reflections from one run
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Run-as identity:** `ob1_mcp` user (which has the reads it needs on
|
||||||
|
`wiki` and `trellis` under v1 permissions).
|
||||||
|
- **Idle behavior:** if a day has no significant activity, the job writes
|
||||||
|
no reflections and logs a no-op. No "no significant activity" placeholder
|
||||||
|
rows.
|
||||||
|
|
||||||
|
### Embedding model
|
||||||
|
|
||||||
|
- Whatever OB1 currently uses for its own captures. Vault indexing,
|
||||||
|
capture embedding, and reflection embedding all use the same model
|
||||||
|
so vectors live in the same space.
|
||||||
|
- Phase 0 confirms which model OB1 is on.
|
||||||
|
- Switching the model later is a deliberate full re-embed operation, not
|
||||||
|
a config flip. The `embedding_model` column on `public.embeddings`
|
||||||
|
supports incremental migration when that day comes.
|
||||||
|
|
||||||
|
### Vault indexing
|
||||||
|
|
||||||
|
- In scope for this plan. Without it, reflections can't cite vault notes
|
||||||
|
semantically.
|
||||||
|
- Backfill: one-time pass over all existing `wiki.notes` rows, embed
|
||||||
|
each, insert into `public.embeddings`.
|
||||||
|
- Incremental: `save_vault_note` in the herby MCP server updated to also
|
||||||
|
embed new content and upsert into `public.embeddings`.
|
||||||
|
|
||||||
|
## Open Items
|
||||||
|
|
||||||
|
- [ ] Final names (target database, schemas, MCP users) — picked at
|
||||||
|
execution time.
|
||||||
|
- [ ] pgvector status — Phase 0 must check `\dx` on each existing
|
||||||
|
database. If installed in OB1's database, no new install needed; if
|
||||||
|
absent, install during consolidation.
|
||||||
|
- [ ] Embedding model + dimension — inherited from whatever OB1 is using.
|
||||||
|
Phase 0 confirms.
|
||||||
|
- [ ] claude.ai chat capture path — no existing mechanism captures these.
|
||||||
|
Out of scope for initial reflection job inputs unless a capture path
|
||||||
|
lands first. If it doesn't, the job runs on Dispatch + Claude Code
|
||||||
|
streams only and chat captures get added later. **Flag for execution:
|
||||||
|
confirm with Travis whether to attempt a capture path in this plan or
|
||||||
|
explicitly defer.**
|
||||||
|
- [ ] Dispatch pre-compact hook timing — Phase 7 of the original ob1
|
||||||
|
deployment plan owns this. If it hasn't shipped, the reflection job's
|
||||||
|
Dispatch input is empty until it does. **Worth confirming current state
|
||||||
|
before starting Phase 4 of this plan.**
|
||||||
|
- [ ] Reflection prompt design — what does the synthesizer actually do?
|
||||||
|
Read N captures + relevant vault notes from the day → extract
|
||||||
|
generalizable lessons → produce structured output. The prompt itself
|
||||||
|
is real work; placeholder for now.
|
||||||
|
- [ ] Noise filtering for captures — what gets included in the daily
|
||||||
|
reflection window? All captures? Captures above some signal threshold?
|
||||||
|
Excludes routine command output? **Design step in Phase 4.**
|
||||||
|
- [ ] Reflection deduplication — re-running the job over overlapping
|
||||||
|
windows could produce near-duplicate lessons. Strategy: confidence-
|
||||||
|
weighted merge, manual review, or first-write-wins? **Design step in
|
||||||
|
Phase 4.**
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase 0 — Inspection and prep
|
||||||
|
|
||||||
|
- [ ] Confirm Postgres instance hosts only `wiki`, `ob1`, `trellis`
|
||||||
|
(nothing else).
|
||||||
|
- [ ] `\dx` on each database — record which has pgvector installed.
|
||||||
|
- [ ] Inspect each database's schema: confirm table list, row counts,
|
||||||
|
any vector columns and their dimensions.
|
||||||
|
- [ ] Identify OB1's current embedding model (env var, code reference,
|
||||||
|
or configuration file).
|
||||||
|
- [ ] Inventory each MCP server's connection configuration: where
|
||||||
|
credentials live, how to update them.
|
||||||
|
- [ ] Confirm backup destination on the NAS is reachable over Tailscale
|
||||||
|
with sufficient space.
|
||||||
|
- [ ] Document current state in a session note before any changes.
|
||||||
|
|
||||||
|
### Phase 1 — Backups and rollback prep
|
||||||
|
|
||||||
|
- [ ] `pg_dumpall` of the full instance to NAS (full safety net).
|
||||||
|
- [ ] Per-database `pg_dump` of `wiki`, `ob1`, `trellis` to NAS
|
||||||
|
(granular restore targets).
|
||||||
|
- [ ] Verify dumps are readable (test restore to a scratch database).
|
||||||
|
- [ ] Write rollback runbook: how to restore from dumps and revert MCP
|
||||||
|
server connection strings if cutover fails.
|
||||||
|
|
||||||
|
### Phase 2 — Consolidation
|
||||||
|
|
||||||
|
- [ ] Pick final names: target database, schema names, MCP user names.
|
||||||
|
- [ ] If OB1's database is the target and is named appropriately, use
|
||||||
|
in place. Otherwise, rename or create new and restore OB1's content.
|
||||||
|
- [ ] Ensure pgvector is installed on the target database
|
||||||
|
(`CREATE EXTENSION IF NOT EXISTS vector`).
|
||||||
|
- [ ] Create `wiki` and `trellis` schemas in the target database.
|
||||||
|
- [ ] Move OB1's existing tables into the `ob1` schema if not already
|
||||||
|
namespaced (`ALTER TABLE ... SET SCHEMA ob1`).
|
||||||
|
- [ ] Restore `wiki` dump into the `wiki` schema using `pg_dump`'s
|
||||||
|
`--schema=wiki` retargeting or a sed pass over the dump SQL.
|
||||||
|
- [ ] Restore `trellis` dump into the `trellis` schema the same way.
|
||||||
|
- [ ] Create the three MCP service users.
|
||||||
|
- [ ] Grant ownership and permissions per the locked decisions.
|
||||||
|
- [ ] Set per-user `search_path` defaults.
|
||||||
|
- [ ] Verify row counts in each schema match pre-migration counts.
|
||||||
|
|
||||||
|
### Phase 3 — MCP server updates
|
||||||
|
|
||||||
|
- [ ] Update herby MCP server: new connection string (target DB,
|
||||||
|
`herby_mcp` user), confirm `search_path` resolution works without
|
||||||
|
qualifying table names.
|
||||||
|
- [ ] Update OB1's MCP server: same shape, `ob1_mcp` user.
|
||||||
|
- [ ] Update trellis MCP server: same shape, `trellis_mcp` user.
|
||||||
|
- [ ] Restart each MCP server, verify health, run one representative
|
||||||
|
tool from each (`list_vault_projects`, OB1 search, `list_threads`).
|
||||||
|
- [ ] **Positive isolation test:** confirm each MCP user is *rejected*
|
||||||
|
when attempting to write outside its owned schema. (E.g., `herby_mcp`
|
||||||
|
attempting `INSERT INTO ob1.captures` fails with permission error.)
|
||||||
|
- [ ] Update Lovebug's connection string if needed.
|
||||||
|
|
||||||
|
### Phase 4 — OB1 reflection layer (foundation)
|
||||||
|
|
||||||
|
- [ ] Create `ob1.reflections` table per the locked schema.
|
||||||
|
- [ ] Create `public.embeddings` table per the locked schema.
|
||||||
|
- [ ] Add the unique constraint on `(source_schema, source_table,
|
||||||
|
source_id, embedding_model)`.
|
||||||
|
- [ ] Add HNSW or IVFFlat index on `public.embeddings.embedding`
|
||||||
|
(model choice depends on row count and recall/speed trade — research
|
||||||
|
current pgvector defaults at execution time).
|
||||||
|
- [ ] Decide noise filtering rules for capture inclusion.
|
||||||
|
- [ ] Design the reflection prompt: input shape (captures from window
|
||||||
|
+ vault context), output shape (structured object matching
|
||||||
|
`ob1.reflections`).
|
||||||
|
- [ ] Decide dedup strategy.
|
||||||
|
|
||||||
|
### Phase 5 — Vault indexing
|
||||||
|
|
||||||
|
- [ ] Confirm embedding model + dimension from Phase 0.
|
||||||
|
- [ ] Write backfill script: walks all `wiki.notes`, embeds each
|
||||||
|
(body + title + tag context — decide chunking), inserts into
|
||||||
|
`public.embeddings`.
|
||||||
|
- [ ] Run backfill, verify row count matches `wiki.notes` count.
|
||||||
|
- [ ] Update herby MCP server's `save_vault_note`: embed new content
|
||||||
|
and upsert into `public.embeddings` on every write.
|
||||||
|
- [ ] Verify: write a new vault note via MCP, confirm embedding row
|
||||||
|
appears in `public.embeddings` referencing the new note.
|
||||||
|
|
||||||
|
### Phase 6 — Reflection job
|
||||||
|
|
||||||
|
- [ ] Implement the reflection job (Python script + cron entry, or
|
||||||
|
systemd timer).
|
||||||
|
- [ ] Job runs as `ob1_mcp` user; connects with its credentials.
|
||||||
|
- [ ] Inputs:
|
||||||
|
- Captures from the last 24 hours (or configured window)
|
||||||
|
- Relevant vault notes pulled via semantic search against the day's
|
||||||
|
captures
|
||||||
|
- [ ] Generates reflection objects via the chosen model.
|
||||||
|
- [ ] Writes to `ob1.reflections`, embeds each, upserts to
|
||||||
|
`public.embeddings`.
|
||||||
|
- [ ] Schedule for 3am local.
|
||||||
|
- [ ] First-run validation: run manually, inspect output, iterate on
|
||||||
|
prompt before letting cron take over.
|
||||||
|
|
||||||
|
### Phase 7 — End-to-end verification
|
||||||
|
|
||||||
|
- [ ] Semantic search query that hits all three content types in one
|
||||||
|
result set: vault notes, captures, reflections. Example query: "what
|
||||||
|
do we know about VLAN tagging" should return relevant rows from
|
||||||
|
multiple sources.
|
||||||
|
- [ ] Reflection citations resolve correctly — pick a recent reflection,
|
||||||
|
follow each citation back to its source row, confirm the join works.
|
||||||
|
- [ ] Permission model holds under stress — try writes from each MCP
|
||||||
|
user into other schemas; all should fail.
|
||||||
|
- [ ] Lovebug can do its job — confirm Lovebug's read patterns still
|
||||||
|
work under the new connection.
|
||||||
|
|
||||||
|
### Phase 8 — Cleanup
|
||||||
|
|
||||||
|
- [ ] Drop the old `wiki` and `trellis` databases (only after multi-day
|
||||||
|
verification period — keep dumps regardless).
|
||||||
|
- [ ] Archive pre-migration dumps to long-term backup location on NAS.
|
||||||
|
- [ ] Update connection-string documentation (KeePassXC, any internal
|
||||||
|
docs).
|
||||||
|
- [ ] Write a session note capturing the final shape, role model, and
|
||||||
|
operational commands (per-schema backup, restore, etc.).
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
### Why consolidate at all
|
||||||
|
|
||||||
|
The three databases grew separately, not by deliberate isolation choice.
|
||||||
|
The reflection layer wants to cross all three constantly (cite vault notes
|
||||||
|
from captures from reflections). Cross-database queries in Postgres are
|
||||||
|
possible via FDW but every cross-DB query is a small tax — connection
|
||||||
|
management, transaction scope, FDW config. Schemas in one database make
|
||||||
|
cross-cutting queries free.
|
||||||
|
|
||||||
|
### Why isolation by user, not by database
|
||||||
|
|
||||||
|
You already separate concerns by *identity* across the stack — `travadmin`
|
||||||
|
vs `herbyadmin` on hosts, Authentik in front of MCP servers, GitHub
|
||||||
|
deploy keys scoped per machine. The pattern is consistent: separate the
|
||||||
|
*who*, not the *where*. Three MCP service users with schema-scoped
|
||||||
|
permissions matches that pattern. Postgres rejects writes outside an
|
||||||
|
MCP's owned schema at the database level — stronger than relying on the
|
||||||
|
MCP's code to behave.
|
||||||
|
|
||||||
|
### Why two layers in OB1
|
||||||
|
|
||||||
|
Same reason the wiki has two layers. Mixing raw captures and synthesized
|
||||||
|
lessons in one table pollutes semantic neighbors and degrades retrieval
|
||||||
|
quality as the reflection corpus grows. The wiki design explicitly
|
||||||
|
rejected this; OB1 deserves the same discipline.
|
||||||
|
|
||||||
|
### Why a shared embeddings table
|
||||||
|
|
||||||
|
Reflection retrieval is fundamentally cross-source. The shared table
|
||||||
|
makes "find similar to X across everything" the default query path.
|
||||||
|
Per-table vector columns would force `UNION ALL` queries across every
|
||||||
|
content type — workable but easy to forget a table when a new content
|
||||||
|
type is added later. Shared table is more extensible.
|
||||||
|
|
||||||
|
### Why vault indexing belongs in this plan
|
||||||
|
|
||||||
|
Without it, reflections can only cite captures and reflections — never
|
||||||
|
vault notes. The "pull anything on LLM memory projects" use case that
|
||||||
|
motivated this whole design needs vault content searchable. Building
|
||||||
|
reflection first and indexing later means the reflection layer is half-
|
||||||
|
useful for weeks.
|
||||||
|
|
||||||
|
### Why permissions are conservative now
|
||||||
|
|
||||||
|
Easier to grant access later than to revoke it after agents have come
|
||||||
|
to expect it. Starting with only OB1 having cross-schema reads (the
|
||||||
|
minimum needed for reflection to work) and revisiting after real usage
|
||||||
|
tells us what we actually need versus what we thought we needed.
|
||||||
|
|
||||||
|
### Risks
|
||||||
|
|
||||||
|
- **Cutover failure**: mitigated by full dumps and rehearsed rollback.
|
||||||
|
- **Embedding model mismatch**: if Phase 0 reveals OB1 is on a model
|
||||||
|
that's expensive to re-embed at scale, the choice of staying vs.
|
||||||
|
switching becomes load-bearing. Currently small corpus means low cost
|
||||||
|
either way; flag if discovered.
|
||||||
|
- **Reflection prompt quality**: bad prompts produce noisy reflections
|
||||||
|
that pollute retrieval. Phase 6's first-run validation gate is the
|
||||||
|
protection — don't let cron take over until manual runs produce good
|
||||||
|
output.
|
||||||
|
- **Permission boundary holes**: positive isolation tests in Phase 3 are
|
||||||
|
the protection. Verify rejection, not just success.
|
||||||
|
|
||||||
|
### What this plan does not do
|
||||||
|
|
||||||
|
- Does not redesign the trellis or wiki MCP servers beyond connection
|
||||||
|
changes.
|
||||||
|
- Does not implement claude.ai chat capture (open item).
|
||||||
|
- Does not address the Dispatch pre-compact hook (lives in the original
|
||||||
|
ob1 deployment plan, Phase 7).
|
||||||
|
- Does not tighten Lovebug's permissions (deferred; natural follow-on).
|
||||||
|
- Does not move Postgres off the current host or change its operational
|
||||||
|
footprint beyond schema reorganization.
|
||||||
Loading…
Reference in New Issue
Block a user