mcp: project-plan — Postgres Consolidation Follow-ups
This commit is contained in:
Lovebug MCP
parent
85130079cb
commit
09e41ed538
130
Sources/Homelab/postgres-consolidation-followups.md
Normal file
130
Sources/Homelab/postgres-consolidation-followups.md
Normal file
@ -0,0 +1,130 @@
|
|||||||
|
---
|
||||||
|
created: '2026-05-17'
|
||||||
|
path: Sources/Homelab
|
||||||
|
project: postgres-consolidation-followups
|
||||||
|
tags:
|
||||||
|
- ob1
|
||||||
|
- mcp
|
||||||
|
- pgvector
|
||||||
|
- embeddings
|
||||||
|
- automation
|
||||||
|
- claude-code
|
||||||
|
type: project-plan
|
||||||
|
---
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Close out the deferred work items from the `postgres-consolidation-reflection-layer` initiative. The major architectural work landed (DB consolidated to petalbrain with three schemas and four service users, chunked embeddings live in public.embeddings, vault MCP write surface fully operational, trellis cutover complete, reflector built). What remains is a mix of small operational tasks, the original plan's Phase 7/8 verification + cleanup, and one architectural decision (n8n parallel-write).
|
||||||
|
|
||||||
|
This plan exists so the follow-ups aren't lost. Most items are independent and can be tackled in any order. Phases are grouping by intent, not strict dependency.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
These are settled from the parent initiative; record them here so future work doesn't re-litigate:
|
||||||
|
|
||||||
|
- **Table naming:** `ob1.thoughts` (not `captures`), `wiki.artifacts` + `wiki.projects` + `wiki.notes`, `trellis.{threads,events,tags,thread_tags}`, `ob1.reflections`
|
||||||
|
- **Embedding storage:** all embeddings in `public.embeddings` (shared across sources); HNSW vector_cosine_ops m=16 ef_construction=64
|
||||||
|
- **Chunking:** 512-token chunks, 64-token overlap, tiktoken cl100k_base, applied at every embedding writer (vault_mcp, ob1-mcp, ob1-reflector; not ob1-enricher which is metadata-only)
|
||||||
|
- **Embedding model:** nomic-embed-text via local Ollama (768-d). Switching requires schema migration + re-embed all.
|
||||||
|
- **Reflection synthesis model:** `claude-sonnet-4-6` via Anthropic Messages API direct (not OpenRouter)
|
||||||
|
- **Vault MCP write surface:** `create_artifact` + `update_artifact` are canonical; `save_note` is a deprecation shim, hidden from MCP exposure
|
||||||
|
- **Reflection dedup:** `UNIQUE (citations_hash, source_window, generated_by)` on `ob1.reflections`, citations_hash = sha256 over sorted canonicalized citation pointers
|
||||||
|
- **Service user permission model:** vault_mcp owns wiki, ob1_mcp owns ob1, trellis_mcp owns trellis; vault_mcp + lovebug get read across schemas as needed; lovebug RW everywhere
|
||||||
|
- **Vault MCP push:** HTTPS + lovebug PAT via env-driven git credential helper (function-form, written via `git config` subprocess to handle `;` in the helper string)
|
||||||
|
- **Local dir:** `/opt/projects/homelab/vault-mcp/` (renamed from herby-dev)
|
||||||
|
|
||||||
|
## Open Items
|
||||||
|
|
||||||
|
Decisions still to make before some of the work below can land:
|
||||||
|
|
||||||
|
- [ ] **n8n Email Scanner Obsidian-write half** — decide: build the parallel-write to Obsidian, build a standalone OB1 workflow instead, or defer entirely
|
||||||
|
- [ ] **claude/competent-knuth-de9483 branch direction** — merge, abandon, or partial cherry-pick? Branch is 7 commits ahead of main with artifact tooling that landed via separate path
|
||||||
|
- [ ] **When to bring `ob1-enricher` back online** — depends on whether the PENDING_QUERY loop fix is in scope or deferred indefinitely
|
||||||
|
- [ ] **When to restart `ob1-watcher`** — currently stopped per directive; ambient capture has been paused for the duration of the debugging chain
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase 1 — Service activation (small, mostly operational)
|
||||||
|
|
||||||
|
- [ ] Reflector `.env` config fix: add `REFLECTOR_CHAT_PROVIDER=anthropic` + `REFLECTOR_ANTHROPIC_MODEL=claude-sonnet-4-6`, comment out stale `REFLECTOR_CHAT_API_BASE` / `REFLECTOR_CHAT_MODEL` (the OpenAI-compat path; misconfig caused the dry-run 404)
|
||||||
|
- [ ] Re-run reflector `--dry-run --window=72h`, inspect output, iterate on the synthesis prompt if quality is off
|
||||||
|
- [ ] First real (non-dry-run) reflection synthesis once dry-run looks good
|
||||||
|
- [ ] Schedule systemd timer for daily 3am reflection job
|
||||||
|
- [ ] Restart `ob1-watcher` to resume ambient capture (writes through chunked path now)
|
||||||
|
- [ ] Fix `ob1-enricher` PENDING_QUERY loop bug — add `pending: false` to merged metadata on successful UPDATE, OR tighten WHERE clause to exclude already-enriched legacy shapes. Then restart enricher to drain the queue.
|
||||||
|
|
||||||
|
### Phase 2 — Cross-source verification (original consolidation plan Phase 7)
|
||||||
|
|
||||||
|
- [ ] Run a semantic search query that hits all three content types in one result set (vault notes + thoughts + reflections). Example: "what do we know about VLAN tagging" returns relevant rows from multiple sources.
|
||||||
|
- [ ] Verify reflection citations resolve correctly — pick a recent reflection, follow each citation back to its source row, confirm the join works
|
||||||
|
- [ ] Permission boundary stress test — try writes from each MCP service user into other schemas; all should fail with the expected permission-denied error
|
||||||
|
- [ ] Lovebug read patterns confirmed under the consolidated connection — confirm `pause_thread`, `start_thread`, `save_vault_note` (well, `create_artifact`), `search_thoughts` all work normally
|
||||||
|
|
||||||
|
### Phase 3 — Cleanup tasks (small, well-scoped, can be batched)
|
||||||
|
|
||||||
|
- [ ] Stale docstring in `reflector/src/ob1_reflector/__main__.py:14-17` (says "INSERT the embedding ... singular" — post-chunking drift)
|
||||||
|
- [ ] 2 remaining pre-existing ruff lint errors in vault_mcp/ (non-auto-fixable; was 11, rename cleanup fixed 9)
|
||||||
|
- [ ] `tests/conftest.py:59` — update `SET search_path TO herbylab` → `wiki, public` for DB-integration tests against petalbrain
|
||||||
|
- [ ] Domain enum casing inconsistency — `get_schema` documents capitalized (`Homelab`) but validator expects lowercase (`homelab`); either update schema docs or update validator to accept both
|
||||||
|
- [ ] Prune stray `branch.claude/amazing-bardeen.remote=origin` from `/vault/.git/config`
|
||||||
|
- [ ] Decide on `vault` Authentik provider display name (SETUP.md:44 says `vault`, the actual slug is `vault-mcp` — cosmetic mismatch)
|
||||||
|
|
||||||
|
### Phase 4 — Original consolidation plan Phase 8 cleanup
|
||||||
|
|
||||||
|
- [ ] Drop old standalone `trellis` DB (kept intact post-cutover for verification window; trellis-mcp now reads/writes against petalbrain.trellis)
|
||||||
|
- [ ] Drop old `herbylab` DB (per original plan, kept post-restore; wiki schema content fully migrated to petalbrain.wiki)
|
||||||
|
- [ ] Archive pre-migration dumps from `/opt/backups/postgres-consolidation/` to long-term NAS location once dumps are verified obsolete
|
||||||
|
- [ ] Update connection-string documentation in KeePassXC: petalbrain + four service users + their passwords
|
||||||
|
- [ ] Write final session note capturing consolidated shape + role model + operational commands (per-schema backup, per-user restore, etc.) — durable record for future-self
|
||||||
|
|
||||||
|
### Phase 5 — Naming consistency (defer cosmetic)
|
||||||
|
|
||||||
|
- [ ] Rename `HERBYLAB_DATABASE_URL` env var to `VAULT_ARTIFACTS_DATABASE_URL` (or similar). Ripples into deploy/.env, compose.yml, scripts, code references. Pure cosmetic since both env vars point at petalbrain via vault_mcp user post-consolidation. Defer unless a fresh consumer trips on the name.
|
||||||
|
- [ ] `alembic/env.py:41` still has `SCHEMA = "herbylab"` constant for migration target. Separate task if artifact alembic migrations should move to `wiki` schema target — would require reconciling existing migration history.
|
||||||
|
|
||||||
|
### Phase 6 — n8n Email Scanner direction
|
||||||
|
|
||||||
|
- [ ] Decide between three options for the Email Scanner workflow's currently-missing Obsidian-write half:
|
||||||
|
- (a) Build the Obsidian-write branch in the existing Email Scanner workflow
|
||||||
|
- (b) Build a standalone OB1 workflow that writes to vault via the new MCP path
|
||||||
|
- (c) Defer until a clear use case emerges
|
||||||
|
- [ ] If (a) or (b): implement and connect
|
||||||
|
|
||||||
|
### Phase 7 — Decisions in flight (mostly bookkeeping)
|
||||||
|
|
||||||
|
- [ ] `claude/competent-knuth-de9483` branch direction:
|
||||||
|
- If merge: rebase onto current main, reconcile any conflicts with the artifact tooling that landed via the rename completion + write-surface flips, decide whether the Trello projection feature ships
|
||||||
|
- If abandon: delete branch, prune `wiki.projects` + `wiki.artifacts` tables if the artifact subsystem isn't going to be developed further (but they're populated now via create_artifact, so keep)
|
||||||
|
- If partial: extract specific commits worth keeping
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
### Why this plan exists
|
||||||
|
|
||||||
|
The parent initiative landed a lot of architectural work but produced a long tail of small follow-ups. Without a plan to hold them, they decay into forgotten lore. This artifact is the holding pen — each item is small, well-scoped, and can be done independently.
|
||||||
|
|
||||||
|
### Why phase order is roughly priority order
|
||||||
|
|
||||||
|
Phase 1 (service activation) gets the most user-visible value: reflector running, watcher capturing, enricher draining metadata backlog. Phase 2 (verification) confirms the consolidation actually does what it was designed to do. Phase 3 (cleanup) is housekeeping. Phase 4 (Phase 8 cleanup from original plan) reclaims disk space + closes documentation gaps. Phases 5-7 are increasingly optional / decision-blocked.
|
||||||
|
|
||||||
|
### CLAUDE.md candidates from the debugging chain
|
||||||
|
|
||||||
|
Worth recording in `~/.claude/CLAUDE.md` or per-project CLAUDE.md files (low-priority but high-value):
|
||||||
|
|
||||||
|
- **Function-form git credential helper pattern:** when configuring git credential helpers programmatically, use `!f() { echo "username=$U"; echo "password=$T"; }; f` form (absorbs git's appended action arg as `$1`). Write via `git config` subprocess (which quotes `;`-containing values) — NOT via GitPython's `set_value` (writes raw, parser truncates at `;`).
|
||||||
|
- **Always rebuild containers with `--no-cache` after code changes** — the image cache trap bit multiple times during this initiative. `docker compose up -d` reuses the existing image unless the build context changes.
|
||||||
|
- **PL/pgSQL trigger function `SET search_path`:** when a trigger function references types or tables in a non-default schema, use `ALTER FUNCTION ... SET search_path = wiki, public` to bind the function's own search_path. Otherwise PL/pgSQL caches the resolution from the FIRST call in each session, which can fail unpredictably depending on what called the function first.
|
||||||
|
- **Anthropic Messages API endpoint shape:** `https://api.anthropic.com/v1/messages` (NO `/chat/completions` suffix). The model name doesn't take the `anthropic/` provider prefix that OpenRouter uses. Request body uses top-level `system` field + `messages` array. Headers: `x-api-key` + `anthropic-version: 2023-06-01`.
|
||||||
|
|
||||||
|
### Risks worth tracking
|
||||||
|
|
||||||
|
- **Old DBs occupying disk** until Phase 4 — minor since the dumps are tiny
|
||||||
|
- **Enricher metadata backlog growing** until the loop fix + restart — ambient capture continues, just gets stuck on metadata extraction
|
||||||
|
- **Watcher off means ambient capture gap** — every day off-line is a day of session content not landing in ob1.thoughts
|
||||||
|
- **HERBYLAB_DATABASE_URL ambiguity** — anyone joining the project trips on the name pointing at petalbrain; minor confusion tax
|
||||||
|
|
||||||
|
### What's deliberately NOT in this plan
|
||||||
|
|
||||||
|
- The parent initiative itself (it's documented in `postgres-consolidation-reflection-layer` and largely complete)
|
||||||
|
- The 13-step vault MCP write surface debugging chain (documented in the trellis thread checkpoint)
|
||||||
|
- Any new features (this plan is purely cleanup + activation of the consolidation work)
|
||||||
Loading…
Reference in New Issue
Block a user