wiki-vault/Sources/Homelab/postgres-consolidation-followups.md

created	path	project	tags	type
2026-05-17	Sources/Homelab	postgres-consolidation-followups	ob1 mcp pgvector embeddings automation claude-code	project-plan

Goal

Close out the deferred work items from the postgres-consolidation-reflection-layer initiative. The major architectural work landed (DB consolidated to petalbrain with three schemas and four service users, chunked embeddings live in public.embeddings, vault MCP write surface fully operational, trellis cutover complete, reflector built). What remains is a mix of small operational tasks, the original plan's Phase 7/8 verification + cleanup, and one architectural decision (n8n parallel-write).

This plan exists so the follow-ups aren't lost. Most items are independent and can be tackled in any order. Phases are grouping by intent, not strict dependency.

Locked Decisions

These are settled from the parent initiative; record them here so future work doesn't re-litigate:

• Table naming: ob1.thoughts (not captures), wiki.artifacts + wiki.projects + wiki.notes, trellis.{threads,events,tags,thread_tags}, ob1.reflections
• Embedding storage: all embeddings in public.embeddings (shared across sources); HNSW vector_cosine_ops m=16 ef_construction=64
• Chunking: 512-token chunks, 64-token overlap, tiktoken cl100k_base, applied at every embedding writer (vault_mcp, ob1-mcp, ob1-reflector; not ob1-enricher which is metadata-only)
• Embedding model: nomic-embed-text via local Ollama (768-d). Switching requires schema migration + re-embed all.
• Reflection synthesis model: claude-sonnet-4-6 via Anthropic Messages API direct (not OpenRouter)
• Vault MCP write surface: create_artifact + update_artifact are canonical; save_note is a deprecation shim, hidden from MCP exposure
• Reflection dedup: UNIQUE (citations_hash, source_window, generated_by) on ob1.reflections, citations_hash = sha256 over sorted canonicalized citation pointers
• Service user permission model: vault_mcp owns wiki, ob1_mcp owns ob1, trellis_mcp owns trellis; vault_mcp + lovebug get read across schemas as needed; lovebug RW everywhere
• Vault MCP push: HTTPS + lovebug PAT via env-driven git credential helper (function-form, written via git config subprocess to handle ; in the helper string)
• Local dir: /opt/projects/homelab/vault-mcp/ (renamed from herby-dev)

Open Items

Decisions still to make before some of the work below can land:

• n8n Email Scanner Obsidian-write half — decide: build the parallel-write to Obsidian, build a standalone OB1 workflow instead, or defer entirely
• claude/competent-knuth-de9483 branch direction — merge, abandon, or partial cherry-pick? Branch is 7 commits ahead of main with artifact tooling that landed via separate path
• When to bring ob1-enricher back online — depends on whether the PENDING_QUERY loop fix is in scope or deferred indefinitely
• When to restart ob1-watcher — currently stopped per directive; ambient capture has been paused for the duration of the debugging chain

Phases

Phase 1 — Service activation (small, mostly operational)

• Reflector .env config fix: add REFLECTOR_CHAT_PROVIDER=anthropic + REFLECTOR_ANTHROPIC_MODEL=claude-sonnet-4-6, comment out stale REFLECTOR_CHAT_API_BASE / REFLECTOR_CHAT_MODEL (the OpenAI-compat path; misconfig caused the dry-run 404)
• Re-run reflector --dry-run --window=72h, inspect output, iterate on the synthesis prompt if quality is off
• First real (non-dry-run) reflection synthesis once dry-run looks good
• Schedule systemd timer for daily 3am reflection job
• Restart ob1-watcher to resume ambient capture (writes through chunked path now)
• Fix ob1-enricher PENDING_QUERY loop bug — add pending: false to merged metadata on successful UPDATE, OR tighten WHERE clause to exclude already-enriched legacy shapes. Then restart enricher to drain the queue.

Phase 2 — Cross-source verification (original consolidation plan Phase 7)

• Run a semantic search query that hits all three content types in one result set (vault notes + thoughts + reflections). Example: "what do we know about VLAN tagging" returns relevant rows from multiple sources.
• Verify reflection citations resolve correctly — pick a recent reflection, follow each citation back to its source row, confirm the join works
• Permission boundary stress test — try writes from each MCP service user into other schemas; all should fail with the expected permission-denied error
• Lovebug read patterns confirmed under the consolidated connection — confirm pause_thread, start_thread, save_vault_note (well, create_artifact), search_thoughts all work normally

Phase 3 — Cleanup tasks (small, well-scoped, can be batched)

• Stale docstring in reflector/src/ob1_reflector/__main__.py:14-17 (says "INSERT the embedding ... singular" — post-chunking drift)
• 2 remaining pre-existing ruff lint errors in vault_mcp/ (non-auto-fixable; was 11, rename cleanup fixed 9)
• tests/conftest.py:59 — update SET search_path TO herbylab → wiki, public for DB-integration tests against petalbrain
• Domain enum casing inconsistency — get_schema documents capitalized (Homelab) but validator expects lowercase (homelab); either update schema docs or update validator to accept both
• Prune stray branch.claude/amazing-bardeen.remote=origin from /vault/.git/config
• Decide on vault Authentik provider display name (SETUP.md:44 says vault, the actual slug is vault-mcp — cosmetic mismatch)

Phase 4 — Original consolidation plan Phase 8 cleanup

• Drop old standalone trellis DB (kept intact post-cutover for verification window; trellis-mcp now reads/writes against petalbrain.trellis)
• Drop old herbylab DB (per original plan, kept post-restore; wiki schema content fully migrated to petalbrain.wiki)
• Archive pre-migration dumps from /opt/backups/postgres-consolidation/ to long-term NAS location once dumps are verified obsolete
• Update connection-string documentation in KeePassXC: petalbrain + four service users + their passwords
• Write final session note capturing consolidated shape + role model + operational commands (per-schema backup, per-user restore, etc.) — durable record for future-self

Phase 5 — Naming consistency (defer cosmetic)

• Rename HERBYLAB_DATABASE_URL env var to VAULT_ARTIFACTS_DATABASE_URL (or similar). Ripples into deploy/.env, compose.yml, scripts, code references. Pure cosmetic since both env vars point at petalbrain via vault_mcp user post-consolidation. Defer unless a fresh consumer trips on the name.
• alembic/env.py:41 still has SCHEMA = "herbylab" constant for migration target. Separate task if artifact alembic migrations should move to wiki schema target — would require reconciling existing migration history.

Phase 6 — n8n Email Scanner direction

• Decide between three options for the Email Scanner workflow's currently-missing Obsidian-write half:
  • (a) Build the Obsidian-write branch in the existing Email Scanner workflow
  • (b) Build a standalone OB1 workflow that writes to vault via the new MCP path
  • (c) Defer until a clear use case emerges
• If (a) or (b): implement and connect

Phase 7 — Decisions in flight (mostly bookkeeping)

• claude/competent-knuth-de9483 branch direction:
  • If merge: rebase onto current main, reconcile any conflicts with the artifact tooling that landed via the rename completion + write-surface flips, decide whether the Trello projection feature ships
  • If abandon: delete branch, prune wiki.projects + wiki.artifacts tables if the artifact subsystem isn't going to be developed further (but they're populated now via create_artifact, so keep)
  • If partial: extract specific commits worth keeping

Notes

Why this plan exists

The parent initiative landed a lot of architectural work but produced a long tail of small follow-ups. Without a plan to hold them, they decay into forgotten lore. This artifact is the holding pen — each item is small, well-scoped, and can be done independently.

Why phase order is roughly priority order

Phase 1 (service activation) gets the most user-visible value: reflector running, watcher capturing, enricher draining metadata backlog. Phase 2 (verification) confirms the consolidation actually does what it was designed to do. Phase 3 (cleanup) is housekeeping. Phase 4 (Phase 8 cleanup from original plan) reclaims disk space + closes documentation gaps. Phases 5-7 are increasingly optional / decision-blocked.

CLAUDE.md candidates from the debugging chain

Worth recording in ~/.claude/CLAUDE.md or per-project CLAUDE.md files (low-priority but high-value):

• Function-form git credential helper pattern: when configuring git credential helpers programmatically, use !f() { echo "username=$U"; echo "password=$T"; }; f form (absorbs git's appended action arg as $1). Write via git config subprocess (which quotes ;-containing values) — NOT via GitPython's set_value (writes raw, parser truncates at ;).
• Always rebuild containers with --no-cache after code changes — the image cache trap bit multiple times during this initiative. docker compose up -d reuses the existing image unless the build context changes.
• PL/pgSQL trigger function SET search_path: when a trigger function references types or tables in a non-default schema, use ALTER FUNCTION ... SET search_path = wiki, public to bind the function's own search_path. Otherwise PL/pgSQL caches the resolution from the FIRST call in each session, which can fail unpredictably depending on what called the function first.
• Anthropic Messages API endpoint shape: https://api.anthropic.com/v1/messages (NO /chat/completions suffix). The model name doesn't take the anthropic/ provider prefix that OpenRouter uses. Request body uses top-level system field + messages array. Headers: x-api-key + anthropic-version: 2023-06-01.

Risks worth tracking

• Old DBs occupying disk until Phase 4 — minor since the dumps are tiny
• Enricher metadata backlog growing until the loop fix + restart — ambient capture continues, just gets stuck on metadata extraction
• Watcher off means ambient capture gap — every day off-line is a day of session content not landing in ob1.thoughts
• HERBYLAB_DATABASE_URL ambiguity — anyone joining the project trips on the name pointing at petalbrain; minor confusion tax

What's deliberately NOT in this plan

• The parent initiative itself (it's documented in postgres-consolidation-reflection-layer and largely complete)
• The 13-step vault MCP write surface debugging chain (documented in the trellis thread checkpoint)
• Any new features (this plan is purely cleanup + activation of the consolidation work)