petal-power/wiki-vault
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
12a57499cb
wiki-vault/Sources/Homelab/2026-05-15-postgres-consolidation-phase0-recon.md
Travis Herbranson 52e3e76250 adding 2 files
2026-05-16 14:32:30 -04:00

13 KiB
Raw Blame History

created path project status tags type updated worker
2026-05-15 Sources/Homelab postgres-consolidation-reflection-layer active
ob1
mcp
pgvector
embeddings
session-note
session-note 2026-05-15 2

Postgres Consolidation — Worker 2, Phase 0 recon

Snapshot of the homelab Postgres instance before any consolidation work. Recorded from homelab-postgres (pgvector/pgvector:pg17), container in /opt/projects/docker/postgres/, port 127.0.0.1:5433, volume ob1_data (external).

Postgres instance

  • Container: homelab-postgres (image pgvector/pgvector:pg17)
  • Volume: ob1_data (external — survives stack down)
  • Network: homelab (alias homelab-postgres, also postgres)
  • Healthy, up 40 hours.
  • POSTGRES_DB default = openbrain.

Databases on instance

Name Owner Size pgvector Notes
herbylab herbylab_user 7958 kB NO Holds the herby-dev artifact store
openbrain postgres 8838 kB YES (0.8.2) OB1 captures live here
trellis trellis_user 8262 kB NO Trellis MCP threads/events
postgres postgres 7478 kB - default; not touched

No other user databases. Plan's "instance hosts only wiki, ob1, trellis" holds — under the actual names herbylab / openbrain / trellis.

Existing roles

  • postgres — Superuser, Create role, Create DB, Replication, Bypass RLS
  • herbylab_user — owns herbylab.herbylab schema
  • trellis_user — owns trellis.trellis schema

No lovebug role exists. No ob1_mcp / vault_mcp / trellis_mcp service users yet — these come in Phase 2.

Schemas, tables, row counts

herbylab DB

  • Schemas: herbylab (owner herbylab_user), public.
  • Tables in herbylab:
    • projects — 0 rows (8192 B; one empty heap page)
    • artifacts — 0 rows (8192 B; one empty heap page)
    • alembic_version — 1 row (9c7e3f1a2b4d)
  • Custom enums: herbylab.project_status, herbylab.artifact_type, herbylab.project_domain.
  • Triggers / functions: projects_notify (NOTIFY on row change), artifacts_set_updated_at.
  • No vector columns.

Why empty (verified, not a missed lookup): the schema was created by alembic migration 9c7e3f1a2b4d on the unmerged branch claude/competent-knuth-de9483 (commit 90bc6a9 Phase 1: herbylab Postgres schema infrastructure). That branch also has commits 8e408c5 Phase 2: polymorphic artifact tools backed by Postgres + vault and 78ec094 Phase 3: Trello projection via n8n which would have wired writes into the schema, but the branch is not merged to main.

The deployed herbydev-mcp runs main's code, which has zero DB access:

  • pyproject.toml declares no DB driver (no psycopg / sqlalchemy / asyncpg).
  • mcp_server/tools/memory.py and mcp_server/tools/tasks.py are empty stub files marked "Phase 2. Do not implement until workflow patterns are established."
  • mcp_server/tools/vault.py writes via core/git_writer.py → filesystem + git commit + push to gitea. No database insertion.
  • HERBYLAB_DATABASE_URL is plumbed through compose env but never read by the running code.

So the DB schema exists from an integration-test run of the unmerged branch's migration; no code path on main ever populates it.

Implication for consolidation: the wiki schema in petalbrain is genuinely empty content-wise. Worker 4's vault-indexing source is the filesystem (/opt/projects/wiki-vault/), not a wiki.notes / wiki.artifacts table. If competent-knuth-de9483 lands later, its artifact tooling will need to be re-pointed at petalbrain.wiki.* instead of herbylab.herbylab.*, and the Phase 4 reflection design should account for wiki.artifacts becoming a populated source over time.

openbrain DB

  • Schemas: public only.
  • Tables in public:
    • thoughts — 70 rows (live; daemons writing today)
  • pgvector 0.8.2 in public.

public.thoughts shape (relevant columns):

id          bigint            PK (nextval thoughts_id_seq)
content     text              NOT NULL
embedding   vector(768)
metadata    jsonb             DEFAULT '{}'
user_id     text              NOT NULL DEFAULT 'travis'
created_at  timestamptz       DEFAULT CURRENT_TIMESTAMP
updated_at  timestamptz       DEFAULT CURRENT_TIMESTAMP

Existing indexes on public.thoughts:

  • thoughts_pkey (id)
  • idx_thoughts_created_at (created_at DESC)
  • idx_thoughts_embedding_hnsw — hnsw (embedding vector_cosine_ops) WITH (m=16, ef_construction=64) — already matches the locked Phase 4 spec for public.embeddings.
  • idx_thoughts_fingerprint — UNIQUE (md5(content), user_id)
  • idx_thoughts_metadata — gin (metadata)
  • idx_thoughts_user_created (user_id, created_at DESC)
  • idx_thoughts_user_id (user_id)

Additional objects:

  • Function match_thoughts(query_embedding vector, match_threshold float, match_count int, filter jsonb, p_user_id text) — bare-vector param (no dim), so dimension changes flow through unchanged.
  • Trigger thoughts_updated_at -> update_updated_at().

Live writer state: 70 rows total, 0 with NULL embedding, 2 still flagged metadata->>'pending' = 'true' (enricher backlog). Most recent row id 255 at 2026-05-16 02:07:23 UTC. Watcher and enricher daemons confirmed live.

trellis DB

  • Schemas: trellis (owner trellis_user), public.
  • Tables in trellis:
    • threads — 13 rows
    • events — 50 rows
    • tags — 42 rows
    • thread_tags — 55 rows
    • alembic_version — 1 row
  • No vector columns.

OB1 embedding model

From ob1-mcp container env (and /opt/projects/homelab/ob1-deploy/compose.yml):

  • EMBEDDING_PROVIDER=ollama
  • OLLAMA_URL=http://ollama:11434
  • EMBEDDING_MODEL=nomic-embed-text (768-d output)
  • CHAT_MODEL=openai/gpt-4o-mini (mcp side)
  • CHAT_MODEL=llama3.2:3b (enricher side)

Migration trail:

  • init.sql originally declared vector(1536) for text-embedding-3-small.
  • migrations/001_embedding_dim_768.sql (applied 2026-05-14) dropped the HNSW index, altered the column to vector(768), recreated the index. Required thoughts to be empty pre-migration.

Conclusion: the shared public.embeddings table goes in as vector(768) to match nomic-embed-text, no model switch needed. HNSW index spec from Travis's overrides (vector_cosine_ops, m=16, ef_construction=64) matches the existing OB1 convention exactly.

MCP server connection configurations

Container Image DB host / DB / user Live?
herbydev-mcp herbydev-mcp homelab-postgres:5432/herbylab as herbylab_user Up 40h
vault-mcp vault-mcp vault-postgres:5432/herbylab as herbylab_user Up 59m
ob1-mcp ob1-mcp homelab-postgres:5432/openbrain as postgres Up 14h
ob1-watcher ob1-watcher (talks to ob1-mcp HTTP, not DB directly) Up 26h
ob1-enricher ob1-enricher (uses OLLAMA_URL; DB password set but no direct connect string visible) Up 14h
trellis-mcp trellis-trellis-mcp homelab-postgres:5432/trellis as trellis_user Up 40h healthy

Connection strings live in:

  • vault-mcp / herbydev-mcp — /opt/projects/homelab/herby-dev/deploy/.env (HERBYLAB_DATABASE_URL)
  • ob1-mcp / ob1-enricher — /opt/projects/homelab/ob1-deploy/.env (POSTGRES_PASSWORD + per-service DB_* env in compose.yml)
  • trellis-mcp — /opt/projects/homelab/trellis-mcp/ (compose env, DATABASE_URL)

Anomaly: vault-mcp references hostname vault-postgres which does not resolve on the homelab network (alias is homelab-postgres

  • postgres). This is consistent with Travis's mid-flight rename rollout — the container is currently running OLD code per the priming, and the connection-string update is part of the unfinished rollout. Not blocking Phase 1, but Phase 3 must coordinate with the rename completion.

Backup destination

Local-only on herbydev (Travis decision, 2026-05-15). The NAS reference in the plan was unintended — disregard it.

  • Destination directory: /opt/backups/postgres-consolidation/ (does not exist yet; Phase 1 step zero is mkdir -p it).
  • Local disk on /: 14 G free of 97 G (86% used).
  • Combined DB size is ~25 MB, so headroom is ample for both Phase 1 bulk dumps and Phase 2-start fresh dumps.
  • Format: pg_dumpall (full instance) + per-database pg_dump in custom format (-Fc), gzipped. Filenames include UTC timestamp.
  • No NAS / Tailscale / off-host destination involved.

Coordination state — running daemons

ob1-watcher and ob1-enricher are both live and writing to openbrain.public.thoughts (latest write ~5 minutes before this note). Per priming, the Phase 2 ALTER TABLE public.thoughts SET SCHEMA ob1 step is the cutover gate: daemons must be paused, table moved + ob1.thoughts shape rewrite by Worker 3, then daemons resumed. Worker 2 stops at the announcement of readiness for that move and waits.

Naming reconciliation (plan vs. reality)

Plan vocabulary wiki / ob1 / trellis will land as the schema names in the consolidated database, per Travis's locked decisions. Source DB → target schema mapping:

  • herbylab DB content → wiki schema (owned by vault_mcp per override; was herby_mcp in the plan)
  • openbrain DB content → ob1 schema; public.thoughts becomes ob1.thoughts (preserving thoughts vocabulary per override; plan said ob1.captures)
  • trellis DB content → trellis schema (owned by trellis_mcp)

Plus shared public.embeddings per Phase 4.

Open: target database name itself. OB1's database is currently openbrain. The plan says "if OB1's database is the target and is named appropriately, use in place. Otherwise, rename or create new." openbrain becomes a misleading name once it also holds wiki + trellis schemas. Worth a deliberate rename (e.g., homelab, petalbrain, or similar) before Phase 2 work begins.

Phase 1 readiness assessment

Ready:

  • Source DB sizes are trivial; dumps will be small and fast.
  • pgvector already present in target DB (openbrain); no extension install needed.
  • Embedding model + dim are confirmed and stable (nomic-embed-text / 768) — no re-embed forced.
  • Existing HNSW spec on public.thoughts matches the locked public.embeddings HNSW spec exactly.

Not ready / needs decision:

  • NAS backup destination Resolved 2026-05-15: local-only at /opt/backups/postgres-consolidation/. No NAS involved.
  • Target database name Resolved 2026-05-15: petalbrain. openbrain renamed before Phase 2 destructive ops begin.
  • wiki schema content scope — herbylab DB schema came from the unmerged claude/competent-knuth-de9483 branch's migration 9c7e3f1a2b4d; main's deployed MCP doesn't open a DB connection at all. Wiki content is filesystem-only under /opt/projects/wiki-vault/. Worker 4's vault-indexing source must walk the filesystem and write into public.embeddings directly. See the herbylab-DB section above for the full evidence trail.
  • Connection-string anomaly in vault-mcp (vault-postgres hostname). Tied to Travis's rename rollout, blocks Phase 3 cutover for that MCP only. Phase 1/2 unaffected.

Conservative call: gate before Phase 1 until the NAS destination and target DB name decisions are made.

Addendum — Phase 2-start fresh backup (Travis directive, 2026-05-15)

In addition to Phase 1 bulk backups, Phase 2 must begin with a fresh pg_dump of each source database immediately before the first destructive operation (no CREATE SCHEMA, no ALTER TABLE, no restore — nothing destructive precedes this step).

Rationale: ob1-watcher is writing to public.thoughts continuously (latest write ~5 min before this note). Time elapses between Phase 1 dumps and Phase 2 cutover, especially with gating. Any rows captured in that window would be lost on rollback if Phase 1 dumps are the only restore point.

Specifics:

  • Run fresh pg_dump on openbrain first — it's the live writer and the most critical. Then herbylab and trellis for symmetry.
  • Destination: /opt/backups/postgres-consolidation/ (same as Phase 1), gzipped custom format.
  • Filename suffix: -phase2-start-<UTC-timestamp> to distinguish from Phase 1 backups.
  • Verify each fresh dump is readable via test restore to a scratch database (same pattern as Phase 1).
  • Only after the three fresh dumps are verified does Phase 2's first destructive operation run.

This is not a replacement for Phase 1 dumps — Phase 1 remains the broad safety net (full instance pg_dumpall + per-DB granular dumps). Phase 2-start dumps are the just-before-disruption snapshot that captures any in-flight writer activity.

Operational note: this is also the natural point to pause ob1-watcher and ob1-enricher. Sequence:

  1. Pause both daemons (docker compose stop ob1-watcher ob1-enricher).
  2. Confirm no in-flight writes (pg_stat_activity quiet on openbrain).
  3. Take the fresh per-DB dumps + verify.
  4. Begin Phase 2 destructive work.
  5. (Worker 3 resumes daemons after ob1.thoughts cutover lands.)