--- created: '2026-05-15' path: Sources/Homelab project: postgres-consolidation-reflection-layer status: active tags: - ob1 - mcp - pgvector - embeddings - session-note type: session-note updated: '2026-05-15' worker: 2 --- # Postgres Consolidation — Worker 2, Phase 0 recon Snapshot of the homelab Postgres instance before any consolidation work. Recorded from `homelab-postgres` (pgvector/pgvector:pg17), container in `/opt/projects/docker/postgres/`, port `127.0.0.1:5433`, volume `ob1_data` (external). ## Postgres instance - Container: `homelab-postgres` (image `pgvector/pgvector:pg17`) - Volume: `ob1_data` (external — survives stack down) - Network: `homelab` (alias `homelab-postgres`, also `postgres`) - Healthy, up 40 hours. - POSTGRES_DB default = `openbrain`. ## Databases on instance | Name | Owner | Size | pgvector | Notes | |-------------|-----------------|---------|----------|-------| | `herbylab` | `herbylab_user` | 7958 kB | NO | Holds the herby-dev artifact store | | `openbrain` | `postgres` | 8838 kB | YES (0.8.2) | OB1 captures live here | | `trellis` | `trellis_user` | 8262 kB | NO | Trellis MCP threads/events | | `postgres` | `postgres` | 7478 kB | - | default; not touched | No other user databases. Plan's "instance hosts only wiki, ob1, trellis" holds — under the actual names `herbylab` / `openbrain` / `trellis`. ## Existing roles - `postgres` — Superuser, Create role, Create DB, Replication, Bypass RLS - `herbylab_user` — owns `herbylab.herbylab` schema - `trellis_user` — owns `trellis.trellis` schema No `lovebug` role exists. No `ob1_mcp` / `vault_mcp` / `trellis_mcp` service users yet — these come in Phase 2. ## Schemas, tables, row counts ### `herbylab` DB - Schemas: `herbylab` (owner `herbylab_user`), `public`. - Tables in `herbylab`: - `projects` — 0 rows (8192 B; one empty heap page) - `artifacts` — 0 rows (8192 B; one empty heap page) - `alembic_version` — 1 row (`9c7e3f1a2b4d`) - Custom enums: `herbylab.project_status`, `herbylab.artifact_type`, `herbylab.project_domain`. - Triggers / functions: `projects_notify` (NOTIFY on row change), `artifacts_set_updated_at`. - No vector columns. **Why empty (verified, not a missed lookup):** the schema was created by alembic migration `9c7e3f1a2b4d` on the unmerged branch `claude/competent-knuth-de9483` (commit `90bc6a9 Phase 1: herbylab Postgres schema infrastructure`). That branch also has commits `8e408c5 Phase 2: polymorphic artifact tools backed by Postgres + vault` and `78ec094 Phase 3: Trello projection via n8n` which would have wired writes into the schema, but **the branch is not merged to `main`.** The deployed `herbydev-mcp` runs `main`'s code, which has zero DB access: - `pyproject.toml` declares no DB driver (no psycopg / sqlalchemy / asyncpg). - `mcp_server/tools/memory.py` and `mcp_server/tools/tasks.py` are empty stub files marked "Phase 2. Do not implement until workflow patterns are established." - `mcp_server/tools/vault.py` writes via `core/git_writer.py` → filesystem + git commit + push to gitea. No database insertion. - `HERBYLAB_DATABASE_URL` is plumbed through compose env but never read by the running code. So the DB schema exists from an integration-test run of the unmerged branch's migration; no code path on `main` ever populates it. Implication for consolidation: the `wiki` schema in `petalbrain` is genuinely empty content-wise. Worker 4's vault-indexing source is the filesystem (`/opt/projects/wiki-vault/`), not a `wiki.notes` / `wiki.artifacts` table. If `competent-knuth-de9483` lands later, its artifact tooling will need to be re-pointed at `petalbrain.wiki.*` instead of `herbylab.herbylab.*`, and the Phase 4 reflection design should account for `wiki.artifacts` becoming a populated source over time. ### `openbrain` DB - Schemas: `public` only. - Tables in `public`: - `thoughts` — 70 rows (live; daemons writing today) - pgvector 0.8.2 in `public`. `public.thoughts` shape (relevant columns): ``` id bigint PK (nextval thoughts_id_seq) content text NOT NULL embedding vector(768) metadata jsonb DEFAULT '{}' user_id text NOT NULL DEFAULT 'travis' created_at timestamptz DEFAULT CURRENT_TIMESTAMP updated_at timestamptz DEFAULT CURRENT_TIMESTAMP ``` Existing indexes on `public.thoughts`: - `thoughts_pkey` (id) - `idx_thoughts_created_at` (created_at DESC) - `idx_thoughts_embedding_hnsw` — **hnsw (embedding vector_cosine_ops) WITH (m=16, ef_construction=64)** — already matches the locked Phase 4 spec for `public.embeddings`. - `idx_thoughts_fingerprint` — UNIQUE (md5(content), user_id) - `idx_thoughts_metadata` — gin (metadata) - `idx_thoughts_user_created` (user_id, created_at DESC) - `idx_thoughts_user_id` (user_id) Additional objects: - Function `match_thoughts(query_embedding vector, match_threshold float, match_count int, filter jsonb, p_user_id text)` — bare-`vector` param (no dim), so dimension changes flow through unchanged. - Trigger `thoughts_updated_at` -> `update_updated_at()`. Live writer state: 70 rows total, 0 with NULL embedding, 2 still flagged `metadata->>'pending' = 'true'` (enricher backlog). Most recent row id 255 at `2026-05-16 02:07:23 UTC`. Watcher and enricher daemons confirmed live. ### `trellis` DB - Schemas: `trellis` (owner `trellis_user`), `public`. - Tables in `trellis`: - `threads` — 13 rows - `events` — 50 rows - `tags` — 42 rows - `thread_tags` — 55 rows - `alembic_version` — 1 row - No vector columns. ## OB1 embedding model From `ob1-mcp` container env (and `/opt/projects/homelab/ob1-deploy/compose.yml`): - `EMBEDDING_PROVIDER=ollama` - `OLLAMA_URL=http://ollama:11434` - `EMBEDDING_MODEL=nomic-embed-text` (768-d output) - `CHAT_MODEL=openai/gpt-4o-mini` (mcp side) - `CHAT_MODEL=llama3.2:3b` (enricher side) Migration trail: - `init.sql` originally declared `vector(1536)` for `text-embedding-3-small`. - `migrations/001_embedding_dim_768.sql` (applied 2026-05-14) dropped the HNSW index, altered the column to `vector(768)`, recreated the index. Required `thoughts` to be empty pre-migration. **Conclusion:** the shared `public.embeddings` table goes in as `vector(768)` to match `nomic-embed-text`, no model switch needed. HNSW index spec from Travis's overrides (`vector_cosine_ops, m=16, ef_construction=64`) matches the existing OB1 convention exactly. ## MCP server connection configurations | Container | Image | DB host / DB / user | Live? | |-----------------|--------------------|---------------------|-------| | `herbydev-mcp` | `herbydev-mcp` | `homelab-postgres:5432/herbylab` as `herbylab_user` | Up 40h | | `vault-mcp` | `vault-mcp` | **`vault-postgres:5432/herbylab` as `herbylab_user`** | Up 59m | | `ob1-mcp` | `ob1-mcp` | `homelab-postgres:5432/openbrain` as `postgres` | Up 14h | | `ob1-watcher` | `ob1-watcher` | (talks to ob1-mcp HTTP, not DB directly) | Up 26h | | `ob1-enricher` | `ob1-enricher` | (uses `OLLAMA_URL`; DB password set but no direct connect string visible) | Up 14h | | `trellis-mcp` | `trellis-trellis-mcp` | `homelab-postgres:5432/trellis` as `trellis_user` | Up 40h healthy | Connection strings live in: - `vault-mcp` / `herbydev-mcp` — `/opt/projects/homelab/herby-dev/deploy/.env` (`HERBYLAB_DATABASE_URL`) - `ob1-mcp` / `ob1-enricher` — `/opt/projects/homelab/ob1-deploy/.env` (`POSTGRES_PASSWORD` + per-service `DB_*` env in `compose.yml`) - `trellis-mcp` — `/opt/projects/homelab/trellis-mcp/` (compose env, `DATABASE_URL`) **Anomaly:** `vault-mcp` references hostname `vault-postgres` which does not resolve on the `homelab` network (alias is `homelab-postgres` + `postgres`). This is consistent with Travis's mid-flight rename rollout — the container is currently running OLD code per the priming, and the connection-string update is part of the unfinished rollout. Not blocking Phase 1, but Phase 3 must coordinate with the rename completion. ## Backup destination **Local-only on herbydev** (Travis decision, 2026-05-15). The NAS reference in the plan was unintended — disregard it. - Destination directory: `/opt/backups/postgres-consolidation/` (does not exist yet; Phase 1 step zero is `mkdir -p` it). - Local disk on `/`: 14 G free of 97 G (86% used). - Combined DB size is ~25 MB, so headroom is ample for both Phase 1 bulk dumps and Phase 2-start fresh dumps. - Format: `pg_dumpall` (full instance) + per-database `pg_dump` in custom format (`-Fc`), gzipped. Filenames include UTC timestamp. - No NAS / Tailscale / off-host destination involved. ## Coordination state — running daemons `ob1-watcher` and `ob1-enricher` are both live and writing to `openbrain.public.thoughts` (latest write ~5 minutes before this note). Per priming, the Phase 2 `ALTER TABLE public.thoughts SET SCHEMA ob1` step is the cutover gate: daemons must be paused, table moved + ob1.thoughts shape rewrite by Worker 3, then daemons resumed. Worker 2 stops at the announcement of readiness for that move and waits. ## Naming reconciliation (plan vs. reality) Plan vocabulary `wiki / ob1 / trellis` will land as the **schema** names in the consolidated database, per Travis's locked decisions. Source DB → target schema mapping: - `herbylab` DB content → `wiki` schema (owned by `vault_mcp` per override; was `herby_mcp` in the plan) - `openbrain` DB content → `ob1` schema; `public.thoughts` becomes `ob1.thoughts` (preserving `thoughts` vocabulary per override; plan said `ob1.captures`) - `trellis` DB content → `trellis` schema (owned by `trellis_mcp`) Plus shared `public.embeddings` per Phase 4. **Open: target database name itself.** OB1's database is currently `openbrain`. The plan says "if OB1's database is the target and is named appropriately, use in place. Otherwise, rename or create new." `openbrain` becomes a misleading name once it also holds wiki + trellis schemas. Worth a deliberate rename (e.g., `homelab`, `petalbrain`, or similar) before Phase 2 work begins. ## Phase 1 readiness assessment Ready: - Source DB sizes are trivial; dumps will be small and fast. - pgvector already present in target DB (`openbrain`); no extension install needed. - Embedding model + dim are confirmed and stable (`nomic-embed-text` / 768) — no re-embed forced. - Existing HNSW spec on `public.thoughts` matches the locked `public.embeddings` HNSW spec exactly. Not ready / needs decision: - ~~NAS backup destination~~ **Resolved 2026-05-15**: local-only at `/opt/backups/postgres-consolidation/`. No NAS involved. - ~~Target database name~~ **Resolved 2026-05-15**: `petalbrain`. `openbrain` renamed before Phase 2 destructive ops begin. - **`wiki` schema content scope** — `herbylab` DB schema came from the unmerged `claude/competent-knuth-de9483` branch's migration `9c7e3f1a2b4d`; `main`'s deployed MCP doesn't open a DB connection at all. Wiki content is filesystem-only under `/opt/projects/wiki-vault/`. Worker 4's vault-indexing source must walk the filesystem and write into `public.embeddings` directly. See the herbylab-DB section above for the full evidence trail. - **Connection-string anomaly in `vault-mcp`** (`vault-postgres` hostname). Tied to Travis's rename rollout, blocks Phase 3 cutover for that MCP only. Phase 1/2 unaffected. Conservative call: gate before Phase 1 until the NAS destination and target DB name decisions are made. ## Addendum — Phase 2-start fresh backup (Travis directive, 2026-05-15) **In addition to Phase 1 bulk backups**, Phase 2 must begin with a fresh `pg_dump` of each source database immediately before the first destructive operation (no `CREATE SCHEMA`, no `ALTER TABLE`, no restore — nothing destructive precedes this step). Rationale: `ob1-watcher` is writing to `public.thoughts` continuously (latest write ~5 min before this note). Time elapses between Phase 1 dumps and Phase 2 cutover, especially with gating. Any rows captured in that window would be lost on rollback if Phase 1 dumps are the only restore point. Specifics: - Run fresh `pg_dump` on `openbrain` first — it's the live writer and the most critical. Then `herbylab` and `trellis` for symmetry. - Destination: `/opt/backups/postgres-consolidation/` (same as Phase 1), gzipped custom format. - Filename suffix: `-phase2-start-` to distinguish from Phase 1 backups. - Verify each fresh dump is readable via test restore to a scratch database (same pattern as Phase 1). - Only after the three fresh dumps are verified does Phase 2's first destructive operation run. This is not a replacement for Phase 1 dumps — Phase 1 remains the broad safety net (full instance `pg_dumpall` + per-DB granular dumps). Phase 2-start dumps are the just-before-disruption snapshot that captures any in-flight writer activity. Operational note: this is also the natural point to pause `ob1-watcher` and `ob1-enricher`. Sequence: 1. Pause both daemons (`docker compose stop ob1-watcher ob1-enricher`). 2. Confirm no in-flight writes (`pg_stat_activity` quiet on `openbrain`). 3. Take the fresh per-DB dumps + verify. 4. Begin Phase 2 destructive work. 5. (Worker 3 resumes daemons after `ob1.thoughts` cutover lands.)