second-brain/postgres-migration-planning.md

Postgres + pgvector migration — implementation notes

Status: shipped (2026-05-24). Originally a planning checklist; rewritten to capture the implemented design and the decisions that landed. Kept in the repo so the next iteration has the "why" without grepping commits.

---

What shipped

• Backing store switched from SQLite to the existing petalbrain Postgres instance (pgvector/pg17), connecting over the homelab docker network at homelab-postgres:5432.
• Relational tables (sources, extractions, wiki_pages) live in a new second_brain schema owned by the lovebug role — no dedicated service role.
• Embeddings reuse the shared public.embeddings table (HNSW vector index already in place). Rows are keyed by (source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text'). For this round we embed extraction summaries only (not transcript chunks, not key_points / claims).
• Embedding pipeline mirrors vault_mcp.core.embeddings: shared embedding_chunking for 512/64-token chunking → Ollama nomic-embed-text (768-d, num_ctx=8192) → delete-before-insert upsert with vector text literals (%s::vector).
• Graceful degradation is the rule: missing DB URL, Ollama down, or any DB error logs a warning and returns None. The relational row and the file-based vault stay canonical.
• Alembic stands up the second_brain schema; public.embeddings is managed out-of-band and intentionally outside the migration tree.
• File-based wiki compiler is unchanged. No dual-write to petalbrain.wiki.
• No SQLite import — we started clean on Postgres.

---

Locked decisions (from the planning round)

Question	Decision
DB instance	Existing petalbrain (homelab-postgres)
Schema	New second_brain schema (owned by lovebug)
Connection	Containerised: homelab-postgres:5432 (NOT host 127.0.0.1:5433)
Pooling	psycopg_pool.ConnectionPool min=1/max=10 — no PgBouncer in front
Driver	psycopg v3, SQLAlchemy uses postgresql+psycopg://
PK style	Int autoincrement (SERIAL) — kept
Timestamps	Naive UTC timestamp without time zone — kept
Embedding scope	Extraction summaries only (Phase 1)
Embedding model	nomic-embed-text (768-d), Ollama via OLLAMA_URL
Migration tool	Alembic, second-brain owns its own tree
public.embeddings	NOT managed by our Alembic — shared with vault-mcp / ob1
File-based wiki	Kept intact; no dual-write into petalbrain.wiki
Old SQLite data	Not imported — clean start

---

Setup (operator runbook)

# 1. One-time, as postgres superuser, bootstrap the schema. lovebug doesn't
#    have CREATE on petalbrain, so this can't be done from the app side.
docker exec homelab-postgres psql -U postgres -d petalbrain -c \
  "CREATE SCHEMA second_brain AUTHORIZATION lovebug;
   GRANT ALL PRIVILEGES ON SCHEMA second_brain TO lovebug;"

# 2. Point the project at the DB (env wins over settings.toml).
export SECOND_BRAIN_DATABASE_URL=\
"postgresql+psycopg://lovebug:<pw>@homelab-postgres:5432/petalbrain"

# 3. Install deps and run the migration.
uv sync
uv run alembic upgrade head

The lovebug password lives in /opt/backups/postgres-consolidation/credentials.env (0600, host-only).

---

Key files (post-migration)

• src/second_brain/models.py — schema-qualified ORM, __table_args__ → second_brain; native Postgres enum types are scoped to the schema too.
• src/second_brain/database.py — psycopg-v3 engine + URL resolution (constructor arg → SECOND_BRAIN_DATABASE_URL → HERBYLAB_DATABASE_URL → [database].url).
• src/second_brain/embeddings/__init__.py — near-verbatim port of vault_mcp/core/embeddings.py adapted to our key tuple.
• alembic.ini + alembic/env.py — schema-pinned Alembic; the env.py guards CREATE SCHEMA behind pg_namespace because lovebug lacks CREATE on the database.
• alembic/versions/4a1e2f6c9d10_v1_second_brain_pipeline_tables.py — v1 migration: enum types + sources + extractions + wiki_pages.
• tests/test_smoke_embedding.py — end-to-end smoke test exercising the full Postgres + Ollama path. Auto-skips when either is unreachable.

---

Known gotchas

1. CREATE SCHEMA IF NOT EXISTS errors as lovebug even when the schema exists. pg_namespace lookup in alembic/env.py is the workaround.
2. +psycopg URL prefix. SQLAlchemy needs it; raw psycopg doesn't. database.py adds it if missing; the embeddings module strips it.
3. Pool conflict. database.py opens an SQLAlchemy pool; embeddings/__init__.py opens an independent psycopg pool. Both share the same DSN; both are small (min=1) so they don't strain the cluster.
4. anthropic-skills:zero-check needs pytest-cov. Added to the new [dependency-groups].dev block so uv sync resolves it.

---

Next iterations (open, not blocked)

• Embed transcript chunks (or at least key_points / claims), not just summaries.
• Promote vector retrieval into the web UI (semantic search across extractions).
• Add a periodic backfill script that re-embeds extractions whose embedding_model no longer matches the current default.
• Decide on the production vault path (/opt/projects/second-brain-vault/?).