second-brain/postgres-migration-planning.md

# Postgres + pgvector migration — implementation notes

Status: **shipped** (2026-05-24). Originally a planning checklist; rewritten
to capture the implemented design and the decisions that landed. Kept in the
repo so the next iteration has the "why" without grepping commits.

---

## What shipped

- Backing store switched from SQLite to the existing **petalbrain** Postgres
  instance (pgvector/pg17), connecting over the homelab docker network at
  `homelab-postgres:5432`.
- Relational tables (`sources`, `extractions`, `wiki_pages`) live in a new
  `second_brain` schema owned by the **`lovebug`** role — no dedicated service
  role.
- Embeddings reuse the **shared `public.embeddings` table** (HNSW vector index
  already in place). Rows are keyed by
  `(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`.
  For this round we embed **extraction summaries only** (not transcript
  chunks, not key_points / claims).
- Embedding pipeline mirrors `vault_mcp.core.embeddings`: shared
  `embedding_chunking` for 512/64-token chunking → Ollama
  `nomic-embed-text` (768-d, `num_ctx=8192`) → delete-before-insert upsert
  with vector text literals (`%s::vector`).
- **Graceful degradation** is the rule: missing DB URL, Ollama down, or any
  DB error logs a warning and returns `None`. The relational row and the
  file-based vault stay canonical.
- **Alembic** stands up the `second_brain` schema; `public.embeddings` is
  managed out-of-band and intentionally outside the migration tree.
- File-based wiki compiler is unchanged. No dual-write to `petalbrain.wiki`.
- **No SQLite import** — we started clean on Postgres.

---

## Locked decisions (from the planning round)

| Question | Decision |
|---|---|
| DB instance | Existing petalbrain (homelab-postgres) |
| Schema | New `second_brain` schema (owned by `lovebug`) |
| Connection | Containerised: `homelab-postgres:5432` (NOT host `127.0.0.1:5433`) |
| Pooling | `psycopg_pool.ConnectionPool` min=1/max=10 — no PgBouncer in front |
| Driver | psycopg v3, SQLAlchemy uses `postgresql+psycopg://` |
| PK style | Int autoincrement (`SERIAL`) — kept |
| Timestamps | Naive UTC `timestamp without time zone` — kept |
| Embedding scope | Extraction summaries only (Phase 1) |
| Embedding model | `nomic-embed-text` (768-d), Ollama via `OLLAMA_URL` |
| Migration tool | Alembic, second-brain owns its own tree |
| `public.embeddings` | NOT managed by our Alembic — shared with vault-mcp / ob1 |
| File-based wiki | Kept intact; no dual-write into petalbrain.wiki |
| Old SQLite data | Not imported — clean start |

---

## Setup (operator runbook)

```bash
# 1. One-time, as postgres superuser, bootstrap the schema. lovebug doesn't
#    have CREATE on petalbrain, so this can't be done from the app side.
docker exec homelab-postgres psql -U postgres -d petalbrain -c \
  "CREATE SCHEMA second_brain AUTHORIZATION lovebug;
   GRANT ALL PRIVILEGES ON SCHEMA second_brain TO lovebug;"

# 2. Point the project at the DB (env wins over settings.toml).
export SECOND_BRAIN_DATABASE_URL=\
"postgresql+psycopg://lovebug:<pw>@homelab-postgres:5432/petalbrain"

# 3. Install deps and run the migration.
uv sync
uv run alembic upgrade head
```

The `lovebug` password lives in `/opt/backups/postgres-consolidation/credentials.env`
(0600, host-only).

---

## Key files (post-migration)

- `src/second_brain/models.py` — schema-qualified ORM, `__table_args__` →
  `second_brain`; native Postgres enum types are scoped to the schema too.
- `src/second_brain/database.py` — psycopg-v3 engine + URL resolution
  (constructor arg → `SECOND_BRAIN_DATABASE_URL` → `HERBYLAB_DATABASE_URL`
  → `[database].url`).
- `src/second_brain/embeddings/__init__.py` — near-verbatim port of
  `vault_mcp/core/embeddings.py` adapted to our key tuple.
- `alembic.ini` + `alembic/env.py` — schema-pinned Alembic; the env.py guards
  `CREATE SCHEMA` behind `pg_namespace` because `lovebug` lacks `CREATE` on
  the database.
- `alembic/versions/4a1e2f6c9d10_v1_second_brain_pipeline_tables.py` — v1
  migration: enum types + `sources` + `extractions` + `wiki_pages`.
- `tests/test_smoke_embedding.py` — end-to-end smoke test exercising the full
  Postgres + Ollama path. Auto-skips when either is unreachable.

---

## Known gotchas

1. **`CREATE SCHEMA IF NOT EXISTS` errors as `lovebug`** even when the schema
   exists. `pg_namespace` lookup in `alembic/env.py` is the workaround.
2. **`+psycopg` URL prefix.** SQLAlchemy needs it; raw psycopg doesn't.
   `database.py` adds it if missing; the embeddings module strips it.
3. **Pool conflict.** `database.py` opens an SQLAlchemy pool; `embeddings/__init__.py`
   opens an independent psycopg pool. Both share the same DSN; both are small
   (min=1) so they don't strain the cluster.
4. **`anthropic-skills:zero-check` needs `pytest-cov`.** Added to the new
   `[dependency-groups].dev` block so `uv sync` resolves it.

---

## Next iterations (open, not blocked)

- Embed transcript chunks (or at least key_points / claims), not just summaries.
- Promote vector retrieval into the web UI (semantic search across extractions).
- Add a periodic backfill script that re-embeds extractions whose
  `embedding_model` no longer matches the current default.
- Decide on the production vault path (`/opt/projects/second-brain-vault/`?).