second-brain/postgres-migration-planning.md

Postgres + pgvector migration — planning questions

Status: awaiting input from Travis before scoping the migration. Trigger: user pivoted away from SQLite — wants to use the existing Postgres + pgvector instance, plus the shared /projects/shared/python/embedding_chunking module.

---

1. Connection / secrets

• Host, port, database name, user
• Where does the connection string live? Options:
  • env var (e.g. SECOND_BRAIN_DB_URL)
  • config/settings.toml (would need a .local.toml override so the password doesn't get committed)
  • .pgpass
  • other
• Is the Postgres instance on herbydev, the NAS, or elsewhere? Tailscale-only or LAN-reachable?
• Driver preference:
  • psycopg v3 sync (default suggestion — pipeline + scheduler are sync today)
  • asyncpg (only worth it if we also rewrite the web layer to async)

2. Schema layout

• Same Postgres instance as vault-mcp's petalbrain.wiki? Yes / No
• If yes — same database with a dedicated schema (e.g. second_brain.sources, second_brain.extractions)? Or its own database entirely?
• Naming conventions to match:
  • snake_case tables — assumed yes
  • created_at / updated_at as timestamptz (vs current naive UTC) — preference?
  • PK type: bigserial / bigint identity / UUID?
• Should second-brain integrate with vault-mcp at all?
  • Option A: standalone — second-brain owns its tables, the wiki compiler still writes Obsidian markdown files
  • Option B: dual-write — published extractions land in petalbrain.wiki via vault-mcp so they show up in the unified wiki
  • Option C: replace the file-based compiler entirely with vault-mcp writes

3. Embeddings

• Public API of /projects/shared/python/embedding_chunking:
  • function signatures
  • what it returns (raw text chunks? chunks + vectors? chunker only, embeddings done separately?)
  • embedding model + vector dimension (drives the vector(N) column type)
  • any batching / rate-limit behavior we need to respect
• Where should embeddings attach?
  • (a) on Extraction rows — semantic search over summaries / key_points / claims
  • (b) on chunked transcripts — chunk-level retrieval for "find the part of the video that says X"
  • (c) both (separate tables: extraction_embeddings + transcript_chunks)
• pgvector index strategy:
  • HNSW — better recall, slower build, recommended default for personal-scale corpus
  • IVFFlat — faster build, needs periodic ANALYZE + a meaningful lists value
• Distance metric: cosine (typical), L2, or inner product?

4. Migration tooling

• Schema is small and not yet in production. Cleanest path:
  • SQLAlchemy Base.metadata.create_all() against Postgres
  • one-off import script to move existing SQLite rows over (if any are worth keeping)
• Alternative: stand up Alembic now while the cost is low. Pays off the first time you ALTER a table in anger; costs ~1 session of setup.
• Travis's call — defaulting to create_all + import script unless you say otherwise.

5. Pooling

• Three consumers share the DB: CLI (process, compile), the FastAPI web server, the scheduler.
• Default plan: SQLAlchemy QueuePool with pool_size=5, max_overflow=5.
• Is PgBouncer already in front of this instance? If yes — switch to NullPool on our side and let PgBouncer pool.

6. Other open items (carryover from prior session)

• Vault location decision: I recommended /opt/projects/second-brain-vault/ as a peer dir. Not yet confirmed.
• NAS mount: is it mounted on herbydev, and at what path? Relevant if media (downloaded videos, transcripts) should live there rather than on the herbydev SSD.

---

What I will produce once these are answered

1. Updated src/second_brain/database.py with a Postgres engine + URL resolution
2. Schema migration: new tables for embeddings (and chunks if we go with option (c))
3. Wiring of embedding_chunking into the extractor (or into a post-extraction step in the scheduler)
4. config/settings.toml.example updates with the new [database] and [embeddings] blocks
5. README / CLAUDE.md updates noting the Postgres dependency
6. One-shot SQLite → Postgres import script (only if there are rows worth keeping)