# Postgres + pgvector migration — planning questions

Status: awaiting input from Travis before scoping the migration.
Trigger: user pivoted away from SQLite — wants to use the existing Postgres + pgvector instance, plus the shared `/projects/shared/python/embedding_chunking` module.

---

## 1. Connection / secrets

- [ ] Host, port, database name, user
- [ ] Where does the connection string live? Options:
  - env var (e.g. `SECOND_BRAIN_DB_URL`)
  - `config/settings.toml` (would need a `.local.toml` override so the password doesn't get committed)
  - `.pgpass`
  - other
- [ ] Is the Postgres instance on `herbydev`, the NAS, or elsewhere? Tailscale-only or LAN-reachable?
- [ ] Driver preference:
  - `psycopg` v3 sync (default suggestion — pipeline + scheduler are sync today)
  - `asyncpg` (only worth it if we also rewrite the web layer to async)

## 2. Schema layout

- [ ] Same Postgres **instance** as `vault-mcp`'s `petalbrain.wiki`? Yes / No
- [ ] If yes — same **database** with a dedicated schema (e.g. `second_brain.sources`, `second_brain.extractions`)? Or its own database entirely?
- [ ] Naming conventions to match:
  - snake_case tables — assumed yes
  - `created_at` / `updated_at` as `timestamptz` (vs current naive UTC) — preference?
  - PK type: `bigserial` / `bigint identity` / UUID?
- [ ] Should second-brain integrate with `vault-mcp` at all?
  - Option A: standalone — second-brain owns its tables, the wiki compiler still writes Obsidian markdown files
  - Option B: dual-write — published extractions land in `petalbrain.wiki` via `vault-mcp` so they show up in the unified wiki
  - Option C: replace the file-based compiler entirely with `vault-mcp` writes

## 3. Embeddings

- [ ] Public API of `/projects/shared/python/embedding_chunking`:
  - function signatures
  - what it returns (raw text chunks? chunks + vectors? chunker only, embeddings done separately?)
  - embedding model + vector dimension (drives the `vector(N)` column type)
  - any batching / rate-limit behavior we need to respect
- [ ] Where should embeddings attach?
  - (a) on `Extraction` rows — semantic search over summaries / key_points / claims
  - (b) on chunked transcripts — chunk-level retrieval for "find the part of the video that says X"
  - (c) both (separate tables: `extraction_embeddings` + `transcript_chunks`)
- [ ] pgvector index strategy:
  - HNSW — better recall, slower build, recommended default for personal-scale corpus
  - IVFFlat — faster build, needs periodic `ANALYZE` + a meaningful `lists` value
- [ ] Distance metric: cosine (typical), L2, or inner product?

## 4. Migration tooling

- [ ] Schema is small and not yet in production. Cleanest path:
  - SQLAlchemy `Base.metadata.create_all()` against Postgres
  - one-off import script to move existing SQLite rows over (if any are worth keeping)
- [ ] Alternative: stand up Alembic now while the cost is low. Pays off the first time you `ALTER` a table in anger; costs ~1 session of setup.
- [ ] Travis's call — defaulting to `create_all` + import script unless you say otherwise.

## 5. Pooling

- [ ] Three consumers share the DB: CLI (`process`, `compile`), the FastAPI web server, the scheduler.
- [ ] Default plan: SQLAlchemy `QueuePool` with `pool_size=5, max_overflow=5`.
- [ ] Is PgBouncer already in front of this instance? If yes — switch to `NullPool` on our side and let PgBouncer pool.

## 6. Other open items (carryover from prior session)

- [ ] Vault location decision: I recommended `/opt/projects/second-brain-vault/` as a peer dir. Not yet confirmed.
- [ ] NAS mount: is it mounted on `herbydev`, and at what path? Relevant if media (downloaded videos, transcripts) should live there rather than on the herbydev SSD.

---

## What I will produce once these are answered

1. Updated `src/second_brain/database.py` with a Postgres engine + URL resolution
2. Schema migration: new tables for embeddings (and chunks if we go with option (c))
3. Wiring of `embedding_chunking` into the extractor (or into a post-extraction step in the scheduler)
4. `config/settings.toml.example` updates with the new `[database]` and `[embeddings]` blocks
5. README / CLAUDE.md updates noting the Postgres dependency
6. One-shot SQLite → Postgres import script (only if there are rows worth keeping)