79 lines
4.2 KiB
Markdown
79 lines
4.2 KiB
Markdown
# Postgres + pgvector migration — planning questions
|
|
|
|
Status: awaiting input from Travis before scoping the migration.
|
|
Trigger: user pivoted away from SQLite — wants to use the existing Postgres + pgvector instance, plus the shared `/projects/shared/python/embedding_chunking` module.
|
|
|
|
---
|
|
|
|
## 1. Connection / secrets
|
|
|
|
- [ ] Host, port, database name, user
|
|
- [ ] Where does the connection string live? Options:
|
|
- env var (e.g. `SECOND_BRAIN_DB_URL`)
|
|
- `config/settings.toml` (would need a `.local.toml` override so the password doesn't get committed)
|
|
- `.pgpass`
|
|
- other
|
|
- [ ] Is the Postgres instance on `herbydev`, the NAS, or elsewhere? Tailscale-only or LAN-reachable?
|
|
- [ ] Driver preference:
|
|
- `psycopg` v3 sync (default suggestion — pipeline + scheduler are sync today)
|
|
- `asyncpg` (only worth it if we also rewrite the web layer to async)
|
|
|
|
## 2. Schema layout
|
|
|
|
- [ ] Same Postgres **instance** as `vault-mcp`'s `petalbrain.wiki`? Yes / No
|
|
- [ ] If yes — same **database** with a dedicated schema (e.g. `second_brain.sources`, `second_brain.extractions`)? Or its own database entirely?
|
|
- [ ] Naming conventions to match:
|
|
- snake_case tables — assumed yes
|
|
- `created_at` / `updated_at` as `timestamptz` (vs current naive UTC) — preference?
|
|
- PK type: `bigserial` / `bigint identity` / UUID?
|
|
- [ ] Should second-brain integrate with `vault-mcp` at all?
|
|
- Option A: standalone — second-brain owns its tables, the wiki compiler still writes Obsidian markdown files
|
|
- Option B: dual-write — published extractions land in `petalbrain.wiki` via `vault-mcp` so they show up in the unified wiki
|
|
- Option C: replace the file-based compiler entirely with `vault-mcp` writes
|
|
|
|
## 3. Embeddings
|
|
|
|
- [ ] Public API of `/projects/shared/python/embedding_chunking`:
|
|
- function signatures
|
|
- what it returns (raw text chunks? chunks + vectors? chunker only, embeddings done separately?)
|
|
- embedding model + vector dimension (drives the `vector(N)` column type)
|
|
- any batching / rate-limit behavior we need to respect
|
|
- [ ] Where should embeddings attach?
|
|
- (a) on `Extraction` rows — semantic search over summaries / key_points / claims
|
|
- (b) on chunked transcripts — chunk-level retrieval for "find the part of the video that says X"
|
|
- (c) both (separate tables: `extraction_embeddings` + `transcript_chunks`)
|
|
- [ ] pgvector index strategy:
|
|
- HNSW — better recall, slower build, recommended default for personal-scale corpus
|
|
- IVFFlat — faster build, needs periodic `ANALYZE` + a meaningful `lists` value
|
|
- [ ] Distance metric: cosine (typical), L2, or inner product?
|
|
|
|
## 4. Migration tooling
|
|
|
|
- [ ] Schema is small and not yet in production. Cleanest path:
|
|
- SQLAlchemy `Base.metadata.create_all()` against Postgres
|
|
- one-off import script to move existing SQLite rows over (if any are worth keeping)
|
|
- [ ] Alternative: stand up Alembic now while the cost is low. Pays off the first time you `ALTER` a table in anger; costs ~1 session of setup.
|
|
- [ ] Travis's call — defaulting to `create_all` + import script unless you say otherwise.
|
|
|
|
## 5. Pooling
|
|
|
|
- [ ] Three consumers share the DB: CLI (`process`, `compile`), the FastAPI web server, the scheduler.
|
|
- [ ] Default plan: SQLAlchemy `QueuePool` with `pool_size=5, max_overflow=5`.
|
|
- [ ] Is PgBouncer already in front of this instance? If yes — switch to `NullPool` on our side and let PgBouncer pool.
|
|
|
|
## 6. Other open items (carryover from prior session)
|
|
|
|
- [ ] Vault location decision: I recommended `/opt/projects/second-brain-vault/` as a peer dir. Not yet confirmed.
|
|
- [ ] NAS mount: is it mounted on `herbydev`, and at what path? Relevant if media (downloaded videos, transcripts) should live there rather than on the herbydev SSD.
|
|
|
|
---
|
|
|
|
## What I will produce once these are answered
|
|
|
|
1. Updated `src/second_brain/database.py` with a Postgres engine + URL resolution
|
|
2. Schema migration: new tables for embeddings (and chunks if we go with option (c))
|
|
3. Wiring of `embedding_chunking` into the extractor (or into a post-extraction step in the scheduler)
|
|
4. `config/settings.toml.example` updates with the new `[database]` and `[embeddings]` blocks
|
|
5. README / CLAUDE.md updates noting the Postgres dependency
|
|
6. One-shot SQLite → Postgres import script (only if there are rows worth keeping)
|