updates to project files

CLAUDE.md

@@ -23,8 +23,9 @@ src/second_brain/
 ├── adapters/      # Pull adapters (youtube.py, article.py)
 ├── context/       # Context assembler
 ├── extractor/     # LLM extraction engine + Pydantic schema
+├── llm/           # claude_cli.py — thin subprocess wrapper around `claude -p`
 ├── compiler/      # Wiki compiler (reads vault, writes pages, git commits)
-├── scheduler/     # Rate-limit smoother, token-aware spacing
+├── scheduler/     # Rate-limit smoother (call-count for CLI, token-budget for API)
 └── web/           # FastAPI + Jinja2 + HTMX UI
 ```
 
@@ -44,9 +45,19 @@ uv run second-brain serve
 - `second-brain compile` — run the wiki compiler
 - `second-brain schedule` — start the overnight scheduler
 
+## Extractor backends
+
+`[extractor].backend` in `config/settings.toml` picks the LLM transport.
+
+- **`cli`** (default) — `src/second_brain/llm/claude_cli.py` shells out to `claude -p --output-format json --model <model> --json-schema <schema>`. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside `tempfile.TemporaryDirectory(prefix="sb-claude-cli-")`, with `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` stripped from env so the CLI uses OAuth instead of an API key, and with `--no-session-persistence --disable-slash-commands --tools "" --setting-sources ""` so host CLAUDE.md / hooks / agents don't leak in.
+- **`api`** — uses the `anthropic` SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (`uv sync --extra api`).
+
+**Schema-validated output gotcha:** when `--json-schema` is used, the parsed object lands at `payload["structured_output"]`, not `payload["result"]`. The wrapper checks both. The engine prefers the structured payload via `LLMExtraction.model_validate(structured)` and falls back to parsing the text field.
+
 ## Environment variables
 
-- `ANTHROPIC_API_KEY` — required for extraction and wiki compilation
+- `ANTHROPIC_API_KEY` — only required when `extractor.backend = "api"`. The default `cli` backend uses your Max OAuth and does not need it.
+- `SECOND_BRAIN_CONFIG` — optional override for the settings file path.
 
 ## Domains
 
@@ -57,6 +68,26 @@ uv run second-brain serve
 
 ## Key files
 
-- `config/settings.toml` — vault path, DB path, scheduler window
+- `config/settings.toml` — vault path, DB path, scheduler window, `[extractor]` backend config
 - `prompts/<domain>.md` — per-domain extraction prompt templates
-- `src/second_brain/extractor/schema.py` — canonical extraction output schema
+- `src/second_brain/extractor/schema.py` — `LLMExtraction` (LLM-only fields, drives the `--json-schema` contract) + `ExtractionResult` (LLMExtraction + SourceMeta filled from the DB)
+- `src/second_brain/llm/claude_cli.py` — CLI subprocess wrapper
+- `src/second_brain/models.py` — SQLAlchemy models + `utcnow()` helper (replaces deprecated `datetime.utcnow`)
+- `postgres-migration-planning.md` — open questions blocking the SQLite → Postgres+pgvector migration
+
+## Conventions / gotchas
+
+- **Naive UTC timestamps everywhere.** Use `from second_brain.models import utcnow` rather than `datetime.utcnow()` (deprecated in 3.12) or `datetime.now()` (local time).
+- **Re-query inside the session.** SQLAlchemy ORM attribute access after the session closes raises `DetachedInstanceError`. The `process` command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates *inside* the `with db.session()` block.
+- **Starlette `TemplateResponse` signature.** Starlette ≥ 0.36 requires `templates.TemplateResponse(request, name, context)`. The old `(name, context)` form raises `TypeError: unhashable type: 'dict'`. Do not put `"request": request` into the context dict — it's auto-injected now.
+- **Vault git repo auto-init.** The wiki compiler calls `_ensure_git_repo()` on every run, which `git init`s the vault and creates an empty seed commit if no `.git` is present. `_git_commit()` also checks `git diff --cached --quiet` before committing to avoid empty commits.
+- **Scheduler throttling shape.** The CLI backend is throttled by `max_calls_per_hour` (Max subscription limits are call-shaped, not token-shaped). The API backend uses `max_tokens_per_hour`. Both share `min_gap_seconds`. The scheduler targets sources in `TRANSCRIBED` state — earlier code mistakenly filtered on `ANALYZED`, making it a no-op.
+
+## In-flight work
+
+- **Postgres + pgvector migration.** SQLite is the current backing store. Travis wants to move to his existing Postgres + pgvector instance and use the shared `/projects/shared/python/embedding_chunking` module for vector indexing. Planning questions are enumerated in `postgres-migration-planning.md` at the project root; no migration work has started yet.
+- **Vault location.** Recommended `/opt/projects/second-brain-vault/` as the production location (peer to the project dir). Not yet confirmed.
+
+## Validation
+
+Per Travis's global CLAUDE.md, run `zero-check` after any code change, and `a-review` after non-trivial diffs (> 50 lines). `a-review` script: `python3 ~/.claude/skills/a-review/run_review.py`, output goes to `reviews/a-review-<date>.md`.

README.md

@@ -43,8 +43,19 @@ Extractions are tagged by domain so the right prompt template is used:
 | `business` | Entrepreneurship, marketing, ops |
 | `homelab` | Self-hosted infra, networking, DevOps |
 
+## Extractor backends
+
+Two backends, selectable via `[extractor].backend` in `config/settings.toml`:
+
+- **`cli`** (default) — shells out to `claude -p --output-format json --json-schema ...`. Runs under your Max OAuth subscription, no per-token billing. The wrapper spawns the CLI in a throwaway `tempfile.TemporaryDirectory` and strips `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` from the env so host CLAUDE.md, hooks, and settings don't leak in, and the subscription is used instead of the API key.
+- **`api`** — uses the `anthropic` Python SDK, requires `ANTHROPIC_API_KEY`. Install with `uv sync --extra api` (the SDK is an optional dependency).
+
 ## Requirements
 
 - Python 3.12+
-- `ANTHROPIC_API_KEY` environment variable
+- Either the `claude` CLI on PATH (default backend) **or** `ANTHROPIC_API_KEY` (api backend)
 - ffmpeg (for Whisper audio extraction)
+
+## In-flight work
+
+- **Postgres + pgvector migration** — SQLite is the current backing store but the project is moving to Travis's Postgres instance. Planning questions live in `postgres-migration-planning.md`; nothing has been migrated yet.

postgres-migration-planning.md

@@ -0,0 +1,78 @@
+# Postgres + pgvector migration — planning questions
+
+Status: awaiting input from Travis before scoping the migration.
+Trigger: user pivoted away from SQLite — wants to use the existing Postgres + pgvector instance, plus the shared `/projects/shared/python/embedding_chunking` module.
+
+---
+
+## 1. Connection / secrets
+
+- [ ] Host, port, database name, user
+- [ ] Where does the connection string live? Options:
+  - env var (e.g. `SECOND_BRAIN_DB_URL`)
+  - `config/settings.toml` (would need a `.local.toml` override so the password doesn't get committed)
+  - `.pgpass`
+  - other
+- [ ] Is the Postgres instance on `herbydev`, the NAS, or elsewhere? Tailscale-only or LAN-reachable?
+- [ ] Driver preference:
+  - `psycopg` v3 sync (default suggestion — pipeline + scheduler are sync today)
+  - `asyncpg` (only worth it if we also rewrite the web layer to async)
+
+## 2. Schema layout
+
+- [ ] Same Postgres **instance** as `vault-mcp`'s `petalbrain.wiki`? Yes / No
+- [ ] If yes — same **database** with a dedicated schema (e.g. `second_brain.sources`, `second_brain.extractions`)? Or its own database entirely?
+- [ ] Naming conventions to match:
+  - snake_case tables — assumed yes
+  - `created_at` / `updated_at` as `timestamptz` (vs current naive UTC) — preference?
+  - PK type: `bigserial` / `bigint identity` / UUID?
+- [ ] Should second-brain integrate with `vault-mcp` at all?
+  - Option A: standalone — second-brain owns its tables, the wiki compiler still writes Obsidian markdown files
+  - Option B: dual-write — published extractions land in `petalbrain.wiki` via `vault-mcp` so they show up in the unified wiki
+  - Option C: replace the file-based compiler entirely with `vault-mcp` writes
+
+## 3. Embeddings
+
+- [ ] Public API of `/projects/shared/python/embedding_chunking`:
+  - function signatures
+  - what it returns (raw text chunks? chunks + vectors? chunker only, embeddings done separately?)
+  - embedding model + vector dimension (drives the `vector(N)` column type)
+  - any batching / rate-limit behavior we need to respect
+- [ ] Where should embeddings attach?
+  - (a) on `Extraction` rows — semantic search over summaries / key_points / claims
+  - (b) on chunked transcripts — chunk-level retrieval for "find the part of the video that says X"
+  - (c) both (separate tables: `extraction_embeddings` + `transcript_chunks`)
+- [ ] pgvector index strategy:
+  - HNSW — better recall, slower build, recommended default for personal-scale corpus
+  - IVFFlat — faster build, needs periodic `ANALYZE` + a meaningful `lists` value
+- [ ] Distance metric: cosine (typical), L2, or inner product?
+
+## 4. Migration tooling
+
+- [ ] Schema is small and not yet in production. Cleanest path:
+  - SQLAlchemy `Base.metadata.create_all()` against Postgres
+  - one-off import script to move existing SQLite rows over (if any are worth keeping)
+- [ ] Alternative: stand up Alembic now while the cost is low. Pays off the first time you `ALTER` a table in anger; costs ~1 session of setup.
+- [ ] Travis's call — defaulting to `create_all` + import script unless you say otherwise.
+
+## 5. Pooling
+
+- [ ] Three consumers share the DB: CLI (`process`, `compile`), the FastAPI web server, the scheduler.
+- [ ] Default plan: SQLAlchemy `QueuePool` with `pool_size=5, max_overflow=5`.
+- [ ] Is PgBouncer already in front of this instance? If yes — switch to `NullPool` on our side and let PgBouncer pool.
+
+## 6. Other open items (carryover from prior session)
+
+- [ ] Vault location decision: I recommended `/opt/projects/second-brain-vault/` as a peer dir. Not yet confirmed.
+- [ ] NAS mount: is it mounted on `herbydev`, and at what path? Relevant if media (downloaded videos, transcripts) should live there rather than on the herbydev SSD.
+
+---
+
+## What I will produce once these are answered
+
+1. Updated `src/second_brain/database.py` with a Postgres engine + URL resolution
+2. Schema migration: new tables for embeddings (and chunks if we go with option (c))
+3. Wiring of `embedding_chunking` into the extractor (or into a post-extraction step in the scheduler)
+4. `config/settings.toml.example` updates with the new `[database]` and `[embeddings]` blocks
+5. README / CLAUDE.md updates noting the Postgres dependency
+6. One-shot SQLite → Postgres import script (only if there are rows worth keeping)