diff --git a/CLAUDE.md b/CLAUDE.md index b0e25f1..51a1adc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,8 +23,9 @@ src/second_brain/ ├── adapters/ # Pull adapters (youtube.py, article.py) ├── context/ # Context assembler ├── extractor/ # LLM extraction engine + Pydantic schema +├── llm/ # claude_cli.py — thin subprocess wrapper around `claude -p` ├── compiler/ # Wiki compiler (reads vault, writes pages, git commits) -├── scheduler/ # Rate-limit smoother, token-aware spacing +├── scheduler/ # Rate-limit smoother (call-count for CLI, token-budget for API) └── web/ # FastAPI + Jinja2 + HTMX UI ``` @@ -44,9 +45,19 @@ uv run second-brain serve - `second-brain compile` — run the wiki compiler - `second-brain schedule` — start the overnight scheduler +## Extractor backends + +`[extractor].backend` in `config/settings.toml` picks the LLM transport. + +- **`cli`** (default) — `src/second_brain/llm/claude_cli.py` shells out to `claude -p --output-format json --model --json-schema `. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside `tempfile.TemporaryDirectory(prefix="sb-claude-cli-")`, with `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` stripped from env so the CLI uses OAuth instead of an API key, and with `--no-session-persistence --disable-slash-commands --tools "" --setting-sources ""` so host CLAUDE.md / hooks / agents don't leak in. +- **`api`** — uses the `anthropic` SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (`uv sync --extra api`). + +**Schema-validated output gotcha:** when `--json-schema` is used, the parsed object lands at `payload["structured_output"]`, not `payload["result"]`. The wrapper checks both. The engine prefers the structured payload via `LLMExtraction.model_validate(structured)` and falls back to parsing the text field. + ## Environment variables -- `ANTHROPIC_API_KEY` — required for extraction and wiki compilation +- `ANTHROPIC_API_KEY` — only required when `extractor.backend = "api"`. The default `cli` backend uses your Max OAuth and does not need it. +- `SECOND_BRAIN_CONFIG` — optional override for the settings file path. ## Domains @@ -57,6 +68,26 @@ uv run second-brain serve ## Key files -- `config/settings.toml` — vault path, DB path, scheduler window +- `config/settings.toml` — vault path, DB path, scheduler window, `[extractor]` backend config - `prompts/.md` — per-domain extraction prompt templates -- `src/second_brain/extractor/schema.py` — canonical extraction output schema +- `src/second_brain/extractor/schema.py` — `LLMExtraction` (LLM-only fields, drives the `--json-schema` contract) + `ExtractionResult` (LLMExtraction + SourceMeta filled from the DB) +- `src/second_brain/llm/claude_cli.py` — CLI subprocess wrapper +- `src/second_brain/models.py` — SQLAlchemy models + `utcnow()` helper (replaces deprecated `datetime.utcnow`) +- `postgres-migration-planning.md` — open questions blocking the SQLite → Postgres+pgvector migration + +## Conventions / gotchas + +- **Naive UTC timestamps everywhere.** Use `from second_brain.models import utcnow` rather than `datetime.utcnow()` (deprecated in 3.12) or `datetime.now()` (local time). +- **Re-query inside the session.** SQLAlchemy ORM attribute access after the session closes raises `DetachedInstanceError`. The `process` command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates *inside* the `with db.session()` block. +- **Starlette `TemplateResponse` signature.** Starlette ≥ 0.36 requires `templates.TemplateResponse(request, name, context)`. The old `(name, context)` form raises `TypeError: unhashable type: 'dict'`. Do not put `"request": request` into the context dict — it's auto-injected now. +- **Vault git repo auto-init.** The wiki compiler calls `_ensure_git_repo()` on every run, which `git init`s the vault and creates an empty seed commit if no `.git` is present. `_git_commit()` also checks `git diff --cached --quiet` before committing to avoid empty commits. +- **Scheduler throttling shape.** The CLI backend is throttled by `max_calls_per_hour` (Max subscription limits are call-shaped, not token-shaped). The API backend uses `max_tokens_per_hour`. Both share `min_gap_seconds`. The scheduler targets sources in `TRANSCRIBED` state — earlier code mistakenly filtered on `ANALYZED`, making it a no-op. + +## In-flight work + +- **Postgres + pgvector migration.** SQLite is the current backing store. Travis wants to move to his existing Postgres + pgvector instance and use the shared `/projects/shared/python/embedding_chunking` module for vector indexing. Planning questions are enumerated in `postgres-migration-planning.md` at the project root; no migration work has started yet. +- **Vault location.** Recommended `/opt/projects/second-brain-vault/` as the production location (peer to the project dir). Not yet confirmed. + +## Validation + +Per Travis's global CLAUDE.md, run `zero-check` after any code change, and `a-review` after non-trivial diffs (> 50 lines). `a-review` script: `python3 ~/.claude/skills/a-review/run_review.py`, output goes to `reviews/a-review-.md`. diff --git a/README.md b/README.md index d7045b3..d417e2e 100644 --- a/README.md +++ b/README.md @@ -43,8 +43,19 @@ Extractions are tagged by domain so the right prompt template is used: | `business` | Entrepreneurship, marketing, ops | | `homelab` | Self-hosted infra, networking, DevOps | +## Extractor backends + +Two backends, selectable via `[extractor].backend` in `config/settings.toml`: + +- **`cli`** (default) — shells out to `claude -p --output-format json --json-schema ...`. Runs under your Max OAuth subscription, no per-token billing. The wrapper spawns the CLI in a throwaway `tempfile.TemporaryDirectory` and strips `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` from the env so host CLAUDE.md, hooks, and settings don't leak in, and the subscription is used instead of the API key. +- **`api`** — uses the `anthropic` Python SDK, requires `ANTHROPIC_API_KEY`. Install with `uv sync --extra api` (the SDK is an optional dependency). + ## Requirements - Python 3.12+ -- `ANTHROPIC_API_KEY` environment variable +- Either the `claude` CLI on PATH (default backend) **or** `ANTHROPIC_API_KEY` (api backend) - ffmpeg (for Whisper audio extraction) + +## In-flight work + +- **Postgres + pgvector migration** — SQLite is the current backing store but the project is moving to Travis's Postgres instance. Planning questions live in `postgres-migration-planning.md`; nothing has been migrated yet. diff --git a/postgres-migration-planning.md b/postgres-migration-planning.md new file mode 100644 index 0000000..99cbac7 --- /dev/null +++ b/postgres-migration-planning.md @@ -0,0 +1,78 @@ +# Postgres + pgvector migration — planning questions + +Status: awaiting input from Travis before scoping the migration. +Trigger: user pivoted away from SQLite — wants to use the existing Postgres + pgvector instance, plus the shared `/projects/shared/python/embedding_chunking` module. + +--- + +## 1. Connection / secrets + +- [ ] Host, port, database name, user +- [ ] Where does the connection string live? Options: + - env var (e.g. `SECOND_BRAIN_DB_URL`) + - `config/settings.toml` (would need a `.local.toml` override so the password doesn't get committed) + - `.pgpass` + - other +- [ ] Is the Postgres instance on `herbydev`, the NAS, or elsewhere? Tailscale-only or LAN-reachable? +- [ ] Driver preference: + - `psycopg` v3 sync (default suggestion — pipeline + scheduler are sync today) + - `asyncpg` (only worth it if we also rewrite the web layer to async) + +## 2. Schema layout + +- [ ] Same Postgres **instance** as `vault-mcp`'s `petalbrain.wiki`? Yes / No +- [ ] If yes — same **database** with a dedicated schema (e.g. `second_brain.sources`, `second_brain.extractions`)? Or its own database entirely? +- [ ] Naming conventions to match: + - snake_case tables — assumed yes + - `created_at` / `updated_at` as `timestamptz` (vs current naive UTC) — preference? + - PK type: `bigserial` / `bigint identity` / UUID? +- [ ] Should second-brain integrate with `vault-mcp` at all? + - Option A: standalone — second-brain owns its tables, the wiki compiler still writes Obsidian markdown files + - Option B: dual-write — published extractions land in `petalbrain.wiki` via `vault-mcp` so they show up in the unified wiki + - Option C: replace the file-based compiler entirely with `vault-mcp` writes + +## 3. Embeddings + +- [ ] Public API of `/projects/shared/python/embedding_chunking`: + - function signatures + - what it returns (raw text chunks? chunks + vectors? chunker only, embeddings done separately?) + - embedding model + vector dimension (drives the `vector(N)` column type) + - any batching / rate-limit behavior we need to respect +- [ ] Where should embeddings attach? + - (a) on `Extraction` rows — semantic search over summaries / key_points / claims + - (b) on chunked transcripts — chunk-level retrieval for "find the part of the video that says X" + - (c) both (separate tables: `extraction_embeddings` + `transcript_chunks`) +- [ ] pgvector index strategy: + - HNSW — better recall, slower build, recommended default for personal-scale corpus + - IVFFlat — faster build, needs periodic `ANALYZE` + a meaningful `lists` value +- [ ] Distance metric: cosine (typical), L2, or inner product? + +## 4. Migration tooling + +- [ ] Schema is small and not yet in production. Cleanest path: + - SQLAlchemy `Base.metadata.create_all()` against Postgres + - one-off import script to move existing SQLite rows over (if any are worth keeping) +- [ ] Alternative: stand up Alembic now while the cost is low. Pays off the first time you `ALTER` a table in anger; costs ~1 session of setup. +- [ ] Travis's call — defaulting to `create_all` + import script unless you say otherwise. + +## 5. Pooling + +- [ ] Three consumers share the DB: CLI (`process`, `compile`), the FastAPI web server, the scheduler. +- [ ] Default plan: SQLAlchemy `QueuePool` with `pool_size=5, max_overflow=5`. +- [ ] Is PgBouncer already in front of this instance? If yes — switch to `NullPool` on our side and let PgBouncer pool. + +## 6. Other open items (carryover from prior session) + +- [ ] Vault location decision: I recommended `/opt/projects/second-brain-vault/` as a peer dir. Not yet confirmed. +- [ ] NAS mount: is it mounted on `herbydev`, and at what path? Relevant if media (downloaded videos, transcripts) should live there rather than on the herbydev SSD. + +--- + +## What I will produce once these are answered + +1. Updated `src/second_brain/database.py` with a Postgres engine + URL resolution +2. Schema migration: new tables for embeddings (and chunks if we go with option (c)) +3. Wiring of `embedding_chunking` into the extractor (or into a post-extraction step in the scheduler) +4. `config/settings.toml.example` updates with the new `[database]` and `[embeddings]` blocks +5. README / CLAUDE.md updates noting the Postgres dependency +6. One-shot SQLite → Postgres import script (only if there are rows worth keeping)