second-brain/CLAUDE.md

# second-brain

A personal knowledge system: video/article extraction pipeline + LLM-maintained wiki.

## Architecture

Three-layer system:

1. **Wiki** — long-term structured memory. Markdown files in an Obsidian vault, maintained by the wiki compiler.
2. **Context assembler** — builds the prompt bundle for each extraction. Phase 1 is thin: domain template + focus field + transcript.
3. **Extractor** — stateless single-shot LLM call per source. Pure function: bundle in, structured extraction out.

## Pipeline stages

```
pending → pulled → transcribed → analyzed → accepted → published
```

## Project layout

```
src/second_brain/
├── adapters/      # Pull adapters (youtube.py, article.py)
├── context/       # Context assembler
├── extractor/     # LLM extraction engine + Pydantic schema
├── llm/           # claude_cli.py — thin subprocess wrapper around `claude -p`
├── compiler/      # Wiki compiler (reads vault, writes pages, git commits)
├── scheduler/     # Rate-limit smoother (call-count for CLI, token-budget for API)
└── web/           # FastAPI + Jinja2 + HTMX UI
```

## Setup

```bash
cp config/settings.toml.example config/settings.toml
# Edit settings.toml with your paths
uv run second-brain serve
```

## CLI commands

- `second-brain add <url>` — queue a source URL (video or article)
- `second-brain process` — run the pipeline on pending items
- `second-brain serve` — launch the web UI (default port 8000)
- `second-brain compile` — run the wiki compiler
- `second-brain schedule` — start the overnight scheduler

## Extractor backends

`[extractor].backend` in `config/settings.toml` picks the LLM transport.

- **`cli`** (default) — `src/second_brain/llm/claude_cli.py` shells out to `claude -p --output-format json --model <model> --json-schema <schema>`. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside `tempfile.TemporaryDirectory(prefix="sb-claude-cli-")`, with `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` stripped from env so the CLI uses OAuth instead of an API key, and with `--no-session-persistence --disable-slash-commands --tools "" --setting-sources ""` so host CLAUDE.md / hooks / agents don't leak in.
- **`api`** — uses the `anthropic` SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (`uv sync --extra api`).

**Schema-validated output gotcha:** when `--json-schema` is used, the parsed object lands at `payload["structured_output"]`, not `payload["result"]`. The wrapper checks both. The engine prefers the structured payload via `LLMExtraction.model_validate(structured)` and falls back to parsing the text field.

## Environment variables

- `ANTHROPIC_API_KEY` — only required when `extractor.backend = "api"`. The default `cli` backend uses your Max OAuth and does not need it.
- `SECOND_BRAIN_CONFIG` — optional override for the settings file path.

## Domains

- `development` — software, programming, systems
- `content` — content creation strategy, video production
- `business` — entrepreneurship, marketing, operations
- `homelab` — self-hosted infrastructure, networking, DevOps

## Key files

- `config/settings.toml` — vault path, DB path, scheduler window, `[extractor]` backend config
- `prompts/<domain>.md` — per-domain extraction prompt templates
- `src/second_brain/extractor/schema.py` — `LLMExtraction` (LLM-only fields, drives the `--json-schema` contract) + `ExtractionResult` (LLMExtraction + SourceMeta filled from the DB)
- `src/second_brain/llm/claude_cli.py` — CLI subprocess wrapper
- `src/second_brain/models.py` — SQLAlchemy models + `utcnow()` helper (replaces deprecated `datetime.utcnow`)
- `postgres-migration-planning.md` — open questions blocking the SQLite → Postgres+pgvector migration

## Conventions / gotchas

- **Naive UTC timestamps everywhere.** Use `from second_brain.models import utcnow` rather than `datetime.utcnow()` (deprecated in 3.12) or `datetime.now()` (local time).
- **Re-query inside the session.** SQLAlchemy ORM attribute access after the session closes raises `DetachedInstanceError`. The `process` command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates *inside* the `with db.session()` block.
- **Starlette `TemplateResponse` signature.** Starlette ≥ 0.36 requires `templates.TemplateResponse(request, name, context)`. The old `(name, context)` form raises `TypeError: unhashable type: 'dict'`. Do not put `"request": request` into the context dict — it's auto-injected now.
- **Vault git repo auto-init.** The wiki compiler calls `_ensure_git_repo()` on every run, which `git init`s the vault and creates an empty seed commit if no `.git` is present. `_git_commit()` also checks `git diff --cached --quiet` before committing to avoid empty commits.
- **Scheduler throttling shape.** The CLI backend is throttled by `max_calls_per_hour` (Max subscription limits are call-shaped, not token-shaped). The API backend uses `max_tokens_per_hour`. Both share `min_gap_seconds`. The scheduler targets sources in `TRANSCRIBED` state — earlier code mistakenly filtered on `ANALYZED`, making it a no-op.

## In-flight work

- **Postgres + pgvector migration.** SQLite is the current backing store. Travis wants to move to his existing Postgres + pgvector instance and use the shared `/projects/shared/python/embedding_chunking` module for vector indexing. Planning questions are enumerated in `postgres-migration-planning.md` at the project root; no migration work has started yet.
- **Vault location.** Recommended `/opt/projects/second-brain-vault/` as the production location (peer to the project dir). Not yet confirmed.

## Validation

Per Travis's global CLAUDE.md, run `zero-check` after any code change, and `a-review` after non-trivial diffs (> 50 lines). `a-review` script: `python3 ~/.claude/skills/a-review/run_review.py`, output goes to `reviews/a-review-<date>.md`.