diff --git a/CLAUDE.md b/CLAUDE.md index 4e30706..f1fb224 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,122 +1,297 @@ # second-brain -A personal knowledge system: video/article extraction pipeline + LLM-maintained wiki. +Personal knowledge pipeline. Two machines coordinate purely through the +shared Postgres queue: the **dev side** (herbys-dev) does article pull ++ all extraction + embedding, and the **tower** (EndeavourOS, RTX 3080) +does video pull + transcription on the GPU. A FastAPI/Jinja/HTMX web +app fronts the queue, settings, and add-to-queue form; the file-based +Obsidian wiki is downstream of `accepted` extractions. -## Architecture - -Three-layer system: - -1. **Wiki** — long-term structured memory. Markdown files in an Obsidian vault, maintained by the wiki compiler. -2. **Context assembler** — builds the prompt bundle for each extraction. Phase 1 is thin: domain template + focus field + transcript. -3. **Extractor** — stateless single-shot LLM call per source. Pure function: bundle in, structured extraction out. - -## Pipeline stages +## Two-machine pipeline ``` -pending → pulled → transcribed → analyzed → accepted → published + add (CLI or POST /sources/add — single video, article, + or YouTube playlist URL → fan-out into N rows) + │ + ▼ + ┌────────── second_brain.sources (the queue) ──────────┐ + │ │ + │ PENDING ── (article)──► dev `second-brain process` + │ trafilatura pull │ + │ PENDING ── (video) ──► tower transcribe-worker │ + │ yt-dlp pull → PULLED │ + │ PULLED ── (video) ──► tower transcribe-worker │ + │ faster-whisper │ + │ both paths land at ──► TRANSCRIBED │ + │ │ + │ TRANSCRIBED ─────────► dev `second-brain schedule` │ + │ Claude CLI extract + │ + │ Ollama embed │ + │ ──► ANALYZED ──► accept in web UI │ + │ ──► ACCEPTED ──► `compile` writes vault│ + │ ──► PUBLISHED │ + │ │ + │ FAILED is the terminal error state │ + └───────────────────────────────────────────────────────┘ ``` -## Project layout +**Source claiming** — `sources.claimed_by` / `claimed_at` columns + +`SELECT ... FOR UPDATE SKIP LOCKED` keep the dev and tower workers from +grabbing the same row. Both stages filter by `source_type`: -``` -src/second_brain/ -├── adapters/ # Pull adapters (youtube.py, article.py) -├── context/ # Context assembler -├── embeddings/ # Chunk + nomic-embed-text + upsert into public.embeddings -├── extractor/ # LLM extraction engine + Pydantic schema -├── llm/ # claude_cli.py — thin subprocess wrapper around `claude -p` -├── compiler/ # Wiki compiler (reads vault, writes pages, git commits) -├── scheduler/ # Rate-limit smoother (call-count for CLI, token-budget for API) -└── web/ # FastAPI + Jinja2 + HTMX UI +- tower transcribe-worker: `status IN (PENDING, PULLED) AND source_type = 'video'` +- dev `process` (article pull): `status = PENDING AND source_type = 'article'` +- dev `schedule` (extract+embed): `status = TRANSCRIBED` (any type) -alembic/ # Schema migrations for the second_brain Postgres schema -``` - -## Setup - -```bash -cp config/settings.toml.example config/settings.toml -# Edit settings.toml — at minimum, point the paths and set SECOND_BRAIN_DATABASE_URL. - -# Connection string format. Containerised (default): -# postgresql+psycopg://lovebug:@homelab-postgres:5432/petalbrain -# Host-side dev runs: -# postgresql+psycopg://lovebug:@127.0.0.1:5433/petalbrain -export SECOND_BRAIN_DATABASE_URL=... - -# One-time, before first migration: bootstrap the schema as postgres -# superuser. lovebug doesn't have CREATE on petalbrain, so this can't be -# done from the app side — same pattern as wiki/trellis/ob1. -# CREATE SCHEMA second_brain AUTHORIZATION lovebug; -# GRANT ALL PRIVILEGES ON SCHEMA second_brain TO lovebug; - -uv sync -uv run alembic upgrade head -uv run second-brain serve -``` +A 30-minute stale-claim reaper runs at worker startup so a crashed +machine doesn't dam the queue. ## CLI commands -- `second-brain add ` — queue a source URL (video or article) -- `second-brain process` — run the pipeline on pending items -- `second-brain serve` — launch the web UI (default port 8000) -- `second-brain compile` — run the wiki compiler -- `second-brain schedule` — start the overnight scheduler +- `second-brain add ` — queue a source. YouTube `playlist?list=…` + URLs auto-expand into one row per video (cap 50, the + `watch?v=…&list=…` form stays a single video — see + `is_youtube_playlist_url` for the rule). +- `second-brain process` — dev-side: pull pending **articles**, extract + + embed any TRANSCRIBED source. Skips videos (tower's job). +- `second-brain schedule` — dev-side overnight extract+embed loop, + honouring `pipeline_settings.extraction_enabled` / active-hours / + max-items. +- `second-brain transcribe-worker` — tower-side daemon. `--once` for + ad-hoc runs. Honours `pipeline_settings.transcription_*` knobs. +- `second-brain serve` — local dev launch of the web UI (port 8000). + Production runs as a container (see `deploy/web/`). +- `second-brain compile` — wiki compiler: ACCEPTED → markdown in + `vault_path`, git-committed. File-based Obsidian vault only; we do + **not** dual-write into petalbrain.wiki. + +## Storage + +**Postgres**: `petalbrain` DB on `homelab-postgres` (pgvector/pg17, docker +container). Connection role: `lovebug`. Dev reaches it at +`127.0.0.1:5433`; tower reaches it over WireGuard at +`db.wg.herbylab.dev:5432`; the web container reaches it at +`homelab-postgres:5432` on the `homelab` docker bridge. **Keep both port +binds** — the host-side 5433 publish and the docker-net 5432 service. + +**Schema `second_brain`** (Alembic-managed in `alembic/`): + +- `sources` — the queue. Int SERIAL PK. Naive UTC `timestamp` + (`without time zone`) — never `timestamptz`. `transcript_text` is the + joined full text the extractor reads; `transcript_segments` is JSONB + with the faster-whisper per-segment / per-word breakdown. + `claimed_by` / `claimed_at` for the queue dance. +- `extractions` — one row per source. JSON columns for the structured + output (summary, key_points, entities, claims, …). +- `pipeline_settings` — **single-row** config (PK pinned to id=1 with a + CHECK constraint). What the dashboard edits and workers read each + iteration: transcription_enabled / active_hours / max_gpu_jobs / + max_video_length / max_items_per_run plus the equivalent extraction_* + fields. DB values override settings.toml. +- `wiki_pages` — bookkeeping for the file-based wiki compiler. + +**Embeddings** live in `public.embeddings` — the **shared** pgvector +table owned by `postgres` and consumed by vault-mcp / ob1 / second-brain. +We do not manage this in Alembic. Rows are keyed by +`(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`, +with `delete-before-insert` semantics on the 4-tuple so a re-embed +never leaves orphans. We embed extraction **summaries only** for now — +not key_points, not transcript chunks. + +## Web app + +FastAPI + Jinja2 + HTMX, in `src/second_brain/web/`. Production lives in +a slim Python 3.12 container (see `deploy/web/`) on the `homelab` docker +network, reaching Postgres + Ollama by service DNS. Published at +`10.0.21.207:8080` and fronted by the Traefik VM (10.0.11.20, +file-provider — **no docker-label routing**) as +`https://brain.herbylab.dev`. + +Routes: + +- `/` — sources list (filterable by domain + status) +- `/queue` — in-flight sources (PENDING / PULLED / TRANSCRIBED) plus the + add-to-queue form +- `/dashboard` — status counts, recent activity, settings snapshot, plus + the add-to-queue form +- `/sources/{id}` — extraction detail + accept/reject HTMX actions +- `/settings` — full HTMX editor for `pipeline_settings` +- `POST /sources/add` — shared endpoint. Detects YouTube playlist URLs + and fans out via yt-dlp `extract_flat`. Reads `HX-Current-URL` to + dispatch the response partial (`_queue_body.html` vs + `_dashboard_body.html`) so each page refreshes its own region inline. + +**No auth.** UI mutates state (accept/reject sources, edit settings) but +has no SSO / CSRF / rate-limiting. The Traefik route is currently +LAN-only by Travis's call — see `deploy/web/README.md` §SECURITY. + +## Transcription + +`src/second_brain/transcribe.py` wraps **faster-whisper** (CTranslate2 +backend). Defaults: model `large-v3`, device `cuda`, compute_type +`float16`. CPU fallback drops to `int8`. `word_timestamps=True` is +always passed; the segment dicts persisted to +`sources.transcript_segments` carry per-word `{start, end, word, +probability}`. + +The tower worker (`src/second_brain/scheduler/transcribe_worker.py`) +runs as a systemd service — see `deploy/tower/README.md` for the full +install runbook. ExecStart goes through +`deploy/tower/run-transcribe-worker.sh` which prepends the venv's +CUDA-12 lib dirs to `LD_LIBRARY_PATH` (see gotcha (b) below). ## Extractor backends -`[extractor].backend` in `config/settings.toml` picks the LLM transport. +`[extractor].backend` in `config/settings.toml`: -- **`cli`** (default) — `src/second_brain/llm/claude_cli.py` shells out to `claude -p --output-format json --model --json-schema `. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside `tempfile.TemporaryDirectory(prefix="sb-claude-cli-")`, with `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` stripped from env so the CLI uses OAuth instead of an API key, and with `--no-session-persistence --disable-slash-commands --tools "" --setting-sources ""` so host CLAUDE.md / hooks / agents don't leak in. -- **`api`** — uses the `anthropic` SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (`uv sync --extra api`). +- `cli` (default) — `src/second_brain/llm/claude_cli.py` shells out to + `claude -p --output-format json --model … --json-schema …`. Runs + under the Max OAuth subscription. Hermetic spawn: throwaway + `tempfile.TemporaryDirectory`, `ANTHROPIC_API_KEY` / `_AUTH_TOKEN` + stripped from env so it uses OAuth, and `--no-session-persistence + --disable-slash-commands --tools "" --setting-sources ""` so host + CLAUDE.md / hooks / agents don't leak in. +- `api` — `anthropic` SDK, lazy-imported. Requires `ANTHROPIC_API_KEY`. + Install with `uv sync --extra api`. -**Schema-validated output gotcha:** when `--json-schema` is used, the parsed object lands at `payload["structured_output"]`, not `payload["result"]`. The wrapper checks both. The engine prefers the structured payload via `LLMExtraction.model_validate(structured)` and falls back to parsing the text field. +Schema-validated CLI output lands at `payload["structured_output"]`, not +`payload["result"]`. The wrapper checks both. ## Environment variables -- `ANTHROPIC_API_KEY` — only required when `extractor.backend = "api"`. The default `cli` backend uses your Max OAuth and does not need it. -- `SECOND_BRAIN_CONFIG` — optional override for the settings file path. -- `SECOND_BRAIN_DATABASE_URL` — Postgres connection string. Falls back to `HERBYLAB_DATABASE_URL` if unset. Required at runtime; both `second_brain.database` and `second_brain.embeddings` read it. Use the SQLAlchemy `postgresql+psycopg://` form; the embeddings module strips the `+psycopg` prefix internally for raw psycopg. -- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default `http://ollama:11434` (homelab docker network); for host-side runs use `http://127.0.0.1:11434`. -- `EMBEDDING_MODEL` — overrides `[embeddings].model`. Default `nomic-embed-text` (768-d). -- `SECOND_BRAIN_DB_POOL_MIN` / `SECOND_BRAIN_DB_POOL_MAX` — SQLAlchemy pool sizing knobs (defaults 1 / 9 overflow). +- `SECOND_BRAIN_DATABASE_URL` — Postgres DSN. Falls back to + `HERBYLAB_DATABASE_URL`. Use `postgresql+psycopg://` for SQLAlchemy; + the embeddings + claim modules strip the `+psycopg` prefix for raw + psycopg. +- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default + `http://ollama:11434` (homelab docker net); host-side runs use + `http://127.0.0.1:11434`. +- `EMBEDDING_MODEL` — default `nomic-embed-text` (768-d). +- `WHISPER_MODEL` / `WHISPER_DEVICE` / `WHISPER_COMPUTE_TYPE` — override + the `[whisper]` block on the tower. +- `ANTHROPIC_API_KEY` — only for `extractor.backend = "api"`. +- `SECOND_BRAIN_CONFIG` — settings file path override. +- `SECOND_BRAIN_WORKER_ID` — stamped into `sources.claimed_by`. +- `SECOND_BRAIN_DB_POOL_MIN` / `_MAX` — SQLAlchemy pool sizing. +- `SECOND_BRAIN_LIVE_NETWORK_TESTS=1` — opt-in for the yt-dlp playlist + test that hits real YouTube. ## Domains -- `development` — software, programming, systems -- `content` — content creation strategy, video production -- `business` — entrepreneurship, marketing, operations -- `homelab` — self-hosted infrastructure, networking, DevOps +`development` / `content` / `business` / `homelab` — drives the +per-domain prompt template loaded by the context assembler. ## Key files -- `config/settings.toml` — vault path, `[database]`, `[extractor]`, `[embeddings]`, `[scheduler]` -- `prompts/.md` — per-domain extraction prompt templates -- `src/second_brain/extractor/schema.py` — `LLMExtraction` (LLM-only fields, drives the `--json-schema` contract) + `ExtractionResult` (LLMExtraction + SourceMeta filled from the DB) -- `src/second_brain/llm/claude_cli.py` — CLI subprocess wrapper -- `src/second_brain/models.py` — SQLAlchemy models (all tables live in the `second_brain` schema) + `utcnow()` helper -- `src/second_brain/database.py` — Postgres engine + URL resolution -- `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert into `public.embeddings`; best-effort with graceful degradation -- `alembic/` + `alembic.ini` — schema migrations for the `second_brain` schema (Alembic does **not** manage `public.embeddings`) -- `postgres-migration-planning.md` — original planning doc; superseded by this file once the migration shipped +- `src/second_brain/models.py` — all ORM in the `second_brain` schema. +- `src/second_brain/sources_service.py` — `add_source`, `add_playlist`, + `is_youtube_playlist_url`, `expand_youtube_playlist`. CLI + web both + call into here so there's no behavior drift between entry points. +- `src/second_brain/claim.py` — the SKIP LOCKED dance + stale reaper. +- `src/second_brain/settings_store.py` — single-row `pipeline_settings` + get/update + the active-window math the workers share. +- `src/second_brain/transcribe.py` — faster-whisper wrapper + segment + serialiser. +- `src/second_brain/scheduler/runner.py` — dev extract scheduler. +- `src/second_brain/scheduler/transcribe_worker.py` — tower poll loop. +- `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert + into `public.embeddings`; best-effort with graceful degradation. +- `alembic/` + `alembic.ini` — schema migrations for the `second_brain` + schema only. `public.embeddings` is **not** ours to migrate. +- `deploy/web/` — Dockerfile + compose.yml + README for the web + container. Travis applies the Knot DNS A-record + Traefik + file-provider router from the README §2. +- `deploy/tower/` — systemd unit + `run-transcribe-worker.sh` wrapper + + env template + EndeavourOS install runbook. +- `prompts/.md` — extraction prompt templates. +- `config/settings.toml` — `[database]`, `[extractor]`, `[embeddings]`, + `[scheduler]`, `[whisper]`. -## Conventions / gotchas +## Load-bearing gotchas -- **Naive UTC timestamps everywhere.** Use `from second_brain.models import utcnow` rather than `datetime.utcnow()` (deprecated in 3.12) or `datetime.now()` (local time). Postgres columns are `timestamp without time zone` — do not switch to `timestamptz`. -- **PK style.** Int autoincrement (`SERIAL`) — locked for compatibility with the existing schema shape. Do not switch to `BIGINT GENERATED ALWAYS AS IDENTITY` or UUID. -- **Re-query inside the session.** SQLAlchemy ORM attribute access after the session closes raises `DetachedInstanceError`. The `process` command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates *inside* the `with db.session()` block. -- **Starlette `TemplateResponse` signature.** Starlette ≥ 0.36 requires `templates.TemplateResponse(request, name, context)`. The old `(name, context)` form raises `TypeError: unhashable type: 'dict'`. Do not put `"request": request` into the context dict — it's auto-injected now. -- **Vault git repo auto-init.** The wiki compiler calls `_ensure_git_repo()` on every run, which `git init`s the vault and creates an empty seed commit if no `.git` is present. `_git_commit()` also checks `git diff --cached --quiet` before committing to avoid empty commits. -- **Scheduler throttling shape.** The CLI backend is throttled by `max_calls_per_hour` (Max subscription limits are call-shaped, not token-shaped). The API backend uses `max_tokens_per_hour`. Both share `min_gap_seconds`. The scheduler targets sources in `TRANSCRIBED` state — earlier code mistakenly filtered on `ANALYZED`, making it a no-op. -- **Alembic + lovebug privilege gap.** `lovebug` owns the `second_brain` schema but does **not** have `CREATE` on the `petalbrain` database. `alembic/env.py` therefore guards `CREATE SCHEMA` behind a `pg_namespace` lookup, and the v1 migration intentionally drops the `IF NOT EXISTS` form (a bare `CREATE SCHEMA IF NOT EXISTS` raises permission denied even when the schema is already present). The schema is bootstrapped once, out-of-band, by `postgres` superuser. -- **public.embeddings is shared, not ours.** Alembic only manages the `second_brain` schema. The vector table itself is owned by `postgres` and shared across vault-mcp / ob1 / second-brain — we just INSERT into it. Rows are keyed by `(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`. -- **Embedding is best-effort.** The pipeline writes the relational `extractions` row first and `sess.flush()`es; only then calls `embed_extraction()`. Any failure (no DB URL, no Ollama, embedding 500) logs at WARNING and returns None — the relational row and file vault stay canonical. -- **File-based wiki, not petalbrain.wiki.** The wiki compiler still writes markdown into the configured `vault_path` and commits via git. We do **not** dual-write into `petalbrain.wiki` — that's vault-mcp's territory and intentionally out of scope. +These have all bitten us; don't relearn them. -## In-flight work +- **(a) Schema bootstrap requires postgres superuser.** `lovebug` does + NOT have `CREATE` on the `petalbrain` database, even though it owns + the `second_brain` schema. The first migration needs a one-time + `CREATE SCHEMA second_brain AUTHORIZATION lovebug; GRANT ALL PRIVILEGES + ON SCHEMA second_brain TO lovebug;` as `postgres`. `alembic/env.py` + guards `CREATE SCHEMA` behind a `pg_namespace` lookup so subsequent + runs as `lovebug` don't trip the permission check. -- **Vault location.** Recommended `/opt/projects/second-brain-vault/` as the production location (peer to the project dir). Not yet confirmed. +- **(b) Arch ships CUDA 13, ctranslate2 needs CUDA 12.** EndeavourOS's + `pacman -S cuda cudnn` installs `libcublas.so.13` / `libcudnn.so.10`, + which `ctranslate2` 4.7.x cannot load — it wants the .12 / .9 ABI. + We pin `nvidia-cublas-cu12` and `nvidia-cudnn-cu12>=9,<10` in the + `tower` optional-dependency group, and the systemd ExecStart calls + `deploy/tower/run-transcribe-worker.sh` which resolves the venv's + wheel-bundled `.so` directories at runtime and prepends them to + `LD_LIBRARY_PATH`. The wrapper uses the namespace packages' + `__path__` (not `__file__`, which is `None` for PEP 420 packages). + systemd does **not** inherit shell exports. + +- **(c) pg_hba needs BOTH subnets.** Two host rules are load-bearing: + ``` + host all all 172.19.0.0/16 scram-sha-256 # homelab docker bridge + host all all 10.99.0.0/24 scram-sha-256 # WireGuard (tower) + ``` + Dropping the 172.19/16 rule locks out the web container + vault-mcp + + ob1-* + trellis-mcp (all on the homelab net). Dropping the 10.99/24 + rule locks out the tower. Both have been removed once each during + unrelated postgres edits — the symptom is `FATAL: no pg_hba.conf + entry for host "…"` in service logs. + +- **(d) Two postgres port binds, both load-bearing.** `homelab-postgres` + publishes `127.0.0.1:5433 → 5432` for dev-host access AND is reachable + on `homelab-postgres:5432` over the docker net. Tower hits it over WG + at `db.wg.herbylab.dev:5432`. Don't consolidate; each path serves a + different consumer set. + +- **(e) Traefik is file-provider on a separate VM (10.0.11.20).** It + cannot read this host's docker labels. The web `compose.yml` therefore + publishes back to a fixed host port (`10.0.21.207:8080`) that Traefik + reaches over LAN, and intentionally carries NO `traefik.*` labels. + Router config lives in Traefik's dynamic-config YAML — see + `deploy/web/README.md` §2b for the snippet. + +- **Code-level conventions, since they bit us early:** + - **Naive UTC timestamps.** Use `from second_brain.models import utcnow` + — never `datetime.utcnow()` (deprecated) or `datetime.now()` (local). + Postgres columns are `timestamp without time zone`. + - **Int autoincrement PKs.** Don't migrate to bigint-identity or UUID; + the existing schema shape is locked. + - **Re-query inside the session.** SQLAlchemy ORM attribute access + after the session closes raises `DetachedInstanceError`. Web routes + and CLI loops collect IDs first, then re-fetch inside the session + block; dict snapshots for templates are built inside `with + db.session()`. + - **Starlette TemplateResponse signature.** `templates.TemplateResponse(request, name, context)` — the old `(name, context)` form raises + `TypeError: unhashable type: 'dict'`. Do not put `"request": + request` into the context dict. + - **Embedding is best-effort.** The extract path flushes the + relational `extractions` row first, then calls `embed_extraction()`. + DB / Ollama failures log and return None; pipeline state stays + canonical in the relational + file vault. + - **Worker LD_LIBRARY_PATH.** See (b). When pre-warming or debugging + faster-whisper manually, run via `deploy/tower/run-transcribe-worker.sh` + or replicate the path probe in your shell. + +## In-flight / parked + +- **Vault location** — recommended `/opt/projects/second-brain-vault/` + as a peer dir to the project. Not yet confirmed. +- **Media + subtitles on NAS** — currently land on the tower's local + SSD. Move to NAS once we have a stable mount path. +- **SSO** — Authentik forward-auth middleware in Traefik is the + obvious next step before opening the UI past LAN. +- **Dev-side extraction systemd timer** — currently invoked manually; + no systemd unit yet by Travis's call. ## Validation -Per Travis's global CLAUDE.md, run `zero-check` after any code change, and `a-review` after non-trivial diffs (> 50 lines). `a-review` script: `python3 ~/.claude/skills/a-review/run_review.py`, output goes to `reviews/a-review-.md`. +Per the user's global CLAUDE.md, run `zero-check` after any code +change. The pipeline's at `~/zero-check-pipeline/validate/run-all.sh +"$(pwd)"`. Run `a-review` after non-trivial diffs (>50 lines): +`python3 ~/.claude/skills/a-review/run_review.py`. Test suite is `uv +run pytest`; live DB + Ollama paths auto-skip when their endpoints +aren't configured.