# second-brain

Personal knowledge pipeline. Two machines coordinate purely through the
shared Postgres queue: the **dev side** (herbys-dev) does article pull
+ all extraction + embedding, and the **tower** (EndeavourOS, RTX 3080)
does video pull + transcription on the GPU. A FastAPI/Jinja/HTMX web
app fronts the queue, settings, and add-to-queue form; the file-based
Obsidian wiki is downstream of `accepted` extractions.

## Two-machine pipeline

```
                add (CLI or POST /sources/add — single video, article,
                     or YouTube playlist URL → fan-out into N rows)
                                  │
                                  ▼
   ┌────────── second_brain.sources (the queue) ──────────┐
   │                                                       │
   │   PENDING  ── (article)──►  dev `second-brain process`
   │                              trafilatura pull         │
   │   PENDING  ── (video)  ──►  tower transcribe-worker   │
   │                              yt-dlp pull → PULLED     │
   │   PULLED   ── (video)  ──►  tower transcribe-worker   │
   │                              faster-whisper           │
   │              both paths land at  ──►  TRANSCRIBED     │
   │                                                       │
   │   TRANSCRIBED ─────────►  dev `second-brain schedule` │
   │                              Claude CLI extract +     │
   │                              Ollama embed             │
   │              ──►  ANALYZED  ──►  accept in web UI     │
   │              ──►  ACCEPTED  ──►  `compile` writes vault│
   │              ──►  PUBLISHED                           │
   │                                                       │
   │   FAILED is the terminal error state                  │
   └───────────────────────────────────────────────────────┘
```

**Source claiming** — `sources.claimed_by` / `claimed_at` columns +
`SELECT ... FOR UPDATE SKIP LOCKED` keep the dev and tower workers from
grabbing the same row. Both stages filter by `source_type`:

- tower transcribe-worker: `status IN (PENDING, PULLED) AND source_type = 'video'`
- dev `process` (article pull): `status = PENDING AND source_type = 'article'`
- dev `schedule` (extract+embed): `status = TRANSCRIBED` (any type)

A 30-minute stale-claim reaper runs at worker startup so a crashed
machine doesn't dam the queue.

## CLI commands

- `second-brain add <url>` — queue a source. YouTube `playlist?list=…`
  URLs auto-expand into one row per video (cap 50, the
  `watch?v=…&list=…` form stays a single video — see
  `is_youtube_playlist_url` for the rule).
- `second-brain process` — dev-side: pull pending **articles**, extract
  + embed any TRANSCRIBED source. Skips videos (tower's job).
- `second-brain schedule` — dev-side overnight extract+embed loop,
  honouring `pipeline_settings.extraction_enabled` / active-hours /
  max-items.
- `second-brain transcribe-worker` — tower-side daemon. `--once` for
  ad-hoc runs. Honours `pipeline_settings.transcription_*` knobs.
- `second-brain serve` — local dev launch of the web UI (port 8000).
  Production runs as a container (see `deploy/web/`).
- `second-brain compile` — wiki compiler: ACCEPTED → markdown in
  `vault_path`, git-committed. File-based Obsidian vault only; we do
  **not** dual-write into petalbrain.wiki.

## Storage

**Postgres**: `petalbrain` DB on `homelab-postgres` (pgvector/pg17, docker
container). Connection role: `lovebug`. Dev reaches it at
`127.0.0.1:5433`; tower reaches it over WireGuard at
`db.wg.herbylab.dev:5432`; the web container reaches it at
`homelab-postgres:5432` on the `homelab` docker bridge. **Keep both port
binds** — the host-side 5433 publish and the docker-net 5432 service.

**Schema `second_brain`** (Alembic-managed in `alembic/`):

- `sources` — the queue. Int SERIAL PK. Naive UTC `timestamp`
  (`without time zone`) — never `timestamptz`. `transcript_text` is the
  joined full text the extractor reads; `transcript_segments` is JSONB
  with the faster-whisper per-segment / per-word breakdown.
  `claimed_by` / `claimed_at` for the queue dance.
- `extractions` — one row per source. JSON columns for the structured
  output (summary, key_points, entities, claims, …).
- `pipeline_settings` — **single-row** config (PK pinned to id=1 with a
  CHECK constraint). What the dashboard edits and workers read each
  iteration: transcription_enabled / active_hours / max_gpu_jobs /
  max_video_length / max_items_per_run plus the equivalent extraction_*
  fields. DB values override settings.toml.
- `wiki_pages` — bookkeeping for the file-based wiki compiler.

**Embeddings** live in `public.embeddings` — the **shared** pgvector
table owned by `postgres` and consumed by vault-mcp / ob1 / second-brain.
We do not manage this in Alembic. Rows are keyed by
`(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`,
with `delete-before-insert` semantics on the 4-tuple so a re-embed
never leaves orphans. We embed extraction **summaries only** for now —
not key_points, not transcript chunks.

## Web app

FastAPI + Jinja2 + HTMX, in `src/second_brain/web/`. Production lives in
a slim Python 3.12 container (see `deploy/web/`) on the `homelab` docker
network, reaching Postgres + Ollama by service DNS. Published at
`10.0.21.207:8080` and fronted by the Traefik VM (10.0.11.20,
file-provider — **no docker-label routing**) as
`https://brain.herbylab.dev`.

Routes:

- `/` — sources list (filterable by domain + status)
- `/queue` — in-flight sources (PENDING / PULLED / TRANSCRIBED) plus the
  add-to-queue form
- `/dashboard` — status counts, recent activity, settings snapshot, plus
  the add-to-queue form
- `/sources/{id}` — extraction detail + accept/reject HTMX actions
- `/settings` — full HTMX editor for `pipeline_settings`
- `POST /sources/add` — shared endpoint. Detects YouTube playlist URLs
  and fans out via yt-dlp `extract_flat`. Reads `HX-Current-URL` to
  dispatch the response partial (`_queue_body.html` vs
  `_dashboard_body.html`) so each page refreshes its own region inline.

**No auth.** UI mutates state (accept/reject sources, edit settings) but
has no SSO / CSRF / rate-limiting. The Traefik route is currently
LAN-only by Travis's call — see `deploy/web/README.md` §SECURITY.

## Transcription

`src/second_brain/transcribe.py` wraps **faster-whisper** (CTranslate2
backend). Defaults: model `large-v3`, device `cuda`, compute_type
`float16`. CPU fallback drops to `int8`. `word_timestamps=True` is
always passed; the segment dicts persisted to
`sources.transcript_segments` carry per-word `{start, end, word,
probability}`.

The tower worker (`src/second_brain/scheduler/transcribe_worker.py`)
runs as a systemd service — see `deploy/tower/README.md` for the full
install runbook. ExecStart goes through
`deploy/tower/run-transcribe-worker.sh` which prepends the venv's
CUDA-12 lib dirs to `LD_LIBRARY_PATH` (see gotcha (b) below).

## Extractor backends

`[extractor].backend` in `config/settings.toml`:

- `cli` (default) — `src/second_brain/llm/claude_cli.py` shells out to
  `claude -p --output-format json --model … --json-schema …`. Runs
  under the Max OAuth subscription. Hermetic spawn: throwaway
  `tempfile.TemporaryDirectory`, `ANTHROPIC_API_KEY` / `_AUTH_TOKEN`
  stripped from env so it uses OAuth, and `--no-session-persistence
  --disable-slash-commands --tools "" --setting-sources ""` so host
  CLAUDE.md / hooks / agents don't leak in.
- `api` — `anthropic` SDK, lazy-imported. Requires `ANTHROPIC_API_KEY`.
  Install with `uv sync --extra api`.

Schema-validated CLI output lands at `payload["structured_output"]`, not
`payload["result"]`. The wrapper checks both.

## Environment variables

- `SECOND_BRAIN_DATABASE_URL` — Postgres DSN. Falls back to
  `HERBYLAB_DATABASE_URL`. Use `postgresql+psycopg://` for SQLAlchemy;
  the embeddings + claim modules strip the `+psycopg` prefix for raw
  psycopg.
- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default
  `http://ollama:11434` (homelab docker net); host-side runs use
  `http://127.0.0.1:11434`.
- `EMBEDDING_MODEL` — default `nomic-embed-text` (768-d).
- `WHISPER_MODEL` / `WHISPER_DEVICE` / `WHISPER_COMPUTE_TYPE` — override
  the `[whisper]` block on the tower.
- `ANTHROPIC_API_KEY` — only for `extractor.backend = "api"`.
- `SECOND_BRAIN_CONFIG` — settings file path override.
- `SECOND_BRAIN_WORKER_ID` — stamped into `sources.claimed_by`.
- `SECOND_BRAIN_DB_POOL_MIN` / `_MAX` — SQLAlchemy pool sizing.
- `SECOND_BRAIN_LIVE_NETWORK_TESTS=1` — opt-in for the yt-dlp playlist
  test that hits real YouTube.

## Domains

`development` / `content` / `business` / `homelab` — drives the
per-domain prompt template loaded by the context assembler.

## Key files

- `src/second_brain/models.py` — all ORM in the `second_brain` schema.
- `src/second_brain/sources_service.py` — `add_source`, `add_playlist`,
  `is_youtube_playlist_url`, `expand_youtube_playlist`. CLI + web both
  call into here so there's no behavior drift between entry points.
- `src/second_brain/claim.py` — the SKIP LOCKED dance + stale reaper.
- `src/second_brain/settings_store.py` — single-row `pipeline_settings`
  get/update + the active-window math the workers share.
- `src/second_brain/transcribe.py` — faster-whisper wrapper + segment
  serialiser.
- `src/second_brain/scheduler/runner.py` — dev extract scheduler.
- `src/second_brain/scheduler/transcribe_worker.py` — tower poll loop.
- `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert
  into `public.embeddings`; best-effort with graceful degradation.
- `alembic/` + `alembic.ini` — schema migrations for the `second_brain`
  schema only. `public.embeddings` is **not** ours to migrate.
- `deploy/web/` — Dockerfile + compose.yml + README for the web
  container. Travis applies the Knot DNS A-record + Traefik
  file-provider router from the README §2.
- `deploy/tower/` — systemd unit + `run-transcribe-worker.sh` wrapper +
  env template + EndeavourOS install runbook.
- `prompts/<domain>.md` — extraction prompt templates.
- `config/settings.toml` — `[database]`, `[extractor]`, `[embeddings]`,
  `[scheduler]`, `[whisper]`.

## Load-bearing gotchas

These have all bitten us; don't relearn them.

- **(a) Schema bootstrap requires postgres superuser.** `lovebug` does
  NOT have `CREATE` on the `petalbrain` database, even though it owns
  the `second_brain` schema. The first migration needs a one-time
  `CREATE SCHEMA second_brain AUTHORIZATION lovebug; GRANT ALL PRIVILEGES
  ON SCHEMA second_brain TO lovebug;` as `postgres`. `alembic/env.py`
  guards `CREATE SCHEMA` behind a `pg_namespace` lookup so subsequent
  runs as `lovebug` don't trip the permission check.

- **(b) Arch ships CUDA 13, ctranslate2 needs CUDA 12.** EndeavourOS's
  `pacman -S cuda cudnn` installs `libcublas.so.13` / `libcudnn.so.10`,
  which `ctranslate2` 4.7.x cannot load — it wants the .12 / .9 ABI.
  We pin `nvidia-cublas-cu12` and `nvidia-cudnn-cu12>=9,<10` in the
  `tower` optional-dependency group, and the systemd ExecStart calls
  `deploy/tower/run-transcribe-worker.sh` which resolves the venv's
  wheel-bundled `.so` directories at runtime and prepends them to
  `LD_LIBRARY_PATH`. The wrapper uses the namespace packages'
  `__path__` (not `__file__`, which is `None` for PEP 420 packages).
  systemd does **not** inherit shell exports.

- **(c) pg_hba needs BOTH subnets.** Two host rules are load-bearing:
  ```
  host all all 172.19.0.0/16  scram-sha-256   # homelab docker bridge
  host all all 10.99.0.0/24   scram-sha-256   # WireGuard (tower)
  ```
  Dropping the 172.19/16 rule locks out the web container + vault-mcp +
  ob1-* + trellis-mcp (all on the homelab net). Dropping the 10.99/24
  rule locks out the tower. Both have been removed once each during
  unrelated postgres edits — the symptom is `FATAL: no pg_hba.conf
  entry for host "…"` in service logs.

- **(d) Two postgres port binds, both load-bearing.** `homelab-postgres`
  publishes `127.0.0.1:5433 → 5432` for dev-host access AND is reachable
  on `homelab-postgres:5432` over the docker net. Tower hits it over WG
  at `db.wg.herbylab.dev:5432`. Don't consolidate; each path serves a
  different consumer set.

- **(e) Traefik is file-provider on a separate VM (10.0.11.20).** It
  cannot read this host's docker labels. The web `compose.yml` therefore
  publishes back to a fixed host port (`10.0.21.207:8080`) that Traefik
  reaches over LAN, and intentionally carries NO `traefik.*` labels.
  Router config lives in Traefik's dynamic-config YAML — see
  `deploy/web/README.md` §2b for the snippet.

- **Code-level conventions, since they bit us early:**
  - **Naive UTC timestamps.** Use `from second_brain.models import utcnow`
    — never `datetime.utcnow()` (deprecated) or `datetime.now()` (local).
    Postgres columns are `timestamp without time zone`.
  - **Int autoincrement PKs.** Don't migrate to bigint-identity or UUID;
    the existing schema shape is locked.
  - **Re-query inside the session.** SQLAlchemy ORM attribute access
    after the session closes raises `DetachedInstanceError`. Web routes
    and CLI loops collect IDs first, then re-fetch inside the session
    block; dict snapshots for templates are built inside `with
    db.session()`.
  - **Starlette TemplateResponse signature.** `templates.TemplateResponse(request, name, context)` — the old `(name, context)` form raises
    `TypeError: unhashable type: 'dict'`. Do not put `"request":
    request` into the context dict.
  - **Embedding is best-effort.** The extract path flushes the
    relational `extractions` row first, then calls `embed_extraction()`.
    DB / Ollama failures log and return None; pipeline state stays
    canonical in the relational + file vault.
  - **Worker LD_LIBRARY_PATH.** See (b). When pre-warming or debugging
    faster-whisper manually, run via `deploy/tower/run-transcribe-worker.sh`
    or replicate the path probe in your shell.

## In-flight / parked

- **Vault location** — recommended `/opt/projects/second-brain-vault/`
  as a peer dir to the project. Not yet confirmed.
- **Media + subtitles on NAS** — currently land on the tower's local
  SSD. Move to NAS once we have a stable mount path.
- **SSO** — Authentik forward-auth middleware in Traefik is the
  obvious next step before opening the UI past LAN.
- **Dev-side extraction systemd timer** — currently invoked manually;
  no systemd unit yet by Travis's call.

## Validation

Per the user's global CLAUDE.md, run `zero-check` after any code
change. The pipeline's at `~/zero-check-pipeline/validate/run-all.sh
"$(pwd)"`. Run `a-review` after non-trivial diffs (>50 lines):
`python3 ~/.claude/skills/a-review/run_review.py`. Test suite is `uv
run pytest`; live DB + Ollama paths auto-skip when their endpoints
aren't configured.