11 changed files with 125 additions and 1137 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -1,297 +1,122 @@
 # second-brain
-Personal knowledge pipeline. Two machines coordinate purely through the
+A personal knowledge system: video/article extraction pipeline + LLM-maintained wiki.
 shared Postgres queue: the **dev side** (herbys-dev) does article pull
 + all extraction + embedding, and the **tower** (EndeavourOS, RTX 3080)
 does video pull + transcription on the GPU. A FastAPI/Jinja/HTMX web
 app fronts the queue, settings, and add-to-queue form; the file-based
 Obsidian wiki is downstream of `accepted` extractions.
-## Two-machine pipeline
+## Architecture
 Three-layer system:
 1. **Wiki** — long-term structured memory. Markdown files in an Obsidian vault, maintained by the wiki compiler.
 2. **Context assembler** — builds the prompt bundle for each extraction. Phase 1 is thin: domain template + focus field + transcript.
 3. **Extractor** — stateless single-shot LLM call per source. Pure function: bundle in, structured extraction out.
 ## Pipeline stages
 ```
-                add (CLI or POST /sources/add — single video, article,
+pending → pulled → transcribed → analyzed → accepted → published
                     or YouTube playlist URL → fan-out into N rows)
                                  │
                                  ▼
   ┌────────── second_brain.sources (the queue) ──────────┐
   │                                                       │
   │   PENDING  ── (article)──►  dev `second-brain process`
   │                              trafilatura pull         │
   │   PENDING  ── (video)  ──►  tower transcribe-worker   │
   │                              yt-dlp pull → PULLED     │
   │   PULLED   ── (video)  ──►  tower transcribe-worker   │
   │                              faster-whisper           │
   │              both paths land at  ──►  TRANSCRIBED     │
   │                                                       │
   │   TRANSCRIBED ─────────►  dev `second-brain schedule` │
   │                              Claude CLI extract +     │
   │                              Ollama embed             │
   │              ──►  ANALYZED  ──►  accept in web UI     │
   │              ──►  ACCEPTED  ──►  `compile` writes vault│
   │              ──►  PUBLISHED                           │
   │                                                       │
   │   FAILED is the terminal error state                  │
   └───────────────────────────────────────────────────────┘
 ```
-**Source claiming** — `sources.claimed_by` / `claimed_at` columns +
+## Project layout
 `SELECT ... FOR UPDATE SKIP LOCKED` keep the dev and tower workers from
 grabbing the same row. Both stages filter by `source_type`:
- tower transcribe-worker: `status IN (PENDING, PULLED) AND source_type = 'video'`
+```
- dev `process` (article pull): `status = PENDING AND source_type = 'article'`
+src/second_brain/
- dev `schedule` (extract+embed): `status = TRANSCRIBED` (any type)
+├── adapters/      # Pull adapters (youtube.py, article.py)
 ├── context/       # Context assembler
 ├── embeddings/    # Chunk + nomic-embed-text + upsert into public.embeddings
 ├── extractor/     # LLM extraction engine + Pydantic schema
 ├── llm/           # claude_cli.py — thin subprocess wrapper around `claude -p`
 ├── compiler/      # Wiki compiler (reads vault, writes pages, git commits)
 ├── scheduler/     # Rate-limit smoother (call-count for CLI, token-budget for API)
 └── web/           # FastAPI + Jinja2 + HTMX UI
-A 30-minute stale-claim reaper runs at worker startup so a crashed
+alembic/           # Schema migrations for the second_brain Postgres schema
-machine doesn't dam the queue.
+```
 ## Setup
 ```bash
 cp config/settings.toml.example config/settings.toml
 # Edit settings.toml — at minimum, point the paths and set SECOND_BRAIN_DATABASE_URL.
 # Connection string format. Containerised (default):
 #   postgresql+psycopg://lovebug:<password>@homelab-postgres:5432/petalbrain
 # Host-side dev runs:
 #   postgresql+psycopg://lovebug:<password>@127.0.0.1:5433/petalbrain
 export SECOND_BRAIN_DATABASE_URL=...
 # One-time, before first migration: bootstrap the schema as postgres
 # superuser. lovebug doesn't have CREATE on petalbrain, so this can't be
 # done from the app side — same pattern as wiki/trellis/ob1.
 #   CREATE SCHEMA second_brain AUTHORIZATION lovebug;
 #   GRANT ALL PRIVILEGES ON SCHEMA second_brain TO lovebug;
 uv sync
 uv run alembic upgrade head
 uv run second-brain serve
 ```
 ## CLI commands
- `second-brain add <url>` — queue a source. YouTube `playlist?list=…`
+- `second-brain add <url>` — queue a source URL (video or article)
-  URLs auto-expand into one row per video (cap 50, the
+- `second-brain process` — run the pipeline on pending items
-  `watch?v=…&list=…` form stays a single video — see
+- `second-brain serve` — launch the web UI (default port 8000)
-  `is_youtube_playlist_url` for the rule).
+- `second-brain compile` — run the wiki compiler
- `second-brain process` — dev-side: pull pending **articles**, extract
+- `second-brain schedule` — start the overnight scheduler
  + embed any TRANSCRIBED source. Skips videos (tower's job).
 - `second-brain schedule` — dev-side overnight extract+embed loop,
  honouring `pipeline_settings.extraction_enabled` / active-hours /
  max-items.
 - `second-brain transcribe-worker` — tower-side daemon. `--once` for
  ad-hoc runs. Honours `pipeline_settings.transcription_*` knobs.
 - `second-brain serve` — local dev launch of the web UI (port 8000).
  Production runs as a container (see `deploy/web/`).
 - `second-brain compile` — wiki compiler: ACCEPTED → markdown in
  `vault_path`, git-committed. File-based Obsidian vault only; we do
  **not** dual-write into petalbrain.wiki.
 ## Storage
 **Postgres**: `petalbrain` DB on `homelab-postgres` (pgvector/pg17, docker
 container). Connection role: `lovebug`. Dev reaches it at
 `127.0.0.1:5433`; tower reaches it over WireGuard at
 `db.wg.herbylab.dev:5432`; the web container reaches it at
 `homelab-postgres:5432` on the `homelab` docker bridge. **Keep both port
 binds** — the host-side 5433 publish and the docker-net 5432 service.
 **Schema `second_brain`** (Alembic-managed in `alembic/`):
 - `sources` — the queue. Int SERIAL PK. Naive UTC `timestamp`
  (`without time zone`) — never `timestamptz`. `transcript_text` is the
  joined full text the extractor reads; `transcript_segments` is JSONB
  with the faster-whisper per-segment / per-word breakdown.
  `claimed_by` / `claimed_at` for the queue dance.
 - `extractions` — one row per source. JSON columns for the structured
  output (summary, key_points, entities, claims, …).
 - `pipeline_settings` — **single-row** config (PK pinned to id=1 with a
  CHECK constraint). What the dashboard edits and workers read each
  iteration: transcription_enabled / active_hours / max_gpu_jobs /
  max_video_length / max_items_per_run plus the equivalent extraction_*
  fields. DB values override settings.toml.
 - `wiki_pages` — bookkeeping for the file-based wiki compiler.
 **Embeddings** live in `public.embeddings` — the **shared** pgvector
 table owned by `postgres` and consumed by vault-mcp / ob1 / second-brain.
 We do not manage this in Alembic. Rows are keyed by
 `(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`,
 with `delete-before-insert` semantics on the 4-tuple so a re-embed
 never leaves orphans. We embed extraction **summaries only** for now —
 not key_points, not transcript chunks.
 ## Web app
 FastAPI + Jinja2 + HTMX, in `src/second_brain/web/`. Production lives in
 a slim Python 3.12 container (see `deploy/web/`) on the `homelab` docker
 network, reaching Postgres + Ollama by service DNS. Published at
 `10.0.21.207:8080` and fronted by the Traefik VM (10.0.11.20,
 file-provider — **no docker-label routing**) as
 `https://brain.herbylab.dev`.
 Routes:
 - `/` — sources list (filterable by domain + status)
 - `/queue` — in-flight sources (PENDING / PULLED / TRANSCRIBED) plus the
  add-to-queue form
 - `/dashboard` — status counts, recent activity, settings snapshot, plus
  the add-to-queue form
 - `/sources/{id}` — extraction detail + accept/reject HTMX actions
 - `/settings` — full HTMX editor for `pipeline_settings`
 - `POST /sources/add` — shared endpoint. Detects YouTube playlist URLs
  and fans out via yt-dlp `extract_flat`. Reads `HX-Current-URL` to
  dispatch the response partial (`_queue_body.html` vs
  `_dashboard_body.html`) so each page refreshes its own region inline.
 **No auth.** UI mutates state (accept/reject sources, edit settings) but
 has no SSO / CSRF / rate-limiting. The Traefik route is currently
 LAN-only by Travis's call — see `deploy/web/README.md` §SECURITY.
 ## Transcription
 `src/second_brain/transcribe.py` wraps **faster-whisper** (CTranslate2
 backend). Defaults: model `large-v3`, device `cuda`, compute_type
 `float16`. CPU fallback drops to `int8`. `word_timestamps=True` is
 always passed; the segment dicts persisted to
 `sources.transcript_segments` carry per-word `{start, end, word,
 probability}`.
 The tower worker (`src/second_brain/scheduler/transcribe_worker.py`)
 runs as a systemd service — see `deploy/tower/README.md` for the full
 install runbook. ExecStart goes through
 `deploy/tower/run-transcribe-worker.sh` which prepends the venv's
 CUDA-12 lib dirs to `LD_LIBRARY_PATH` (see gotcha (b) below).
 ## Extractor backends
-`[extractor].backend` in `config/settings.toml`:
+`[extractor].backend` in `config/settings.toml` picks the LLM transport.
- `cli` (default) — `src/second_brain/llm/claude_cli.py` shells out to
+- **`cli`** (default) — `src/second_brain/llm/claude_cli.py` shells out to `claude -p --output-format json --model <model> --json-schema <schema>`. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside `tempfile.TemporaryDirectory(prefix="sb-claude-cli-")`, with `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` stripped from env so the CLI uses OAuth instead of an API key, and with `--no-session-persistence --disable-slash-commands --tools "" --setting-sources ""` so host CLAUDE.md / hooks / agents don't leak in.
-  `claude -p --output-format json --model … --json-schema …`. Runs
+- **`api`** — uses the `anthropic` SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (`uv sync --extra api`).
  under the Max OAuth subscription. Hermetic spawn: throwaway
  `tempfile.TemporaryDirectory`, `ANTHROPIC_API_KEY` / `_AUTH_TOKEN`
  stripped from env so it uses OAuth, and `--no-session-persistence
  --disable-slash-commands --tools "" --setting-sources ""` so host
  CLAUDE.md / hooks / agents don't leak in.
 - `api` — `anthropic` SDK, lazy-imported. Requires `ANTHROPIC_API_KEY`.
  Install with `uv sync --extra api`.
-Schema-validated CLI output lands at `payload["structured_output"]`, not
+**Schema-validated output gotcha:** when `--json-schema` is used, the parsed object lands at `payload["structured_output"]`, not `payload["result"]`. The wrapper checks both. The engine prefers the structured payload via `LLMExtraction.model_validate(structured)` and falls back to parsing the text field.
 `payload["result"]`. The wrapper checks both.
 ## Environment variables
- `SECOND_BRAIN_DATABASE_URL` — Postgres DSN. Falls back to
+- `ANTHROPIC_API_KEY` — only required when `extractor.backend = "api"`. The default `cli` backend uses your Max OAuth and does not need it.
-  `HERBYLAB_DATABASE_URL`. Use `postgresql+psycopg://` for SQLAlchemy;
+- `SECOND_BRAIN_CONFIG` — optional override for the settings file path.
-  the embeddings + claim modules strip the `+psycopg` prefix for raw
+- `SECOND_BRAIN_DATABASE_URL` — Postgres connection string. Falls back to `HERBYLAB_DATABASE_URL` if unset. Required at runtime; both `second_brain.database` and `second_brain.embeddings` read it. Use the SQLAlchemy `postgresql+psycopg://` form; the embeddings module strips the `+psycopg` prefix internally for raw psycopg.
-  psycopg.
+- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default `http://ollama:11434` (homelab docker network); for host-side runs use `http://127.0.0.1:11434`.
- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default
+- `EMBEDDING_MODEL` — overrides `[embeddings].model`. Default `nomic-embed-text` (768-d).
-  `http://ollama:11434` (homelab docker net); host-side runs use
+- `SECOND_BRAIN_DB_POOL_MIN` / `SECOND_BRAIN_DB_POOL_MAX` — SQLAlchemy pool sizing knobs (defaults 1 / 9 overflow).
  `http://127.0.0.1:11434`.
 - `EMBEDDING_MODEL` — default `nomic-embed-text` (768-d).
 - `WHISPER_MODEL` / `WHISPER_DEVICE` / `WHISPER_COMPUTE_TYPE` — override
  the `[whisper]` block on the tower.
 - `ANTHROPIC_API_KEY` — only for `extractor.backend = "api"`.
 - `SECOND_BRAIN_CONFIG` — settings file path override.
 - `SECOND_BRAIN_WORKER_ID` — stamped into `sources.claimed_by`.
 - `SECOND_BRAIN_DB_POOL_MIN` / `_MAX` — SQLAlchemy pool sizing.
 - `SECOND_BRAIN_LIVE_NETWORK_TESTS=1` — opt-in for the yt-dlp playlist
  test that hits real YouTube.
 ## Domains
-`development` / `content` / `business` / `homelab` — drives the
+- `development` — software, programming, systems
-per-domain prompt template loaded by the context assembler.
+- `content` — content creation strategy, video production
 - `business` — entrepreneurship, marketing, operations
 - `homelab` — self-hosted infrastructure, networking, DevOps
 ## Key files
- `src/second_brain/models.py` — all ORM in the `second_brain` schema.
+- `config/settings.toml` — vault path, `[database]`, `[extractor]`, `[embeddings]`, `[scheduler]`
- `src/second_brain/sources_service.py` — `add_source`, `add_playlist`,
+- `prompts/<domain>.md` — per-domain extraction prompt templates
-  `is_youtube_playlist_url`, `expand_youtube_playlist`. CLI + web both
+- `src/second_brain/extractor/schema.py` — `LLMExtraction` (LLM-only fields, drives the `--json-schema` contract) + `ExtractionResult` (LLMExtraction + SourceMeta filled from the DB)
-  call into here so there's no behavior drift between entry points.
+- `src/second_brain/llm/claude_cli.py` — CLI subprocess wrapper
- `src/second_brain/claim.py` — the SKIP LOCKED dance + stale reaper.
+- `src/second_brain/models.py` — SQLAlchemy models (all tables live in the `second_brain` schema) + `utcnow()` helper
- `src/second_brain/settings_store.py` — single-row `pipeline_settings`
+- `src/second_brain/database.py` — Postgres engine + URL resolution
-  get/update + the active-window math the workers share.
+- `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert into `public.embeddings`; best-effort with graceful degradation
- `src/second_brain/transcribe.py` — faster-whisper wrapper + segment
+- `alembic/` + `alembic.ini` — schema migrations for the `second_brain` schema (Alembic does **not** manage `public.embeddings`)
-  serialiser.
+- `postgres-migration-planning.md` — original planning doc; superseded by this file once the migration shipped
 - `src/second_brain/scheduler/runner.py` — dev extract scheduler.
 - `src/second_brain/scheduler/transcribe_worker.py` — tower poll loop.
 - `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert
  into `public.embeddings`; best-effort with graceful degradation.
 - `alembic/` + `alembic.ini` — schema migrations for the `second_brain`
  schema only. `public.embeddings` is **not** ours to migrate.
 - `deploy/web/` — Dockerfile + compose.yml + README for the web
  container. Travis applies the Knot DNS A-record + Traefik
  file-provider router from the README §2.
 - `deploy/tower/` — systemd unit + `run-transcribe-worker.sh` wrapper +
  env template + EndeavourOS install runbook.
 - `prompts/<domain>.md` — extraction prompt templates.
 - `config/settings.toml` — `[database]`, `[extractor]`, `[embeddings]`,
  `[scheduler]`, `[whisper]`.
-## Load-bearing gotchas
+## Conventions / gotchas
-These have all bitten us; don't relearn them.
+- **Naive UTC timestamps everywhere.** Use `from second_brain.models import utcnow` rather than `datetime.utcnow()` (deprecated in 3.12) or `datetime.now()` (local time). Postgres columns are `timestamp without time zone` — do not switch to `timestamptz`.
 - **PK style.** Int autoincrement (`SERIAL`) — locked for compatibility with the existing schema shape. Do not switch to `BIGINT GENERATED ALWAYS AS IDENTITY` or UUID.
 - **Re-query inside the session.** SQLAlchemy ORM attribute access after the session closes raises `DetachedInstanceError`. The `process` command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates *inside* the `with db.session()` block.
 - **Starlette `TemplateResponse` signature.** Starlette ≥ 0.36 requires `templates.TemplateResponse(request, name, context)`. The old `(name, context)` form raises `TypeError: unhashable type: 'dict'`. Do not put `"request": request` into the context dict — it's auto-injected now.
 - **Vault git repo auto-init.** The wiki compiler calls `_ensure_git_repo()` on every run, which `git init`s the vault and creates an empty seed commit if no `.git` is present. `_git_commit()` also checks `git diff --cached --quiet` before committing to avoid empty commits.
 - **Scheduler throttling shape.** The CLI backend is throttled by `max_calls_per_hour` (Max subscription limits are call-shaped, not token-shaped). The API backend uses `max_tokens_per_hour`. Both share `min_gap_seconds`. The scheduler targets sources in `TRANSCRIBED` state — earlier code mistakenly filtered on `ANALYZED`, making it a no-op.
 - **Alembic + lovebug privilege gap.** `lovebug` owns the `second_brain` schema but does **not** have `CREATE` on the `petalbrain` database. `alembic/env.py` therefore guards `CREATE SCHEMA` behind a `pg_namespace` lookup, and the v1 migration intentionally drops the `IF NOT EXISTS` form (a bare `CREATE SCHEMA IF NOT EXISTS` raises permission denied even when the schema is already present). The schema is bootstrapped once, out-of-band, by `postgres` superuser.
 - **public.embeddings is shared, not ours.** Alembic only manages the `second_brain` schema. The vector table itself is owned by `postgres` and shared across vault-mcp / ob1 / second-brain — we just INSERT into it. Rows are keyed by `(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`.
 - **Embedding is best-effort.** The pipeline writes the relational `extractions` row first and `sess.flush()`es; only then calls `embed_extraction()`. Any failure (no DB URL, no Ollama, embedding 500) logs at WARNING and returns None — the relational row and file vault stay canonical.
 - **File-based wiki, not petalbrain.wiki.** The wiki compiler still writes markdown into the configured `vault_path` and commits via git. We do **not** dual-write into `petalbrain.wiki` — that's vault-mcp's territory and intentionally out of scope.
- **(a) Schema bootstrap requires postgres superuser.** `lovebug` does
+## In-flight work
  NOT have `CREATE` on the `petalbrain` database, even though it owns
  the `second_brain` schema. The first migration needs a one-time
  `CREATE SCHEMA second_brain AUTHORIZATION lovebug; GRANT ALL PRIVILEGES
  ON SCHEMA second_brain TO lovebug;` as `postgres`. `alembic/env.py`
  guards `CREATE SCHEMA` behind a `pg_namespace` lookup so subsequent
  runs as `lovebug` don't trip the permission check.
- **(b) Arch ships CUDA 13, ctranslate2 needs CUDA 12.** EndeavourOS's
+- **Vault location.** Recommended `/opt/projects/second-brain-vault/` as the production location (peer to the project dir). Not yet confirmed.
  `pacman -S cuda cudnn` installs `libcublas.so.13` / `libcudnn.so.10`,
  which `ctranslate2` 4.7.x cannot load — it wants the .12 / .9 ABI.
  We pin `nvidia-cublas-cu12` and `nvidia-cudnn-cu12>=9,<10` in the
  `tower` optional-dependency group, and the systemd ExecStart calls
  `deploy/tower/run-transcribe-worker.sh` which resolves the venv's
  wheel-bundled `.so` directories at runtime and prepends them to
  `LD_LIBRARY_PATH`. The wrapper uses the namespace packages'
  `__path__` (not `__file__`, which is `None` for PEP 420 packages).
  systemd does **not** inherit shell exports.
 - **(c) pg_hba needs BOTH subnets.** Two host rules are load-bearing:
  ```
  host all all 172.19.0.0/16  scram-sha-256   # homelab docker bridge
  host all all 10.99.0.0/24   scram-sha-256   # WireGuard (tower)
  ```
  Dropping the 172.19/16 rule locks out the web container + vault-mcp +
  ob1-* + trellis-mcp (all on the homelab net). Dropping the 10.99/24
  rule locks out the tower. Both have been removed once each during
  unrelated postgres edits — the symptom is `FATAL: no pg_hba.conf
  entry for host "…"` in service logs.
 - **(d) Two postgres port binds, both load-bearing.** `homelab-postgres`
  publishes `127.0.0.1:5433 → 5432` for dev-host access AND is reachable
  on `homelab-postgres:5432` over the docker net. Tower hits it over WG
  at `db.wg.herbylab.dev:5432`. Don't consolidate; each path serves a
  different consumer set.
 - **(e) Traefik is file-provider on a separate VM (10.0.11.20).** It
  cannot read this host's docker labels. The web `compose.yml` therefore
  publishes back to a fixed host port (`10.0.21.207:8080`) that Traefik
  reaches over LAN, and intentionally carries NO `traefik.*` labels.
  Router config lives in Traefik's dynamic-config YAML — see
  `deploy/web/README.md` §2b for the snippet.
 - **Code-level conventions, since they bit us early:**
  - **Naive UTC timestamps.** Use `from second_brain.models import utcnow`
    — never `datetime.utcnow()` (deprecated) or `datetime.now()` (local).
    Postgres columns are `timestamp without time zone`.
  - **Int autoincrement PKs.** Don't migrate to bigint-identity or UUID;
    the existing schema shape is locked.
  - **Re-query inside the session.** SQLAlchemy ORM attribute access
    after the session closes raises `DetachedInstanceError`. Web routes
    and CLI loops collect IDs first, then re-fetch inside the session
    block; dict snapshots for templates are built inside `with
    db.session()`.
  - **Starlette TemplateResponse signature.** `templates.TemplateResponse(request, name, context)` — the old `(name, context)` form raises
    `TypeError: unhashable type: 'dict'`. Do not put `"request":
    request` into the context dict.
  - **Embedding is best-effort.** The extract path flushes the
    relational `extractions` row first, then calls `embed_extraction()`.
    DB / Ollama failures log and return None; pipeline state stays
    canonical in the relational + file vault.
  - **Worker LD_LIBRARY_PATH.** See (b). When pre-warming or debugging
    faster-whisper manually, run via `deploy/tower/run-transcribe-worker.sh`
    or replicate the path probe in your shell.
 ## In-flight / parked
 - **Vault location** — recommended `/opt/projects/second-brain-vault/`
  as a peer dir to the project. Not yet confirmed.
 - **Media + subtitles on NAS** — currently land on the tower's local
  SSD. Move to NAS once we have a stable mount path.
 - **SSO** — Authentik forward-auth middleware in Traefik is the
  obvious next step before opening the UI past LAN.
 - **Dev-side extraction systemd timer** — currently invoked manually;
  no systemd unit yet by Travis's call.
 ## Validation
-Per the user's global CLAUDE.md, run `zero-check` after any code
+Per Travis's global CLAUDE.md, run `zero-check` after any code change, and `a-review` after non-trivial diffs (> 50 lines). `a-review` script: `python3 ~/.claude/skills/a-review/run_review.py`, output goes to `reviews/a-review-<date>.md`.
 change. The pipeline's at `~/zero-check-pipeline/validate/run-all.sh
 "$(pwd)"`. Run `a-review` after non-trivial diffs (>50 lines):
 `python3 ~/.claude/skills/a-review/run_review.py`. Test suite is `uv
 run pytest`; live DB + Ollama paths auto-skip when their endpoints
 aren't configured.
--- a/deploy/tower/README.md
+++ b/deploy/tower/README.md
@ -50,9 +50,9 @@ your CLI PATH.
 ```bash
 sudo mkdir -p /opt/projects/homelab
-sudo chown trucktrav:trucktrav /opt/projects/homelab
+sudo chown herbyadmin:herbyadmin /opt/projects/homelab
 cd /opt/projects/homelab
-git clone gitea-pbs-trucktrav:petal-power/second-brain.git
+git clone gitea-lovebug:petal-power/second-brain.git
 cd second-brain
 # Install the runtime deps + the `tower` extra (faster-whisper / CTranslate2).
@ -90,7 +90,7 @@ The worker assumes the tunnel is already configured and that
 | Peer | WG IP |
 |------|-------|
 | hub (herbydev) | `10.99.0.1` |
-| tower | `10.99.0.3` |
+| tower | `10.99.0.2` |
 `db.wg.herbylab.dev` resolves to `10.99.0.1`. Confirm connectivity from
 the tower:
--- a/deploy/tower/second-brain-transcribe.service
+++ b/deploy/tower/second-brain-transcribe.service
@ -16,8 +16,8 @@ StartLimitBurst=5
 [Service]
 Type=simple
-User=trucktrav
+User=herbyadmin
-Group=trucktrav
+Group=herbyadmin
 # Adjust to wherever you `git clone`d the repo on the tower.
 WorkingDirectory=/opt/projects/homelab/second-brain
--- a/reviews/a-review-2026-05-25.md
+++ b/reviews/a-review-2026-05-25.md
@ -1,37 +0,0 @@
 # a-review — 2026-05-25
 **Model:** gemini
 **Diff base:** `HEAD~1` (5 files changed, 615 insertions(+), 30 deletions(-))
 **Session reference:** none
 **Context:** CLAUDE.md + session notes + expanded diff
 ---
 Excellent work. The changes are well-structured, follow existing project conventions, and include a comprehensive test suite. This is a high-quality contribution. I have only one minor finding and one informational note.
 ### [WARNING] Logic Error
 *   **File:** `src/second_brain/web/templates/source_detail.html`
 *   **Location:** Line 151
 *   **Issue:** The "Re-extract only" button is conditionally rendered with `{% if source.has_transcript and source.source_type == 'video' %}`. This is too restrictive. The backend service function `reset_source_for_retry` correctly allows an "extract" retry for any source that has content in `transcript_text`, which includes processed articles. Limiting this feature to only videos in the UI prevents a valid and useful action for non-video sources.
 *   **Suggestion:** Change the conditional to `{% if source.has_transcript %}`. This will correctly show the button for any source (article or video) that has been pulled and has content ready for extraction, which matches the backend's capability.
 ```html
    <!-- Change this -->
    {% if source.has_transcript and source.source_type == 'video' %}
    <!-- To this -->
    {% if source.has_transcript %}
 ```
 ### [INFO] Security
 *   **Files:** `src/second_brain/web/app.py`, `src/second_brain/web/templates/source_detail.html`
 *   **Observation:** The changes introduce new endpoints (`/edit`, `/retry`) that modify database state. According to `CLAUDE.md`, the application has "No auth" and relies on being deployed in a trusted, LAN-only environment. These new endpoints increase the number of ways an unauthorized user on the network could alter data.
 *   **Context:** This is not a new vulnerability, but an expansion of the existing trust model. As long as the application remains on a trusted network, this is acceptable per the project's documented security posture. It's worth keeping in mind that no CSRF protection is being used for these HTMX `POST` actions.
 ### Summary
 The overall code quality is very high. The changes are thoughtfully implemented, with a clean separation between the CLI, the web layer, and a shared service layer. The refactoring in `web/app.py` to create `_render_source_detail` is a good move for maintainability. The addition of a thorough, live-DB test suite for the new service functions is commendable and demonstrates a commitment to quality.
 Once the minor template logic issue is resolved, this change will be in perfect shape.
--- a/src/second_brain/config.py
+++ b/src/second_brain/config.py
@ -85,28 +85,6 @@ def _merge(default: dict, override: dict) -> dict:
    return out
 def _overlay_sections(base: dict, overlay: dict) -> dict:
    """Section-level overlay: one level deep.
    For each key in `overlay`, if both `base[key]` and `overlay[key]` are
    dicts, merge their keys (overlay wins). Otherwise the overlay value
    replaces the base value outright. Sections present in only one of
    the two pass through untouched.
    Use this — not `_merge` — when combining `settings.toml` with
    `settings.local.toml`, so a local `[database] url = "..."` doesn't
    silently wipe other `[database]` keys defined in the base file.
    """
    out = dict(base)
    for key, ov_val in (overlay or {}).items():
        base_val = out.get(key)
        if isinstance(base_val, dict) and isinstance(ov_val, dict):
            out[key] = {**base_val, **ov_val}
        else:
            out[key] = ov_val
    return out
 class Config:
    """Holds resolved configuration values."""
@ -197,24 +175,7 @@ _config_instance: Config | None = None
 def load_config(path: Path | None = None) -> Config:
-    """Load (and cache) configuration from a TOML file.
+    """Load (and cache) configuration from a TOML file."""
    After loading the base file, look for a sibling `settings.local.toml`
    and overlay it (section-level merge — see `_overlay_sections`). The
    local file is gitignored and intended for host-specific secrets
    (e.g. `[database].url` for a dev shell) without forking the tracked
    base.
    Path resolution:
      - explicit `path=` arg wins
      - else `SECOND_BRAIN_CONFIG` env var
      - else walks up from CWD for `config/settings.toml` or `settings.toml`
    The local-overlay sibling is looked up next to whichever base file
    was chosen, so `SECOND_BRAIN_CONFIG=/etc/sb/settings.toml` will also
    pick up `/etc/sb/settings.local.toml` if present. If no base file is
    found at all, no overlay is attempted.
    """
    global _config_instance
    if _config_instance is not None:
        return _config_instance
@ -236,15 +197,6 @@ def load_config(path: Path | None = None) -> Config:
        with open(path, "rb") as fh:
            raw = tomllib.load(fh)
        # Section-level overlay from a sibling settings.local.toml.
        # Filename is the one already covered in .gitignore — host
        # secrets live here and never get committed.
        local_path = path.parent / "settings.local.toml"
        if local_path.exists():
            with open(local_path, "rb") as fh:
                local_raw = tomllib.load(fh)
            raw = _overlay_sections(raw, local_raw)
    _config_instance = Config(raw)
    return _config_instance
--- a/src/second_brain/main.py
+++ b/src/second_brain/main.py
@ -4,7 +4,6 @@ second-brain CLI entry point.
 Commands:
  add       — queue a source URL (video or article)
  process   — run the full pipeline on pending/pulled/transcribed sources
  retry     — reset a source so the pipeline re-processes it
  serve     — launch the FastAPI web UI
  compile   — run the wiki compiler on accepted sources
  schedule  — start the overnight scheduler
@ -308,47 +307,6 @@ def serve(host: str, port: int, reload: bool) -> None:
    )
 # ---------------------------------------------------------------------------
 # retry
 # ---------------------------------------------------------------------------
@cli.command()
@click.argument("source_id", type=int)
@click.option(
    "--mode",
    type=click.Choice(["full", "extract"]),
    default="full",
    show_default=True,
    help="`full` resets to PENDING (re-pull + re-process); "
    "`extract` resets to TRANSCRIBED (requires existing transcript).",
 )
 def retry(source_id: int, mode: str) -> None:
    """Reset a source so the pipeline re-processes it.
    Clears the claim pair and error_message and rewinds status to the
    appropriate stage. Parity with the dashboard re-run buttons.
    """
    from second_brain.sources_service import reset_source_for_retry
    db = get_database()
    try:
        with db.session() as sess:
            source = reset_source_for_retry(
                sess, source_id=source_id, mode=mode
            )
            new_status = source.status.value
            url = source.url
    except LookupError as exc:
        click.echo(f"[retry] {exc}", err=True)
        sys.exit(2)
    except ValueError as exc:
        click.echo(f"[retry] {exc}", err=True)
        sys.exit(2)
    click.echo(f"[retry] id={source_id} status → {new_status}  ({url})")
 # ---------------------------------------------------------------------------
 # compile
 # ---------------------------------------------------------------------------
--- a/src/second_brain/sources_service.py
+++ b/src/second_brain/sources_service.py
@ -275,126 +275,14 @@ def add_playlist(
    return result
 # ---------------------------------------------------------------------------
 # Edit + retry helpers (shared by web routes and CLI)
 # ---------------------------------------------------------------------------
 # Fields the edit-mode form is allowed to mutate. URL is intentionally
 # excluded — it's the UNIQUE dedupe key and editing it risks collisions
 # plus breaks the embeddings 4-tuple identity (source_id is stable but
 # semantic identity drifts).
 EDITABLE_METADATA_FIELDS = ("domain", "title", "focus")
 def update_source_metadata(
    sess: Session,
    *,
    source_id: int,
    domain: Optional[str] = None,
    title: Optional[str] = None,
    focus: Optional[str] = None,
 ) -> Source:
    """Update editable metadata on a source row.
    Empty strings collapse to NULL for title/focus (matches `add_source`
    semantics). Domain is required to stay in the configured set —
    bumping the constraint is a separate decision.
    Raises ValueError on validation failures and LookupError if the row
    is missing. Caller owns the transaction boundary.
    """
    source = sess.query(Source).filter(Source.id == source_id).first()
    if source is None:
        raise LookupError(f"source id={source_id} not found")
    if domain is not None:
        if domain not in DOMAINS:
            raise ValueError(
                f"unknown domain {domain!r}; valid: {', '.join(DOMAINS)}"
            )
        source.domain = domain
    if title is not None:
        source.title = title.strip() or None
    if focus is not None:
        source.focus = focus.strip() or None
    source.updated_at = utcnow()
    sess.flush()
    return source
 # Retry modes — keep the surface small. Anything else is a separate ask.
 RETRY_FULL = "full"
 RETRY_EXTRACT = "extract"
 RETRY_MODES = (RETRY_FULL, RETRY_EXTRACT)
 def reset_source_for_retry(
    sess: Session,
    *,
    source_id: int,
    mode: str = RETRY_FULL,
 ) -> Source:
    """Reset a source so the pipeline re-processes it.
    Modes:
      - "full": status → PENDING. Article pull or video download +
        transcribe will happen again on the appropriate worker.
      - "extract": status → TRANSCRIBED. Requires the row to already
        carry a transcript (transcript_text). The dev scheduler will
        re-run extraction without re-pulling the source. Cheap.
    Always clears `claimed_by` / `claimed_at` / `error_message` so the
    queue dance can pick it up cleanly and stale errors don't bleed
    into the UI.
    Raises ValueError on bad mode / missing transcript for extract mode,
    and LookupError if the row is missing.
    """
    if mode not in RETRY_MODES:
        raise ValueError(
            f"unknown retry mode {mode!r}; valid: {', '.join(RETRY_MODES)}"
        )
    source = sess.query(Source).filter(Source.id == source_id).first()
    if source is None:
        raise LookupError(f"source id={source_id} not found")
    if mode == RETRY_EXTRACT:
        if not (source.transcript_text and source.transcript_text.strip()):
            raise ValueError(
                "extract-only retry requires an existing transcript; "
                "use full retry instead"
            )
        source.status = SourceStatus.TRANSCRIBED
    else:
        source.status = SourceStatus.PENDING
    source.claimed_by = None
    source.claimed_at = None
    source.error_message = None
    source.updated_at = utcnow()
    sess.flush()
    return source
 __all__ = [
    "AddResult",
    "DEFAULT_PLAYLIST_MAX_ITEMS",
    "EDITABLE_METADATA_FIELDS",
    "PlaylistAddResult",
    "RETRY_EXTRACT",
    "RETRY_FULL",
    "RETRY_MODES",
    "add_playlist",
    "add_source",
    "expand_youtube_playlist",
    "is_article_url",
    "is_youtube_playlist_url",
    "reset_source_for_retry",
    "update_source_metadata",
    "validate_url",
 ]
--- a/src/second_brain/web/app.py
+++ b/src/second_brain/web/app.py
@ -31,14 +31,9 @@ from second_brain.settings_store import (
 )
 from second_brain.sources_service import (
    DEFAULT_PLAYLIST_MAX_ITEMS,
    RETRY_EXTRACT,
    RETRY_FULL,
    RETRY_MODES,
    add_playlist,
    add_source,
    is_youtube_playlist_url,
    reset_source_for_retry,
    update_source_metadata,
 )
 # ---------------------------------------------------------------------------
@ -125,33 +120,16 @@ async def index(
    )
-def _render_source_detail(
+@app.get("/sources/{source_id}", response_class=HTMLResponse)
-    request: Request,
+async def source_detail(request: Request, source_id: int):
-    source_id: int,
+    """Source detail page with extraction results and accept/reject actions."""
    *,
    edit: bool = False,
    flash: Optional[str] = None,
    error: Optional[str] = None,
    form_overrides: Optional[dict] = None,
 ) -> HTMLResponse:
    """Single render path for /sources/{id} — used by GET + every HTMX
    mutation endpoint so each action gets a consistent re-render.
    `form_overrides` lets a failed edit-save bounce show the user's
    in-progress values instead of the row's stored values.
    """
    db = get_database()
    with db.session() as sess:
        source = sess.query(Source).filter(Source.id == source_id).first()
        if not source:
            raise HTTPException(status_code=404, detail="Source not found")
        source_data = _source_to_dict(source)
-        extraction_data = (
+        extraction_data = _extraction_to_dict(source.extraction) if source.extraction else None
            _extraction_to_dict(source.extraction) if source.extraction else None
        )
    if form_overrides:
        source_data = {**source_data, **form_overrides}
    return templates.TemplateResponse(
        request,
@ -159,25 +137,10 @@ def _render_source_detail(
        {
            "source": source_data,
            "extraction": extraction_data,
            "edit_mode": edit,
            "flash": flash,
            "error": error,
            "domains": DOMAINS,
        },
    )
@app.get("/sources/{source_id}", response_class=HTMLResponse)
 async def source_detail(request: Request, source_id: int, edit: bool = False):
    """Source detail page with extraction results and accept/reject actions.
    `?edit=1` opens the metadata editor inline. Editing exposes domain,
    title, and focus — URL stays read-only since it's the UNIQUE dedupe
    key for the queue and the embeddings identity tuple.
    """
    return _render_source_detail(request, source_id, edit=edit)
 _IN_FLIGHT_STATUSES = [
    SourceStatus.PENDING,
    SourceStatus.PULLED,
@ -229,11 +192,17 @@ async def accept_source(request: Request, source_id: int):
        if not source:
            raise HTTPException(status_code=404, detail="Source not found")
        source.status = SourceStatus.ACCEPTED
        source_data = _source_to_dict(source)
        extraction_data = _extraction_to_dict(source.extraction) if source.extraction else None
-    return _render_source_detail(
+    return templates.TemplateResponse(
        request,
-        source_id,
+        "source_detail.html",
-        flash="Accepted — will be included in next wiki compile.",
+        {
            "source": source_data,
            "extraction": extraction_data,
            "flash": "Accepted — will be included in next wiki compile.",
        },
    )
@ -246,80 +215,19 @@ async def reject_source(request: Request, source_id: int):
        if not source:
            raise HTTPException(status_code=404, detail="Source not found")
        source.status = SourceStatus.FAILED
        source_data = _source_to_dict(source)
        extraction_data = _extraction_to_dict(source.extraction) if source.extraction else None
-    return _render_source_detail(request, source_id, flash="Rejected.")
+    return templates.TemplateResponse(
@app.post("/sources/{source_id}/edit", response_class=HTMLResponse)
 async def edit_source_metadata(
    request: Request,
    source_id: int,
    domain: str = Form(...),
    title: Optional[str] = Form(None),
    focus: Optional[str] = Form(None),
 ):
    """HTMX edit-mode save. Updates domain/title/focus only — URL is
    intentionally not editable (see service-layer comment)."""
    db = get_database()
    try:
        with db.session() as sess:
            update_source_metadata(
                sess,
                source_id=source_id,
                domain=domain,
                title=title,
                focus=focus,
            )
    except LookupError as exc:
        raise HTTPException(status_code=404, detail=str(exc)) from exc
    except ValueError as exc:
        # Bounce back to edit mode with the user's in-progress values so
        # they don't have to retype them.
        return _render_source_detail(
        request,
-            source_id,
+        "source_detail.html",
-            edit=True,
+        {
-            error=str(exc),
+            "source": source_data,
-            form_overrides={
+            "extraction": extraction_data,
-                "domain": domain,
+            "flash": "Rejected.",
                "title": title or "",
                "focus": focus or "",
        },
    )
    return _render_source_detail(request, source_id, flash="Saved.")
@app.post("/sources/{source_id}/retry", response_class=HTMLResponse)
 async def retry_source(
    request: Request,
    source_id: int,
    mode: str = Form(RETRY_FULL),
 ):
    """HTMX retry/re-run. `mode=full` → PENDING (re-pull + re-process).
    `mode=extract` → TRANSCRIBED (re-extract only; requires an existing
    transcript). Always clears the claim pair and error_message."""
    if mode not in RETRY_MODES:
        raise HTTPException(status_code=400, detail=f"unknown mode {mode!r}")
    db = get_database()
    try:
        with db.session() as sess:
            source = reset_source_for_retry(
                sess, source_id=source_id, mode=mode
            )
            new_status = source.status.value
    except LookupError as exc:
        raise HTTPException(status_code=404, detail=str(exc)) from exc
    except ValueError as exc:
        return _render_source_detail(request, source_id, error=str(exc))
    if mode == RETRY_EXTRACT:
        flash = f"Re-queued for extraction (status → {new_status})."
    else:
        flash = f"Re-queued from scratch (status → {new_status})."
    return _render_source_detail(request, source_id, flash=flash)
 # ---------------------------------------------------------------------------
 # JSON API endpoints
@ -378,7 +286,6 @@ def _source_to_dict(s: Source) -> dict:
        "focus": s.focus,
        "ingested_at": s.ingested_at.strftime("%Y-%m-%d %H:%M") if s.ingested_at else "",
        "has_extraction": s.extraction is not None,
        "has_transcript": bool((s.transcript_text or "").strip()),
        "error_message": s.error_message,
    }
--- a/src/second_brain/web/templates/source_detail.html
+++ b/src/second_brain/web/templates/source_detail.html
@ -11,19 +11,9 @@
        </svg>
        Back to sources
    </a>
-    <div class="flex items-start justify-between gap-4">
+    <h2 class="text-2xl font-semibold text-gray-900">{{ source.title }}</h2>
        <div class="min-w-0">
            <h2 class="text-2xl font-semibold text-gray-900 break-words">{{ source.title }}</h2>
    <a href="{{ source.url }}" target="_blank"
       class="text-sm text-indigo-500 hover:text-indigo-700 break-all">{{ source.url }}</a>
        </div>
        {% if not edit_mode %}
        <a href="/sources/{{ source.id }}?edit=1"
           class="shrink-0 px-3 py-1.5 text-sm border border-gray-300 text-gray-700 rounded hover:bg-gray-50 transition-colors">
            Edit
        </a>
        {% endif %}
    </div>
 </div>
 {% if flash %}
@ -31,54 +21,7 @@
    {{ flash }}
 </div>
 {% endif %}
 {% if error %}
 <div class="mb-4 px-4 py-3 bg-red-50 border border-red-200 text-red-800 rounded text-sm">
    {{ error }}
 </div>
 {% endif %}
 {% if edit_mode %}
 <!-- Edit mode: domain / title / focus. URL is intentionally read-only -->
 <form hx-post="/sources/{{ source.id }}/edit"
      hx-target="body"
      hx-swap="innerHTML"
      class="mb-8 bg-white rounded-lg border border-gray-200 p-5 space-y-4">
    <div>
        <label class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">URL <span class="font-normal normal-case text-gray-400">(read-only — dedupe key)</span></label>
        <input type="text" value="{{ source.url }}" disabled
               class="w-full px-3 py-2 border border-gray-200 bg-gray-50 text-gray-500 text-sm rounded">
    </div>
    <div>
        <label for="edit-title" class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">Title</label>
        <input type="text" id="edit-title" name="title" value="{{ source.title or '' }}"
               class="w-full px-3 py-2 border border-gray-300 text-sm rounded focus:outline-none focus:ring-2 focus:ring-indigo-300">
    </div>
    <div>
        <label for="edit-domain" class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">Domain</label>
        <select id="edit-domain" name="domain"
                class="w-full px-3 py-2 border border-gray-300 text-sm rounded focus:outline-none focus:ring-2 focus:ring-indigo-300">
            {% for d in domains %}
            <option value="{{ d }}" {% if d == source.domain %}selected{% endif %}>{{ d }}</option>
            {% endfor %}
        </select>
    </div>
    <div>
        <label for="edit-focus" class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">Focus</label>
        <textarea id="edit-focus" name="focus" rows="3"
                  class="w-full px-3 py-2 border border-gray-300 text-sm rounded focus:outline-none focus:ring-2 focus:ring-indigo-300">{{ source.focus or '' }}</textarea>
    </div>
    <div class="flex gap-2 pt-1">
        <button type="submit"
                class="px-4 py-2 bg-indigo-600 text-white text-sm font-semibold rounded hover:bg-indigo-700 transition-colors">
            Save
        </button>
        <a href="/sources/{{ source.id }}"
           class="px-4 py-2 border border-gray-300 text-gray-700 text-sm font-semibold rounded hover:bg-gray-50 transition-colors">
            Cancel
        </a>
    </div>
 </form>
 {% else %}
 <!-- Meta row -->
 <div class="flex flex-wrap gap-2 mb-6">
    {% set status_colors = {
@ -117,9 +60,9 @@
 </div>
 {% endif %}
-<!-- Actions row: accept/reject (when analyzed) + retry (always) -->
+<!-- Accept / Reject actions (only when analyzed) -->
-<div class="flex flex-wrap gap-3 mb-8">
+{% if source.status == 'analyzed' %}
-    {% if source.status == 'analyzed' %}
+<div class="flex gap-3 mb-8">
    <button hx-post="/sources/{{ source.id }}/accept"
            hx-target="body"
            hx-swap="innerHTML"
@ -132,31 +75,6 @@
            class="px-5 py-2 bg-red-500 text-white text-sm font-semibold rounded hover:bg-red-600 transition-colors">
        Reject
    </button>
    {% endif %}
    <!-- Re-run: full reset → PENDING. Re-pulls and re-processes from scratch. -->
    <button hx-post="/sources/{{ source.id }}/retry"
            hx-vals='{"mode": "full"}'
            hx-target="body"
            hx-swap="innerHTML"
            hx-confirm="Re-run this source from scratch? Status will reset to PENDING and the pipeline will re-pull{% if source.source_type == 'video' %} and re-transcribe (expensive){% endif %}."
            class="px-5 py-2 bg-amber-500 text-white text-sm font-semibold rounded hover:bg-amber-600 transition-colors">
        Re-run from scratch
    </button>
    {% if source.has_transcript %}
    <!-- Light retry: skip the re-pull, just re-run LLM extraction. Works
         for any source with stored body text (videos have transcripts,
         articles have the trafilatura-extracted body). -->
    <button hx-post="/sources/{{ source.id }}/retry"
            hx-vals='{"mode": "extract"}'
            hx-target="body"
            hx-swap="innerHTML"
            hx-confirm="Re-extract only — keep the existing transcript/body and re-run the LLM extraction?"
            class="px-5 py-2 bg-amber-100 text-amber-800 text-sm font-semibold rounded hover:bg-amber-200 border border-amber-300 transition-colors">
        Re-extract only
    </button>
    {% endif %}
 </div>
 {% endif %}
--- a/tests/test_config_overlay.py
+++ b/tests/test_config_overlay.py
@ -1,165 +0,0 @@
 """Tests for the settings.local.toml overlay loader.
 The overlay is the dev-side path for keeping host-specific secrets
 (notably `[database].url`) out of the tracked `settings.toml` without
 forcing every interactive shell to export env vars. The footgun this
 test suite guards against: a one-key local override (e.g. a single
 `[database] url = "..."`) silently wiping every other key in that
 section. The merge is section-level, not shallow.
 """
 from __future__ import annotations
 from pathlib import Path
 import pytest
 # ---------------------------------------------------------------------------
 # Pure-Python: _overlay_sections
 # ---------------------------------------------------------------------------
 def test_overlay_preserves_sibling_keys_in_overlapping_section():
    """The exact regression target — local `url` must not wipe `pool_size`."""
    from second_brain.config import _overlay_sections
    base = {"database": {"url": "", "pool_size": 5}}
    local = {"database": {"url": "postgresql://x"}}
    out = _overlay_sections(base, local)
    assert out["database"]["url"] == "postgresql://x"
    assert out["database"]["pool_size"] == 5
 def test_overlay_section_only_in_base_survives():
    from second_brain.config import _overlay_sections
    base = {"database": {"url": "a"}, "scheduler": {"interval": 30}}
    local = {"database": {"url": "b"}}
    out = _overlay_sections(base, local)
    assert out["scheduler"] == {"interval": 30}
    assert out["database"]["url"] == "b"
 def test_overlay_section_only_in_local_appears():
    from second_brain.config import _overlay_sections
    base = {"database": {"url": "a"}}
    local = {"embeddings": {"ollama_url": "http://x"}}
    out = _overlay_sections(base, local)
    assert out["database"] == {"url": "a"}
    assert out["embeddings"] == {"ollama_url": "http://x"}
 def test_overlay_scalar_in_overlay_replaces_dict_in_base():
    """Non-dict overlay values replace outright — no merge attempt."""
    from second_brain.config import _overlay_sections
    base = {"feature": {"a": 1, "b": 2}}
    local = {"feature": "off"}  # scalar wins outright
    out = _overlay_sections(base, local)
    assert out["feature"] == "off"
 def test_overlay_dict_in_overlay_replaces_scalar_in_base():
    from second_brain.config import _overlay_sections
    base = {"feature": "off"}
    local = {"feature": {"a": 1}}
    out = _overlay_sections(base, local)
    assert out["feature"] == {"a": 1}
 def test_overlay_empty_overlay_is_identity():
    from second_brain.config import _overlay_sections
    base = {"database": {"url": "a", "pool_size": 5}}
    assert _overlay_sections(base, {}) == base
    assert _overlay_sections(base, None) == base
 def test_overlay_only_merges_one_level():
    """Doc: the overlay is section-level. Nested dicts inside a section
    replace, not deep-merge. This pins the documented depth."""
    from second_brain.config import _overlay_sections
    base = {"db": {"url": "a", "pool": {"min": 1, "max": 10}}}
    local = {"db": {"pool": {"max": 20}}}
    out = _overlay_sections(base, local)
    # pool replaced wholesale — `min` does NOT survive. This is deliberate.
    assert out["db"]["pool"] == {"max": 20}
    assert out["db"]["url"] == "a"
 # ---------------------------------------------------------------------------
 # load_config: end-to-end with real TOML files on disk
 # ---------------------------------------------------------------------------
 def _write(path: Path, body: str) -> None:
    path.write_text(body, encoding="utf-8")
@pytest.fixture(autouse=True)
 def _reset_config():
    """Drop the cached singleton before/after each test so overlay state
    doesn't leak between cases."""
    from second_brain.config import reset_config_singleton
    reset_config_singleton()
    yield
    reset_config_singleton()
 def test_load_config_picks_up_sibling_local_overlay(tmp_path):
    from second_brain.config import load_config
    base = tmp_path / "settings.toml"
    local = tmp_path / "settings.local.toml"
    _write(
        base,
        '[database]\nurl = ""\n\n[embeddings]\nollama_url = "http://ollama:11434"\nmodel = "nomic-embed-text"\n',
    )
    _write(
        local,
        '[database]\nurl = "postgresql+psycopg://lovebug:x@127.0.0.1:5433/petalbrain"\n\n[embeddings]\nollama_url = "http://127.0.0.1:11434"\n',
    )
    cfg = load_config(base)
    # url from local wins; default model survives (not in local).
    assert cfg.database["url"].startswith("postgresql+psycopg://")
    assert cfg.embeddings["ollama_url"] == "http://127.0.0.1:11434"
    assert cfg.embeddings["model"] == "nomic-embed-text"
 def test_load_config_no_local_overlay_is_inert(tmp_path):
    from second_brain.config import load_config
    base = tmp_path / "settings.toml"
    _write(base, '[database]\nurl = "from-base"\n')
    cfg = load_config(base)
    assert cfg.database["url"] == "from-base"
 def test_load_config_local_overlay_at_explicit_path(tmp_path):
    """Doc: when an explicit (or SECOND_BRAIN_CONFIG) path is used, the
    overlay sibling is resolved next to THAT path — not next to the
    repo's config dir."""
    from second_brain.config import load_config
    subdir = tmp_path / "etc"
    subdir.mkdir()
    base = subdir / "settings.toml"
    local = subdir / "settings.local.toml"
    _write(base, '[database]\nurl = "base"\n')
    _write(local, '[database]\nurl = "overlay-wins"\n')
    cfg = load_config(base)
    assert cfg.database["url"] == "overlay-wins"
--- a/tests/test_source_edit_retry.py
+++ b/tests/test_source_edit_retry.py
@ -1,258 +0,0 @@
 """Tests for the source edit-metadata + retry-reset service helpers.
 Live-DB pattern matches `tests/test_claim.py` / `tests/test_playlist.py`:
 both groups auto-skip when no petalbrain connection is configured. The
 service layer is where edit/retry logic lives (the web routes + CLI are
 thin wrappers around it), so this is where the meaningful coverage is.
 """
 from __future__ import annotations
 import os
 import uuid
 import psycopg
 import pytest
 def _db_reachable() -> bool:
    url = os.environ.get("SECOND_BRAIN_DATABASE_URL") or os.environ.get(
        "HERBYLAB_DATABASE_URL"
    )
    if not url:
        return False
    raw = url.replace("postgresql+psycopg://", "postgresql://", 1)
    try:
        with psycopg.connect(raw, connect_timeout=2):
            return True
    except Exception:
        return False
 pytestmark = pytest.mark.skipif(
    not _db_reachable(),
    reason="needs SECOND_BRAIN_DATABASE_URL pointed at a real petalbrain",
 )
 def _seed_source(db, **overrides):
    """Insert a Source row and return (id, url). Caller cleans up by id."""
    from second_brain.models import Source, SourceStatus, SourceType, utcnow
    tag = uuid.uuid4().hex[:8]
    defaults = dict(
        url=f"https://example.test/edit-retry/{tag}",
        title=f"seed-{tag}",
        domain="development",
        source_type=SourceType.ARTICLE,
        status=SourceStatus.PENDING,
        ingested_at=utcnow(),
    )
    defaults.update(overrides)
    with db.session() as sess:
        src = Source(**defaults)
        sess.add(src)
        sess.flush()
        return src.id, src.url
 def _cleanup(db, source_id):
    from second_brain.models import Source
    with db.session() as sess:
        sess.query(Source).filter(Source.id == source_id).delete(
            synchronize_session=False
        )
 # ---------------------------------------------------------------------------
 # update_source_metadata
 # ---------------------------------------------------------------------------
 def test_update_metadata_writes_domain_title_focus():
    from second_brain.database import Database, reset_database_singleton
    from second_brain.models import Source
    from second_brain.sources_service import update_source_metadata
    reset_database_singleton()
    db = Database()
    sid, _ = _seed_source(db)
    try:
        with db.session() as sess:
            update_source_metadata(
                sess,
                source_id=sid,
                domain="homelab",
                title="new title",
                focus="new focus directive",
            )
        with db.session() as sess:
            row = sess.query(Source).filter(Source.id == sid).first()
            assert row.domain == "homelab"
            assert row.title == "new title"
            assert row.focus == "new focus directive"
            assert row.updated_at is not None
    finally:
        _cleanup(db, sid)
        reset_database_singleton()
 def test_update_metadata_blank_strings_become_null():
    from second_brain.database import Database, reset_database_singleton
    from second_brain.models import Source
    from second_brain.sources_service import update_source_metadata
    reset_database_singleton()
    db = Database()
    sid, _ = _seed_source(db)
    try:
        with db.session() as sess:
            update_source_metadata(
                sess, source_id=sid, domain="development", title="   ", focus=""
            )
        with db.session() as sess:
            row = sess.query(Source).filter(Source.id == sid).first()
            assert row.title is None
            assert row.focus is None
    finally:
        _cleanup(db, sid)
        reset_database_singleton()
 def test_update_metadata_rejects_unknown_domain():
    from second_brain.database import Database, reset_database_singleton
    from second_brain.sources_service import update_source_metadata
    reset_database_singleton()
    db = Database()
    sid, _ = _seed_source(db)
    try:
        with db.session() as sess:
            with pytest.raises(ValueError, match="unknown domain"):
                update_source_metadata(sess, source_id=sid, domain="not-a-domain")
    finally:
        _cleanup(db, sid)
        reset_database_singleton()
 def test_update_metadata_missing_source_raises_lookup():
    from second_brain.database import Database, reset_database_singleton
    from second_brain.sources_service import update_source_metadata
    reset_database_singleton()
    db = Database()
    try:
        with db.session() as sess:
            with pytest.raises(LookupError):
                update_source_metadata(
                    sess, source_id=-99999, domain="development"
                )
    finally:
        reset_database_singleton()
 # ---------------------------------------------------------------------------
 # reset_source_for_retry
 # ---------------------------------------------------------------------------
 def test_retry_full_resets_to_pending_and_clears_claim():
    from second_brain.database import Database, reset_database_singleton
    from second_brain.models import Source, SourceStatus, utcnow
    from second_brain.sources_service import reset_source_for_retry
    reset_database_singleton()
    db = Database()
    sid, _ = _seed_source(
        db,
        status=SourceStatus.FAILED,
        error_message="boom",
        claimed_by="worker-x",
        claimed_at=utcnow(),
    )
    try:
        with db.session() as sess:
            reset_source_for_retry(sess, source_id=sid, mode="full")
        with db.session() as sess:
            row = sess.query(Source).filter(Source.id == sid).first()
            assert row.status == SourceStatus.PENDING
            assert row.claimed_by is None
            assert row.claimed_at is None
            assert row.error_message is None
    finally:
        _cleanup(db, sid)
        reset_database_singleton()
 def test_retry_extract_sets_transcribed_when_transcript_present():
    from second_brain.database import Database, reset_database_singleton
    from second_brain.models import Source, SourceStatus
    from second_brain.sources_service import reset_source_for_retry
    reset_database_singleton()
    db = Database()
    sid, _ = _seed_source(
        db,
        status=SourceStatus.ANALYZED,
        transcript_text="word " * 50,
        error_message="prior failure",
    )
    try:
        with db.session() as sess:
            reset_source_for_retry(sess, source_id=sid, mode="extract")
        with db.session() as sess:
            row = sess.query(Source).filter(Source.id == sid).first()
            assert row.status == SourceStatus.TRANSCRIBED
            assert row.error_message is None
            assert row.transcript_text  # transcript preserved
    finally:
        _cleanup(db, sid)
        reset_database_singleton()
 def test_retry_extract_without_transcript_raises():
    from second_brain.database import Database, reset_database_singleton
    from second_brain.models import SourceStatus
    from second_brain.sources_service import reset_source_for_retry
    reset_database_singleton()
    db = Database()
    sid, _ = _seed_source(db, status=SourceStatus.PENDING, transcript_text=None)
    try:
        with db.session() as sess:
            with pytest.raises(ValueError, match="requires an existing transcript"):
                reset_source_for_retry(sess, source_id=sid, mode="extract")
    finally:
        _cleanup(db, sid)
        reset_database_singleton()
 def test_retry_unknown_mode_raises():
    from second_brain.database import Database, reset_database_singleton
    from second_brain.sources_service import reset_source_for_retry
    reset_database_singleton()
    db = Database()
    sid, _ = _seed_source(db)
    try:
        with db.session() as sess:
            with pytest.raises(ValueError, match="unknown retry mode"):
                reset_source_for_retry(sess, source_id=sid, mode="bogus")
    finally:
        _cleanup(db, sid)
        reset_database_singleton()
 def test_retry_missing_source_raises_lookup():
    from second_brain.database import Database, reset_database_singleton
    from second_brain.sources_service import reset_source_for_retry
    reset_database_singleton()
    db = Database()
    try:
        with db.session() as sess:
            with pytest.raises(LookupError):
                reset_source_for_retry(sess, source_id=-99999, mode="full")
    finally:
        reset_database_singleton()