Compare commits

..

No commits in common. "main" and "claude/laughing-shirley-52f4a6" have entirely different histories.

11 changed files with 125 additions and 1137 deletions

353
CLAUDE.md
View File

@ -1,297 +1,122 @@
# second-brain
Personal knowledge pipeline. Two machines coordinate purely through the
shared Postgres queue: the **dev side** (herbys-dev) does article pull
+ all extraction + embedding, and the **tower** (EndeavourOS, RTX 3080)
does video pull + transcription on the GPU. A FastAPI/Jinja/HTMX web
app fronts the queue, settings, and add-to-queue form; the file-based
Obsidian wiki is downstream of `accepted` extractions.
A personal knowledge system: video/article extraction pipeline + LLM-maintained wiki.
## Two-machine pipeline
## Architecture
Three-layer system:
1. **Wiki** — long-term structured memory. Markdown files in an Obsidian vault, maintained by the wiki compiler.
2. **Context assembler** — builds the prompt bundle for each extraction. Phase 1 is thin: domain template + focus field + transcript.
3. **Extractor** — stateless single-shot LLM call per source. Pure function: bundle in, structured extraction out.
## Pipeline stages
```
add (CLI or POST /sources/add — single video, article,
or YouTube playlist URL → fan-out into N rows)
┌────────── second_brain.sources (the queue) ──────────┐
│ │
│ PENDING ── (article)──► dev `second-brain process`
│ trafilatura pull │
│ PENDING ── (video) ──► tower transcribe-worker │
│ yt-dlp pull → PULLED │
│ PULLED ── (video) ──► tower transcribe-worker │
│ faster-whisper │
│ both paths land at ──► TRANSCRIBED │
│ │
│ TRANSCRIBED ─────────► dev `second-brain schedule`
│ Claude CLI extract + │
│ Ollama embed │
│ ──► ANALYZED ──► accept in web UI │
│ ──► ACCEPTED ──► `compile` writes vault│
│ ──► PUBLISHED │
│ │
│ FAILED is the terminal error state │
└───────────────────────────────────────────────────────┘
pending → pulled → transcribed → analyzed → accepted → published
```
**Source claiming** — `sources.claimed_by` / `claimed_at` columns +
`SELECT ... FOR UPDATE SKIP LOCKED` keep the dev and tower workers from
grabbing the same row. Both stages filter by `source_type`:
## Project layout
- tower transcribe-worker: `status IN (PENDING, PULLED) AND source_type = 'video'`
- dev `process` (article pull): `status = PENDING AND source_type = 'article'`
- dev `schedule` (extract+embed): `status = TRANSCRIBED` (any type)
```
src/second_brain/
├── adapters/ # Pull adapters (youtube.py, article.py)
├── context/ # Context assembler
├── embeddings/ # Chunk + nomic-embed-text + upsert into public.embeddings
├── extractor/ # LLM extraction engine + Pydantic schema
├── llm/ # claude_cli.py — thin subprocess wrapper around `claude -p`
├── compiler/ # Wiki compiler (reads vault, writes pages, git commits)
├── scheduler/ # Rate-limit smoother (call-count for CLI, token-budget for API)
└── web/ # FastAPI + Jinja2 + HTMX UI
A 30-minute stale-claim reaper runs at worker startup so a crashed
machine doesn't dam the queue.
alembic/ # Schema migrations for the second_brain Postgres schema
```
## Setup
```bash
cp config/settings.toml.example config/settings.toml
# Edit settings.toml — at minimum, point the paths and set SECOND_BRAIN_DATABASE_URL.
# Connection string format. Containerised (default):
# postgresql+psycopg://lovebug:<password>@homelab-postgres:5432/petalbrain
# Host-side dev runs:
# postgresql+psycopg://lovebug:<password>@127.0.0.1:5433/petalbrain
export SECOND_BRAIN_DATABASE_URL=...
# One-time, before first migration: bootstrap the schema as postgres
# superuser. lovebug doesn't have CREATE on petalbrain, so this can't be
# done from the app side — same pattern as wiki/trellis/ob1.
# CREATE SCHEMA second_brain AUTHORIZATION lovebug;
# GRANT ALL PRIVILEGES ON SCHEMA second_brain TO lovebug;
uv sync
uv run alembic upgrade head
uv run second-brain serve
```
## CLI commands
- `second-brain add <url>` — queue a source. YouTube `playlist?list=…`
URLs auto-expand into one row per video (cap 50, the
`watch?v=…&list=…` form stays a single video — see
`is_youtube_playlist_url` for the rule).
- `second-brain process` — dev-side: pull pending **articles**, extract
+ embed any TRANSCRIBED source. Skips videos (tower's job).
- `second-brain schedule` — dev-side overnight extract+embed loop,
honouring `pipeline_settings.extraction_enabled` / active-hours /
max-items.
- `second-brain transcribe-worker` — tower-side daemon. `--once` for
ad-hoc runs. Honours `pipeline_settings.transcription_*` knobs.
- `second-brain serve` — local dev launch of the web UI (port 8000).
Production runs as a container (see `deploy/web/`).
- `second-brain compile` — wiki compiler: ACCEPTED → markdown in
`vault_path`, git-committed. File-based Obsidian vault only; we do
**not** dual-write into petalbrain.wiki.
## Storage
**Postgres**: `petalbrain` DB on `homelab-postgres` (pgvector/pg17, docker
container). Connection role: `lovebug`. Dev reaches it at
`127.0.0.1:5433`; tower reaches it over WireGuard at
`db.wg.herbylab.dev:5432`; the web container reaches it at
`homelab-postgres:5432` on the `homelab` docker bridge. **Keep both port
binds** — the host-side 5433 publish and the docker-net 5432 service.
**Schema `second_brain`** (Alembic-managed in `alembic/`):
- `sources` — the queue. Int SERIAL PK. Naive UTC `timestamp`
(`without time zone`) — never `timestamptz`. `transcript_text` is the
joined full text the extractor reads; `transcript_segments` is JSONB
with the faster-whisper per-segment / per-word breakdown.
`claimed_by` / `claimed_at` for the queue dance.
- `extractions` — one row per source. JSON columns for the structured
output (summary, key_points, entities, claims, …).
- `pipeline_settings`**single-row** config (PK pinned to id=1 with a
CHECK constraint). What the dashboard edits and workers read each
iteration: transcription_enabled / active_hours / max_gpu_jobs /
max_video_length / max_items_per_run plus the equivalent extraction_*
fields. DB values override settings.toml.
- `wiki_pages` — bookkeeping for the file-based wiki compiler.
**Embeddings** live in `public.embeddings` — the **shared** pgvector
table owned by `postgres` and consumed by vault-mcp / ob1 / second-brain.
We do not manage this in Alembic. Rows are keyed by
`(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`,
with `delete-before-insert` semantics on the 4-tuple so a re-embed
never leaves orphans. We embed extraction **summaries only** for now —
not key_points, not transcript chunks.
## Web app
FastAPI + Jinja2 + HTMX, in `src/second_brain/web/`. Production lives in
a slim Python 3.12 container (see `deploy/web/`) on the `homelab` docker
network, reaching Postgres + Ollama by service DNS. Published at
`10.0.21.207:8080` and fronted by the Traefik VM (10.0.11.20,
file-provider — **no docker-label routing**) as
`https://brain.herbylab.dev`.
Routes:
- `/` — sources list (filterable by domain + status)
- `/queue` — in-flight sources (PENDING / PULLED / TRANSCRIBED) plus the
add-to-queue form
- `/dashboard` — status counts, recent activity, settings snapshot, plus
the add-to-queue form
- `/sources/{id}` — extraction detail + accept/reject HTMX actions
- `/settings` — full HTMX editor for `pipeline_settings`
- `POST /sources/add` — shared endpoint. Detects YouTube playlist URLs
and fans out via yt-dlp `extract_flat`. Reads `HX-Current-URL` to
dispatch the response partial (`_queue_body.html` vs
`_dashboard_body.html`) so each page refreshes its own region inline.
**No auth.** UI mutates state (accept/reject sources, edit settings) but
has no SSO / CSRF / rate-limiting. The Traefik route is currently
LAN-only by Travis's call — see `deploy/web/README.md` §SECURITY.
## Transcription
`src/second_brain/transcribe.py` wraps **faster-whisper** (CTranslate2
backend). Defaults: model `large-v3`, device `cuda`, compute_type
`float16`. CPU fallback drops to `int8`. `word_timestamps=True` is
always passed; the segment dicts persisted to
`sources.transcript_segments` carry per-word `{start, end, word,
probability}`.
The tower worker (`src/second_brain/scheduler/transcribe_worker.py`)
runs as a systemd service — see `deploy/tower/README.md` for the full
install runbook. ExecStart goes through
`deploy/tower/run-transcribe-worker.sh` which prepends the venv's
CUDA-12 lib dirs to `LD_LIBRARY_PATH` (see gotcha (b) below).
- `second-brain add <url>` — queue a source URL (video or article)
- `second-brain process` — run the pipeline on pending items
- `second-brain serve` — launch the web UI (default port 8000)
- `second-brain compile` — run the wiki compiler
- `second-brain schedule` — start the overnight scheduler
## Extractor backends
`[extractor].backend` in `config/settings.toml`:
`[extractor].backend` in `config/settings.toml` picks the LLM transport.
- `cli` (default) — `src/second_brain/llm/claude_cli.py` shells out to
`claude -p --output-format json --model … --json-schema …`. Runs
under the Max OAuth subscription. Hermetic spawn: throwaway
`tempfile.TemporaryDirectory`, `ANTHROPIC_API_KEY` / `_AUTH_TOKEN`
stripped from env so it uses OAuth, and `--no-session-persistence
--disable-slash-commands --tools "" --setting-sources ""` so host
CLAUDE.md / hooks / agents don't leak in.
- `api``anthropic` SDK, lazy-imported. Requires `ANTHROPIC_API_KEY`.
Install with `uv sync --extra api`.
- **`cli`** (default) — `src/second_brain/llm/claude_cli.py` shells out to `claude -p --output-format json --model <model> --json-schema <schema>`. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside `tempfile.TemporaryDirectory(prefix="sb-claude-cli-")`, with `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` stripped from env so the CLI uses OAuth instead of an API key, and with `--no-session-persistence --disable-slash-commands --tools "" --setting-sources ""` so host CLAUDE.md / hooks / agents don't leak in.
- **`api`** — uses the `anthropic` SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (`uv sync --extra api`).
Schema-validated CLI output lands at `payload["structured_output"]`, not
`payload["result"]`. The wrapper checks both.
**Schema-validated output gotcha:** when `--json-schema` is used, the parsed object lands at `payload["structured_output"]`, not `payload["result"]`. The wrapper checks both. The engine prefers the structured payload via `LLMExtraction.model_validate(structured)` and falls back to parsing the text field.
## Environment variables
- `SECOND_BRAIN_DATABASE_URL` — Postgres DSN. Falls back to
`HERBYLAB_DATABASE_URL`. Use `postgresql+psycopg://` for SQLAlchemy;
the embeddings + claim modules strip the `+psycopg` prefix for raw
psycopg.
- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default
`http://ollama:11434` (homelab docker net); host-side runs use
`http://127.0.0.1:11434`.
- `EMBEDDING_MODEL` — default `nomic-embed-text` (768-d).
- `WHISPER_MODEL` / `WHISPER_DEVICE` / `WHISPER_COMPUTE_TYPE` — override
the `[whisper]` block on the tower.
- `ANTHROPIC_API_KEY` — only for `extractor.backend = "api"`.
- `SECOND_BRAIN_CONFIG` — settings file path override.
- `SECOND_BRAIN_WORKER_ID` — stamped into `sources.claimed_by`.
- `SECOND_BRAIN_DB_POOL_MIN` / `_MAX` — SQLAlchemy pool sizing.
- `SECOND_BRAIN_LIVE_NETWORK_TESTS=1` — opt-in for the yt-dlp playlist
test that hits real YouTube.
- `ANTHROPIC_API_KEY` — only required when `extractor.backend = "api"`. The default `cli` backend uses your Max OAuth and does not need it.
- `SECOND_BRAIN_CONFIG` — optional override for the settings file path.
- `SECOND_BRAIN_DATABASE_URL` — Postgres connection string. Falls back to `HERBYLAB_DATABASE_URL` if unset. Required at runtime; both `second_brain.database` and `second_brain.embeddings` read it. Use the SQLAlchemy `postgresql+psycopg://` form; the embeddings module strips the `+psycopg` prefix internally for raw psycopg.
- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default `http://ollama:11434` (homelab docker network); for host-side runs use `http://127.0.0.1:11434`.
- `EMBEDDING_MODEL` — overrides `[embeddings].model`. Default `nomic-embed-text` (768-d).
- `SECOND_BRAIN_DB_POOL_MIN` / `SECOND_BRAIN_DB_POOL_MAX` — SQLAlchemy pool sizing knobs (defaults 1 / 9 overflow).
## Domains
`development` / `content` / `business` / `homelab` — drives the
per-domain prompt template loaded by the context assembler.
- `development` — software, programming, systems
- `content` — content creation strategy, video production
- `business` — entrepreneurship, marketing, operations
- `homelab` — self-hosted infrastructure, networking, DevOps
## Key files
- `src/second_brain/models.py` — all ORM in the `second_brain` schema.
- `src/second_brain/sources_service.py``add_source`, `add_playlist`,
`is_youtube_playlist_url`, `expand_youtube_playlist`. CLI + web both
call into here so there's no behavior drift between entry points.
- `src/second_brain/claim.py` — the SKIP LOCKED dance + stale reaper.
- `src/second_brain/settings_store.py` — single-row `pipeline_settings`
get/update + the active-window math the workers share.
- `src/second_brain/transcribe.py` — faster-whisper wrapper + segment
serialiser.
- `src/second_brain/scheduler/runner.py` — dev extract scheduler.
- `src/second_brain/scheduler/transcribe_worker.py` — tower poll loop.
- `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert
into `public.embeddings`; best-effort with graceful degradation.
- `alembic/` + `alembic.ini` — schema migrations for the `second_brain`
schema only. `public.embeddings` is **not** ours to migrate.
- `deploy/web/` — Dockerfile + compose.yml + README for the web
container. Travis applies the Knot DNS A-record + Traefik
file-provider router from the README §2.
- `deploy/tower/` — systemd unit + `run-transcribe-worker.sh` wrapper +
env template + EndeavourOS install runbook.
- `prompts/<domain>.md` — extraction prompt templates.
- `config/settings.toml``[database]`, `[extractor]`, `[embeddings]`,
`[scheduler]`, `[whisper]`.
- `config/settings.toml` — vault path, `[database]`, `[extractor]`, `[embeddings]`, `[scheduler]`
- `prompts/<domain>.md` — per-domain extraction prompt templates
- `src/second_brain/extractor/schema.py``LLMExtraction` (LLM-only fields, drives the `--json-schema` contract) + `ExtractionResult` (LLMExtraction + SourceMeta filled from the DB)
- `src/second_brain/llm/claude_cli.py` — CLI subprocess wrapper
- `src/second_brain/models.py` — SQLAlchemy models (all tables live in the `second_brain` schema) + `utcnow()` helper
- `src/second_brain/database.py` — Postgres engine + URL resolution
- `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert into `public.embeddings`; best-effort with graceful degradation
- `alembic/` + `alembic.ini` — schema migrations for the `second_brain` schema (Alembic does **not** manage `public.embeddings`)
- `postgres-migration-planning.md` — original planning doc; superseded by this file once the migration shipped
## Load-bearing gotchas
## Conventions / gotchas
These have all bitten us; don't relearn them.
- **Naive UTC timestamps everywhere.** Use `from second_brain.models import utcnow` rather than `datetime.utcnow()` (deprecated in 3.12) or `datetime.now()` (local time). Postgres columns are `timestamp without time zone` — do not switch to `timestamptz`.
- **PK style.** Int autoincrement (`SERIAL`) — locked for compatibility with the existing schema shape. Do not switch to `BIGINT GENERATED ALWAYS AS IDENTITY` or UUID.
- **Re-query inside the session.** SQLAlchemy ORM attribute access after the session closes raises `DetachedInstanceError`. The `process` command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates *inside* the `with db.session()` block.
- **Starlette `TemplateResponse` signature.** Starlette ≥ 0.36 requires `templates.TemplateResponse(request, name, context)`. The old `(name, context)` form raises `TypeError: unhashable type: 'dict'`. Do not put `"request": request` into the context dict — it's auto-injected now.
- **Vault git repo auto-init.** The wiki compiler calls `_ensure_git_repo()` on every run, which `git init`s the vault and creates an empty seed commit if no `.git` is present. `_git_commit()` also checks `git diff --cached --quiet` before committing to avoid empty commits.
- **Scheduler throttling shape.** The CLI backend is throttled by `max_calls_per_hour` (Max subscription limits are call-shaped, not token-shaped). The API backend uses `max_tokens_per_hour`. Both share `min_gap_seconds`. The scheduler targets sources in `TRANSCRIBED` state — earlier code mistakenly filtered on `ANALYZED`, making it a no-op.
- **Alembic + lovebug privilege gap.** `lovebug` owns the `second_brain` schema but does **not** have `CREATE` on the `petalbrain` database. `alembic/env.py` therefore guards `CREATE SCHEMA` behind a `pg_namespace` lookup, and the v1 migration intentionally drops the `IF NOT EXISTS` form (a bare `CREATE SCHEMA IF NOT EXISTS` raises permission denied even when the schema is already present). The schema is bootstrapped once, out-of-band, by `postgres` superuser.
- **public.embeddings is shared, not ours.** Alembic only manages the `second_brain` schema. The vector table itself is owned by `postgres` and shared across vault-mcp / ob1 / second-brain — we just INSERT into it. Rows are keyed by `(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`.
- **Embedding is best-effort.** The pipeline writes the relational `extractions` row first and `sess.flush()`es; only then calls `embed_extraction()`. Any failure (no DB URL, no Ollama, embedding 500) logs at WARNING and returns None — the relational row and file vault stay canonical.
- **File-based wiki, not petalbrain.wiki.** The wiki compiler still writes markdown into the configured `vault_path` and commits via git. We do **not** dual-write into `petalbrain.wiki` — that's vault-mcp's territory and intentionally out of scope.
- **(a) Schema bootstrap requires postgres superuser.** `lovebug` does
NOT have `CREATE` on the `petalbrain` database, even though it owns
the `second_brain` schema. The first migration needs a one-time
`CREATE SCHEMA second_brain AUTHORIZATION lovebug; GRANT ALL PRIVILEGES
ON SCHEMA second_brain TO lovebug;` as `postgres`. `alembic/env.py`
guards `CREATE SCHEMA` behind a `pg_namespace` lookup so subsequent
runs as `lovebug` don't trip the permission check.
## In-flight work
- **(b) Arch ships CUDA 13, ctranslate2 needs CUDA 12.** EndeavourOS's
`pacman -S cuda cudnn` installs `libcublas.so.13` / `libcudnn.so.10`,
which `ctranslate2` 4.7.x cannot load — it wants the .12 / .9 ABI.
We pin `nvidia-cublas-cu12` and `nvidia-cudnn-cu12>=9,<10` in the
`tower` optional-dependency group, and the systemd ExecStart calls
`deploy/tower/run-transcribe-worker.sh` which resolves the venv's
wheel-bundled `.so` directories at runtime and prepends them to
`LD_LIBRARY_PATH`. The wrapper uses the namespace packages'
`__path__` (not `__file__`, which is `None` for PEP 420 packages).
systemd does **not** inherit shell exports.
- **(c) pg_hba needs BOTH subnets.** Two host rules are load-bearing:
```
host all all 172.19.0.0/16 scram-sha-256 # homelab docker bridge
host all all 10.99.0.0/24 scram-sha-256 # WireGuard (tower)
```
Dropping the 172.19/16 rule locks out the web container + vault-mcp +
ob1-* + trellis-mcp (all on the homelab net). Dropping the 10.99/24
rule locks out the tower. Both have been removed once each during
unrelated postgres edits — the symptom is `FATAL: no pg_hba.conf
entry for host "…"` in service logs.
- **(d) Two postgres port binds, both load-bearing.** `homelab-postgres`
publishes `127.0.0.1:5433 → 5432` for dev-host access AND is reachable
on `homelab-postgres:5432` over the docker net. Tower hits it over WG
at `db.wg.herbylab.dev:5432`. Don't consolidate; each path serves a
different consumer set.
- **(e) Traefik is file-provider on a separate VM (10.0.11.20).** It
cannot read this host's docker labels. The web `compose.yml` therefore
publishes back to a fixed host port (`10.0.21.207:8080`) that Traefik
reaches over LAN, and intentionally carries NO `traefik.*` labels.
Router config lives in Traefik's dynamic-config YAML — see
`deploy/web/README.md` §2b for the snippet.
- **Code-level conventions, since they bit us early:**
- **Naive UTC timestamps.** Use `from second_brain.models import utcnow`
— never `datetime.utcnow()` (deprecated) or `datetime.now()` (local).
Postgres columns are `timestamp without time zone`.
- **Int autoincrement PKs.** Don't migrate to bigint-identity or UUID;
the existing schema shape is locked.
- **Re-query inside the session.** SQLAlchemy ORM attribute access
after the session closes raises `DetachedInstanceError`. Web routes
and CLI loops collect IDs first, then re-fetch inside the session
block; dict snapshots for templates are built inside `with
db.session()`.
- **Starlette TemplateResponse signature.** `templates.TemplateResponse(request, name, context)` — the old `(name, context)` form raises
`TypeError: unhashable type: 'dict'`. Do not put `"request":
request` into the context dict.
- **Embedding is best-effort.** The extract path flushes the
relational `extractions` row first, then calls `embed_extraction()`.
DB / Ollama failures log and return None; pipeline state stays
canonical in the relational + file vault.
- **Worker LD_LIBRARY_PATH.** See (b). When pre-warming or debugging
faster-whisper manually, run via `deploy/tower/run-transcribe-worker.sh`
or replicate the path probe in your shell.
## In-flight / parked
- **Vault location** — recommended `/opt/projects/second-brain-vault/`
as a peer dir to the project. Not yet confirmed.
- **Media + subtitles on NAS** — currently land on the tower's local
SSD. Move to NAS once we have a stable mount path.
- **SSO** — Authentik forward-auth middleware in Traefik is the
obvious next step before opening the UI past LAN.
- **Dev-side extraction systemd timer** — currently invoked manually;
no systemd unit yet by Travis's call.
- **Vault location.** Recommended `/opt/projects/second-brain-vault/` as the production location (peer to the project dir). Not yet confirmed.
## Validation
Per the user's global CLAUDE.md, run `zero-check` after any code
change. The pipeline's at `~/zero-check-pipeline/validate/run-all.sh
"$(pwd)"`. Run `a-review` after non-trivial diffs (>50 lines):
`python3 ~/.claude/skills/a-review/run_review.py`. Test suite is `uv
run pytest`; live DB + Ollama paths auto-skip when their endpoints
aren't configured.
Per Travis's global CLAUDE.md, run `zero-check` after any code change, and `a-review` after non-trivial diffs (> 50 lines). `a-review` script: `python3 ~/.claude/skills/a-review/run_review.py`, output goes to `reviews/a-review-<date>.md`.

View File

@ -50,9 +50,9 @@ your CLI PATH.
```bash
sudo mkdir -p /opt/projects/homelab
sudo chown trucktrav:trucktrav /opt/projects/homelab
sudo chown herbyadmin:herbyadmin /opt/projects/homelab
cd /opt/projects/homelab
git clone gitea-pbs-trucktrav:petal-power/second-brain.git
git clone gitea-lovebug:petal-power/second-brain.git
cd second-brain
# Install the runtime deps + the `tower` extra (faster-whisper / CTranslate2).
@ -90,7 +90,7 @@ The worker assumes the tunnel is already configured and that
| Peer | WG IP |
|------|-------|
| hub (herbydev) | `10.99.0.1` |
| tower | `10.99.0.3` |
| tower | `10.99.0.2` |
`db.wg.herbylab.dev` resolves to `10.99.0.1`. Confirm connectivity from
the tower:

View File

@ -16,8 +16,8 @@ StartLimitBurst=5
[Service]
Type=simple
User=trucktrav
Group=trucktrav
User=herbyadmin
Group=herbyadmin
# Adjust to wherever you `git clone`d the repo on the tower.
WorkingDirectory=/opt/projects/homelab/second-brain

View File

@ -1,37 +0,0 @@
# a-review — 2026-05-25
**Model:** gemini
**Diff base:** `HEAD~1` (5 files changed, 615 insertions(+), 30 deletions(-))
**Session reference:** none
**Context:** CLAUDE.md + session notes + expanded diff
---
Excellent work. The changes are well-structured, follow existing project conventions, and include a comprehensive test suite. This is a high-quality contribution. I have only one minor finding and one informational note.
### [WARNING] Logic Error
* **File:** `src/second_brain/web/templates/source_detail.html`
* **Location:** Line 151
* **Issue:** The "Re-extract only" button is conditionally rendered with `{% if source.has_transcript and source.source_type == 'video' %}`. This is too restrictive. The backend service function `reset_source_for_retry` correctly allows an "extract" retry for any source that has content in `transcript_text`, which includes processed articles. Limiting this feature to only videos in the UI prevents a valid and useful action for non-video sources.
* **Suggestion:** Change the conditional to `{% if source.has_transcript %}`. This will correctly show the button for any source (article or video) that has been pulled and has content ready for extraction, which matches the backend's capability.
```html
<!-- Change this -->
{% if source.has_transcript and source.source_type == 'video' %}
<!-- To this -->
{% if source.has_transcript %}
```
### [INFO] Security
* **Files:** `src/second_brain/web/app.py`, `src/second_brain/web/templates/source_detail.html`
* **Observation:** The changes introduce new endpoints (`/edit`, `/retry`) that modify database state. According to `CLAUDE.md`, the application has "No auth" and relies on being deployed in a trusted, LAN-only environment. These new endpoints increase the number of ways an unauthorized user on the network could alter data.
* **Context:** This is not a new vulnerability, but an expansion of the existing trust model. As long as the application remains on a trusted network, this is acceptable per the project's documented security posture. It's worth keeping in mind that no CSRF protection is being used for these HTMX `POST` actions.
### Summary
The overall code quality is very high. The changes are thoughtfully implemented, with a clean separation between the CLI, the web layer, and a shared service layer. The refactoring in `web/app.py` to create `_render_source_detail` is a good move for maintainability. The addition of a thorough, live-DB test suite for the new service functions is commendable and demonstrates a commitment to quality.
Once the minor template logic issue is resolved, this change will be in perfect shape.

View File

@ -85,28 +85,6 @@ def _merge(default: dict, override: dict) -> dict:
return out
def _overlay_sections(base: dict, overlay: dict) -> dict:
"""Section-level overlay: one level deep.
For each key in `overlay`, if both `base[key]` and `overlay[key]` are
dicts, merge their keys (overlay wins). Otherwise the overlay value
replaces the base value outright. Sections present in only one of
the two pass through untouched.
Use this not `_merge` when combining `settings.toml` with
`settings.local.toml`, so a local `[database] url = "..."` doesn't
silently wipe other `[database]` keys defined in the base file.
"""
out = dict(base)
for key, ov_val in (overlay or {}).items():
base_val = out.get(key)
if isinstance(base_val, dict) and isinstance(ov_val, dict):
out[key] = {**base_val, **ov_val}
else:
out[key] = ov_val
return out
class Config:
"""Holds resolved configuration values."""
@ -197,24 +175,7 @@ _config_instance: Config | None = None
def load_config(path: Path | None = None) -> Config:
"""Load (and cache) configuration from a TOML file.
After loading the base file, look for a sibling `settings.local.toml`
and overlay it (section-level merge see `_overlay_sections`). The
local file is gitignored and intended for host-specific secrets
(e.g. `[database].url` for a dev shell) without forking the tracked
base.
Path resolution:
- explicit `path=` arg wins
- else `SECOND_BRAIN_CONFIG` env var
- else walks up from CWD for `config/settings.toml` or `settings.toml`
The local-overlay sibling is looked up next to whichever base file
was chosen, so `SECOND_BRAIN_CONFIG=/etc/sb/settings.toml` will also
pick up `/etc/sb/settings.local.toml` if present. If no base file is
found at all, no overlay is attempted.
"""
"""Load (and cache) configuration from a TOML file."""
global _config_instance
if _config_instance is not None:
return _config_instance
@ -236,15 +197,6 @@ def load_config(path: Path | None = None) -> Config:
with open(path, "rb") as fh:
raw = tomllib.load(fh)
# Section-level overlay from a sibling settings.local.toml.
# Filename is the one already covered in .gitignore — host
# secrets live here and never get committed.
local_path = path.parent / "settings.local.toml"
if local_path.exists():
with open(local_path, "rb") as fh:
local_raw = tomllib.load(fh)
raw = _overlay_sections(raw, local_raw)
_config_instance = Config(raw)
return _config_instance

View File

@ -4,7 +4,6 @@ second-brain CLI entry point.
Commands:
add queue a source URL (video or article)
process run the full pipeline on pending/pulled/transcribed sources
retry reset a source so the pipeline re-processes it
serve launch the FastAPI web UI
compile run the wiki compiler on accepted sources
schedule start the overnight scheduler
@ -308,47 +307,6 @@ def serve(host: str, port: int, reload: bool) -> None:
)
# ---------------------------------------------------------------------------
# retry
# ---------------------------------------------------------------------------
@cli.command()
@click.argument("source_id", type=int)
@click.option(
"--mode",
type=click.Choice(["full", "extract"]),
default="full",
show_default=True,
help="`full` resets to PENDING (re-pull + re-process); "
"`extract` resets to TRANSCRIBED (requires existing transcript).",
)
def retry(source_id: int, mode: str) -> None:
"""Reset a source so the pipeline re-processes it.
Clears the claim pair and error_message and rewinds status to the
appropriate stage. Parity with the dashboard re-run buttons.
"""
from second_brain.sources_service import reset_source_for_retry
db = get_database()
try:
with db.session() as sess:
source = reset_source_for_retry(
sess, source_id=source_id, mode=mode
)
new_status = source.status.value
url = source.url
except LookupError as exc:
click.echo(f"[retry] {exc}", err=True)
sys.exit(2)
except ValueError as exc:
click.echo(f"[retry] {exc}", err=True)
sys.exit(2)
click.echo(f"[retry] id={source_id} status → {new_status} ({url})")
# ---------------------------------------------------------------------------
# compile
# ---------------------------------------------------------------------------

View File

@ -275,126 +275,14 @@ def add_playlist(
return result
# ---------------------------------------------------------------------------
# Edit + retry helpers (shared by web routes and CLI)
# ---------------------------------------------------------------------------
# Fields the edit-mode form is allowed to mutate. URL is intentionally
# excluded — it's the UNIQUE dedupe key and editing it risks collisions
# plus breaks the embeddings 4-tuple identity (source_id is stable but
# semantic identity drifts).
EDITABLE_METADATA_FIELDS = ("domain", "title", "focus")
def update_source_metadata(
sess: Session,
*,
source_id: int,
domain: Optional[str] = None,
title: Optional[str] = None,
focus: Optional[str] = None,
) -> Source:
"""Update editable metadata on a source row.
Empty strings collapse to NULL for title/focus (matches `add_source`
semantics). Domain is required to stay in the configured set
bumping the constraint is a separate decision.
Raises ValueError on validation failures and LookupError if the row
is missing. Caller owns the transaction boundary.
"""
source = sess.query(Source).filter(Source.id == source_id).first()
if source is None:
raise LookupError(f"source id={source_id} not found")
if domain is not None:
if domain not in DOMAINS:
raise ValueError(
f"unknown domain {domain!r}; valid: {', '.join(DOMAINS)}"
)
source.domain = domain
if title is not None:
source.title = title.strip() or None
if focus is not None:
source.focus = focus.strip() or None
source.updated_at = utcnow()
sess.flush()
return source
# Retry modes — keep the surface small. Anything else is a separate ask.
RETRY_FULL = "full"
RETRY_EXTRACT = "extract"
RETRY_MODES = (RETRY_FULL, RETRY_EXTRACT)
def reset_source_for_retry(
sess: Session,
*,
source_id: int,
mode: str = RETRY_FULL,
) -> Source:
"""Reset a source so the pipeline re-processes it.
Modes:
- "full": status PENDING. Article pull or video download +
transcribe will happen again on the appropriate worker.
- "extract": status TRANSCRIBED. Requires the row to already
carry a transcript (transcript_text). The dev scheduler will
re-run extraction without re-pulling the source. Cheap.
Always clears `claimed_by` / `claimed_at` / `error_message` so the
queue dance can pick it up cleanly and stale errors don't bleed
into the UI.
Raises ValueError on bad mode / missing transcript for extract mode,
and LookupError if the row is missing.
"""
if mode not in RETRY_MODES:
raise ValueError(
f"unknown retry mode {mode!r}; valid: {', '.join(RETRY_MODES)}"
)
source = sess.query(Source).filter(Source.id == source_id).first()
if source is None:
raise LookupError(f"source id={source_id} not found")
if mode == RETRY_EXTRACT:
if not (source.transcript_text and source.transcript_text.strip()):
raise ValueError(
"extract-only retry requires an existing transcript; "
"use full retry instead"
)
source.status = SourceStatus.TRANSCRIBED
else:
source.status = SourceStatus.PENDING
source.claimed_by = None
source.claimed_at = None
source.error_message = None
source.updated_at = utcnow()
sess.flush()
return source
__all__ = [
"AddResult",
"DEFAULT_PLAYLIST_MAX_ITEMS",
"EDITABLE_METADATA_FIELDS",
"PlaylistAddResult",
"RETRY_EXTRACT",
"RETRY_FULL",
"RETRY_MODES",
"add_playlist",
"add_source",
"expand_youtube_playlist",
"is_article_url",
"is_youtube_playlist_url",
"reset_source_for_retry",
"update_source_metadata",
"validate_url",
]

View File

@ -31,14 +31,9 @@ from second_brain.settings_store import (
)
from second_brain.sources_service import (
DEFAULT_PLAYLIST_MAX_ITEMS,
RETRY_EXTRACT,
RETRY_FULL,
RETRY_MODES,
add_playlist,
add_source,
is_youtube_playlist_url,
reset_source_for_retry,
update_source_metadata,
)
# ---------------------------------------------------------------------------
@ -125,33 +120,16 @@ async def index(
)
def _render_source_detail(
request: Request,
source_id: int,
*,
edit: bool = False,
flash: Optional[str] = None,
error: Optional[str] = None,
form_overrides: Optional[dict] = None,
) -> HTMLResponse:
"""Single render path for /sources/{id} — used by GET + every HTMX
mutation endpoint so each action gets a consistent re-render.
`form_overrides` lets a failed edit-save bounce show the user's
in-progress values instead of the row's stored values.
"""
@app.get("/sources/{source_id}", response_class=HTMLResponse)
async def source_detail(request: Request, source_id: int):
"""Source detail page with extraction results and accept/reject actions."""
db = get_database()
with db.session() as sess:
source = sess.query(Source).filter(Source.id == source_id).first()
if not source:
raise HTTPException(status_code=404, detail="Source not found")
source_data = _source_to_dict(source)
extraction_data = (
_extraction_to_dict(source.extraction) if source.extraction else None
)
if form_overrides:
source_data = {**source_data, **form_overrides}
extraction_data = _extraction_to_dict(source.extraction) if source.extraction else None
return templates.TemplateResponse(
request,
@ -159,25 +137,10 @@ def _render_source_detail(
{
"source": source_data,
"extraction": extraction_data,
"edit_mode": edit,
"flash": flash,
"error": error,
"domains": DOMAINS,
},
)
@app.get("/sources/{source_id}", response_class=HTMLResponse)
async def source_detail(request: Request, source_id: int, edit: bool = False):
"""Source detail page with extraction results and accept/reject actions.
`?edit=1` opens the metadata editor inline. Editing exposes domain,
title, and focus URL stays read-only since it's the UNIQUE dedupe
key for the queue and the embeddings identity tuple.
"""
return _render_source_detail(request, source_id, edit=edit)
_IN_FLIGHT_STATUSES = [
SourceStatus.PENDING,
SourceStatus.PULLED,
@ -229,11 +192,17 @@ async def accept_source(request: Request, source_id: int):
if not source:
raise HTTPException(status_code=404, detail="Source not found")
source.status = SourceStatus.ACCEPTED
source_data = _source_to_dict(source)
extraction_data = _extraction_to_dict(source.extraction) if source.extraction else None
return _render_source_detail(
return templates.TemplateResponse(
request,
source_id,
flash="Accepted — will be included in next wiki compile.",
"source_detail.html",
{
"source": source_data,
"extraction": extraction_data,
"flash": "Accepted — will be included in next wiki compile.",
},
)
@ -246,80 +215,19 @@ async def reject_source(request: Request, source_id: int):
if not source:
raise HTTPException(status_code=404, detail="Source not found")
source.status = SourceStatus.FAILED
source_data = _source_to_dict(source)
extraction_data = _extraction_to_dict(source.extraction) if source.extraction else None
return _render_source_detail(request, source_id, flash="Rejected.")
@app.post("/sources/{source_id}/edit", response_class=HTMLResponse)
async def edit_source_metadata(
request: Request,
source_id: int,
domain: str = Form(...),
title: Optional[str] = Form(None),
focus: Optional[str] = Form(None),
):
"""HTMX edit-mode save. Updates domain/title/focus only — URL is
intentionally not editable (see service-layer comment)."""
db = get_database()
try:
with db.session() as sess:
update_source_metadata(
sess,
source_id=source_id,
domain=domain,
title=title,
focus=focus,
)
except LookupError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
except ValueError as exc:
# Bounce back to edit mode with the user's in-progress values so
# they don't have to retype them.
return _render_source_detail(
return templates.TemplateResponse(
request,
source_id,
edit=True,
error=str(exc),
form_overrides={
"domain": domain,
"title": title or "",
"focus": focus or "",
"source_detail.html",
{
"source": source_data,
"extraction": extraction_data,
"flash": "Rejected.",
},
)
return _render_source_detail(request, source_id, flash="Saved.")
@app.post("/sources/{source_id}/retry", response_class=HTMLResponse)
async def retry_source(
request: Request,
source_id: int,
mode: str = Form(RETRY_FULL),
):
"""HTMX retry/re-run. `mode=full` → PENDING (re-pull + re-process).
`mode=extract` TRANSCRIBED (re-extract only; requires an existing
transcript). Always clears the claim pair and error_message."""
if mode not in RETRY_MODES:
raise HTTPException(status_code=400, detail=f"unknown mode {mode!r}")
db = get_database()
try:
with db.session() as sess:
source = reset_source_for_retry(
sess, source_id=source_id, mode=mode
)
new_status = source.status.value
except LookupError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
except ValueError as exc:
return _render_source_detail(request, source_id, error=str(exc))
if mode == RETRY_EXTRACT:
flash = f"Re-queued for extraction (status → {new_status})."
else:
flash = f"Re-queued from scratch (status → {new_status})."
return _render_source_detail(request, source_id, flash=flash)
# ---------------------------------------------------------------------------
# JSON API endpoints
@ -378,7 +286,6 @@ def _source_to_dict(s: Source) -> dict:
"focus": s.focus,
"ingested_at": s.ingested_at.strftime("%Y-%m-%d %H:%M") if s.ingested_at else "",
"has_extraction": s.extraction is not None,
"has_transcript": bool((s.transcript_text or "").strip()),
"error_message": s.error_message,
}

View File

@ -11,19 +11,9 @@
</svg>
Back to sources
</a>
<div class="flex items-start justify-between gap-4">
<div class="min-w-0">
<h2 class="text-2xl font-semibold text-gray-900 break-words">{{ source.title }}</h2>
<h2 class="text-2xl font-semibold text-gray-900">{{ source.title }}</h2>
<a href="{{ source.url }}" target="_blank"
class="text-sm text-indigo-500 hover:text-indigo-700 break-all">{{ source.url }}</a>
</div>
{% if not edit_mode %}
<a href="/sources/{{ source.id }}?edit=1"
class="shrink-0 px-3 py-1.5 text-sm border border-gray-300 text-gray-700 rounded hover:bg-gray-50 transition-colors">
Edit
</a>
{% endif %}
</div>
</div>
{% if flash %}
@ -31,54 +21,7 @@
{{ flash }}
</div>
{% endif %}
{% if error %}
<div class="mb-4 px-4 py-3 bg-red-50 border border-red-200 text-red-800 rounded text-sm">
{{ error }}
</div>
{% endif %}
{% if edit_mode %}
<!-- Edit mode: domain / title / focus. URL is intentionally read-only -->
<form hx-post="/sources/{{ source.id }}/edit"
hx-target="body"
hx-swap="innerHTML"
class="mb-8 bg-white rounded-lg border border-gray-200 p-5 space-y-4">
<div>
<label class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">URL <span class="font-normal normal-case text-gray-400">(read-only — dedupe key)</span></label>
<input type="text" value="{{ source.url }}" disabled
class="w-full px-3 py-2 border border-gray-200 bg-gray-50 text-gray-500 text-sm rounded">
</div>
<div>
<label for="edit-title" class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">Title</label>
<input type="text" id="edit-title" name="title" value="{{ source.title or '' }}"
class="w-full px-3 py-2 border border-gray-300 text-sm rounded focus:outline-none focus:ring-2 focus:ring-indigo-300">
</div>
<div>
<label for="edit-domain" class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">Domain</label>
<select id="edit-domain" name="domain"
class="w-full px-3 py-2 border border-gray-300 text-sm rounded focus:outline-none focus:ring-2 focus:ring-indigo-300">
{% for d in domains %}
<option value="{{ d }}" {% if d == source.domain %}selected{% endif %}>{{ d }}</option>
{% endfor %}
</select>
</div>
<div>
<label for="edit-focus" class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">Focus</label>
<textarea id="edit-focus" name="focus" rows="3"
class="w-full px-3 py-2 border border-gray-300 text-sm rounded focus:outline-none focus:ring-2 focus:ring-indigo-300">{{ source.focus or '' }}</textarea>
</div>
<div class="flex gap-2 pt-1">
<button type="submit"
class="px-4 py-2 bg-indigo-600 text-white text-sm font-semibold rounded hover:bg-indigo-700 transition-colors">
Save
</button>
<a href="/sources/{{ source.id }}"
class="px-4 py-2 border border-gray-300 text-gray-700 text-sm font-semibold rounded hover:bg-gray-50 transition-colors">
Cancel
</a>
</div>
</form>
{% else %}
<!-- Meta row -->
<div class="flex flex-wrap gap-2 mb-6">
{% set status_colors = {
@ -117,9 +60,9 @@
</div>
{% endif %}
<!-- Actions row: accept/reject (when analyzed) + retry (always) -->
<div class="flex flex-wrap gap-3 mb-8">
{% if source.status == 'analyzed' %}
<!-- Accept / Reject actions (only when analyzed) -->
{% if source.status == 'analyzed' %}
<div class="flex gap-3 mb-8">
<button hx-post="/sources/{{ source.id }}/accept"
hx-target="body"
hx-swap="innerHTML"
@ -132,31 +75,6 @@
class="px-5 py-2 bg-red-500 text-white text-sm font-semibold rounded hover:bg-red-600 transition-colors">
Reject
</button>
{% endif %}
<!-- Re-run: full reset → PENDING. Re-pulls and re-processes from scratch. -->
<button hx-post="/sources/{{ source.id }}/retry"
hx-vals='{"mode": "full"}'
hx-target="body"
hx-swap="innerHTML"
hx-confirm="Re-run this source from scratch? Status will reset to PENDING and the pipeline will re-pull{% if source.source_type == 'video' %} and re-transcribe (expensive){% endif %}."
class="px-5 py-2 bg-amber-500 text-white text-sm font-semibold rounded hover:bg-amber-600 transition-colors">
Re-run from scratch
</button>
{% if source.has_transcript %}
<!-- Light retry: skip the re-pull, just re-run LLM extraction. Works
for any source with stored body text (videos have transcripts,
articles have the trafilatura-extracted body). -->
<button hx-post="/sources/{{ source.id }}/retry"
hx-vals='{"mode": "extract"}'
hx-target="body"
hx-swap="innerHTML"
hx-confirm="Re-extract only — keep the existing transcript/body and re-run the LLM extraction?"
class="px-5 py-2 bg-amber-100 text-amber-800 text-sm font-semibold rounded hover:bg-amber-200 border border-amber-300 transition-colors">
Re-extract only
</button>
{% endif %}
</div>
{% endif %}

View File

@ -1,165 +0,0 @@
"""Tests for the settings.local.toml overlay loader.
The overlay is the dev-side path for keeping host-specific secrets
(notably `[database].url`) out of the tracked `settings.toml` without
forcing every interactive shell to export env vars. The footgun this
test suite guards against: a one-key local override (e.g. a single
`[database] url = "..."`) silently wiping every other key in that
section. The merge is section-level, not shallow.
"""
from __future__ import annotations
from pathlib import Path
import pytest
# ---------------------------------------------------------------------------
# Pure-Python: _overlay_sections
# ---------------------------------------------------------------------------
def test_overlay_preserves_sibling_keys_in_overlapping_section():
"""The exact regression target — local `url` must not wipe `pool_size`."""
from second_brain.config import _overlay_sections
base = {"database": {"url": "", "pool_size": 5}}
local = {"database": {"url": "postgresql://x"}}
out = _overlay_sections(base, local)
assert out["database"]["url"] == "postgresql://x"
assert out["database"]["pool_size"] == 5
def test_overlay_section_only_in_base_survives():
from second_brain.config import _overlay_sections
base = {"database": {"url": "a"}, "scheduler": {"interval": 30}}
local = {"database": {"url": "b"}}
out = _overlay_sections(base, local)
assert out["scheduler"] == {"interval": 30}
assert out["database"]["url"] == "b"
def test_overlay_section_only_in_local_appears():
from second_brain.config import _overlay_sections
base = {"database": {"url": "a"}}
local = {"embeddings": {"ollama_url": "http://x"}}
out = _overlay_sections(base, local)
assert out["database"] == {"url": "a"}
assert out["embeddings"] == {"ollama_url": "http://x"}
def test_overlay_scalar_in_overlay_replaces_dict_in_base():
"""Non-dict overlay values replace outright — no merge attempt."""
from second_brain.config import _overlay_sections
base = {"feature": {"a": 1, "b": 2}}
local = {"feature": "off"} # scalar wins outright
out = _overlay_sections(base, local)
assert out["feature"] == "off"
def test_overlay_dict_in_overlay_replaces_scalar_in_base():
from second_brain.config import _overlay_sections
base = {"feature": "off"}
local = {"feature": {"a": 1}}
out = _overlay_sections(base, local)
assert out["feature"] == {"a": 1}
def test_overlay_empty_overlay_is_identity():
from second_brain.config import _overlay_sections
base = {"database": {"url": "a", "pool_size": 5}}
assert _overlay_sections(base, {}) == base
assert _overlay_sections(base, None) == base
def test_overlay_only_merges_one_level():
"""Doc: the overlay is section-level. Nested dicts inside a section
replace, not deep-merge. This pins the documented depth."""
from second_brain.config import _overlay_sections
base = {"db": {"url": "a", "pool": {"min": 1, "max": 10}}}
local = {"db": {"pool": {"max": 20}}}
out = _overlay_sections(base, local)
# pool replaced wholesale — `min` does NOT survive. This is deliberate.
assert out["db"]["pool"] == {"max": 20}
assert out["db"]["url"] == "a"
# ---------------------------------------------------------------------------
# load_config: end-to-end with real TOML files on disk
# ---------------------------------------------------------------------------
def _write(path: Path, body: str) -> None:
path.write_text(body, encoding="utf-8")
@pytest.fixture(autouse=True)
def _reset_config():
"""Drop the cached singleton before/after each test so overlay state
doesn't leak between cases."""
from second_brain.config import reset_config_singleton
reset_config_singleton()
yield
reset_config_singleton()
def test_load_config_picks_up_sibling_local_overlay(tmp_path):
from second_brain.config import load_config
base = tmp_path / "settings.toml"
local = tmp_path / "settings.local.toml"
_write(
base,
'[database]\nurl = ""\n\n[embeddings]\nollama_url = "http://ollama:11434"\nmodel = "nomic-embed-text"\n',
)
_write(
local,
'[database]\nurl = "postgresql+psycopg://lovebug:x@127.0.0.1:5433/petalbrain"\n\n[embeddings]\nollama_url = "http://127.0.0.1:11434"\n',
)
cfg = load_config(base)
# url from local wins; default model survives (not in local).
assert cfg.database["url"].startswith("postgresql+psycopg://")
assert cfg.embeddings["ollama_url"] == "http://127.0.0.1:11434"
assert cfg.embeddings["model"] == "nomic-embed-text"
def test_load_config_no_local_overlay_is_inert(tmp_path):
from second_brain.config import load_config
base = tmp_path / "settings.toml"
_write(base, '[database]\nurl = "from-base"\n')
cfg = load_config(base)
assert cfg.database["url"] == "from-base"
def test_load_config_local_overlay_at_explicit_path(tmp_path):
"""Doc: when an explicit (or SECOND_BRAIN_CONFIG) path is used, the
overlay sibling is resolved next to THAT path not next to the
repo's config dir."""
from second_brain.config import load_config
subdir = tmp_path / "etc"
subdir.mkdir()
base = subdir / "settings.toml"
local = subdir / "settings.local.toml"
_write(base, '[database]\nurl = "base"\n')
_write(local, '[database]\nurl = "overlay-wins"\n')
cfg = load_config(base)
assert cfg.database["url"] == "overlay-wins"

View File

@ -1,258 +0,0 @@
"""Tests for the source edit-metadata + retry-reset service helpers.
Live-DB pattern matches `tests/test_claim.py` / `tests/test_playlist.py`:
both groups auto-skip when no petalbrain connection is configured. The
service layer is where edit/retry logic lives (the web routes + CLI are
thin wrappers around it), so this is where the meaningful coverage is.
"""
from __future__ import annotations
import os
import uuid
import psycopg
import pytest
def _db_reachable() -> bool:
url = os.environ.get("SECOND_BRAIN_DATABASE_URL") or os.environ.get(
"HERBYLAB_DATABASE_URL"
)
if not url:
return False
raw = url.replace("postgresql+psycopg://", "postgresql://", 1)
try:
with psycopg.connect(raw, connect_timeout=2):
return True
except Exception:
return False
pytestmark = pytest.mark.skipif(
not _db_reachable(),
reason="needs SECOND_BRAIN_DATABASE_URL pointed at a real petalbrain",
)
def _seed_source(db, **overrides):
"""Insert a Source row and return (id, url). Caller cleans up by id."""
from second_brain.models import Source, SourceStatus, SourceType, utcnow
tag = uuid.uuid4().hex[:8]
defaults = dict(
url=f"https://example.test/edit-retry/{tag}",
title=f"seed-{tag}",
domain="development",
source_type=SourceType.ARTICLE,
status=SourceStatus.PENDING,
ingested_at=utcnow(),
)
defaults.update(overrides)
with db.session() as sess:
src = Source(**defaults)
sess.add(src)
sess.flush()
return src.id, src.url
def _cleanup(db, source_id):
from second_brain.models import Source
with db.session() as sess:
sess.query(Source).filter(Source.id == source_id).delete(
synchronize_session=False
)
# ---------------------------------------------------------------------------
# update_source_metadata
# ---------------------------------------------------------------------------
def test_update_metadata_writes_domain_title_focus():
from second_brain.database import Database, reset_database_singleton
from second_brain.models import Source
from second_brain.sources_service import update_source_metadata
reset_database_singleton()
db = Database()
sid, _ = _seed_source(db)
try:
with db.session() as sess:
update_source_metadata(
sess,
source_id=sid,
domain="homelab",
title="new title",
focus="new focus directive",
)
with db.session() as sess:
row = sess.query(Source).filter(Source.id == sid).first()
assert row.domain == "homelab"
assert row.title == "new title"
assert row.focus == "new focus directive"
assert row.updated_at is not None
finally:
_cleanup(db, sid)
reset_database_singleton()
def test_update_metadata_blank_strings_become_null():
from second_brain.database import Database, reset_database_singleton
from second_brain.models import Source
from second_brain.sources_service import update_source_metadata
reset_database_singleton()
db = Database()
sid, _ = _seed_source(db)
try:
with db.session() as sess:
update_source_metadata(
sess, source_id=sid, domain="development", title=" ", focus=""
)
with db.session() as sess:
row = sess.query(Source).filter(Source.id == sid).first()
assert row.title is None
assert row.focus is None
finally:
_cleanup(db, sid)
reset_database_singleton()
def test_update_metadata_rejects_unknown_domain():
from second_brain.database import Database, reset_database_singleton
from second_brain.sources_service import update_source_metadata
reset_database_singleton()
db = Database()
sid, _ = _seed_source(db)
try:
with db.session() as sess:
with pytest.raises(ValueError, match="unknown domain"):
update_source_metadata(sess, source_id=sid, domain="not-a-domain")
finally:
_cleanup(db, sid)
reset_database_singleton()
def test_update_metadata_missing_source_raises_lookup():
from second_brain.database import Database, reset_database_singleton
from second_brain.sources_service import update_source_metadata
reset_database_singleton()
db = Database()
try:
with db.session() as sess:
with pytest.raises(LookupError):
update_source_metadata(
sess, source_id=-99999, domain="development"
)
finally:
reset_database_singleton()
# ---------------------------------------------------------------------------
# reset_source_for_retry
# ---------------------------------------------------------------------------
def test_retry_full_resets_to_pending_and_clears_claim():
from second_brain.database import Database, reset_database_singleton
from second_brain.models import Source, SourceStatus, utcnow
from second_brain.sources_service import reset_source_for_retry
reset_database_singleton()
db = Database()
sid, _ = _seed_source(
db,
status=SourceStatus.FAILED,
error_message="boom",
claimed_by="worker-x",
claimed_at=utcnow(),
)
try:
with db.session() as sess:
reset_source_for_retry(sess, source_id=sid, mode="full")
with db.session() as sess:
row = sess.query(Source).filter(Source.id == sid).first()
assert row.status == SourceStatus.PENDING
assert row.claimed_by is None
assert row.claimed_at is None
assert row.error_message is None
finally:
_cleanup(db, sid)
reset_database_singleton()
def test_retry_extract_sets_transcribed_when_transcript_present():
from second_brain.database import Database, reset_database_singleton
from second_brain.models import Source, SourceStatus
from second_brain.sources_service import reset_source_for_retry
reset_database_singleton()
db = Database()
sid, _ = _seed_source(
db,
status=SourceStatus.ANALYZED,
transcript_text="word " * 50,
error_message="prior failure",
)
try:
with db.session() as sess:
reset_source_for_retry(sess, source_id=sid, mode="extract")
with db.session() as sess:
row = sess.query(Source).filter(Source.id == sid).first()
assert row.status == SourceStatus.TRANSCRIBED
assert row.error_message is None
assert row.transcript_text # transcript preserved
finally:
_cleanup(db, sid)
reset_database_singleton()
def test_retry_extract_without_transcript_raises():
from second_brain.database import Database, reset_database_singleton
from second_brain.models import SourceStatus
from second_brain.sources_service import reset_source_for_retry
reset_database_singleton()
db = Database()
sid, _ = _seed_source(db, status=SourceStatus.PENDING, transcript_text=None)
try:
with db.session() as sess:
with pytest.raises(ValueError, match="requires an existing transcript"):
reset_source_for_retry(sess, source_id=sid, mode="extract")
finally:
_cleanup(db, sid)
reset_database_singleton()
def test_retry_unknown_mode_raises():
from second_brain.database import Database, reset_database_singleton
from second_brain.sources_service import reset_source_for_retry
reset_database_singleton()
db = Database()
sid, _ = _seed_source(db)
try:
with db.session() as sess:
with pytest.raises(ValueError, match="unknown retry mode"):
reset_source_for_retry(sess, source_id=sid, mode="bogus")
finally:
_cleanup(db, sid)
reset_database_singleton()
def test_retry_missing_source_raises_lookup():
from second_brain.database import Database, reset_database_singleton
from second_brain.sources_service import reset_source_for_retry
reset_database_singleton()
db = Database()
try:
with db.session() as sess:
with pytest.raises(LookupError):
reset_source_for_retry(sess, source_id=-99999, mode="full")
finally:
reset_database_singleton()