Compare commits
9 Commits
claude/lau
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
8d29f2fb08 | ||
|
|
d40f25c99d | ||
|
|
22f4a0915f | ||
|
|
5131f3ccf6 | ||
|
|
93f71ebb14 | ||
|
|
24055ad686 | ||
|
|
373e137993 | ||
|
|
3d6bd6385f | ||
|
|
fbbb1a13a0 |
353
CLAUDE.md
353
CLAUDE.md
@ -1,122 +1,297 @@
|
||||
# second-brain
|
||||
|
||||
A personal knowledge system: video/article extraction pipeline + LLM-maintained wiki.
|
||||
Personal knowledge pipeline. Two machines coordinate purely through the
|
||||
shared Postgres queue: the **dev side** (herbys-dev) does article pull
|
||||
+ all extraction + embedding, and the **tower** (EndeavourOS, RTX 3080)
|
||||
does video pull + transcription on the GPU. A FastAPI/Jinja/HTMX web
|
||||
app fronts the queue, settings, and add-to-queue form; the file-based
|
||||
Obsidian wiki is downstream of `accepted` extractions.
|
||||
|
||||
## Architecture
|
||||
|
||||
Three-layer system:
|
||||
|
||||
1. **Wiki** — long-term structured memory. Markdown files in an Obsidian vault, maintained by the wiki compiler.
|
||||
2. **Context assembler** — builds the prompt bundle for each extraction. Phase 1 is thin: domain template + focus field + transcript.
|
||||
3. **Extractor** — stateless single-shot LLM call per source. Pure function: bundle in, structured extraction out.
|
||||
|
||||
## Pipeline stages
|
||||
## Two-machine pipeline
|
||||
|
||||
```
|
||||
pending → pulled → transcribed → analyzed → accepted → published
|
||||
add (CLI or POST /sources/add — single video, article,
|
||||
or YouTube playlist URL → fan-out into N rows)
|
||||
│
|
||||
▼
|
||||
┌────────── second_brain.sources (the queue) ──────────┐
|
||||
│ │
|
||||
│ PENDING ── (article)──► dev `second-brain process`
|
||||
│ trafilatura pull │
|
||||
│ PENDING ── (video) ──► tower transcribe-worker │
|
||||
│ yt-dlp pull → PULLED │
|
||||
│ PULLED ── (video) ──► tower transcribe-worker │
|
||||
│ faster-whisper │
|
||||
│ both paths land at ──► TRANSCRIBED │
|
||||
│ │
|
||||
│ TRANSCRIBED ─────────► dev `second-brain schedule` │
|
||||
│ Claude CLI extract + │
|
||||
│ Ollama embed │
|
||||
│ ──► ANALYZED ──► accept in web UI │
|
||||
│ ──► ACCEPTED ──► `compile` writes vault│
|
||||
│ ──► PUBLISHED │
|
||||
│ │
|
||||
│ FAILED is the terminal error state │
|
||||
└───────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Project layout
|
||||
**Source claiming** — `sources.claimed_by` / `claimed_at` columns +
|
||||
`SELECT ... FOR UPDATE SKIP LOCKED` keep the dev and tower workers from
|
||||
grabbing the same row. Both stages filter by `source_type`:
|
||||
|
||||
```
|
||||
src/second_brain/
|
||||
├── adapters/ # Pull adapters (youtube.py, article.py)
|
||||
├── context/ # Context assembler
|
||||
├── embeddings/ # Chunk + nomic-embed-text + upsert into public.embeddings
|
||||
├── extractor/ # LLM extraction engine + Pydantic schema
|
||||
├── llm/ # claude_cli.py — thin subprocess wrapper around `claude -p`
|
||||
├── compiler/ # Wiki compiler (reads vault, writes pages, git commits)
|
||||
├── scheduler/ # Rate-limit smoother (call-count for CLI, token-budget for API)
|
||||
└── web/ # FastAPI + Jinja2 + HTMX UI
|
||||
- tower transcribe-worker: `status IN (PENDING, PULLED) AND source_type = 'video'`
|
||||
- dev `process` (article pull): `status = PENDING AND source_type = 'article'`
|
||||
- dev `schedule` (extract+embed): `status = TRANSCRIBED` (any type)
|
||||
|
||||
alembic/ # Schema migrations for the second_brain Postgres schema
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
cp config/settings.toml.example config/settings.toml
|
||||
# Edit settings.toml — at minimum, point the paths and set SECOND_BRAIN_DATABASE_URL.
|
||||
|
||||
# Connection string format. Containerised (default):
|
||||
# postgresql+psycopg://lovebug:<password>@homelab-postgres:5432/petalbrain
|
||||
# Host-side dev runs:
|
||||
# postgresql+psycopg://lovebug:<password>@127.0.0.1:5433/petalbrain
|
||||
export SECOND_BRAIN_DATABASE_URL=...
|
||||
|
||||
# One-time, before first migration: bootstrap the schema as postgres
|
||||
# superuser. lovebug doesn't have CREATE on petalbrain, so this can't be
|
||||
# done from the app side — same pattern as wiki/trellis/ob1.
|
||||
# CREATE SCHEMA second_brain AUTHORIZATION lovebug;
|
||||
# GRANT ALL PRIVILEGES ON SCHEMA second_brain TO lovebug;
|
||||
|
||||
uv sync
|
||||
uv run alembic upgrade head
|
||||
uv run second-brain serve
|
||||
```
|
||||
A 30-minute stale-claim reaper runs at worker startup so a crashed
|
||||
machine doesn't dam the queue.
|
||||
|
||||
## CLI commands
|
||||
|
||||
- `second-brain add <url>` — queue a source URL (video or article)
|
||||
- `second-brain process` — run the pipeline on pending items
|
||||
- `second-brain serve` — launch the web UI (default port 8000)
|
||||
- `second-brain compile` — run the wiki compiler
|
||||
- `second-brain schedule` — start the overnight scheduler
|
||||
- `second-brain add <url>` — queue a source. YouTube `playlist?list=…`
|
||||
URLs auto-expand into one row per video (cap 50, the
|
||||
`watch?v=…&list=…` form stays a single video — see
|
||||
`is_youtube_playlist_url` for the rule).
|
||||
- `second-brain process` — dev-side: pull pending **articles**, extract
|
||||
+ embed any TRANSCRIBED source. Skips videos (tower's job).
|
||||
- `second-brain schedule` — dev-side overnight extract+embed loop,
|
||||
honouring `pipeline_settings.extraction_enabled` / active-hours /
|
||||
max-items.
|
||||
- `second-brain transcribe-worker` — tower-side daemon. `--once` for
|
||||
ad-hoc runs. Honours `pipeline_settings.transcription_*` knobs.
|
||||
- `second-brain serve` — local dev launch of the web UI (port 8000).
|
||||
Production runs as a container (see `deploy/web/`).
|
||||
- `second-brain compile` — wiki compiler: ACCEPTED → markdown in
|
||||
`vault_path`, git-committed. File-based Obsidian vault only; we do
|
||||
**not** dual-write into petalbrain.wiki.
|
||||
|
||||
## Storage
|
||||
|
||||
**Postgres**: `petalbrain` DB on `homelab-postgres` (pgvector/pg17, docker
|
||||
container). Connection role: `lovebug`. Dev reaches it at
|
||||
`127.0.0.1:5433`; tower reaches it over WireGuard at
|
||||
`db.wg.herbylab.dev:5432`; the web container reaches it at
|
||||
`homelab-postgres:5432` on the `homelab` docker bridge. **Keep both port
|
||||
binds** — the host-side 5433 publish and the docker-net 5432 service.
|
||||
|
||||
**Schema `second_brain`** (Alembic-managed in `alembic/`):
|
||||
|
||||
- `sources` — the queue. Int SERIAL PK. Naive UTC `timestamp`
|
||||
(`without time zone`) — never `timestamptz`. `transcript_text` is the
|
||||
joined full text the extractor reads; `transcript_segments` is JSONB
|
||||
with the faster-whisper per-segment / per-word breakdown.
|
||||
`claimed_by` / `claimed_at` for the queue dance.
|
||||
- `extractions` — one row per source. JSON columns for the structured
|
||||
output (summary, key_points, entities, claims, …).
|
||||
- `pipeline_settings` — **single-row** config (PK pinned to id=1 with a
|
||||
CHECK constraint). What the dashboard edits and workers read each
|
||||
iteration: transcription_enabled / active_hours / max_gpu_jobs /
|
||||
max_video_length / max_items_per_run plus the equivalent extraction_*
|
||||
fields. DB values override settings.toml.
|
||||
- `wiki_pages` — bookkeeping for the file-based wiki compiler.
|
||||
|
||||
**Embeddings** live in `public.embeddings` — the **shared** pgvector
|
||||
table owned by `postgres` and consumed by vault-mcp / ob1 / second-brain.
|
||||
We do not manage this in Alembic. Rows are keyed by
|
||||
`(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`,
|
||||
with `delete-before-insert` semantics on the 4-tuple so a re-embed
|
||||
never leaves orphans. We embed extraction **summaries only** for now —
|
||||
not key_points, not transcript chunks.
|
||||
|
||||
## Web app
|
||||
|
||||
FastAPI + Jinja2 + HTMX, in `src/second_brain/web/`. Production lives in
|
||||
a slim Python 3.12 container (see `deploy/web/`) on the `homelab` docker
|
||||
network, reaching Postgres + Ollama by service DNS. Published at
|
||||
`10.0.21.207:8080` and fronted by the Traefik VM (10.0.11.20,
|
||||
file-provider — **no docker-label routing**) as
|
||||
`https://brain.herbylab.dev`.
|
||||
|
||||
Routes:
|
||||
|
||||
- `/` — sources list (filterable by domain + status)
|
||||
- `/queue` — in-flight sources (PENDING / PULLED / TRANSCRIBED) plus the
|
||||
add-to-queue form
|
||||
- `/dashboard` — status counts, recent activity, settings snapshot, plus
|
||||
the add-to-queue form
|
||||
- `/sources/{id}` — extraction detail + accept/reject HTMX actions
|
||||
- `/settings` — full HTMX editor for `pipeline_settings`
|
||||
- `POST /sources/add` — shared endpoint. Detects YouTube playlist URLs
|
||||
and fans out via yt-dlp `extract_flat`. Reads `HX-Current-URL` to
|
||||
dispatch the response partial (`_queue_body.html` vs
|
||||
`_dashboard_body.html`) so each page refreshes its own region inline.
|
||||
|
||||
**No auth.** UI mutates state (accept/reject sources, edit settings) but
|
||||
has no SSO / CSRF / rate-limiting. The Traefik route is currently
|
||||
LAN-only by Travis's call — see `deploy/web/README.md` §SECURITY.
|
||||
|
||||
## Transcription
|
||||
|
||||
`src/second_brain/transcribe.py` wraps **faster-whisper** (CTranslate2
|
||||
backend). Defaults: model `large-v3`, device `cuda`, compute_type
|
||||
`float16`. CPU fallback drops to `int8`. `word_timestamps=True` is
|
||||
always passed; the segment dicts persisted to
|
||||
`sources.transcript_segments` carry per-word `{start, end, word,
|
||||
probability}`.
|
||||
|
||||
The tower worker (`src/second_brain/scheduler/transcribe_worker.py`)
|
||||
runs as a systemd service — see `deploy/tower/README.md` for the full
|
||||
install runbook. ExecStart goes through
|
||||
`deploy/tower/run-transcribe-worker.sh` which prepends the venv's
|
||||
CUDA-12 lib dirs to `LD_LIBRARY_PATH` (see gotcha (b) below).
|
||||
|
||||
## Extractor backends
|
||||
|
||||
`[extractor].backend` in `config/settings.toml` picks the LLM transport.
|
||||
`[extractor].backend` in `config/settings.toml`:
|
||||
|
||||
- **`cli`** (default) — `src/second_brain/llm/claude_cli.py` shells out to `claude -p --output-format json --model <model> --json-schema <schema>`. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside `tempfile.TemporaryDirectory(prefix="sb-claude-cli-")`, with `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` stripped from env so the CLI uses OAuth instead of an API key, and with `--no-session-persistence --disable-slash-commands --tools "" --setting-sources ""` so host CLAUDE.md / hooks / agents don't leak in.
|
||||
- **`api`** — uses the `anthropic` SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (`uv sync --extra api`).
|
||||
- `cli` (default) — `src/second_brain/llm/claude_cli.py` shells out to
|
||||
`claude -p --output-format json --model … --json-schema …`. Runs
|
||||
under the Max OAuth subscription. Hermetic spawn: throwaway
|
||||
`tempfile.TemporaryDirectory`, `ANTHROPIC_API_KEY` / `_AUTH_TOKEN`
|
||||
stripped from env so it uses OAuth, and `--no-session-persistence
|
||||
--disable-slash-commands --tools "" --setting-sources ""` so host
|
||||
CLAUDE.md / hooks / agents don't leak in.
|
||||
- `api` — `anthropic` SDK, lazy-imported. Requires `ANTHROPIC_API_KEY`.
|
||||
Install with `uv sync --extra api`.
|
||||
|
||||
**Schema-validated output gotcha:** when `--json-schema` is used, the parsed object lands at `payload["structured_output"]`, not `payload["result"]`. The wrapper checks both. The engine prefers the structured payload via `LLMExtraction.model_validate(structured)` and falls back to parsing the text field.
|
||||
Schema-validated CLI output lands at `payload["structured_output"]`, not
|
||||
`payload["result"]`. The wrapper checks both.
|
||||
|
||||
## Environment variables
|
||||
|
||||
- `ANTHROPIC_API_KEY` — only required when `extractor.backend = "api"`. The default `cli` backend uses your Max OAuth and does not need it.
|
||||
- `SECOND_BRAIN_CONFIG` — optional override for the settings file path.
|
||||
- `SECOND_BRAIN_DATABASE_URL` — Postgres connection string. Falls back to `HERBYLAB_DATABASE_URL` if unset. Required at runtime; both `second_brain.database` and `second_brain.embeddings` read it. Use the SQLAlchemy `postgresql+psycopg://` form; the embeddings module strips the `+psycopg` prefix internally for raw psycopg.
|
||||
- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default `http://ollama:11434` (homelab docker network); for host-side runs use `http://127.0.0.1:11434`.
|
||||
- `EMBEDDING_MODEL` — overrides `[embeddings].model`. Default `nomic-embed-text` (768-d).
|
||||
- `SECOND_BRAIN_DB_POOL_MIN` / `SECOND_BRAIN_DB_POOL_MAX` — SQLAlchemy pool sizing knobs (defaults 1 / 9 overflow).
|
||||
- `SECOND_BRAIN_DATABASE_URL` — Postgres DSN. Falls back to
|
||||
`HERBYLAB_DATABASE_URL`. Use `postgresql+psycopg://` for SQLAlchemy;
|
||||
the embeddings + claim modules strip the `+psycopg` prefix for raw
|
||||
psycopg.
|
||||
- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default
|
||||
`http://ollama:11434` (homelab docker net); host-side runs use
|
||||
`http://127.0.0.1:11434`.
|
||||
- `EMBEDDING_MODEL` — default `nomic-embed-text` (768-d).
|
||||
- `WHISPER_MODEL` / `WHISPER_DEVICE` / `WHISPER_COMPUTE_TYPE` — override
|
||||
the `[whisper]` block on the tower.
|
||||
- `ANTHROPIC_API_KEY` — only for `extractor.backend = "api"`.
|
||||
- `SECOND_BRAIN_CONFIG` — settings file path override.
|
||||
- `SECOND_BRAIN_WORKER_ID` — stamped into `sources.claimed_by`.
|
||||
- `SECOND_BRAIN_DB_POOL_MIN` / `_MAX` — SQLAlchemy pool sizing.
|
||||
- `SECOND_BRAIN_LIVE_NETWORK_TESTS=1` — opt-in for the yt-dlp playlist
|
||||
test that hits real YouTube.
|
||||
|
||||
## Domains
|
||||
|
||||
- `development` — software, programming, systems
|
||||
- `content` — content creation strategy, video production
|
||||
- `business` — entrepreneurship, marketing, operations
|
||||
- `homelab` — self-hosted infrastructure, networking, DevOps
|
||||
`development` / `content` / `business` / `homelab` — drives the
|
||||
per-domain prompt template loaded by the context assembler.
|
||||
|
||||
## Key files
|
||||
|
||||
- `config/settings.toml` — vault path, `[database]`, `[extractor]`, `[embeddings]`, `[scheduler]`
|
||||
- `prompts/<domain>.md` — per-domain extraction prompt templates
|
||||
- `src/second_brain/extractor/schema.py` — `LLMExtraction` (LLM-only fields, drives the `--json-schema` contract) + `ExtractionResult` (LLMExtraction + SourceMeta filled from the DB)
|
||||
- `src/second_brain/llm/claude_cli.py` — CLI subprocess wrapper
|
||||
- `src/second_brain/models.py` — SQLAlchemy models (all tables live in the `second_brain` schema) + `utcnow()` helper
|
||||
- `src/second_brain/database.py` — Postgres engine + URL resolution
|
||||
- `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert into `public.embeddings`; best-effort with graceful degradation
|
||||
- `alembic/` + `alembic.ini` — schema migrations for the `second_brain` schema (Alembic does **not** manage `public.embeddings`)
|
||||
- `postgres-migration-planning.md` — original planning doc; superseded by this file once the migration shipped
|
||||
- `src/second_brain/models.py` — all ORM in the `second_brain` schema.
|
||||
- `src/second_brain/sources_service.py` — `add_source`, `add_playlist`,
|
||||
`is_youtube_playlist_url`, `expand_youtube_playlist`. CLI + web both
|
||||
call into here so there's no behavior drift between entry points.
|
||||
- `src/second_brain/claim.py` — the SKIP LOCKED dance + stale reaper.
|
||||
- `src/second_brain/settings_store.py` — single-row `pipeline_settings`
|
||||
get/update + the active-window math the workers share.
|
||||
- `src/second_brain/transcribe.py` — faster-whisper wrapper + segment
|
||||
serialiser.
|
||||
- `src/second_brain/scheduler/runner.py` — dev extract scheduler.
|
||||
- `src/second_brain/scheduler/transcribe_worker.py` — tower poll loop.
|
||||
- `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert
|
||||
into `public.embeddings`; best-effort with graceful degradation.
|
||||
- `alembic/` + `alembic.ini` — schema migrations for the `second_brain`
|
||||
schema only. `public.embeddings` is **not** ours to migrate.
|
||||
- `deploy/web/` — Dockerfile + compose.yml + README for the web
|
||||
container. Travis applies the Knot DNS A-record + Traefik
|
||||
file-provider router from the README §2.
|
||||
- `deploy/tower/` — systemd unit + `run-transcribe-worker.sh` wrapper +
|
||||
env template + EndeavourOS install runbook.
|
||||
- `prompts/<domain>.md` — extraction prompt templates.
|
||||
- `config/settings.toml` — `[database]`, `[extractor]`, `[embeddings]`,
|
||||
`[scheduler]`, `[whisper]`.
|
||||
|
||||
## Conventions / gotchas
|
||||
## Load-bearing gotchas
|
||||
|
||||
- **Naive UTC timestamps everywhere.** Use `from second_brain.models import utcnow` rather than `datetime.utcnow()` (deprecated in 3.12) or `datetime.now()` (local time). Postgres columns are `timestamp without time zone` — do not switch to `timestamptz`.
|
||||
- **PK style.** Int autoincrement (`SERIAL`) — locked for compatibility with the existing schema shape. Do not switch to `BIGINT GENERATED ALWAYS AS IDENTITY` or UUID.
|
||||
- **Re-query inside the session.** SQLAlchemy ORM attribute access after the session closes raises `DetachedInstanceError`. The `process` command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates *inside* the `with db.session()` block.
|
||||
- **Starlette `TemplateResponse` signature.** Starlette ≥ 0.36 requires `templates.TemplateResponse(request, name, context)`. The old `(name, context)` form raises `TypeError: unhashable type: 'dict'`. Do not put `"request": request` into the context dict — it's auto-injected now.
|
||||
- **Vault git repo auto-init.** The wiki compiler calls `_ensure_git_repo()` on every run, which `git init`s the vault and creates an empty seed commit if no `.git` is present. `_git_commit()` also checks `git diff --cached --quiet` before committing to avoid empty commits.
|
||||
- **Scheduler throttling shape.** The CLI backend is throttled by `max_calls_per_hour` (Max subscription limits are call-shaped, not token-shaped). The API backend uses `max_tokens_per_hour`. Both share `min_gap_seconds`. The scheduler targets sources in `TRANSCRIBED` state — earlier code mistakenly filtered on `ANALYZED`, making it a no-op.
|
||||
- **Alembic + lovebug privilege gap.** `lovebug` owns the `second_brain` schema but does **not** have `CREATE` on the `petalbrain` database. `alembic/env.py` therefore guards `CREATE SCHEMA` behind a `pg_namespace` lookup, and the v1 migration intentionally drops the `IF NOT EXISTS` form (a bare `CREATE SCHEMA IF NOT EXISTS` raises permission denied even when the schema is already present). The schema is bootstrapped once, out-of-band, by `postgres` superuser.
|
||||
- **public.embeddings is shared, not ours.** Alembic only manages the `second_brain` schema. The vector table itself is owned by `postgres` and shared across vault-mcp / ob1 / second-brain — we just INSERT into it. Rows are keyed by `(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`.
|
||||
- **Embedding is best-effort.** The pipeline writes the relational `extractions` row first and `sess.flush()`es; only then calls `embed_extraction()`. Any failure (no DB URL, no Ollama, embedding 500) logs at WARNING and returns None — the relational row and file vault stay canonical.
|
||||
- **File-based wiki, not petalbrain.wiki.** The wiki compiler still writes markdown into the configured `vault_path` and commits via git. We do **not** dual-write into `petalbrain.wiki` — that's vault-mcp's territory and intentionally out of scope.
|
||||
These have all bitten us; don't relearn them.
|
||||
|
||||
## In-flight work
|
||||
- **(a) Schema bootstrap requires postgres superuser.** `lovebug` does
|
||||
NOT have `CREATE` on the `petalbrain` database, even though it owns
|
||||
the `second_brain` schema. The first migration needs a one-time
|
||||
`CREATE SCHEMA second_brain AUTHORIZATION lovebug; GRANT ALL PRIVILEGES
|
||||
ON SCHEMA second_brain TO lovebug;` as `postgres`. `alembic/env.py`
|
||||
guards `CREATE SCHEMA` behind a `pg_namespace` lookup so subsequent
|
||||
runs as `lovebug` don't trip the permission check.
|
||||
|
||||
- **Vault location.** Recommended `/opt/projects/second-brain-vault/` as the production location (peer to the project dir). Not yet confirmed.
|
||||
- **(b) Arch ships CUDA 13, ctranslate2 needs CUDA 12.** EndeavourOS's
|
||||
`pacman -S cuda cudnn` installs `libcublas.so.13` / `libcudnn.so.10`,
|
||||
which `ctranslate2` 4.7.x cannot load — it wants the .12 / .9 ABI.
|
||||
We pin `nvidia-cublas-cu12` and `nvidia-cudnn-cu12>=9,<10` in the
|
||||
`tower` optional-dependency group, and the systemd ExecStart calls
|
||||
`deploy/tower/run-transcribe-worker.sh` which resolves the venv's
|
||||
wheel-bundled `.so` directories at runtime and prepends them to
|
||||
`LD_LIBRARY_PATH`. The wrapper uses the namespace packages'
|
||||
`__path__` (not `__file__`, which is `None` for PEP 420 packages).
|
||||
systemd does **not** inherit shell exports.
|
||||
|
||||
- **(c) pg_hba needs BOTH subnets.** Two host rules are load-bearing:
|
||||
```
|
||||
host all all 172.19.0.0/16 scram-sha-256 # homelab docker bridge
|
||||
host all all 10.99.0.0/24 scram-sha-256 # WireGuard (tower)
|
||||
```
|
||||
Dropping the 172.19/16 rule locks out the web container + vault-mcp +
|
||||
ob1-* + trellis-mcp (all on the homelab net). Dropping the 10.99/24
|
||||
rule locks out the tower. Both have been removed once each during
|
||||
unrelated postgres edits — the symptom is `FATAL: no pg_hba.conf
|
||||
entry for host "…"` in service logs.
|
||||
|
||||
- **(d) Two postgres port binds, both load-bearing.** `homelab-postgres`
|
||||
publishes `127.0.0.1:5433 → 5432` for dev-host access AND is reachable
|
||||
on `homelab-postgres:5432` over the docker net. Tower hits it over WG
|
||||
at `db.wg.herbylab.dev:5432`. Don't consolidate; each path serves a
|
||||
different consumer set.
|
||||
|
||||
- **(e) Traefik is file-provider on a separate VM (10.0.11.20).** It
|
||||
cannot read this host's docker labels. The web `compose.yml` therefore
|
||||
publishes back to a fixed host port (`10.0.21.207:8080`) that Traefik
|
||||
reaches over LAN, and intentionally carries NO `traefik.*` labels.
|
||||
Router config lives in Traefik's dynamic-config YAML — see
|
||||
`deploy/web/README.md` §2b for the snippet.
|
||||
|
||||
- **Code-level conventions, since they bit us early:**
|
||||
- **Naive UTC timestamps.** Use `from second_brain.models import utcnow`
|
||||
— never `datetime.utcnow()` (deprecated) or `datetime.now()` (local).
|
||||
Postgres columns are `timestamp without time zone`.
|
||||
- **Int autoincrement PKs.** Don't migrate to bigint-identity or UUID;
|
||||
the existing schema shape is locked.
|
||||
- **Re-query inside the session.** SQLAlchemy ORM attribute access
|
||||
after the session closes raises `DetachedInstanceError`. Web routes
|
||||
and CLI loops collect IDs first, then re-fetch inside the session
|
||||
block; dict snapshots for templates are built inside `with
|
||||
db.session()`.
|
||||
- **Starlette TemplateResponse signature.** `templates.TemplateResponse(request, name, context)` — the old `(name, context)` form raises
|
||||
`TypeError: unhashable type: 'dict'`. Do not put `"request":
|
||||
request` into the context dict.
|
||||
- **Embedding is best-effort.** The extract path flushes the
|
||||
relational `extractions` row first, then calls `embed_extraction()`.
|
||||
DB / Ollama failures log and return None; pipeline state stays
|
||||
canonical in the relational + file vault.
|
||||
- **Worker LD_LIBRARY_PATH.** See (b). When pre-warming or debugging
|
||||
faster-whisper manually, run via `deploy/tower/run-transcribe-worker.sh`
|
||||
or replicate the path probe in your shell.
|
||||
|
||||
## In-flight / parked
|
||||
|
||||
- **Vault location** — recommended `/opt/projects/second-brain-vault/`
|
||||
as a peer dir to the project. Not yet confirmed.
|
||||
- **Media + subtitles on NAS** — currently land on the tower's local
|
||||
SSD. Move to NAS once we have a stable mount path.
|
||||
- **SSO** — Authentik forward-auth middleware in Traefik is the
|
||||
obvious next step before opening the UI past LAN.
|
||||
- **Dev-side extraction systemd timer** — currently invoked manually;
|
||||
no systemd unit yet by Travis's call.
|
||||
|
||||
## Validation
|
||||
|
||||
Per Travis's global CLAUDE.md, run `zero-check` after any code change, and `a-review` after non-trivial diffs (> 50 lines). `a-review` script: `python3 ~/.claude/skills/a-review/run_review.py`, output goes to `reviews/a-review-<date>.md`.
|
||||
Per the user's global CLAUDE.md, run `zero-check` after any code
|
||||
change. The pipeline's at `~/zero-check-pipeline/validate/run-all.sh
|
||||
"$(pwd)"`. Run `a-review` after non-trivial diffs (>50 lines):
|
||||
`python3 ~/.claude/skills/a-review/run_review.py`. Test suite is `uv
|
||||
run pytest`; live DB + Ollama paths auto-skip when their endpoints
|
||||
aren't configured.
|
||||
|
||||
@ -50,9 +50,9 @@ your CLI PATH.
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /opt/projects/homelab
|
||||
sudo chown herbyadmin:herbyadmin /opt/projects/homelab
|
||||
sudo chown trucktrav:trucktrav /opt/projects/homelab
|
||||
cd /opt/projects/homelab
|
||||
git clone gitea-lovebug:petal-power/second-brain.git
|
||||
git clone gitea-pbs-trucktrav:petal-power/second-brain.git
|
||||
cd second-brain
|
||||
|
||||
# Install the runtime deps + the `tower` extra (faster-whisper / CTranslate2).
|
||||
@ -90,7 +90,7 @@ The worker assumes the tunnel is already configured and that
|
||||
| Peer | WG IP |
|
||||
|------|-------|
|
||||
| hub (herbydev) | `10.99.0.1` |
|
||||
| tower | `10.99.0.2` |
|
||||
| tower | `10.99.0.3` |
|
||||
|
||||
`db.wg.herbylab.dev` resolves to `10.99.0.1`. Confirm connectivity from
|
||||
the tower:
|
||||
|
||||
@ -16,8 +16,8 @@ StartLimitBurst=5
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=herbyadmin
|
||||
Group=herbyadmin
|
||||
User=trucktrav
|
||||
Group=trucktrav
|
||||
|
||||
# Adjust to wherever you `git clone`d the repo on the tower.
|
||||
WorkingDirectory=/opt/projects/homelab/second-brain
|
||||
|
||||
37
reviews/a-review-2026-05-25.md
Normal file
37
reviews/a-review-2026-05-25.md
Normal file
@ -0,0 +1,37 @@
|
||||
# a-review — 2026-05-25
|
||||
|
||||
**Model:** gemini
|
||||
**Diff base:** `HEAD~1` (5 files changed, 615 insertions(+), 30 deletions(-))
|
||||
**Session reference:** none
|
||||
**Context:** CLAUDE.md + session notes + expanded diff
|
||||
|
||||
---
|
||||
|
||||
Excellent work. The changes are well-structured, follow existing project conventions, and include a comprehensive test suite. This is a high-quality contribution. I have only one minor finding and one informational note.
|
||||
|
||||
### [WARNING] Logic Error
|
||||
|
||||
* **File:** `src/second_brain/web/templates/source_detail.html`
|
||||
* **Location:** Line 151
|
||||
* **Issue:** The "Re-extract only" button is conditionally rendered with `{% if source.has_transcript and source.source_type == 'video' %}`. This is too restrictive. The backend service function `reset_source_for_retry` correctly allows an "extract" retry for any source that has content in `transcript_text`, which includes processed articles. Limiting this feature to only videos in the UI prevents a valid and useful action for non-video sources.
|
||||
* **Suggestion:** Change the conditional to `{% if source.has_transcript %}`. This will correctly show the button for any source (article or video) that has been pulled and has content ready for extraction, which matches the backend's capability.
|
||||
|
||||
```html
|
||||
<!-- Change this -->
|
||||
{% if source.has_transcript and source.source_type == 'video' %}
|
||||
|
||||
<!-- To this -->
|
||||
{% if source.has_transcript %}
|
||||
```
|
||||
|
||||
### [INFO] Security
|
||||
|
||||
* **Files:** `src/second_brain/web/app.py`, `src/second_brain/web/templates/source_detail.html`
|
||||
* **Observation:** The changes introduce new endpoints (`/edit`, `/retry`) that modify database state. According to `CLAUDE.md`, the application has "No auth" and relies on being deployed in a trusted, LAN-only environment. These new endpoints increase the number of ways an unauthorized user on the network could alter data.
|
||||
* **Context:** This is not a new vulnerability, but an expansion of the existing trust model. As long as the application remains on a trusted network, this is acceptable per the project's documented security posture. It's worth keeping in mind that no CSRF protection is being used for these HTMX `POST` actions.
|
||||
|
||||
### Summary
|
||||
|
||||
The overall code quality is very high. The changes are thoughtfully implemented, with a clean separation between the CLI, the web layer, and a shared service layer. The refactoring in `web/app.py` to create `_render_source_detail` is a good move for maintainability. The addition of a thorough, live-DB test suite for the new service functions is commendable and demonstrates a commitment to quality.
|
||||
|
||||
Once the minor template logic issue is resolved, this change will be in perfect shape.
|
||||
@ -85,6 +85,28 @@ def _merge(default: dict, override: dict) -> dict:
|
||||
return out
|
||||
|
||||
|
||||
def _overlay_sections(base: dict, overlay: dict) -> dict:
|
||||
"""Section-level overlay: one level deep.
|
||||
|
||||
For each key in `overlay`, if both `base[key]` and `overlay[key]` are
|
||||
dicts, merge their keys (overlay wins). Otherwise the overlay value
|
||||
replaces the base value outright. Sections present in only one of
|
||||
the two pass through untouched.
|
||||
|
||||
Use this — not `_merge` — when combining `settings.toml` with
|
||||
`settings.local.toml`, so a local `[database] url = "..."` doesn't
|
||||
silently wipe other `[database]` keys defined in the base file.
|
||||
"""
|
||||
out = dict(base)
|
||||
for key, ov_val in (overlay or {}).items():
|
||||
base_val = out.get(key)
|
||||
if isinstance(base_val, dict) and isinstance(ov_val, dict):
|
||||
out[key] = {**base_val, **ov_val}
|
||||
else:
|
||||
out[key] = ov_val
|
||||
return out
|
||||
|
||||
|
||||
class Config:
|
||||
"""Holds resolved configuration values."""
|
||||
|
||||
@ -175,7 +197,24 @@ _config_instance: Config | None = None
|
||||
|
||||
|
||||
def load_config(path: Path | None = None) -> Config:
|
||||
"""Load (and cache) configuration from a TOML file."""
|
||||
"""Load (and cache) configuration from a TOML file.
|
||||
|
||||
After loading the base file, look for a sibling `settings.local.toml`
|
||||
and overlay it (section-level merge — see `_overlay_sections`). The
|
||||
local file is gitignored and intended for host-specific secrets
|
||||
(e.g. `[database].url` for a dev shell) without forking the tracked
|
||||
base.
|
||||
|
||||
Path resolution:
|
||||
- explicit `path=` arg wins
|
||||
- else `SECOND_BRAIN_CONFIG` env var
|
||||
- else walks up from CWD for `config/settings.toml` or `settings.toml`
|
||||
|
||||
The local-overlay sibling is looked up next to whichever base file
|
||||
was chosen, so `SECOND_BRAIN_CONFIG=/etc/sb/settings.toml` will also
|
||||
pick up `/etc/sb/settings.local.toml` if present. If no base file is
|
||||
found at all, no overlay is attempted.
|
||||
"""
|
||||
global _config_instance
|
||||
if _config_instance is not None:
|
||||
return _config_instance
|
||||
@ -197,6 +236,15 @@ def load_config(path: Path | None = None) -> Config:
|
||||
with open(path, "rb") as fh:
|
||||
raw = tomllib.load(fh)
|
||||
|
||||
# Section-level overlay from a sibling settings.local.toml.
|
||||
# Filename is the one already covered in .gitignore — host
|
||||
# secrets live here and never get committed.
|
||||
local_path = path.parent / "settings.local.toml"
|
||||
if local_path.exists():
|
||||
with open(local_path, "rb") as fh:
|
||||
local_raw = tomllib.load(fh)
|
||||
raw = _overlay_sections(raw, local_raw)
|
||||
|
||||
_config_instance = Config(raw)
|
||||
return _config_instance
|
||||
|
||||
|
||||
@ -4,6 +4,7 @@ second-brain CLI entry point.
|
||||
Commands:
|
||||
add — queue a source URL (video or article)
|
||||
process — run the full pipeline on pending/pulled/transcribed sources
|
||||
retry — reset a source so the pipeline re-processes it
|
||||
serve — launch the FastAPI web UI
|
||||
compile — run the wiki compiler on accepted sources
|
||||
schedule — start the overnight scheduler
|
||||
@ -307,6 +308,47 @@ def serve(host: str, port: int, reload: bool) -> None:
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# retry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@cli.command()
|
||||
@click.argument("source_id", type=int)
|
||||
@click.option(
|
||||
"--mode",
|
||||
type=click.Choice(["full", "extract"]),
|
||||
default="full",
|
||||
show_default=True,
|
||||
help="`full` resets to PENDING (re-pull + re-process); "
|
||||
"`extract` resets to TRANSCRIBED (requires existing transcript).",
|
||||
)
|
||||
def retry(source_id: int, mode: str) -> None:
|
||||
"""Reset a source so the pipeline re-processes it.
|
||||
|
||||
Clears the claim pair and error_message and rewinds status to the
|
||||
appropriate stage. Parity with the dashboard re-run buttons.
|
||||
"""
|
||||
from second_brain.sources_service import reset_source_for_retry
|
||||
|
||||
db = get_database()
|
||||
try:
|
||||
with db.session() as sess:
|
||||
source = reset_source_for_retry(
|
||||
sess, source_id=source_id, mode=mode
|
||||
)
|
||||
new_status = source.status.value
|
||||
url = source.url
|
||||
except LookupError as exc:
|
||||
click.echo(f"[retry] {exc}", err=True)
|
||||
sys.exit(2)
|
||||
except ValueError as exc:
|
||||
click.echo(f"[retry] {exc}", err=True)
|
||||
sys.exit(2)
|
||||
|
||||
click.echo(f"[retry] id={source_id} status → {new_status} ({url})")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# compile
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@ -275,14 +275,126 @@ def add_playlist(
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Edit + retry helpers (shared by web routes and CLI)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Fields the edit-mode form is allowed to mutate. URL is intentionally
|
||||
# excluded — it's the UNIQUE dedupe key and editing it risks collisions
|
||||
# plus breaks the embeddings 4-tuple identity (source_id is stable but
|
||||
# semantic identity drifts).
|
||||
EDITABLE_METADATA_FIELDS = ("domain", "title", "focus")
|
||||
|
||||
|
||||
def update_source_metadata(
|
||||
sess: Session,
|
||||
*,
|
||||
source_id: int,
|
||||
domain: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
focus: Optional[str] = None,
|
||||
) -> Source:
|
||||
"""Update editable metadata on a source row.
|
||||
|
||||
Empty strings collapse to NULL for title/focus (matches `add_source`
|
||||
semantics). Domain is required to stay in the configured set —
|
||||
bumping the constraint is a separate decision.
|
||||
|
||||
Raises ValueError on validation failures and LookupError if the row
|
||||
is missing. Caller owns the transaction boundary.
|
||||
"""
|
||||
source = sess.query(Source).filter(Source.id == source_id).first()
|
||||
if source is None:
|
||||
raise LookupError(f"source id={source_id} not found")
|
||||
|
||||
if domain is not None:
|
||||
if domain not in DOMAINS:
|
||||
raise ValueError(
|
||||
f"unknown domain {domain!r}; valid: {', '.join(DOMAINS)}"
|
||||
)
|
||||
source.domain = domain
|
||||
|
||||
if title is not None:
|
||||
source.title = title.strip() or None
|
||||
|
||||
if focus is not None:
|
||||
source.focus = focus.strip() or None
|
||||
|
||||
source.updated_at = utcnow()
|
||||
sess.flush()
|
||||
return source
|
||||
|
||||
|
||||
# Retry modes — keep the surface small. Anything else is a separate ask.
|
||||
RETRY_FULL = "full"
|
||||
RETRY_EXTRACT = "extract"
|
||||
RETRY_MODES = (RETRY_FULL, RETRY_EXTRACT)
|
||||
|
||||
|
||||
def reset_source_for_retry(
|
||||
sess: Session,
|
||||
*,
|
||||
source_id: int,
|
||||
mode: str = RETRY_FULL,
|
||||
) -> Source:
|
||||
"""Reset a source so the pipeline re-processes it.
|
||||
|
||||
Modes:
|
||||
- "full": status → PENDING. Article pull or video download +
|
||||
transcribe will happen again on the appropriate worker.
|
||||
- "extract": status → TRANSCRIBED. Requires the row to already
|
||||
carry a transcript (transcript_text). The dev scheduler will
|
||||
re-run extraction without re-pulling the source. Cheap.
|
||||
|
||||
Always clears `claimed_by` / `claimed_at` / `error_message` so the
|
||||
queue dance can pick it up cleanly and stale errors don't bleed
|
||||
into the UI.
|
||||
|
||||
Raises ValueError on bad mode / missing transcript for extract mode,
|
||||
and LookupError if the row is missing.
|
||||
"""
|
||||
if mode not in RETRY_MODES:
|
||||
raise ValueError(
|
||||
f"unknown retry mode {mode!r}; valid: {', '.join(RETRY_MODES)}"
|
||||
)
|
||||
|
||||
source = sess.query(Source).filter(Source.id == source_id).first()
|
||||
if source is None:
|
||||
raise LookupError(f"source id={source_id} not found")
|
||||
|
||||
if mode == RETRY_EXTRACT:
|
||||
if not (source.transcript_text and source.transcript_text.strip()):
|
||||
raise ValueError(
|
||||
"extract-only retry requires an existing transcript; "
|
||||
"use full retry instead"
|
||||
)
|
||||
source.status = SourceStatus.TRANSCRIBED
|
||||
else:
|
||||
source.status = SourceStatus.PENDING
|
||||
|
||||
source.claimed_by = None
|
||||
source.claimed_at = None
|
||||
source.error_message = None
|
||||
source.updated_at = utcnow()
|
||||
sess.flush()
|
||||
return source
|
||||
|
||||
|
||||
__all__ = [
|
||||
"AddResult",
|
||||
"DEFAULT_PLAYLIST_MAX_ITEMS",
|
||||
"EDITABLE_METADATA_FIELDS",
|
||||
"PlaylistAddResult",
|
||||
"RETRY_EXTRACT",
|
||||
"RETRY_FULL",
|
||||
"RETRY_MODES",
|
||||
"add_playlist",
|
||||
"add_source",
|
||||
"expand_youtube_playlist",
|
||||
"is_article_url",
|
||||
"is_youtube_playlist_url",
|
||||
"reset_source_for_retry",
|
||||
"update_source_metadata",
|
||||
"validate_url",
|
||||
]
|
||||
|
||||
@ -31,9 +31,14 @@ from second_brain.settings_store import (
|
||||
)
|
||||
from second_brain.sources_service import (
|
||||
DEFAULT_PLAYLIST_MAX_ITEMS,
|
||||
RETRY_EXTRACT,
|
||||
RETRY_FULL,
|
||||
RETRY_MODES,
|
||||
add_playlist,
|
||||
add_source,
|
||||
is_youtube_playlist_url,
|
||||
reset_source_for_retry,
|
||||
update_source_metadata,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -120,16 +125,33 @@ async def index(
|
||||
)
|
||||
|
||||
|
||||
@app.get("/sources/{source_id}", response_class=HTMLResponse)
|
||||
async def source_detail(request: Request, source_id: int):
|
||||
"""Source detail page with extraction results and accept/reject actions."""
|
||||
def _render_source_detail(
|
||||
request: Request,
|
||||
source_id: int,
|
||||
*,
|
||||
edit: bool = False,
|
||||
flash: Optional[str] = None,
|
||||
error: Optional[str] = None,
|
||||
form_overrides: Optional[dict] = None,
|
||||
) -> HTMLResponse:
|
||||
"""Single render path for /sources/{id} — used by GET + every HTMX
|
||||
mutation endpoint so each action gets a consistent re-render.
|
||||
|
||||
`form_overrides` lets a failed edit-save bounce show the user's
|
||||
in-progress values instead of the row's stored values.
|
||||
"""
|
||||
db = get_database()
|
||||
with db.session() as sess:
|
||||
source = sess.query(Source).filter(Source.id == source_id).first()
|
||||
if not source:
|
||||
raise HTTPException(status_code=404, detail="Source not found")
|
||||
source_data = _source_to_dict(source)
|
||||
extraction_data = _extraction_to_dict(source.extraction) if source.extraction else None
|
||||
extraction_data = (
|
||||
_extraction_to_dict(source.extraction) if source.extraction else None
|
||||
)
|
||||
|
||||
if form_overrides:
|
||||
source_data = {**source_data, **form_overrides}
|
||||
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
@ -137,10 +159,25 @@ async def source_detail(request: Request, source_id: int):
|
||||
{
|
||||
"source": source_data,
|
||||
"extraction": extraction_data,
|
||||
"edit_mode": edit,
|
||||
"flash": flash,
|
||||
"error": error,
|
||||
"domains": DOMAINS,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.get("/sources/{source_id}", response_class=HTMLResponse)
|
||||
async def source_detail(request: Request, source_id: int, edit: bool = False):
|
||||
"""Source detail page with extraction results and accept/reject actions.
|
||||
|
||||
`?edit=1` opens the metadata editor inline. Editing exposes domain,
|
||||
title, and focus — URL stays read-only since it's the UNIQUE dedupe
|
||||
key for the queue and the embeddings identity tuple.
|
||||
"""
|
||||
return _render_source_detail(request, source_id, edit=edit)
|
||||
|
||||
|
||||
_IN_FLIGHT_STATUSES = [
|
||||
SourceStatus.PENDING,
|
||||
SourceStatus.PULLED,
|
||||
@ -192,17 +229,11 @@ async def accept_source(request: Request, source_id: int):
|
||||
if not source:
|
||||
raise HTTPException(status_code=404, detail="Source not found")
|
||||
source.status = SourceStatus.ACCEPTED
|
||||
source_data = _source_to_dict(source)
|
||||
extraction_data = _extraction_to_dict(source.extraction) if source.extraction else None
|
||||
|
||||
return templates.TemplateResponse(
|
||||
return _render_source_detail(
|
||||
request,
|
||||
"source_detail.html",
|
||||
{
|
||||
"source": source_data,
|
||||
"extraction": extraction_data,
|
||||
"flash": "Accepted — will be included in next wiki compile.",
|
||||
},
|
||||
source_id,
|
||||
flash="Accepted — will be included in next wiki compile.",
|
||||
)
|
||||
|
||||
|
||||
@ -215,19 +246,80 @@ async def reject_source(request: Request, source_id: int):
|
||||
if not source:
|
||||
raise HTTPException(status_code=404, detail="Source not found")
|
||||
source.status = SourceStatus.FAILED
|
||||
source_data = _source_to_dict(source)
|
||||
extraction_data = _extraction_to_dict(source.extraction) if source.extraction else None
|
||||
|
||||
return templates.TemplateResponse(
|
||||
return _render_source_detail(request, source_id, flash="Rejected.")
|
||||
|
||||
|
||||
@app.post("/sources/{source_id}/edit", response_class=HTMLResponse)
|
||||
async def edit_source_metadata(
|
||||
request: Request,
|
||||
source_id: int,
|
||||
domain: str = Form(...),
|
||||
title: Optional[str] = Form(None),
|
||||
focus: Optional[str] = Form(None),
|
||||
):
|
||||
"""HTMX edit-mode save. Updates domain/title/focus only — URL is
|
||||
intentionally not editable (see service-layer comment)."""
|
||||
db = get_database()
|
||||
try:
|
||||
with db.session() as sess:
|
||||
update_source_metadata(
|
||||
sess,
|
||||
source_id=source_id,
|
||||
domain=domain,
|
||||
title=title,
|
||||
focus=focus,
|
||||
)
|
||||
except LookupError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except ValueError as exc:
|
||||
# Bounce back to edit mode with the user's in-progress values so
|
||||
# they don't have to retype them.
|
||||
return _render_source_detail(
|
||||
request,
|
||||
"source_detail.html",
|
||||
{
|
||||
"source": source_data,
|
||||
"extraction": extraction_data,
|
||||
"flash": "Rejected.",
|
||||
source_id,
|
||||
edit=True,
|
||||
error=str(exc),
|
||||
form_overrides={
|
||||
"domain": domain,
|
||||
"title": title or "",
|
||||
"focus": focus or "",
|
||||
},
|
||||
)
|
||||
|
||||
return _render_source_detail(request, source_id, flash="Saved.")
|
||||
|
||||
|
||||
@app.post("/sources/{source_id}/retry", response_class=HTMLResponse)
|
||||
async def retry_source(
|
||||
request: Request,
|
||||
source_id: int,
|
||||
mode: str = Form(RETRY_FULL),
|
||||
):
|
||||
"""HTMX retry/re-run. `mode=full` → PENDING (re-pull + re-process).
|
||||
`mode=extract` → TRANSCRIBED (re-extract only; requires an existing
|
||||
transcript). Always clears the claim pair and error_message."""
|
||||
if mode not in RETRY_MODES:
|
||||
raise HTTPException(status_code=400, detail=f"unknown mode {mode!r}")
|
||||
|
||||
db = get_database()
|
||||
try:
|
||||
with db.session() as sess:
|
||||
source = reset_source_for_retry(
|
||||
sess, source_id=source_id, mode=mode
|
||||
)
|
||||
new_status = source.status.value
|
||||
except LookupError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except ValueError as exc:
|
||||
return _render_source_detail(request, source_id, error=str(exc))
|
||||
|
||||
if mode == RETRY_EXTRACT:
|
||||
flash = f"Re-queued for extraction (status → {new_status})."
|
||||
else:
|
||||
flash = f"Re-queued from scratch (status → {new_status})."
|
||||
return _render_source_detail(request, source_id, flash=flash)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# JSON API endpoints
|
||||
@ -286,6 +378,7 @@ def _source_to_dict(s: Source) -> dict:
|
||||
"focus": s.focus,
|
||||
"ingested_at": s.ingested_at.strftime("%Y-%m-%d %H:%M") if s.ingested_at else "",
|
||||
"has_extraction": s.extraction is not None,
|
||||
"has_transcript": bool((s.transcript_text or "").strip()),
|
||||
"error_message": s.error_message,
|
||||
}
|
||||
|
||||
|
||||
@ -11,17 +11,74 @@
|
||||
</svg>
|
||||
Back to sources
|
||||
</a>
|
||||
<h2 class="text-2xl font-semibold text-gray-900">{{ source.title }}</h2>
|
||||
<div class="flex items-start justify-between gap-4">
|
||||
<div class="min-w-0">
|
||||
<h2 class="text-2xl font-semibold text-gray-900 break-words">{{ source.title }}</h2>
|
||||
<a href="{{ source.url }}" target="_blank"
|
||||
class="text-sm text-indigo-500 hover:text-indigo-700 break-all">{{ source.url }}</a>
|
||||
</div>
|
||||
{% if not edit_mode %}
|
||||
<a href="/sources/{{ source.id }}?edit=1"
|
||||
class="shrink-0 px-3 py-1.5 text-sm border border-gray-300 text-gray-700 rounded hover:bg-gray-50 transition-colors">
|
||||
Edit
|
||||
</a>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{% if flash %}
|
||||
<div class="mb-4 px-4 py-3 bg-green-50 border border-green-200 text-green-800 rounded text-sm">
|
||||
{{ flash }}
|
||||
</div>
|
||||
{% endif %}
|
||||
{% if error %}
|
||||
<div class="mb-4 px-4 py-3 bg-red-50 border border-red-200 text-red-800 rounded text-sm">
|
||||
{{ error }}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if edit_mode %}
|
||||
<!-- Edit mode: domain / title / focus. URL is intentionally read-only -->
|
||||
<form hx-post="/sources/{{ source.id }}/edit"
|
||||
hx-target="body"
|
||||
hx-swap="innerHTML"
|
||||
class="mb-8 bg-white rounded-lg border border-gray-200 p-5 space-y-4">
|
||||
<div>
|
||||
<label class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">URL <span class="font-normal normal-case text-gray-400">(read-only — dedupe key)</span></label>
|
||||
<input type="text" value="{{ source.url }}" disabled
|
||||
class="w-full px-3 py-2 border border-gray-200 bg-gray-50 text-gray-500 text-sm rounded">
|
||||
</div>
|
||||
<div>
|
||||
<label for="edit-title" class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">Title</label>
|
||||
<input type="text" id="edit-title" name="title" value="{{ source.title or '' }}"
|
||||
class="w-full px-3 py-2 border border-gray-300 text-sm rounded focus:outline-none focus:ring-2 focus:ring-indigo-300">
|
||||
</div>
|
||||
<div>
|
||||
<label for="edit-domain" class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">Domain</label>
|
||||
<select id="edit-domain" name="domain"
|
||||
class="w-full px-3 py-2 border border-gray-300 text-sm rounded focus:outline-none focus:ring-2 focus:ring-indigo-300">
|
||||
{% for d in domains %}
|
||||
<option value="{{ d }}" {% if d == source.domain %}selected{% endif %}>{{ d }}</option>
|
||||
{% endfor %}
|
||||
</select>
|
||||
</div>
|
||||
<div>
|
||||
<label for="edit-focus" class="block text-xs font-semibold text-gray-500 uppercase tracking-wide mb-1">Focus</label>
|
||||
<textarea id="edit-focus" name="focus" rows="3"
|
||||
class="w-full px-3 py-2 border border-gray-300 text-sm rounded focus:outline-none focus:ring-2 focus:ring-indigo-300">{{ source.focus or '' }}</textarea>
|
||||
</div>
|
||||
<div class="flex gap-2 pt-1">
|
||||
<button type="submit"
|
||||
class="px-4 py-2 bg-indigo-600 text-white text-sm font-semibold rounded hover:bg-indigo-700 transition-colors">
|
||||
Save
|
||||
</button>
|
||||
<a href="/sources/{{ source.id }}"
|
||||
class="px-4 py-2 border border-gray-300 text-gray-700 text-sm font-semibold rounded hover:bg-gray-50 transition-colors">
|
||||
Cancel
|
||||
</a>
|
||||
</div>
|
||||
</form>
|
||||
{% else %}
|
||||
<!-- Meta row -->
|
||||
<div class="flex flex-wrap gap-2 mb-6">
|
||||
{% set status_colors = {
|
||||
@ -60,9 +117,9 @@
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
<!-- Accept / Reject actions (only when analyzed) -->
|
||||
<!-- Actions row: accept/reject (when analyzed) + retry (always) -->
|
||||
<div class="flex flex-wrap gap-3 mb-8">
|
||||
{% if source.status == 'analyzed' %}
|
||||
<div class="flex gap-3 mb-8">
|
||||
<button hx-post="/sources/{{ source.id }}/accept"
|
||||
hx-target="body"
|
||||
hx-swap="innerHTML"
|
||||
@ -75,6 +132,31 @@
|
||||
class="px-5 py-2 bg-red-500 text-white text-sm font-semibold rounded hover:bg-red-600 transition-colors">
|
||||
Reject
|
||||
</button>
|
||||
{% endif %}
|
||||
|
||||
<!-- Re-run: full reset → PENDING. Re-pulls and re-processes from scratch. -->
|
||||
<button hx-post="/sources/{{ source.id }}/retry"
|
||||
hx-vals='{"mode": "full"}'
|
||||
hx-target="body"
|
||||
hx-swap="innerHTML"
|
||||
hx-confirm="Re-run this source from scratch? Status will reset to PENDING and the pipeline will re-pull{% if source.source_type == 'video' %} and re-transcribe (expensive){% endif %}."
|
||||
class="px-5 py-2 bg-amber-500 text-white text-sm font-semibold rounded hover:bg-amber-600 transition-colors">
|
||||
Re-run from scratch
|
||||
</button>
|
||||
|
||||
{% if source.has_transcript %}
|
||||
<!-- Light retry: skip the re-pull, just re-run LLM extraction. Works
|
||||
for any source with stored body text (videos have transcripts,
|
||||
articles have the trafilatura-extracted body). -->
|
||||
<button hx-post="/sources/{{ source.id }}/retry"
|
||||
hx-vals='{"mode": "extract"}'
|
||||
hx-target="body"
|
||||
hx-swap="innerHTML"
|
||||
hx-confirm="Re-extract only — keep the existing transcript/body and re-run the LLM extraction?"
|
||||
class="px-5 py-2 bg-amber-100 text-amber-800 text-sm font-semibold rounded hover:bg-amber-200 border border-amber-300 transition-colors">
|
||||
Re-extract only
|
||||
</button>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
|
||||
165
tests/test_config_overlay.py
Normal file
165
tests/test_config_overlay.py
Normal file
@ -0,0 +1,165 @@
|
||||
"""Tests for the settings.local.toml overlay loader.
|
||||
|
||||
The overlay is the dev-side path for keeping host-specific secrets
|
||||
(notably `[database].url`) out of the tracked `settings.toml` without
|
||||
forcing every interactive shell to export env vars. The footgun this
|
||||
test suite guards against: a one-key local override (e.g. a single
|
||||
`[database] url = "..."`) silently wiping every other key in that
|
||||
section. The merge is section-level, not shallow.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pure-Python: _overlay_sections
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_overlay_preserves_sibling_keys_in_overlapping_section():
|
||||
"""The exact regression target — local `url` must not wipe `pool_size`."""
|
||||
from second_brain.config import _overlay_sections
|
||||
|
||||
base = {"database": {"url": "", "pool_size": 5}}
|
||||
local = {"database": {"url": "postgresql://x"}}
|
||||
|
||||
out = _overlay_sections(base, local)
|
||||
assert out["database"]["url"] == "postgresql://x"
|
||||
assert out["database"]["pool_size"] == 5
|
||||
|
||||
|
||||
def test_overlay_section_only_in_base_survives():
|
||||
from second_brain.config import _overlay_sections
|
||||
|
||||
base = {"database": {"url": "a"}, "scheduler": {"interval": 30}}
|
||||
local = {"database": {"url": "b"}}
|
||||
|
||||
out = _overlay_sections(base, local)
|
||||
assert out["scheduler"] == {"interval": 30}
|
||||
assert out["database"]["url"] == "b"
|
||||
|
||||
|
||||
def test_overlay_section_only_in_local_appears():
|
||||
from second_brain.config import _overlay_sections
|
||||
|
||||
base = {"database": {"url": "a"}}
|
||||
local = {"embeddings": {"ollama_url": "http://x"}}
|
||||
|
||||
out = _overlay_sections(base, local)
|
||||
assert out["database"] == {"url": "a"}
|
||||
assert out["embeddings"] == {"ollama_url": "http://x"}
|
||||
|
||||
|
||||
def test_overlay_scalar_in_overlay_replaces_dict_in_base():
|
||||
"""Non-dict overlay values replace outright — no merge attempt."""
|
||||
from second_brain.config import _overlay_sections
|
||||
|
||||
base = {"feature": {"a": 1, "b": 2}}
|
||||
local = {"feature": "off"} # scalar wins outright
|
||||
|
||||
out = _overlay_sections(base, local)
|
||||
assert out["feature"] == "off"
|
||||
|
||||
|
||||
def test_overlay_dict_in_overlay_replaces_scalar_in_base():
|
||||
from second_brain.config import _overlay_sections
|
||||
|
||||
base = {"feature": "off"}
|
||||
local = {"feature": {"a": 1}}
|
||||
|
||||
out = _overlay_sections(base, local)
|
||||
assert out["feature"] == {"a": 1}
|
||||
|
||||
|
||||
def test_overlay_empty_overlay_is_identity():
|
||||
from second_brain.config import _overlay_sections
|
||||
|
||||
base = {"database": {"url": "a", "pool_size": 5}}
|
||||
assert _overlay_sections(base, {}) == base
|
||||
assert _overlay_sections(base, None) == base
|
||||
|
||||
|
||||
def test_overlay_only_merges_one_level():
|
||||
"""Doc: the overlay is section-level. Nested dicts inside a section
|
||||
replace, not deep-merge. This pins the documented depth."""
|
||||
from second_brain.config import _overlay_sections
|
||||
|
||||
base = {"db": {"url": "a", "pool": {"min": 1, "max": 10}}}
|
||||
local = {"db": {"pool": {"max": 20}}}
|
||||
|
||||
out = _overlay_sections(base, local)
|
||||
# pool replaced wholesale — `min` does NOT survive. This is deliberate.
|
||||
assert out["db"]["pool"] == {"max": 20}
|
||||
assert out["db"]["url"] == "a"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# load_config: end-to-end with real TOML files on disk
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _write(path: Path, body: str) -> None:
|
||||
path.write_text(body, encoding="utf-8")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_config():
|
||||
"""Drop the cached singleton before/after each test so overlay state
|
||||
doesn't leak between cases."""
|
||||
from second_brain.config import reset_config_singleton
|
||||
|
||||
reset_config_singleton()
|
||||
yield
|
||||
reset_config_singleton()
|
||||
|
||||
|
||||
def test_load_config_picks_up_sibling_local_overlay(tmp_path):
|
||||
from second_brain.config import load_config
|
||||
|
||||
base = tmp_path / "settings.toml"
|
||||
local = tmp_path / "settings.local.toml"
|
||||
_write(
|
||||
base,
|
||||
'[database]\nurl = ""\n\n[embeddings]\nollama_url = "http://ollama:11434"\nmodel = "nomic-embed-text"\n',
|
||||
)
|
||||
_write(
|
||||
local,
|
||||
'[database]\nurl = "postgresql+psycopg://lovebug:x@127.0.0.1:5433/petalbrain"\n\n[embeddings]\nollama_url = "http://127.0.0.1:11434"\n',
|
||||
)
|
||||
|
||||
cfg = load_config(base)
|
||||
# url from local wins; default model survives (not in local).
|
||||
assert cfg.database["url"].startswith("postgresql+psycopg://")
|
||||
assert cfg.embeddings["ollama_url"] == "http://127.0.0.1:11434"
|
||||
assert cfg.embeddings["model"] == "nomic-embed-text"
|
||||
|
||||
|
||||
def test_load_config_no_local_overlay_is_inert(tmp_path):
|
||||
from second_brain.config import load_config
|
||||
|
||||
base = tmp_path / "settings.toml"
|
||||
_write(base, '[database]\nurl = "from-base"\n')
|
||||
|
||||
cfg = load_config(base)
|
||||
assert cfg.database["url"] == "from-base"
|
||||
|
||||
|
||||
def test_load_config_local_overlay_at_explicit_path(tmp_path):
|
||||
"""Doc: when an explicit (or SECOND_BRAIN_CONFIG) path is used, the
|
||||
overlay sibling is resolved next to THAT path — not next to the
|
||||
repo's config dir."""
|
||||
from second_brain.config import load_config
|
||||
|
||||
subdir = tmp_path / "etc"
|
||||
subdir.mkdir()
|
||||
base = subdir / "settings.toml"
|
||||
local = subdir / "settings.local.toml"
|
||||
_write(base, '[database]\nurl = "base"\n')
|
||||
_write(local, '[database]\nurl = "overlay-wins"\n')
|
||||
|
||||
cfg = load_config(base)
|
||||
assert cfg.database["url"] == "overlay-wins"
|
||||
258
tests/test_source_edit_retry.py
Normal file
258
tests/test_source_edit_retry.py
Normal file
@ -0,0 +1,258 @@
|
||||
"""Tests for the source edit-metadata + retry-reset service helpers.
|
||||
|
||||
Live-DB pattern matches `tests/test_claim.py` / `tests/test_playlist.py`:
|
||||
both groups auto-skip when no petalbrain connection is configured. The
|
||||
service layer is where edit/retry logic lives (the web routes + CLI are
|
||||
thin wrappers around it), so this is where the meaningful coverage is.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import uuid
|
||||
|
||||
import psycopg
|
||||
import pytest
|
||||
|
||||
|
||||
def _db_reachable() -> bool:
|
||||
url = os.environ.get("SECOND_BRAIN_DATABASE_URL") or os.environ.get(
|
||||
"HERBYLAB_DATABASE_URL"
|
||||
)
|
||||
if not url:
|
||||
return False
|
||||
raw = url.replace("postgresql+psycopg://", "postgresql://", 1)
|
||||
try:
|
||||
with psycopg.connect(raw, connect_timeout=2):
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not _db_reachable(),
|
||||
reason="needs SECOND_BRAIN_DATABASE_URL pointed at a real petalbrain",
|
||||
)
|
||||
|
||||
|
||||
def _seed_source(db, **overrides):
|
||||
"""Insert a Source row and return (id, url). Caller cleans up by id."""
|
||||
from second_brain.models import Source, SourceStatus, SourceType, utcnow
|
||||
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
defaults = dict(
|
||||
url=f"https://example.test/edit-retry/{tag}",
|
||||
title=f"seed-{tag}",
|
||||
domain="development",
|
||||
source_type=SourceType.ARTICLE,
|
||||
status=SourceStatus.PENDING,
|
||||
ingested_at=utcnow(),
|
||||
)
|
||||
defaults.update(overrides)
|
||||
with db.session() as sess:
|
||||
src = Source(**defaults)
|
||||
sess.add(src)
|
||||
sess.flush()
|
||||
return src.id, src.url
|
||||
|
||||
|
||||
def _cleanup(db, source_id):
|
||||
from second_brain.models import Source
|
||||
|
||||
with db.session() as sess:
|
||||
sess.query(Source).filter(Source.id == source_id).delete(
|
||||
synchronize_session=False
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# update_source_metadata
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_update_metadata_writes_domain_title_focus():
|
||||
from second_brain.database import Database, reset_database_singleton
|
||||
from second_brain.models import Source
|
||||
from second_brain.sources_service import update_source_metadata
|
||||
|
||||
reset_database_singleton()
|
||||
db = Database()
|
||||
sid, _ = _seed_source(db)
|
||||
try:
|
||||
with db.session() as sess:
|
||||
update_source_metadata(
|
||||
sess,
|
||||
source_id=sid,
|
||||
domain="homelab",
|
||||
title="new title",
|
||||
focus="new focus directive",
|
||||
)
|
||||
with db.session() as sess:
|
||||
row = sess.query(Source).filter(Source.id == sid).first()
|
||||
assert row.domain == "homelab"
|
||||
assert row.title == "new title"
|
||||
assert row.focus == "new focus directive"
|
||||
assert row.updated_at is not None
|
||||
finally:
|
||||
_cleanup(db, sid)
|
||||
reset_database_singleton()
|
||||
|
||||
|
||||
def test_update_metadata_blank_strings_become_null():
|
||||
from second_brain.database import Database, reset_database_singleton
|
||||
from second_brain.models import Source
|
||||
from second_brain.sources_service import update_source_metadata
|
||||
|
||||
reset_database_singleton()
|
||||
db = Database()
|
||||
sid, _ = _seed_source(db)
|
||||
try:
|
||||
with db.session() as sess:
|
||||
update_source_metadata(
|
||||
sess, source_id=sid, domain="development", title=" ", focus=""
|
||||
)
|
||||
with db.session() as sess:
|
||||
row = sess.query(Source).filter(Source.id == sid).first()
|
||||
assert row.title is None
|
||||
assert row.focus is None
|
||||
finally:
|
||||
_cleanup(db, sid)
|
||||
reset_database_singleton()
|
||||
|
||||
|
||||
def test_update_metadata_rejects_unknown_domain():
|
||||
from second_brain.database import Database, reset_database_singleton
|
||||
from second_brain.sources_service import update_source_metadata
|
||||
|
||||
reset_database_singleton()
|
||||
db = Database()
|
||||
sid, _ = _seed_source(db)
|
||||
try:
|
||||
with db.session() as sess:
|
||||
with pytest.raises(ValueError, match="unknown domain"):
|
||||
update_source_metadata(sess, source_id=sid, domain="not-a-domain")
|
||||
finally:
|
||||
_cleanup(db, sid)
|
||||
reset_database_singleton()
|
||||
|
||||
|
||||
def test_update_metadata_missing_source_raises_lookup():
|
||||
from second_brain.database import Database, reset_database_singleton
|
||||
from second_brain.sources_service import update_source_metadata
|
||||
|
||||
reset_database_singleton()
|
||||
db = Database()
|
||||
try:
|
||||
with db.session() as sess:
|
||||
with pytest.raises(LookupError):
|
||||
update_source_metadata(
|
||||
sess, source_id=-99999, domain="development"
|
||||
)
|
||||
finally:
|
||||
reset_database_singleton()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# reset_source_for_retry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_retry_full_resets_to_pending_and_clears_claim():
|
||||
from second_brain.database import Database, reset_database_singleton
|
||||
from second_brain.models import Source, SourceStatus, utcnow
|
||||
from second_brain.sources_service import reset_source_for_retry
|
||||
|
||||
reset_database_singleton()
|
||||
db = Database()
|
||||
sid, _ = _seed_source(
|
||||
db,
|
||||
status=SourceStatus.FAILED,
|
||||
error_message="boom",
|
||||
claimed_by="worker-x",
|
||||
claimed_at=utcnow(),
|
||||
)
|
||||
try:
|
||||
with db.session() as sess:
|
||||
reset_source_for_retry(sess, source_id=sid, mode="full")
|
||||
with db.session() as sess:
|
||||
row = sess.query(Source).filter(Source.id == sid).first()
|
||||
assert row.status == SourceStatus.PENDING
|
||||
assert row.claimed_by is None
|
||||
assert row.claimed_at is None
|
||||
assert row.error_message is None
|
||||
finally:
|
||||
_cleanup(db, sid)
|
||||
reset_database_singleton()
|
||||
|
||||
|
||||
def test_retry_extract_sets_transcribed_when_transcript_present():
|
||||
from second_brain.database import Database, reset_database_singleton
|
||||
from second_brain.models import Source, SourceStatus
|
||||
from second_brain.sources_service import reset_source_for_retry
|
||||
|
||||
reset_database_singleton()
|
||||
db = Database()
|
||||
sid, _ = _seed_source(
|
||||
db,
|
||||
status=SourceStatus.ANALYZED,
|
||||
transcript_text="word " * 50,
|
||||
error_message="prior failure",
|
||||
)
|
||||
try:
|
||||
with db.session() as sess:
|
||||
reset_source_for_retry(sess, source_id=sid, mode="extract")
|
||||
with db.session() as sess:
|
||||
row = sess.query(Source).filter(Source.id == sid).first()
|
||||
assert row.status == SourceStatus.TRANSCRIBED
|
||||
assert row.error_message is None
|
||||
assert row.transcript_text # transcript preserved
|
||||
finally:
|
||||
_cleanup(db, sid)
|
||||
reset_database_singleton()
|
||||
|
||||
|
||||
def test_retry_extract_without_transcript_raises():
|
||||
from second_brain.database import Database, reset_database_singleton
|
||||
from second_brain.models import SourceStatus
|
||||
from second_brain.sources_service import reset_source_for_retry
|
||||
|
||||
reset_database_singleton()
|
||||
db = Database()
|
||||
sid, _ = _seed_source(db, status=SourceStatus.PENDING, transcript_text=None)
|
||||
try:
|
||||
with db.session() as sess:
|
||||
with pytest.raises(ValueError, match="requires an existing transcript"):
|
||||
reset_source_for_retry(sess, source_id=sid, mode="extract")
|
||||
finally:
|
||||
_cleanup(db, sid)
|
||||
reset_database_singleton()
|
||||
|
||||
|
||||
def test_retry_unknown_mode_raises():
|
||||
from second_brain.database import Database, reset_database_singleton
|
||||
from second_brain.sources_service import reset_source_for_retry
|
||||
|
||||
reset_database_singleton()
|
||||
db = Database()
|
||||
sid, _ = _seed_source(db)
|
||||
try:
|
||||
with db.session() as sess:
|
||||
with pytest.raises(ValueError, match="unknown retry mode"):
|
||||
reset_source_for_retry(sess, source_id=sid, mode="bogus")
|
||||
finally:
|
||||
_cleanup(db, sid)
|
||||
reset_database_singleton()
|
||||
|
||||
|
||||
def test_retry_missing_source_raises_lookup():
|
||||
from second_brain.database import Database, reset_database_singleton
|
||||
from second_brain.sources_service import reset_source_for_retry
|
||||
|
||||
reset_database_singleton()
|
||||
db = Database()
|
||||
try:
|
||||
with db.session() as sess:
|
||||
with pytest.raises(LookupError):
|
||||
reset_source_for_retry(sess, source_id=-99999, mode="full")
|
||||
finally:
|
||||
reset_database_singleton()
|
||||
Loading…
Reference in New Issue
Block a user