Two commits since the last merge:
- web: add-to-queue form on /queue too, with per-page HTMX dispatch.
Shared form partial reused by /dashboard and /queue; the POST
/sources/add endpoint dispatches the response partial by inspecting
HX-Current-URL so each page's own list/counts refresh inline. Same
service call, same flash behavior.
- tower: pin CUDA-12 wheels + LD_LIBRARY_PATH wrapper for the systemd
service. ctranslate2 4.7.2 needs CUDA 12 + cuDNN 9 but Arch ships
CUDA 13, so we pin nvidia-cublas-cu12 + nvidia-cudnn-cu12>=9,<10 in
the `tower` extra and add deploy/tower/run-transcribe-worker.sh,
which resolves the venv's lib dirs at runtime and prepends them to
LD_LIBRARY_PATH before exec-ing the worker. systemd doesn't inherit
shell exports, so this is what makes the service survive reboots
and Python upgrades. ExecStart= now points at the wrapper, and the
StartLimit* directives moved into [Unit] where modern systemd
expects them.
Travis's tower-side corrections on main (`User=trucktrav`,
`gitea-pbs-trucktrav` clone alias, tower WG IP 10.99.0.3) auto-merge
cleanly against these branch edits — different lines in the same files.
Arch / EndeavourOS now ships CUDA 13 (libcublas.so.13); ctranslate2
4.7.2 (which faster-whisper rides) wants CUDA 12 + cuDNN 9 and won't
load against the system libs. Travis got it working ad-hoc with a
shell export of LD_LIBRARY_PATH + manual pip install, but systemd
doesn't inherit either, so the next reboot would refire the
libcublas.so.12 load error.
Making it permanent + reproducible:
- pyproject `tower` extra now pins the CUDA-12 runtime as pip wheels
alongside faster-whisper:
nvidia-cublas-cu12; sys_platform == 'linux'
nvidia-cudnn-cu12>=9,<10; sys_platform == 'linux'
uv.lock resolves nvidia-cublas-cu12 12.9.2.10 + nvidia-cudnn-cu12
9.22.0.52. Dev side (no --extra tower) stays clean — verified by a
no-extra `uv sync` followed by `uv pip list | grep nvidia` returning
empty.
- deploy/tower/run-transcribe-worker.sh (new, +x): computes the venv's
CUDA-12 lib dirs at runtime via `uv run python` (resolving
nvidia.cublas / nvidia.cudnn through __path__ — they're PEP 420
namespace packages with no __file__), prepends them to
LD_LIBRARY_PATH, then execs `uv run second-brain transcribe-worker`.
No hard-coded python3.XX path so it survives Python upgrades. If the
wheels aren't installed it aborts with a clear "uv sync --extra
tower" hint instead of a silent libcublas load failure deep inside
ctranslate2.
- second-brain-transcribe.service: ExecStart now points at the
wrapper. Also moves StartLimitIntervalSec / StartLimitBurst from
[Service] into [Unit] where modern systemd expects them
(systemd-analyze verify previously flagged the misplaced keys as
silently ignored). Restart=always, EnvironmentFile, After=/Wants=
wg-quick@wg-lan.service, User=herbyadmin all unchanged.
- second-brain-transcribe.env.example: trimmed to just
SECOND_BRAIN_DATABASE_URL with the placeholder spelled out, plus a
clear pointer to the ready-to-scp env file generated on herbys-dev
at /opt/backups/postgres-consolidation/second-brain-transcribe.env
(mode 0600, regeneratable from credentials.env without ever echoing
the password). The committed example never carries a real secret.
- deploy/tower/README.md: documents the CUDA-13-vs-CUDA-12 gotcha
upfront ("don't `pacman -S cuda cudnn`"), the wrapper-based
ExecStart, the scp-from-dev EnvironmentFile recipe with the
password-regen one-liner, and the EnvironmentFile-vs-shell-export
note.
Verified locally on dev (no GPU):
- uv.lock resolves with the new tower deps.
- A throwaway venv installed with the same `nvidia-cublas-cu12
nvidia-cudnn-cu12>=9,<10` pins produces lib dirs containing
libcublas.so.12 and libcudnn.so.9 via the wrapper's path probe.
- systemd-analyze verify is clean except the expected
"/opt/projects/... not executable on this host" warning (the
wrapper exists only in the tower's checkout).
- 27 passed / 2 skipped in pytest; zero-check 5/5.
GPU large-v3 + live service start under systemd remain tower-only
validation steps.
Travis's original ask put the form on the "queue page" but it landed
only on /dashboard. Putting it on /queue too without duplicating any
logic:
- _add_to_queue_form.html — extracted the form markup into a shared
partial. Callers set `{% set swap_target = "#<id>-body" %}` before
including it; that's the only knob that differs between the two
pages. Same POST endpoint, same field set, same flash behaviour.
- _dashboard_body.html — now {% include %}s the shared partial with
swap_target="#dashboard-body". Net change: a six-line include
replacing the inline form.
- _queue_body.html (new) — `<div id="queue-body">` wrapping the shared
form (swap_target="#queue-body") + the in-flight source list (same
card markup as the Sources view, with the wrap-fix already applied).
Empty state copy matches the queue context.
- queue.html (new) — extends base.html, page-titles "Queue", and
includes _queue_body.html. /queue no longer reuses index.html.
- /queue route — renders queue.html via a new _build_queue_context
helper. Same in-flight filter as before (PENDING/PULLED/TRANSCRIBED).
- POST /sources/add — reads HX-Current-URL (HTMX sends it on every
request; falls back to Referer) and picks the response partial:
/queue → _queue_body.html, anything else → _dashboard_body.html. One
endpoint, one service call, two render branches — neither side
reaches into the other's state.
Live-verified through the rebuilt container:
- GET /queue (LAN + brain.herbylab.dev) — form present.
- GET /dashboard — form still present (unchanged behaviour).
- GET / — form still absent (Sources view stays clean).
- POST from /queue → response carries id="queue-body" (and not
dashboard-body); the freshly-added row appears in the swapped list.
- POST from /dashboard → response carries id="dashboard-body".
- Flash scenarios from /queue: queued / duplicate / playlist 13 videos
/ validation error — all four render correctly.
The four follow-up commits since the last merge:
- web: dashboard "Add to queue" form with HTMX save. Service-layer
add_source extracted so the CLI and the new POST share validation,
dedupe, and source_type detection — no behaviour drift.
- web: source-list cards drop `truncate` in favour of break-words /
break-all so long titles + long URLs wrap and the card grows. Status
/ domain / timestamp badges stay aligned via min-w-0 + flex-1 on the
left column.
- transcripts: v4 migration adds sources.transcript_segments JSONB.
Tower worker now persists faster-whisper's segment-level output
(start/end/text + word_timestamps when available) into the new
column alongside transcript_text (which stays canonical for the
extractor). Best-effort: a malformed JSONB write doesn't abort the
status=TRANSCRIBED commit.
- playlists: queue-time YouTube playlist fan-out. is_youtube_playlist_url
strictly matches /playlist?list=… on a known YouTube host (a
watch?v=…&list=… URL stays a single-add, documented choice). yt-dlp
extract_flat enumerates entries; add_playlist loops them through
add_source so the existing UNIQUE dedupe path handles repeats. CLI
and web both auto-detect — no flag needed. Default ceiling 50
entries.
Pasting a YouTube playlist URL into either entry point now expands
into one source row per video. Single-video URLs and non-YouTube URLs
keep their existing behaviour untouched.
Service module additions:
- is_youtube_playlist_url(url): strict detector. Only `/playlist?list=…`
on a known YouTube host (youtube.com / m / music / no-www) counts.
A `watch?v=…&list=…` URL is ambiguous (user usually pasted a single
video that happens to sit inside a playlist) and intentionally falls
through to single-add. To fan out, paste the canonical playlist URL.
- expand_youtube_playlist(url, *, max_items=50): yt-dlp with
extract_flat=True, playlistend=max_items, skip_download. Builds a
canonical https://www.youtube.com/watch?v={id} URL per entry and
silently drops placeholders for private/removed videos.
- add_playlist(sess, *, url, domain, focus, max_items, expander=None):
loops expansion entries through add_source so URL validation,
source_type detection, and the UNIQUE dedupe path stay identical to
the single-add flow. Per-entry titles win over any caller-supplied
title (a single playlist title would be wrong for N videos). Partial
failures don't abort the batch — failed entries are tallied with
up-to-10 (url, reason) tuples for the flash. `expander` is an
injection seam for tests so the suite never hits YouTube unless
explicitly opted in.
- DEFAULT_PLAYLIST_MAX_ITEMS = 50 — shared ceiling, no throttle change.
CLI: `second-brain add <playlist-url>` auto-detects and reports
`expanded / added / duplicates / failed`. No new flag needed.
Web: POST /sources/add same detection. _playlist_flash() builds the
HTMX flash — "Queued N videos (M duplicates skipped, F failed)"
with sensible plural forms and graceful omission of zero counters.
Tests:
- 15 pure-Python detection cases (positives + negatives, including the
ambiguous watch?v=…&list=… rule).
- 3 DB-backed add_playlist tests (with a mocked expander, so no
network): count aggregation across new + pre-seeded duplicates,
bad-entry tolerance, and the empty-playlist case.
- 1 opt-in live-network test gated on SECOND_BRAIN_LIVE_NETWORK_TESTS=1
exercising expand_youtube_playlist against a real public playlist.
Live-verified end to end:
- web POST of a real 13-entry public playlist queued 13 video rows
with titles, flash showed "Queued 13 videos".
- re-POST returned "Queued 0 videos (13 duplicates skipped)".
- watch?v=…&list=… correctly stayed a single-add.
- CLI parity confirmed against the same playlist.
Tower worker now persists faster-whisper's segment-level output
(start/end/text + word-level timing when available) alongside the
existing joined `transcript_text`. The text column stays the canonical
input the extractor reads — this is additive.
Changes:
- alembic v4: sources.transcript_segments JSONB NULL. JSONB rather than
JSON so future equality/containment queries are indexable without a
re-migration. Same lovebug-no-CREATE-on-petalbrain guard as prior
migrations.
- ORM model: Optional[list] mapped to JSONB (postgresql dialect).
- transcribe.py:
- Always pass word_timestamps=True to faster-whisper.transcribe.
- New segment_to_dict() flattens the upstream NamedTuple-shaped
Segment/Word into JSON-safe plain dicts so the JSONB write doesn't
drag faster-whisper into any reader.
- Per-word defensive conversion: a single malformed word can't drop
the surrounding segment.
- transcribe_worker._advance: after a successful transcribe, persist
segments into source.transcript_segments inside a try/except. If the
JSONB write fails (oversize row, malformed dict, etc.) we log a
warning and still commit transcript_text + status=TRANSCRIBED — the
pipeline never crashes over the additive index.
- Tests: three new unit tests against fake Segment/Word objects cover
the happy path (word entries serialise), the no-words case
(`segment.words is None` → empty list), and the malformed-word skip.
json.dumps(d) asserts JSONB-binding compatibility.
Live-verified: migration applied clean against petalbrain (`\d sources`
shows transcript_segments jsonb); ORM round-trip writes and reads the
sample payload identically. GPU large-v3 word-timestamp behaviour is
unchanged from upstream — only the tower can validate that hot path.
Travis prefers the cards to auto-resize to their content rather than
ellipsis-clip the title + URL into a single line. Two places use the
same card markup — both updated:
- src/second_brain/web/templates/index.html — main sources list.
- src/second_brain/web/templates/_dashboard_body.html — recent activity
on the dashboard.
Changes are minimal and identical on both:
- title h3/p: drop `truncate`, add `break-words` so normal long titles
wrap at word boundaries.
- URL p: drop `truncate`, add `break-all` because URLs are typically
one unbreakable token and `break-words` alone wouldn't split them.
- `min-w-0 flex-1` on the left container kept — it lets the flex child
shrink so the right column's status / domain / timestamp badges stay
pinned and never get pushed off-screen by a long URL.
- The optional Focus line also gets `break-words` defensively.
Verified live on the rebuilt web container:
- /dashboard: 0 truncate hits in the response, RepoWise URL renders
intact with break-all applied.
- /?status=analyzed: 0 truncate hits, title + URL both carry the
new wrap classes.
Extracts the `second-brain add` CLI's queueing logic into a service
module (sources_service.add_source) so the CLI and the new web POST
route share the same validation, dedupe, and source_type heuristic —
no behaviour drift between the two entry points.
UI:
- The dashboard body (Pipeline counts + Settings snapshot + Recent
activity) moves into _dashboard_body.html, wrapped in
`<div id="dashboard-body">`. HTMX targets that id for swap.
- A new "Add to queue" section sits at the top of the partial: URL
(required, type=url), Domain (select matching DOMAINS), Title and
Focus (both optional, match the CLI flags). hx-post=/sources/add,
hx-target=#dashboard-body, hx-swap=outerHTML — same pattern as the
settings save form.
Route:
- POST /sources/add calls sources_service.add_source inside a single
transaction, then re-renders _dashboard_body.html so the pipeline
counts and recent-activity list update in place. Flash slots above
the form report:
✓ Queued <type>: <url> on a fresh add
✓ Already queued (id=…, status=…) on a dupe (matches CLI text)
✗ <validation message> on bad URL / unknown domain
CLI:
- `second-brain add` now delegates to the same service. Field values
for the success echo are captured inside the session block so a
post-commit detached-instance access can't fail. Bad input exits 2
with a clear stderr message instead of raising.
Live-verified through the web container on herbys-dev: dashboard
renders the form, happy add lands a row, dup-detect matches the CLI
phrasing, two validation errors surface as red flashes, swap target
id survives across swaps, no tracebacks in uvicorn logs.
Dockerfile, compose.yml, env template, and runbook for the second-brain
web container. Targets herbys-dev (10.0.21.207); Traefik (file-provider
on 10.0.11.20) reaches the published host port at 10.0.21.207:8080.
Image:
- python:3.12-slim + the official uv binary copied from the upstream
image, plus apt-installed git + ca-certificates so the Gitea VCS pin
for embedding-chunking resolves at build time.
- uv sync --frozen --no-dev --no-install-project, source copy, then a
second uv sync to install the project itself. Two-step so the lock
install layer caches independently of source edits.
- No ffmpeg / claude CLI / faster-whisper — web role doesn't need any
of them. Extraction runs on the dev host's CLI; transcription on the
tower.
- Drops to uid 1000 (`app`) before CMD. Uvicorn binds 0.0.0.0:8000
inside the container, with --proxy-headers + --forwarded-allow-ips=*
so Traefik's X-Forwarded-* survive.
compose.yml:
- Joins the existing external `homelab` bridge network so the container
reaches homelab-postgres:5432 and ollama:11434 by service DNS.
- Publishes the uvicorn port at 10.0.21.207:8080 (LAN IP bound, not
0.0.0.0) for Traefik on the separate VM to reach. NO traefik.* labels
— file-provider Traefik can't read them.
- env_file: .env (0600, gitignored) — SECOND_BRAIN_DATABASE_URL points
at homelab-postgres:5432 (containerised), NOT the host's 127.0.0.1:5433
port-map.
- restart: unless-stopped.
README.md:
- Build + bring-up commands.
- SECURITY note: no SSO / no CSRF / mutating endpoints — Traefik route
must be LAN-only for now (Travis's call).
- The two infra steps Travis applies himself, with ready-to-paste
snippets:
- Knot DNS: brain.herbylab.dev → 10.0.11.20.
- Traefik dynamic config: file-provider router + service block.
- Verification checklist for both before-and-after-DNS states.
Live-verified on herbys-dev: container Up, dashboard returns 200 with
real status counts from petalbrain, settings save round-trip works,
no tracebacks in logs.
Leftover ffmpeg-era imports from the earlier draft of the CPU transcribe
smoke test; the final numpy-array-bypass version doesn't need them.
Ruff autofix.
deploy/tower/:
- second-brain-transcribe.service — systemd unit. User=herbyadmin,
Type=simple, After=/Wants= wg-quick@wg-lan.service so the WG tunnel
must come up first. Restart=always with a StartLimitBurst guard.
- second-brain-transcribe.env.example — env file template documenting
the SECOND_BRAIN_DATABASE_URL form for db.wg.herbylab.dev (10.99.0.1)
and the optional WHISPER_* overrides.
- README.md — EndeavourOS install steps (nvidia/cuda/cudnn, ffmpeg, uv
+ tower extra, model pre-warm), WG topology reference, validation
checklist for what to confirm once the tunnel is live, and a
follow-ups section flagging the local-disk → NAS media migration as
out-of-scope-for-this-round.
Tests:
- tests/test_claim.py — live-DB race test. Two threads call
claim_next_source against a single PULLED video row; SKIP LOCKED
must give exactly one of them the row, the other gets None. Also
asserts the claimed_by/at columns land + release nulls them.
Auto-skips when no SECOND_BRAIN_DATABASE_URL is set.
- tests/test_transcribe.py — pure-Python coverage of resolve_settings
(cpu→int8, cuda→float16, env-over-block) and write_srt; plus a CPU
smoke test that synthesizes a numpy audio array and runs the `tiny`
model on cpu/int8 (auto-skipped when faster-whisper isn't installed,
i.e. on the dev side without --extra tower).
Splits pull+transcribe (now tower-side, eager) from extract+embed
(stays on the dev scheduler). Three machine-coordination pieces land
together because they reference each other:
- v3 migration adds sources.claimed_by + claimed_at — observability +
stale-claim recovery columns. The actual race-safety primitive is
`SELECT ... FOR UPDATE SKIP LOCKED` in the new claim helper, so two
machines can poll the queue without doubling work.
- src/second_brain/claim.py owns the claim dance (claim_next_source,
release_claim, reap_stale_claims). Both stage gates filter by
source_type so the tower never grabs articles and the dev side never
grabs videos.
- src/second_brain/transcribe.py wraps faster-whisper (lazy-imported so
it stays out of the dev install). resolve_settings() reads env >
[whisper] block > defaults, falling back to int8 on cpu / float16 on
cuda when compute_type is unspecified. Default model large-v3.
- src/second_brain/scheduler/transcribe_worker.py is the long-running
poll loop. Reads pipeline_settings every iteration so the dashboard's
enable/window/max-items/max-video-length take effect within one
cycle. Reaps stale claims at startup. SIGTERM-clean. DB-unreachable
backs off with a log line; never crash-loops.
- adapters/youtube.py drops the torch-whisper transcribe path; pull
stays. Removes openai-whisper from the default deps and gates
faster-whisper behind a new `tower` extra (uv sync --extra tower).
- main.py: new `second-brain transcribe-worker` (--once for ad-hoc).
`process` now article-only on the pull side but still picks up
TRANSCRIBED of any source_type for the extract step.
Live-verified: migration applies clean, transcribe-worker --once
honors transcription_enabled=false gate.
The dev-side scheduler now snapshots the singleton pipeline_settings row
at the top of run() and applies three DB-driven gates:
- extraction_enabled = false → skip the entire run with a single log line.
Lets Travis pause extraction from the dashboard without touching the
config file or restarting anything.
- extraction_active_hours_start/_end → fast-fail if outside the window
AND (when set) override the static settings.toml [scheduler].window_*
bounds so the inner loop also respects the dashboard's choice. Either
side being NULL means "no constraint on that side" — matches the form's
"leave blank = always active" semantics.
- extraction_max_items_per_run → hard cap on processed items per
invocation. 0/null means unlimited (existing behavior).
Snapshot semantics are deliberate: a mid-run toggle doesn't half-apply,
same way max_calls_per_hour is tracked locally. Workers re-read on the
next invocation.
Adds three new routes on the FastAPI web app:
- GET /dashboard — pipeline counts by status (per-status drill-down
via filtered /?status=… links), recent activity
(latest 10 sources), and a settings snapshot
card with an "edit →" shortcut.
- GET /settings — full editing form for pipeline_settings.
- POST /settings/save — HTMX endpoint that validates the partial form,
upserts the row, and returns the rerendered
card with a "Saved." or error flash. The card
is its own _settings_card.html partial so the
HTMX swap targets only the form region.
Form parsing lives in settings_store helpers (parse_time_or_none,
parse_int_or_none) — keeps the route thin and the empty-string-to-NULL
normalisation in one place. Validation rejects negative counts and
sub-1 GPU concurrency.
The base template grew a Dashboard + Settings nav so the new routes are
reachable without typing URLs. Added python-multipart to deps because
FastAPI's Form(...) raises at app import without it.
Introduces second_brain.pipeline_settings — the single-row config row
the upcoming web dashboard edits and the workers read at the start of
each run. Pinned to id=1 by a CHECK constraint so upserts-by-PK keep
the table singleton, and the migration seeds the row with the table's
column defaults via INSERT ... ON CONFLICT DO NOTHING.
Two consumer groups carved out:
- transcription_* fields persist now; future tower-side worker reads them.
- extraction_* fields will be read by the existing scheduler in the next
commit, which is the actual behavior change Travis cares about today.
The settings_store helper centralises get/update + form-parsing
(time-of-day, int-or-none) and active-window math so the routes and the
scheduler don't reimplement them.
Without this, every `second-brain process` run prints
couldn't stop thread 'pool-1-worker-N' within 5.0 seconds
during interpreter shutdown — psycopg_pool's background workers don't
get a clean stop signal before Python's threading shutdown deadline.
Registering close_pool via atexit the first time we open the pool fixes
it without changing any API. `close_pool` is already idempotent, so the
explicit teardown path in the smoke test (which calls it directly) and
the atexit path coexist safely.
a-review (gemini, full migration diff) flagged that the smoke test had
the embedding model name baked into the SQL count query, decoupling it
from the application's configured value. Read it back from
`config.embeddings.model` (env override still wins) so the test stays
valid if the default ever moves off nomic-embed-text.
CLAUDE.md updates:
- new src/second_brain/embeddings/ entry in the project layout
- setup section now lists the postgres-superuser bootstrap (CREATE SCHEMA
AUTHORIZATION lovebug) and the alembic step
- env var table now covers SECOND_BRAIN_DATABASE_URL, OLLAMA_URL,
EMBEDDING_MODEL, and the pool sizing knobs
- conventions section gained five new gotchas (the lovebug CREATE gap,
public.embeddings ownership, best-effort embedding, etc.)
postgres-migration-planning.md flipped from "planning questions" to a
"what shipped" runbook — locked decisions table, operator setup steps,
known gotchas, and the next-iteration backlog.
Ruff autofix removed pre-existing F401 unused imports across adapters,
context, compiler, extractor, scheduler, web, and main. Also reordered
models.py so its SQLAlchemy imports sit at the top of the file
(E402 was triggered by the inline `utcnow` helper definition).
Added a [dependency-groups] dev block (pytest, pytest-cov, ruff) so the
zero-check test gauntlet — which runs `pytest --cov --cov-report=term-missing`
— can resolve its plugins without a manual `uv pip install pytest-cov`.
The runtime role (lovebug) doesn't have CREATE on the petalbrain database
even though it owns the second_brain schema, so a bare
`CREATE SCHEMA IF NOT EXISTS` errors out with permission denied. Gate
the bootstrap on a pg_namespace lookup so we only attempt the create
when the schema is genuinely missing — operators bootstrap it once as
postgres superuser, alembic just respects it afterward.
The smoke test exercises the full Postgres + embedding path against a
live DB + Ollama (autoskipped otherwise): writes an extraction, embeds
the summary, asserts public.embeddings has the expected row count, and
re-embeds to verify the delete-before-insert idempotency.
Swap the SQLite backing store for petalbrain Postgres + pgvector, modeled
on vault-mcp. All second-brain relational tables now live in the
`second_brain` schema (owned by the lovebug role); embeddings are written
to the shared public.embeddings table.
Locked design decisions (per Travis):
- DB: existing petalbrain Postgres, second_brain schema, lovebug role.
- Connection: containerized homelab-postgres:5432, plain psycopg_pool
(min=1/max=10), no PgBouncer.
- ORM stays SQLAlchemy; int autoincrement PKs + naive UTC DateTime.
- Embeddings: reuse shared public.embeddings keyed by
(source_schema='second_brain', source_table='extractions', source_id,
model='nomic-embed-text'). Summaries only for this round.
- Pipeline: chunk_text → Ollama nomic-embed-text → delete-before-insert
upsert, with graceful degradation (no DB / no Ollama → log + skip).
- Alembic stands up second-brain's own schema; public.embeddings stays
out-of-band.
- File-based wiki compiler is unchanged.
No SQLite data import — starting clean.
This commit is the scaffolding only; `alembic upgrade head` and a smoke
test of the embedding path are the next checkpoint.
Covers the usual Python noise (__pycache__, .venv, tooling caches),
common secrets paths (.env, config/settings.local.toml), editor/OS
junk, and stray SQLite files in case anyone points db_path at a
repo-local path. Nothing to scrub from history — none of these were
ever tracked.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Run extraction under the Max OAuth subscription via `claude -p` instead
of the per-token Anthropic API. The new src/second_brain/llm/claude_cli.py
spawns the CLI in a hermetic tempdir so the host project's CLAUDE.md,
hooks, MCP config, and settings don't leak into the prompt. Uses
--json-schema with LLMExtraction.model_json_schema() so the CLI guarantees
valid structured output — replaces the brittle markdown-fence stripping
in the old engine. The Anthropic SDK is preserved as an optional "api"
backend selectable via config.
While in here, fix a handful of blockers that the smoke test surfaced:
- scheduler filtered ANALYZED instead of TRANSCRIBED, so it never
actually advanced any sources
- process command read sources in a closed session, raising
DetachedInstanceError before any work happened
- config.prompts_dir walked one parent too many, resolving outside
the project and forcing the fallback prompt for every domain
- compiler called git rev-parse against a vault that was never
git-init'd; now auto-inits with an empty seed commit and skips empty
commits cleanly
- datetime.utcnow() deprecated in 3.12+ — single utcnow() helper in
models.py keeps naive UTC semantics so no DB migration is needed
- sess.query(...).get() deprecated in SA 2.x → sess.get(...)
- dead `import anthropic` removed from compiler
Smoke test (article → process → accept → compile) succeeds end-to-end
with ANTHROPIC_API_KEY unset. a-review run saved under reviews/.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
