petal-power/second-brain
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
repo for storing the second brain project
24 Commits 4 Branches 0 Tags 1.1 MiB
Python 82.4%
HTML 15.5%
Shell 1%
Dockerfile 0.9%
Mako 0.2%
b2e2359651
Go to file
Travis Herbranson b2e2359651 tower: pin CUDA-12 wheels + LD_LIBRARY_PATH wrapper for the systemd service 
Arch / EndeavourOS now ships CUDA 13 (libcublas.so.13); ctranslate2
4.7.2 (which faster-whisper rides) wants CUDA 12 + cuDNN 9 and won't
load against the system libs. Travis got it working ad-hoc with a
shell export of LD_LIBRARY_PATH + manual pip install, but systemd
doesn't inherit either, so the next reboot would refire the
libcublas.so.12 load error.

Making it permanent + reproducible:

- pyproject `tower` extra now pins the CUDA-12 runtime as pip wheels
  alongside faster-whisper:
    nvidia-cublas-cu12; sys_platform == 'linux'
    nvidia-cudnn-cu12>=9,<10; sys_platform == 'linux'
  uv.lock resolves nvidia-cublas-cu12 12.9.2.10 + nvidia-cudnn-cu12
  9.22.0.52. Dev side (no --extra tower) stays clean — verified by a
  no-extra `uv sync` followed by `uv pip list | grep nvidia` returning
  empty.

- deploy/tower/run-transcribe-worker.sh (new, +x): computes the venv's
  CUDA-12 lib dirs at runtime via `uv run python` (resolving
  nvidia.cublas / nvidia.cudnn through __path__ — they're PEP 420
  namespace packages with no __file__), prepends them to
  LD_LIBRARY_PATH, then execs `uv run second-brain transcribe-worker`.
  No hard-coded python3.XX path so it survives Python upgrades. If the
  wheels aren't installed it aborts with a clear "uv sync --extra
  tower" hint instead of a silent libcublas load failure deep inside
  ctranslate2.

- second-brain-transcribe.service: ExecStart now points at the
  wrapper. Also moves StartLimitIntervalSec / StartLimitBurst from
  [Service] into [Unit] where modern systemd expects them
  (systemd-analyze verify previously flagged the misplaced keys as
  silently ignored). Restart=always, EnvironmentFile, After=/Wants=
  wg-quick@wg-lan.service, User=herbyadmin all unchanged.

- second-brain-transcribe.env.example: trimmed to just
  SECOND_BRAIN_DATABASE_URL with the placeholder spelled out, plus a
  clear pointer to the ready-to-scp env file generated on herbys-dev
  at /opt/backups/postgres-consolidation/second-brain-transcribe.env
  (mode 0600, regeneratable from credentials.env without ever echoing
  the password). The committed example never carries a real secret.

- deploy/tower/README.md: documents the CUDA-13-vs-CUDA-12 gotcha
  upfront ("don't `pacman -S cuda cudnn`"), the wrapper-based
  ExecStart, the scp-from-dev EnvironmentFile recipe with the
  password-regen one-liner, and the EnvironmentFile-vs-shell-export
  note.

Verified locally on dev (no GPU):
- uv.lock resolves with the new tower deps.
- A throwaway venv installed with the same `nvidia-cublas-cu12
  nvidia-cudnn-cu12>=9,<10` pins produces lib dirs containing
  libcublas.so.12 and libcudnn.so.9 via the wrapper's path probe.
- systemd-analyze verify is clean except the expected
  "/opt/projects/... not executable on this host" warning (the
  wrapper exists only in the tower's checkout).
- 27 passed / 2 skipped in pytest; zero-check 5/5.

GPU large-v3 + live service start under systemd remain tower-only
validation steps.
2026-05-25 15:39:13 -04:00
alembic transcripts: capture segment-level output into a new JSONB column 2026-05-25 13:40:49 -04:00
config postgres migration: schema, models, embeddings, alembic 2026-05-24 22:46:48 -04:00
deploy tower: pin CUDA-12 wheels + LD_LIBRARY_PATH wrapper for the systemd service 2026-05-25 15:39:13 -04:00
prompts project init 2026-05-22 19:08:22 -04:00
reviews docs: update CLAUDE.md + migration plan to reflect shipped Postgres design 2026-05-24 22:57:31 -04:00
src/second_brain web: add-to-queue form on /queue too, with per-page HTMX dispatch 2026-05-25 14:21:02 -04:00
tests playlists: queue-time YouTube fan-out via yt-dlp extract_flat 2026-05-25 13:59:35 -04:00
.gitignore add .gitignore 2026-05-24 21:10:53 -04:00
alembic.ini postgres migration: schema, models, embeddings, alembic 2026-05-24 22:46:48 -04:00
CLAUDE.md docs: update CLAUDE.md + migration plan to reflect shipped Postgres design 2026-05-24 22:57:31 -04:00
postgres-migration-planning.md docs: update CLAUDE.md + migration plan to reflect shipped Postgres design 2026-05-24 22:57:31 -04:00
pyproject.toml tower: pin CUDA-12 wheels + LD_LIBRARY_PATH wrapper for the systemd service 2026-05-25 15:39:13 -04:00
README.md updates to project files 2026-05-24 22:23:59 -04:00
uv.lock tower: pin CUDA-12 wheels + LD_LIBRARY_PATH wrapper for the systemd service 2026-05-25 15:39:13 -04:00

README.md

second-brain

A personal knowledge system built on a video/article extraction pipeline and an LLM-maintained wiki (inspired by the Karpathy LLM Wiki pattern).

What it does

  1. Pull — download YouTube videos or fetch web articles
  2. Transcribe — extract transcripts via Whisper (videos) or trafilatura (articles)
  3. Extract — single-shot Claude call produces structured notes per source
  4. Review — web UI to accept or reject extractions
  5. Compile — wiki compiler folds accepted extractions into a living Obsidian vault

Quick start

# Install dependencies
uv sync

# Copy and edit config
cp config/settings.toml config/settings.local.toml

# Queue a source
uv run second-brain add https://www.youtube.com/watch?v=...

# Run the pipeline
uv run second-brain process

# Review in the web UI
uv run second-brain serve

# Compile to wiki
uv run second-brain compile

Domains

Extractions are tagged by domain so the right prompt template is used:

Domain Focus
development Software, programming, systems
content Content creation, video production
business Entrepreneurship, marketing, ops
homelab Self-hosted infra, networking, DevOps

Extractor backends

Two backends, selectable via [extractor].backend in config/settings.toml:

  • cli (default) — shells out to claude -p --output-format json --json-schema .... Runs under your Max OAuth subscription, no per-token billing. The wrapper spawns the CLI in a throwaway tempfile.TemporaryDirectory and strips ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN from the env so host CLAUDE.md, hooks, and settings don't leak in, and the subscription is used instead of the API key.
  • api — uses the anthropic Python SDK, requires ANTHROPIC_API_KEY. Install with uv sync --extra api (the SDK is an optional dependency).

Requirements

  • Python 3.12+
  • Either the claude CLI on PATH (default backend) or ANTHROPIC_API_KEY (api backend)
  • ffmpeg (for Whisper audio extraction)

In-flight work

  • Postgres + pgvector migration — SQLite is the current backing store but the project is moving to Travis's Postgres instance. Planning questions live in postgres-migration-planning.md; nothing has been migrated yet.