second-brain/deploy/tower/README.md

Tower-side transcribe worker — deploy notes

Target host: EndeavourOS tower (Arch base), RTX 3080. Runs the second-brain transcribe-worker daemon under systemd. Reaches the petalbrain Postgres over WireGuard at db.wg.herbylab.dev:5432.

This worker only does pull + transcribe for VIDEO sources. Articles and all extraction/embedding stay on the dev side.

---

1. System packages

# Build / runtime tooling.
sudo pacman -S --needed base-devel git ffmpeg python uv

# NVIDIA driver only — see the CUDA-12 gotcha below before installing
# the distro's `cuda` / `cudnn` packages.
sudo pacman -S --needed nvidia nvidia-utils

# Confirm the GPU is visible.
nvidia-smi

⚠ CUDA-12 gotcha — do NOT install pacman's cuda / cudnn

Arch / EndeavourOS now ships CUDA 13 (libcublas.so.13). The ctranslate2 4.7.x backend that faster-whisper 1.x rides on top of wants CUDA 12 (libcublas.so.12) + cuDNN 9 (libcudnn.so.9) and will not load against the system's CUDA-13 libs.

Rather than fight the distro, this project pins the CUDA-12 runtime as pip wheels in the venv via the tower extra (nvidia-cublas-cu12, nvidia-cudnn-cu12>=9,<10). The uv sync --extra tower step in §2 brings them in automatically; the systemd ExecStart wrapper (deploy/tower/run-transcribe-worker.sh) prepends those wheel-bundled .so directories to LD_LIBRARY_PATH at start time.

If you've already installed pacman's cuda / cudnn, you can leave them — the wrapper's LD_LIBRARY_PATH prepend wins for this service.

yt-dlp comes in via the Python project (uv sync step below), so the distro yt-dlp is optional. Leave it off the host unless you want it on your CLI PATH.

---

2. Project checkout + venv

sudo mkdir -p /opt/projects/homelab
sudo chown trucktrav:trucktrav /opt/projects/homelab
cd /opt/projects/homelab
git clone gitea-pbs-trucktrav:petal-power/second-brain.git
cd second-brain

# Install the runtime deps + the `tower` extra (faster-whisper / CTranslate2).
# The dev side does NOT install --extra tower; only the tower needs it.
uv sync --extra tower

uv sync --extra tower will pull ~1 GB of pinned CUDA-12 wheels (nvidia-cublas-cu12, nvidia-cudnn-cu12, plus faster-whisper / ctranslate2). First-run model download (~3 GB for large-v3) lands under ~/.cache/huggingface/ the first time the worker transcribes; pre-warm if you want:

# Run the wrapper so the CUDA-12 lib paths are pinned exactly the way
# the systemd service will see them — this is the cheapest end-to-end
# sanity check that the venv + LD_LIBRARY_PATH plumbing is correct.
deploy/tower/run-transcribe-worker.sh --once

Or just instantiate the model directly:

LD_LIBRARY_PATH="$(uv run python -c 'import os, nvidia.cublas, nvidia.cudnn; print(":".join(os.path.join(os.path.dirname(m.__file__),"lib") for m in [nvidia.cublas, nvidia.cudnn]))')" \
    uv run python -c "from faster_whisper import WhisperModel; WhisperModel('large-v3', device='cuda', compute_type='float16')"

---

3. WireGuard

The worker assumes the tunnel is already configured and that wg-quick@wg-lan.service brings it up at boot. Topology:

Peer	WG IP
hub (herbydev)	10.99.0.1
tower	10.99.0.3

db.wg.herbylab.dev resolves to 10.99.0.1. Confirm connectivity from the tower:

ping -c2 db.wg.herbylab.dev
nc -vz db.wg.herbylab.dev 5432

Tunnel setup itself is out of scope for this README — that's configured separately as part of Travis's infra.

---

4. EnvironmentFile

systemd reads this BEFORE dropping to User=herbyadmin. The file is not a shell init script — export VAR=… in your shell does not apply here, and changes require systemctl restart to take effect.

There's a ready-to-scp file on herbys-dev with the real lovebug password already populated, generated from /opt/backups/postgres-consolidation/credentials.env:

/opt/backups/postgres-consolidation/second-brain-transcribe.env   (mode 0600)

Pull it across instead of hand-copying the password:

# from the tower
scp herbys-dev:/opt/backups/postgres-consolidation/second-brain-transcribe.env /tmp/sb-env
sudo install -m 0600 -o root -g root /tmp/sb-env /etc/default/second-brain-transcribe
rm /tmp/sb-env

If you ever need to regenerate the dev-host file (e.g. after a password rotation), the recipe is:

# on herbys-dev — never echoes the password, mode 0600
PW=$(grep ^LOVEBUG_PG_PASSWORD= /opt/backups/postgres-consolidation/credentials.env | cut -d= -f2-)
umask 077
echo "SECOND_BRAIN_DATABASE_URL=postgresql://lovebug:${PW}@db.wg.herbylab.dev:5432/petalbrain" \
    > /opt/backups/postgres-consolidation/second-brain-transcribe.env
unset PW

The committed deploy/tower/second-brain-transcribe.env.example documents the format and never carries a real secret.

---

5. systemd unit

sudo install -m 0644 deploy/tower/second-brain-transcribe.service \
    /etc/systemd/system/second-brain-transcribe.service
sudo systemctl daemon-reload
sudo systemctl enable --now second-brain-transcribe.service

The unit declares After= + Wants= on wg-quick@wg-lan.service, so the worker starts after the tunnel comes up at boot. If the tunnel disappears mid-run the worker keeps polling and logs DB-unreachable warnings; it doesn't crash-loop.

ExecStart= calls the in-repo deploy/tower/run-transcribe-worker.sh wrapper, which resolves the venv's CUDA-12 lib dirs at runtime and prepends them to LD_LIBRARY_PATH before exec-ing uv run. systemd does NOT inherit your shell's LD_LIBRARY_PATH, so the wrapper is how the CUDA-12 pin survives across reboots and python upgrades.

Day-to-day commands

# What's it doing right now?
systemctl status second-brain-transcribe.service
journalctl -u second-brain-transcribe.service -f

# Pause without uninstalling — toggle from the dashboard at
# https://brain.herbylab.dev/settings  (or whichever host).
# Or stop the unit:
sudo systemctl stop second-brain-transcribe.service

# Pick up code updates:
cd /opt/projects/homelab/second-brain
git pull
uv sync --extra tower
sudo systemctl restart second-brain-transcribe.service

---

6. Validation — what to confirm once WG is live

These can only be verified on the tower itself:

1. WG reachability. nc -vz db.wg.herbylab.dev 5432 succeeds.
2. DB auth. uv run alembic current returns the latest revision id.
3. GPU large-v3 path. With a short test video queued via second-brain add <url> on the dev side, run uv run second-brain transcribe-worker --once on the tower. Watch journalctl — first run pulls the model from HuggingFace (slow), subsequent runs are fast. The source row should end up in TRANSCRIBED with transcript_text populated and claimed_by cleared.
4. Race safety smoke test. Run two --once invocations in parallel against an empty queue, then with one PENDING video. Only one should claim it; the other should see no work.
5. Settings round-trip. Flip transcription_enabled off on the dashboard; within one poll cycle (~15s) the worker logs transcription_enabled=false — sleeping. Flip back on, the next PENDING video gets picked up.

---

Follow-ups (out of scope for this round)

• Media + subtitles directory currently lives on the tower's local disk (under ~/.local/share/second-brain/). Move to a NAS mount later so the dev side can compile/inspect the SRTs without a cross-machine fetch. Open question on the right mount point + cred flow — see the project's postgres-migration-planning.md for the related vault-location decision Travis still owes.
• Tower-side observability. No metrics endpoint yet; journald is the only signal. A /api/worker-status heartbeat the dashboard could read would be nice once two workers exist.
• GPU concurrency > 1. transcription_max_concurrent_gpu_jobs exists in pipeline_settings but the worker only runs one job at a time. Scale-out would need a per-job semaphore (or just running N worker instances with distinct SECOND_BRAIN_WORKER_IDs).