second-brain/deploy/tower/README.md

Tower-side transcribe worker — deploy notes

Target host: EndeavourOS tower (Arch base), RTX 3080. Runs the second-brain transcribe-worker daemon under systemd. Reaches the petalbrain Postgres over WireGuard at db.wg.herbylab.dev:5432.

This worker only does pull + transcribe for VIDEO sources. Articles and all extraction/embedding stay on the dev side.

---

1. System packages

# Build / runtime tooling.
sudo pacman -S --needed base-devel git ffmpeg python uv

# NVIDIA — driver, CUDA, cuDNN. faster-whisper / CTranslate2 needs cuDNN
# at runtime; missing-symbol errors at first transcribe are usually a
# cuDNN install gap. Use whichever driver/cuda channel matches your
# Arch derivative; on stock EndeavourOS the AUR/community packages work:
sudo pacman -S --needed nvidia nvidia-utils cuda cudnn

# Confirm the GPU is visible.
nvidia-smi

yt-dlp comes in via the Python project (uv sync step below), so the distro yt-dlp is optional. Leave it off the host unless you want it on your CLI PATH.

---

2. Project checkout + venv

sudo mkdir -p /opt/projects/homelab
sudo chown herbyadmin:herbyadmin /opt/projects/homelab
cd /opt/projects/homelab
git clone gitea-lovebug:petal-power/second-brain.git
cd second-brain

# Install the runtime deps + the `tower` extra (faster-whisper / CTranslate2).
# The dev side does NOT install --extra tower; only the tower needs it.
uv sync --extra tower

faster-whisper will pull a couple hundred MB of CUDA wheels. First-run model download (~3 GB for large-v3) lands under ~/.cache/huggingface/ the first time the worker transcribes; pre-warm it if you want:

uv run python -c "from faster_whisper import WhisperModel; WhisperModel('large-v3', device='cuda', compute_type='float16')"

---

3. WireGuard

The worker assumes the tunnel is already configured and that wg-quick@wg-lan.service brings it up at boot. Topology:

Peer	WG IP
hub (herbydev)	10.99.0.1
tower	10.99.0.2

db.wg.herbylab.dev resolves to 10.99.0.1. Confirm connectivity from the tower:

ping -c2 db.wg.herbylab.dev
nc -vz db.wg.herbylab.dev 5432

Tunnel setup itself is out of scope for this README — that's configured separately as part of Travis's infra.

---

4. EnvironmentFile

sudo install -m 0600 -o root -g root \
    deploy/tower/second-brain-transcribe.env.example \
    /etc/default/second-brain-transcribe
sudo $EDITOR /etc/default/second-brain-transcribe   # paste the lovebug password

The lovebug password lives in /opt/backups/postgres-consolidation/credentials.env on the dev host. Copy it across through whatever channel you trust (personal Bitwarden, scp over WG, etc.) — do not commit it.

---

5. systemd unit

sudo install -m 0644 deploy/tower/second-brain-transcribe.service \
    /etc/systemd/system/second-brain-transcribe.service
sudo systemctl daemon-reload
sudo systemctl enable --now second-brain-transcribe.service

The unit declares After= + Wants= on wg-quick@wg-lan.service, so the worker starts after the tunnel comes up at boot. If the tunnel disappears mid-run the worker keeps polling and logs DB-unreachable warnings; it doesn't crash-loop.

Day-to-day commands

# What's it doing right now?
systemctl status second-brain-transcribe.service
journalctl -u second-brain-transcribe.service -f

# Pause without uninstalling — toggle from the dashboard at
# https://brain.herbylab.dev/settings  (or whichever host).
# Or stop the unit:
sudo systemctl stop second-brain-transcribe.service

# Pick up code updates:
cd /opt/projects/homelab/second-brain
git pull
uv sync --extra tower
sudo systemctl restart second-brain-transcribe.service

---

6. Validation — what to confirm once WG is live

These can only be verified on the tower itself:

1. WG reachability. nc -vz db.wg.herbylab.dev 5432 succeeds.
2. DB auth. uv run alembic current returns the latest revision id.
3. GPU large-v3 path. With a short test video queued via second-brain add <url> on the dev side, run uv run second-brain transcribe-worker --once on the tower. Watch journalctl — first run pulls the model from HuggingFace (slow), subsequent runs are fast. The source row should end up in TRANSCRIBED with transcript_text populated and claimed_by cleared.
4. Race safety smoke test. Run two --once invocations in parallel against an empty queue, then with one PENDING video. Only one should claim it; the other should see no work.
5. Settings round-trip. Flip transcription_enabled off on the dashboard; within one poll cycle (~15s) the worker logs transcription_enabled=false — sleeping. Flip back on, the next PENDING video gets picked up.

---

Follow-ups (out of scope for this round)

• Media + subtitles directory currently lives on the tower's local disk (under ~/.local/share/second-brain/). Move to a NAS mount later so the dev side can compile/inspect the SRTs without a cross-machine fetch. Open question on the right mount point + cred flow — see the project's postgres-migration-planning.md for the related vault-location decision Travis still owes.
• Tower-side observability. No metrics endpoint yet; journald is the only signal. A /api/worker-status heartbeat the dashboard could read would be nice once two workers exist.
• GPU concurrency > 1. transcription_max_concurrent_gpu_jobs exists in pipeline_settings but the worker only runs one job at a time. Scale-out would need a per-job semaphore (or just running N worker instances with distinct SECOND_BRAIN_WORKER_IDs).