second-brain/deploy/tower/README.md

# Tower-side transcribe worker — deploy notes

Target host: **EndeavourOS tower** (Arch base), RTX 3080. Runs the
`second-brain transcribe-worker` daemon under systemd. Reaches the
petalbrain Postgres over WireGuard at `db.wg.herbylab.dev:5432`.

This worker only does **pull + transcribe** for VIDEO sources. Articles
and all extraction/embedding stay on the dev side.

---

## 1. System packages

```bash
# Build / runtime tooling.
sudo pacman -S --needed base-devel git ffmpeg python uv

# NVIDIA driver only — see the CUDA-12 gotcha below before installing
# the distro's `cuda` / `cudnn` packages.
sudo pacman -S --needed nvidia nvidia-utils

# Confirm the GPU is visible.
nvidia-smi
```

### ⚠ CUDA-12 gotcha — do NOT install pacman's `cuda` / `cudnn`

Arch / EndeavourOS now ships **CUDA 13** (`libcublas.so.13`). The
`ctranslate2` 4.7.x backend that `faster-whisper` 1.x rides on top of
wants **CUDA 12** (`libcublas.so.12`) + **cuDNN 9** (`libcudnn.so.9`)
and will not load against the system's CUDA-13 libs.

Rather than fight the distro, this project pins the CUDA-12 runtime as
pip wheels in the venv via the `tower` extra (`nvidia-cublas-cu12`,
`nvidia-cudnn-cu12>=9,<10`). The `uv sync --extra tower` step in §2
brings them in automatically; the systemd `ExecStart` wrapper
(`deploy/tower/run-transcribe-worker.sh`) prepends those wheel-bundled
`.so` directories to `LD_LIBRARY_PATH` at start time.

If you've already installed pacman's `cuda` / `cudnn`, you can leave
them — the wrapper's `LD_LIBRARY_PATH` prepend wins for this service.

yt-dlp comes in via the Python project (`uv sync` step below), so the
distro yt-dlp is *optional*. Leave it off the host unless you want it on
your CLI PATH.

---

## 2. Project checkout + venv

```bash
sudo mkdir -p /opt/projects/homelab
sudo chown trucktrav:trucktrav /opt/projects/homelab
cd /opt/projects/homelab
git clone gitea-pbs-trucktrav:petal-power/second-brain.git
cd second-brain

# Install the runtime deps + the `tower` extra (faster-whisper / CTranslate2).
# The dev side does NOT install --extra tower; only the tower needs it.
uv sync --extra tower
```

`uv sync --extra tower` will pull ~1 GB of pinned CUDA-12 wheels
(`nvidia-cublas-cu12`, `nvidia-cudnn-cu12`, plus `faster-whisper` /
`ctranslate2`). First-run model download (~3 GB for large-v3) lands
under `~/.cache/huggingface/` the first time the worker transcribes;
pre-warm if you want:

```bash
# Run the wrapper so the CUDA-12 lib paths are pinned exactly the way
# the systemd service will see them — this is the cheapest end-to-end
# sanity check that the venv + LD_LIBRARY_PATH plumbing is correct.
deploy/tower/run-transcribe-worker.sh --once
```

Or just instantiate the model directly:

```bash
LD_LIBRARY_PATH="$(uv run python -c 'import os, nvidia.cublas, nvidia.cudnn; print(":".join(os.path.join(os.path.dirname(m.__file__),"lib") for m in [nvidia.cublas, nvidia.cudnn]))')" \
    uv run python -c "from faster_whisper import WhisperModel; WhisperModel('large-v3', device='cuda', compute_type='float16')"
```

---

## 3. WireGuard

The worker assumes the tunnel is already configured and that
`wg-quick@wg-lan.service` brings it up at boot. Topology:

| Peer | WG IP |
|------|-------|
| hub (herbydev) | `10.99.0.1` |
| tower | `10.99.0.3` |

`db.wg.herbylab.dev` resolves to `10.99.0.1`. Confirm connectivity from
the tower:

```bash
ping -c2 db.wg.herbylab.dev
nc -vz db.wg.herbylab.dev 5432
```

Tunnel setup itself is **out of scope for this README** — that's
configured separately as part of Travis's infra.

---

## 4. EnvironmentFile

systemd reads this BEFORE dropping to `User=herbyadmin`. The file is
**not** a shell init script — `export VAR=…` in your shell does not
apply here, and changes require `systemctl restart` to take effect.

There's a ready-to-scp file on **herbys-dev** with the real lovebug
password already populated, generated from
`/opt/backups/postgres-consolidation/credentials.env`:

```
/opt/backups/postgres-consolidation/second-brain-transcribe.env   (mode 0600)
```

Pull it across instead of hand-copying the password:

```bash
# from the tower
scp herbys-dev:/opt/backups/postgres-consolidation/second-brain-transcribe.env /tmp/sb-env
sudo install -m 0600 -o root -g root /tmp/sb-env /etc/default/second-brain-transcribe
rm /tmp/sb-env
```

If you ever need to regenerate the dev-host file (e.g. after a password
rotation), the recipe is:

```bash
# on herbys-dev — never echoes the password, mode 0600
PW=$(grep ^LOVEBUG_PG_PASSWORD= /opt/backups/postgres-consolidation/credentials.env | cut -d= -f2-)
umask 077
echo "SECOND_BRAIN_DATABASE_URL=postgresql://lovebug:${PW}@db.wg.herbylab.dev:5432/petalbrain" \
    > /opt/backups/postgres-consolidation/second-brain-transcribe.env
unset PW
```

The committed `deploy/tower/second-brain-transcribe.env.example`
documents the format and never carries a real secret.

---

## 5. systemd unit

```bash
sudo install -m 0644 deploy/tower/second-brain-transcribe.service \
    /etc/systemd/system/second-brain-transcribe.service
sudo systemctl daemon-reload
sudo systemctl enable --now second-brain-transcribe.service
```

The unit declares `After=` + `Wants=` on `wg-quick@wg-lan.service`, so
the worker starts after the tunnel comes up at boot. If the tunnel
disappears mid-run the worker keeps polling and logs DB-unreachable
warnings; it doesn't crash-loop.

`ExecStart=` calls the in-repo `deploy/tower/run-transcribe-worker.sh`
wrapper, which resolves the venv's CUDA-12 lib dirs at runtime and
prepends them to `LD_LIBRARY_PATH` before exec-ing `uv run`. systemd
does NOT inherit your shell's `LD_LIBRARY_PATH`, so the wrapper is
how the CUDA-12 pin survives across reboots and python upgrades.

### Day-to-day commands

```bash
# What's it doing right now?
systemctl status second-brain-transcribe.service
journalctl -u second-brain-transcribe.service -f

# Pause without uninstalling — toggle from the dashboard at
# https://brain.herbylab.dev/settings  (or whichever host).
# Or stop the unit:
sudo systemctl stop second-brain-transcribe.service

# Pick up code updates:
cd /opt/projects/homelab/second-brain
git pull
uv sync --extra tower
sudo systemctl restart second-brain-transcribe.service
```

---

## 6. Validation — what to confirm once WG is live

These can only be verified on the tower itself:

1. **WG reachability.** `nc -vz db.wg.herbylab.dev 5432` succeeds.
2. **DB auth.** `uv run alembic current` returns the latest revision id.
3. **GPU large-v3 path.** With a short test video queued via
   `second-brain add <url>` on the dev side, run `uv run second-brain transcribe-worker --once`
   on the tower. Watch `journalctl` — first run pulls the model from
   HuggingFace (slow), subsequent runs are fast. The source row should
   end up in `TRANSCRIBED` with `transcript_text` populated and
   `claimed_by` cleared.
4. **Race safety smoke test.** Run two `--once` invocations in parallel
   against an empty queue, then with one PENDING video. Only one should
   claim it; the other should see no work.
5. **Settings round-trip.** Flip `transcription_enabled` off on the
   dashboard; within one poll cycle (~15s) the worker logs
   `transcription_enabled=false — sleeping`. Flip back on, the next
   PENDING video gets picked up.

---

## Follow-ups (out of scope for this round)

- **Media + subtitles directory currently lives on the tower's local
  disk** (under `~/.local/share/second-brain/`). Move to a NAS mount
  later so the dev side can compile/inspect the SRTs without a
  cross-machine fetch. Open question on the right mount point + cred
  flow — see the project's `postgres-migration-planning.md` for the
  related vault-location decision Travis still owes.
- **Tower-side observability.** No metrics endpoint yet; journald is
  the only signal. A `/api/worker-status` heartbeat the dashboard could
  read would be nice once two workers exist.
- **GPU concurrency > 1.** `transcription_max_concurrent_gpu_jobs`
  exists in `pipeline_settings` but the worker only runs one job at a
  time. Scale-out would need a per-job semaphore (or just running N
  worker instances with distinct `SECOND_BRAIN_WORKER_ID`s).