second-brain/deploy/tower/README.md

# Tower-side transcribe worker — deploy notes

Target host: **EndeavourOS tower** (Arch base), RTX 3080. Runs the
`second-brain transcribe-worker` daemon under systemd. Reaches the
petalbrain Postgres over WireGuard at `db.wg.herbylab.dev:5432`.

This worker only does **pull + transcribe** for VIDEO sources. Articles
and all extraction/embedding stay on the dev side.

---

## 1. System packages

```bash
# Build / runtime tooling.
sudo pacman -S --needed base-devel git ffmpeg python uv

# NVIDIA — driver, CUDA, cuDNN. faster-whisper / CTranslate2 needs cuDNN
# at runtime; missing-symbol errors at first transcribe are usually a
# cuDNN install gap. Use whichever driver/cuda channel matches your
# Arch derivative; on stock EndeavourOS the AUR/community packages work:
sudo pacman -S --needed nvidia nvidia-utils cuda cudnn

# Confirm the GPU is visible.
nvidia-smi
```

yt-dlp comes in via the Python project (`uv sync` step below), so the
distro yt-dlp is *optional*. Leave it off the host unless you want it on
your CLI PATH.

---

## 2. Project checkout + venv

```bash
sudo mkdir -p /opt/projects/homelab
sudo chown trucktrav:trucktrav /opt/projects/homelab
cd /opt/projects/homelab
git clone gitea-pbs-trucktrav:petal-power/second-brain.git
cd second-brain

# Install the runtime deps + the `tower` extra (faster-whisper / CTranslate2).
# The dev side does NOT install --extra tower; only the tower needs it.
uv sync --extra tower
```

`faster-whisper` will pull a couple hundred MB of CUDA wheels. First-run
model download (~3 GB for large-v3) lands under `~/.cache/huggingface/`
the first time the worker transcribes; pre-warm it if you want:

```bash
uv run python -c "from faster_whisper import WhisperModel; WhisperModel('large-v3', device='cuda', compute_type='float16')"
```

---

## 3. WireGuard

The worker assumes the tunnel is already configured and that
`wg-quick@wg-lan.service` brings it up at boot. Topology:

| Peer | WG IP |
|------|-------|
| hub (herbydev) | `10.99.0.1` |
| tower | `10.99.0.3` |

`db.wg.herbylab.dev` resolves to `10.99.0.1`. Confirm connectivity from
the tower:

```bash
ping -c2 db.wg.herbylab.dev
nc -vz db.wg.herbylab.dev 5432
```

Tunnel setup itself is **out of scope for this README** — that's
configured separately as part of Travis's infra.

---

## 4. EnvironmentFile

```bash
sudo install -m 0600 -o root -g root \
    deploy/tower/second-brain-transcribe.env.example \
    /etc/default/second-brain-transcribe
sudo $EDITOR /etc/default/second-brain-transcribe   # paste the lovebug password
```

The lovebug password lives in `/opt/backups/postgres-consolidation/credentials.env`
on the dev host. Copy it across through whatever channel you trust
(personal Bitwarden, scp over WG, etc.) — do not commit it.

---

## 5. systemd unit

```bash
sudo install -m 0644 deploy/tower/second-brain-transcribe.service \
    /etc/systemd/system/second-brain-transcribe.service
sudo systemctl daemon-reload
sudo systemctl enable --now second-brain-transcribe.service
```

The unit declares `After=` + `Wants=` on `wg-quick@wg-lan.service`, so
the worker starts after the tunnel comes up at boot. If the tunnel
disappears mid-run the worker keeps polling and logs DB-unreachable
warnings; it doesn't crash-loop.

### Day-to-day commands

```bash
# What's it doing right now?
systemctl status second-brain-transcribe.service
journalctl -u second-brain-transcribe.service -f

# Pause without uninstalling — toggle from the dashboard at
# https://brain.herbylab.dev/settings  (or whichever host).
# Or stop the unit:
sudo systemctl stop second-brain-transcribe.service

# Pick up code updates:
cd /opt/projects/homelab/second-brain
git pull
uv sync --extra tower
sudo systemctl restart second-brain-transcribe.service
```

---

## 6. Validation — what to confirm once WG is live

These can only be verified on the tower itself:

1. **WG reachability.** `nc -vz db.wg.herbylab.dev 5432` succeeds.
2. **DB auth.** `uv run alembic current` returns the latest revision id.
3. **GPU large-v3 path.** With a short test video queued via
   `second-brain add <url>` on the dev side, run `uv run second-brain transcribe-worker --once`
   on the tower. Watch `journalctl` — first run pulls the model from
   HuggingFace (slow), subsequent runs are fast. The source row should
   end up in `TRANSCRIBED` with `transcript_text` populated and
   `claimed_by` cleared.
4. **Race safety smoke test.** Run two `--once` invocations in parallel
   against an empty queue, then with one PENDING video. Only one should
   claim it; the other should see no work.
5. **Settings round-trip.** Flip `transcription_enabled` off on the
   dashboard; within one poll cycle (~15s) the worker logs
   `transcription_enabled=false — sleeping`. Flip back on, the next
   PENDING video gets picked up.

---

## Follow-ups (out of scope for this round)

- **Media + subtitles directory currently lives on the tower's local
  disk** (under `~/.local/share/second-brain/`). Move to a NAS mount
  later so the dev side can compile/inspect the SRTs without a
  cross-machine fetch. Open question on the right mount point + cred
  flow — see the project's `postgres-migration-planning.md` for the
  related vault-location decision Travis still owes.
- **Tower-side observability.** No metrics endpoint yet; journald is
  the only signal. A `/api/worker-status` heartbeat the dashboard could
  read would be nice once two workers exist.
- **GPU concurrency > 1.** `transcription_max_concurrent_gpu_jobs`
  exists in `pipeline_settings` but the worker only runs one job at a
  time. Scale-out would need a per-job semaphore (or just running N
  worker instances with distinct `SECOND_BRAIN_WORKER_ID`s).