mcp: session-notes — WireGuard Postgres Access — Initial Deploy & Peer How-To

Sources/Homelab/2026-05-25-wireguard-postgres-access-initial-deploy.md

@@ -0,0 +1,131 @@
+---
+created: '2026-05-25'
+path: Sources/Homelab
+project: wireguard-postgres-access
+tags:
+- wireguard
+- postgres
+- networking
+- knot
+- dns
+- homelab
+type: session-notes
+---
+
+## Outcome
+
+Stood up WireGuard hub-and-spoke overlay on herbys-dev (`wg-lan` interface, `10.99.0.0/24`). First peer (ThinkPad as POC, standing in for tower) handshakes and pings hub successfully. Knot DNS serves `db.wg.herbylab.dev` and `herbys-dev.wg.herbylab.dev` → `10.99.0.1`. Postgres rebind handled by Lovebug out of band.
+
+## What got built
+
+- **Hub** on herbys-dev: `wg-lan` interface at `10.99.0.1/24`, listening UDP/51820, enabled via `wg-quick@wg-lan` systemd unit
+- **First peer** (ThinkPad): `10.99.0.2`, dials hub via `10.0.21.207:51820`, persistent keepalive 25s
+- **OpenWrt firewall**: Lab→Private UDP/51820 allowed (scoped to source IP)
+- **Knot zone update**: `herbylab.dev` zone gained a `$ORIGIN wg.herbylab.dev.` section with two A records, serial bumped
+- **Per-host key generation pattern**: private keys generated and locked to each host; only public keys cross machines
+
+## Learnings
+
+- **`$(cat secret_file)` splice pattern** for moving on-disk secrets into config files without crossing the terminal. Keeps secrets out of scrollback and shell history.
+- **`read -s` (bash) / `read -s -P` (fish)** for prompting secrets without echo when the secret is coming from a human, not a file.
+- **WireGuard `AllowedIPs` is both a routing rule and an ACL** — on the hub it's per-peer (`/32` per peer); on the client it's "what to route through the tunnel" (`/24` for the whole overlay subnet).
+- **`PersistentKeepalive`** is needed when the peer is behind NAT or stateful firewalls (inter-VLAN through OpenWrt counts). Without it, the return path can age out.
+- **DNS zone files — three ways to handle subdomains:**
+  1. Flat records in parent zone (`db.wg IN A ...`) — simplest, fine for small zones
+  2. `$ORIGIN` shift mid-file — visual grouping, same zone
+  3. Separate zone file with NS delegation — real separation, production pattern
+  Chose Way 2 here. Way 3 is overkill until there's a real reason to split.
+- **Serial number must increment on every zone change** — and the file edit doesn't take effect until `knotc zone-reload`. `knotc zone-status` shows the *loaded* serial, not the file's serial.
+- **DNS caching layers** to consider when fixes don't propagate: Pi-hole, local stub resolver (systemd-resolved if present), application caches. TTL bounds the worst case.
+
+## Decisions
+
+- **Way 2 (origin shift)** chosen over separate zone for `wg.herbylab.dev`. Two records doesn't justify a new zone file.
+- **POC on laptop, not tower** — proved the shape works before touching the actual target rig. Tower peer to be added later as a follow-up.
+- **Knot record naming**: `db.wg.herbylab.dev` (service-oriented) and `herbys-dev.wg.herbylab.dev` (host-oriented), both pointing at `10.99.0.1`. Use whichever fits the connection string.
+
+---
+
+## How to add a new peer to wg-lan
+
+### 1. Generate keypair on the new peer
+
+```
+sudo mkdir -p /etc/wireguard
+cd /etc/wireguard
+sudo sh -c 'umask 077; wg genkey | tee <peer>_private.key | wg pubkey > <peer>_public.key'
+sudo cat /etc/wireguard/<peer>_public.key
+```
+
+Capture the public key output. Replace `<peer>` with a short identifier (e.g., `tower`, `nas`).
+
+### 2. Register the peer's pubkey on the hub (herbys-dev)
+
+Append to `/etc/wireguard/wg-lan.conf`:
+
+```
+[Peer]
+# <peer-name>
+PublicKey = <peer-public-key>
+AllowedIPs = 10.99.0.<N>/32
+```
+
+Where `<N>` is the next free address (`.2` taken, start at `.3` for new peers).
+
+Reload the hub without dropping existing tunnels:
+
+```
+sudo wg syncconf wg-lan <(sudo wg-quick strip wg-lan)
+```
+
+(`wg syncconf` applies the new config without bouncing the interface — existing peers stay connected.)
+
+### 3. Write the peer's config
+
+On the new peer:
+
+```
+sudo sh -c 'cat > /etc/wireguard/wg-lan.conf << EOF
+[Interface]
+PrivateKey = $(cat /etc/wireguard/<peer>_private.key)
+Address = 10.99.0.<N>/24
+
+[Peer]
+# herbys-dev hub
+PublicKey = nbLpIXOmbaIXeJySVLZ+8sCQkR9Qz2uQzAfcIkTACmE=
+Endpoint = 10.0.21.207:51820
+AllowedIPs = 10.99.0.0/24
+PersistentKeepalive = 25
+EOF'
+sudo chmod 600 /etc/wireguard/wg-lan.conf
+```
+
+### 4. OpenWrt firewall (if peer is on a new VLAN)
+
+If the new peer is on a VLAN that doesn't already have a Lab→Private:UDP/51820 rule, add one in LuCI. Scope source to the peer's IP if possible.
+
+### 5. Bring tunnel up and verify
+
+```
+sudo systemctl enable --now wg-quick@wg-lan
+sudo wg show
+ping -c 3 10.99.0.1
+```
+
+`wg show` should list the hub peer with a recent handshake and non-zero transfer after the ping.
+
+---
+
+## Open follow-ups
+
+- Add actual tower (not laptop) as a peer
+- Confirm Postgres boot ordering — `wg-lan` must be up before Docker brings Postgres up (add `After=wg-quick@wg-lan.service` to Docker unit, or similar)
+- Document key rotation procedure (what to do if a peer private key leaks)
+- Sort out laptop's DNS resolver — `systemd-resolved` not present, need to know what's actually resolving for deliberate cache flushes
+- Ansible role: extend in-progress role to template hub config from peer registry once the manual pattern is stable
+
+## References
+
+- Project plan: `wireguard-postgres-access`
+- Knot zone: `/var/lib/knot/herbylab.dev.zone`
+- Hub config: `/etc/wireguard/wg-lan.conf` on herbys-dev (10.0.21.207)