diff --git a/Sources/Homelab/2026-05-21-traefik-routes-for-pihole-openwrt-us-test-authy-static-ip-cutove.md b/Sources/Homelab/2026-05-21-traefik-routes-for-pihole-openwrt-us-test-authy-static-ip-cutove.md new file mode 100644 index 0000000..d064f59 --- /dev/null +++ b/Sources/Homelab/2026-05-21-traefik-routes-for-pihole-openwrt-us-test-authy-static-ip-cutove.md @@ -0,0 +1,43 @@ +--- +created: '2026-05-21' +path: Sources/Homelab +project: traefik-deployment +tags: +- traefik +- dns +- pihole +- proxmox +- knot +type: session-notes +--- + +## Outcome + +- Two more services routed through Traefik on `*.herbylab.dev`: `pihole` and `openwrt`. LE wildcard cert + Knot internal resolution working for both. +- `us-test-authy` (VM 100) recovered from a broken graphical-login + no-network state and pinned to static `10.0.21.207` via netplan + OpenWrt static lease. +- TigerVNC under `herbyadmin` restored on `:1`. + +## Topics Covered + +- **pihole behind Traefik — redirect loop fix.** `pihole.herbylab.dev/admin/login` was bouncing to itself. Cause was an unnecessary `addPrefix: "/admin"` middleware on the router. Pi-hole already serves the UI at `/admin/`, so the middleware was rewriting `/admin/login` → `/admin/admin/login`, which Pi-hole then redirected back to `/admin/login`. Dropping the middleware fixed it. +- **openwrt LuCI behind Traefik.** Added router for `openwrt.herbylab.dev` pointing to `https://10.0.11.1:443`. Needed `serversTransport` with `insecureSkipVerify: true` because LuCI's cert is self-signed for the IP. Public cert chain (LE wildcard) is unaffected. +- **us-test-authy boot failure — network conflict.** Earlier today the VM started failing graphical login with "Failed to start session". Root cause traced via live-ISO mount of `/dev/mapper/ubuntu--vg-ubuntu--lv`: `systemd-networkd-wait-online` timed out at boot, then VNC, lightdm-keyring, pulseaudio all cascaded from the missing network. Underlying conflict: `NetworkManager` and `systemd-networkd` both managing `ens18`. Masked NM via `systemctl --root=/mnt mask NetworkManager`. +- **DHCP not completing after NM mask.** tcpdump on `ens18` showed DHCP DISCOVER going out but no OFFER coming back — an L2/VLAN issue, not a daemon conflict. VM NIC config had no VLAN tag; PVE `vmbr0` not VLAN-aware. Rather than untangle the tag/PVID interaction tonight, dropped DHCP and set netplan to static `10.0.21.207/24` gw `10.0.21.1`. Added matching static lease in OpenWrt for the MAC as belt-and-suspenders. +- **PVE noVNC F-keys.** Browser intercepts Ctrl+Alt+Fn, blocking TTY drop. Workaround: switched the VM to `vga: qxl`, used SPICE via `remote-viewer`. The generated `.vv` had `proxy=http://pve.herbylab.dev:3128` which doesn't resolve — sed-replaced with PVE's IP and it connected. +- **TigerVNC restart.** Stale X11 lock files from `travadmin` blocked `herbyadmin`'s `vncserver@1.service` from claiming display `:1`. Documented fix from previous session: `vncserver -kill :1` as travadmin, remove `/tmp/.X1-lock` and `/tmp/.X11-unix/X1`, then `reset-failed` + `start` under herbyadmin. + +## Key Learnings + +- **`addPrefix` is a footgun behind reverse proxies.** It's the right tool when a backend serves at `/` and you want to mount it under a path. It's wrong when the backend already serves under that path — you'll double-prefix and create self-redirect loops. Pi-hole's UI lives at `/admin/` natively; no rewriting needed. +- **`systemd-networkd-wait-online` failures cascade silently.** A 2-minute boot delay plus broken display sessions, broken keyring, broken VNC — all of them are downstream of "the network never came up." The lightdm "Failed to start session" message is misleading; the real fault is several layers below. +- **No "default tag" on PVE NICs.** A VM NIC with no `tag=` is untagged on the bridge. Which VLAN that untagged frame ends up on depends entirely on OpenWrt's PVID for the trunk port the PVE NIC is plugged into. Historically this VM worked by accident — the PVID happened to be 21, so untagged ingress landed on Private. Don't rely on it. Either tag explicitly or go static. +- **DHCP debugging shortcut: tcpdump first, daemon configs second.** Watching for DISCOVER/OFFER tells you within seconds whether the problem is the client, the L2 path, or the server. Saves the netplan/networkd config-spelunking time. +- **PVE SPICE `.vv` proxy hostname.** PVE bakes whatever hostname is set in datacenter config into the SPICE file. If that hostname doesn't resolve on the client, the SPICE viewer fails with a generic "unable to connect" error. Quick fix: `sed -i 's|||g'` the file. Proper fix: set a resolvable hostname in PVE. + +## Follow-ons + +- [ ] Make `vmbr0` VLAN-aware on PVE and explicitly tag the VM NIC (`tag=21`). Tonight we punted to static IP; proper fix is to bring this VM into the same VLAN model as Traefik. +- [ ] Pick the right hostname for PVE in the datacenter config so SPICE `.vv` files generate with a resolvable host (likely add `pve.herbylab.dev` A record in Knot). +- [ ] Decide on a long-term split for display `:1` between `travadmin` and `herbyadmin`, or commit to herbyadmin-only running `vncserver` to prevent the lock collision recurring. +- [ ] Add A record for `us-test-authy.herbylab.dev → 10.0.21.207` in Knot so saved VNC viewer profiles and SSH configs stop needing the IP. +- [ ] Remove the CD/DVD ISO attachment from VM 100 and reset boot order back to `scsi0` only. \ No newline at end of file