wiki-vault/Sources/Homelab/2026-05-21-traefik-routes-for-pihole-openwrt-us-test-authy-static-ip-cutove.md

created	path	project	tags	type
2026-05-21	Sources/Homelab	traefik-deployment	traefik dns pihole proxmox knot	session-notes

Outcome

• Two more services routed through Traefik on *.herbylab.dev: pihole and openwrt. LE wildcard cert + Knot internal resolution working for both.
• us-test-authy (VM 100) recovered from a broken graphical-login + no-network state and pinned to static 10.0.21.207 via netplan + OpenWrt static lease.
• TigerVNC under herbyadmin restored on :1.

Topics Covered

• pihole behind Traefik — redirect loop fix. pihole.herbylab.dev/admin/login was bouncing to itself. Cause was an unnecessary addPrefix: "/admin" middleware on the router. Pi-hole already serves the UI at /admin/, so the middleware was rewriting /admin/login → /admin/admin/login, which Pi-hole then redirected back to /admin/login. Dropping the middleware fixed it.
• openwrt LuCI behind Traefik. Added router for openwrt.herbylab.dev pointing to https://10.0.11.1:443. Needed serversTransport with insecureSkipVerify: true because LuCI's cert is self-signed for the IP. Public cert chain (LE wildcard) is unaffected.
• us-test-authy boot failure — network conflict. Earlier today the VM started failing graphical login with "Failed to start session". Root cause traced via live-ISO mount of /dev/mapper/ubuntu--vg-ubuntu--lv: systemd-networkd-wait-online timed out at boot, then VNC, lightdm-keyring, pulseaudio all cascaded from the missing network. Underlying conflict: NetworkManager and systemd-networkd both managing ens18. Masked NM via systemctl --root=/mnt mask NetworkManager.
• DHCP not completing after NM mask. tcpdump on ens18 showed DHCP DISCOVER going out but no OFFER coming back — an L2/VLAN issue, not a daemon conflict. VM NIC config had no VLAN tag; PVE vmbr0 not VLAN-aware. Rather than untangle the tag/PVID interaction tonight, dropped DHCP and set netplan to static 10.0.21.207/24 gw 10.0.21.1. Added matching static lease in OpenWrt for the MAC as belt-and-suspenders.
• PVE noVNC F-keys. Browser intercepts Ctrl+Alt+Fn, blocking TTY drop. Workaround: switched the VM to vga: qxl, used SPICE via remote-viewer. The generated .vv had proxy=http://pve.herbylab.dev:3128 which doesn't resolve — sed-replaced with PVE's IP and it connected.
• TigerVNC restart. Stale X11 lock files from travadmin blocked herbyadmin's vncserver@1.service from claiming display :1. Documented fix from previous session: vncserver -kill :1 as travadmin, remove /tmp/.X1-lock and /tmp/.X11-unix/X1, then reset-failed + start under herbyadmin.

Key Learnings

• addPrefix is a footgun behind reverse proxies. It's the right tool when a backend serves at / and you want to mount it under a path. It's wrong when the backend already serves under that path — you'll double-prefix and create self-redirect loops. Pi-hole's UI lives at /admin/ natively; no rewriting needed.
• systemd-networkd-wait-online failures cascade silently. A 2-minute boot delay plus broken display sessions, broken keyring, broken VNC — all of them are downstream of "the network never came up." The lightdm "Failed to start session" message is misleading; the real fault is several layers below.
• No "default tag" on PVE NICs. A VM NIC with no tag= is untagged on the bridge. Which VLAN that untagged frame ends up on depends entirely on OpenWrt's PVID for the trunk port the PVE NIC is plugged into. Historically this VM worked by accident — the PVID happened to be 21, so untagged ingress landed on Private. Don't rely on it. Either tag explicitly or go static.
• DHCP debugging shortcut: tcpdump first, daemon configs second. Watching for DISCOVER/OFFER tells you within seconds whether the problem is the client, the L2 path, or the server. Saves the netplan/networkd config-spelunking time.
• PVE SPICE .vv proxy hostname. PVE bakes whatever hostname is set in datacenter config into the SPICE file. If that hostname doesn't resolve on the client, the SPICE viewer fails with a generic "unable to connect" error. Quick fix: sed -i 's|<hostname>|<IP>|g' the file. Proper fix: set a resolvable hostname in PVE.

Follow-ons

• Make vmbr0 VLAN-aware on PVE and explicitly tag the VM NIC (tag=21). Tonight we punted to static IP; proper fix is to bring this VM into the same VLAN model as Traefik.
• Pick the right hostname for PVE in the datacenter config so SPICE .vv files generate with a resolvable host (likely add pve.herbylab.dev A record in Knot).
• Decide on a long-term split for display :1 between travadmin and herbyadmin, or commit to herbyadmin-only running vncserver to prevent the lock collision recurring.
• Add A record for us-test-authy.herbylab.dev → 10.0.21.207 in Knot so saved VNC viewer profiles and SSH configs stop needing the IP.
• Remove the CD/DVD ISO attachment from VM 100 and reset boot order back to scsi0 only.