wiki-vault/Sources/Homelab/2026-06-08-petal-dispatch-tailscale.md

created	path	project	tags	type
2026-06-08	Sources/Homelab	petal-dispatch-p4	tailscale split-dns knot subnet-router petal-dispatch remote-access homelab p4	session-notes

Outcome

Off-LAN remote access to *.herbylab.dev over Tailscale is working end to end. The petal-dispatch Android app and web client both connect to https://dispatch.herbylab.dev/ from off-LAN (verified on hotel wifi), with SSE live and session context (session_uuid) carrying across devices. No app changes required — the hardcoded URL just works. This was the prerequisite for the phone app off-LAN, answering the open question of whether remote access plumbing was needed (it was).

Chose Path A (split-DNS so original URLs work everywhere) over Path B (.ts.net MagicDNS hostname). A is the permanent answer; B was a lighter-touch stopgap. Committed to A and landed it.

Working configuration

1. Tailscale on knot (LXC 103, unprivileged)

• Required /dev/net/tun passthrough in /etc/pve/lxc/103.conf (on PVE host):

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

• pct restart 103, then install via the standard script, then tailscale up.

2. knot listens on its tailnet IP

• Added a third listen line in /etc/knot/knot.conf server block alongside the LAN and loopback addresses:

listen: <knot-tailnet-100.x>@53

• This is config-file-based, so it is persistent across knot restarts and reboots — not a runtime setting that needs re-applying.
• knot requires full systemctl restart knot to re-bind sockets (knotc reload does not).

3. Tailscale admin DNS (split DNS)

• Restricted/custom nameserver → knot's tailnet IP, restricted to domain herbylab.dev.
• "Override local DNS" on.

4. Subnet router (because knot returns LAN IPs)

• knot returns 10.0.11.20 (Traefik VM LAN IP), which tailnet clients can't reach directly.
• herbys-dev advertises the LAN: sudo tailscale up --advertise-routes=10.0.0.0/16 (/16 covers both 10.0.11.x and 10.0.21.x).
• Route must be approved in the Tailscale admin machines page — this is a separate gate from advertising.
• Clients accept with sudo tailscale up --accept-routes.

Key learnings

• tailscale up is all-or-nothing on prefs. A bare tailscale up or a systemctl restart tailscaled silently drops any flag not repeated — --advertise-routes on the subnet router, --accept-routes on the client. Always repeat the full flag set. This was the single biggest time sink: the architecture was correct the whole time; advertised routes had been silently cleared by intervening bare restarts.
• Route approval is a distinct step from route advertising. A node can advertise a subnet and still show empty AllowedIPs until the route is approved in admin. No amount of --accept-routes on the client pulls a route that isn't authorized server-side. Verify with tailscale status --json | grep -A20 <node> → look for the subnet in AllowedIPs.
• DNS working ≠ routing working. Split-DNS resolving correctly (dig returns the right IP via 100.100.100.100) is independent of whether the returned IP is reachable. The DNS-works/ping-fails split is the expected signature of "LAN IP returned, subnet router/approval not yet complete," not a bug.
• Diagnosis order that works: confirm client connected (tailscale status) → confirm route present in client table (ip route | grep tailscale) → if absent, confirm the route is being advertised AND approved (AllowedIPs on the advertising node) before touching the client again.

Reachability note (deferred, not done)

This path uses herbys-dev as subnet router, which means inter-VLAN forwarding (herbys-dev is on a different VLAN than 10.0.11.20). It works today. A cleaner future option exists: since the Traefik VM is itself a tailnet node (100.86.14.2), pointing knot's service records at Traefik's tailnet IP would eliminate the subnet router, IP forwarding, and the VLAN hop entirely (Path B-style records, no MagicDNS dependency). Not needed now; noted for if the subnet-router path ever gets fragile.

Follow-up trigger

• Infra node Tailscale keys are on the default ~6-month expiry (herbys-dev, knot, traefik). When they expire, remote access dies silently until re-auth. Decided to leave as-is for now rather than disable expiry. Trigger: ~November 2026 — re-auth or disable key expiry before then. Breadcrumb lives here so a cold debug session finds the cause fast.