wiki-vault/Sources/Homelab/2026-05-11-traefik-phase1-phase2-wildcard-canary.md

created	path	project	status	tags	type	updated
2026-05-11	Sources/Homelab	traefik-deployment	active	homelab traefik tls cloudflare dns docker	session-notes	2026-05-11

Traefik — Phase 1 close-out + Phase 2 canary complete

Outcome

Closed Phase 1.6 (wildcard cert) and shipped all of Phase 2 (canary, both surfaces, dashboard lockdown) in one session. The full Traefik architecture is now running end-to-end. Every service from here forward inherits this pattern.

What's live:

• Traefik 3.7 on VM 102 (10.0.21.181), Docker + file + ACME providers
• LE prod wildcard *.herbylab.dev issued via Cloudflare DNS-01, in acme.json, ~90 day expiry
• whoami.herbylab.dev reachable on both surfaces with real certs
• traefik.herbylab.dev — dashboard routed through Traefik itself (TLS + basic auth), no insecure :8080 exposed
• New cloudflared tunnel traefik-vm on the Traefik VM, parallel to the existing cloudflared-mcp on HerbyDev (no migration yet)

Topics Covered

• Traefik VM vs LXC clarification (it's VM 102, not an LXC — Knot is the LXC at 103)
• Docker compose layout under /opt/traefik/, /opt/whoami/, /opt/cloudflared/ — Traefik infra + services as separate compose projects sharing the external traefik Docker network
• Static + dynamic config split (traefik.yml + dynamic/ file provider directory)
• ACME staging → prod flip (wipe acme.json, remove caServer, recreate)
• Cross-host routing model: same-host services use Docker labels; HerbyDev services will use file provider when migrated
• cloudflared token method, second tunnel running parallel to the existing one
• Dashboard exposure tiers (private+basicauth vs Authentik SSO vs disabled) — picked basicauth for now, Authentik comes with MCP migration

Key Learnings

• .env is read at compose command time, not container runtime. An empty CF_DNS_API_TOKEN produced a Cloudflare 6111: Invalid format for Authorization header error — the auth header was Bearer with nothing after it. Diagnostic: docker compose exec traefik printenv CF_DNS_API_TOKEN shows what the container actually sees.
• CF edge cert ≠ origin cert. Public clients always see Cloudflare's edge cert (Google Trust Services). The LE wildcard lives on Traefik behind cloudflared and is never seen by public traffic. This is expected and correct; tripped on it briefly when reading curl output.
• cloudflared → traefik:80 causes a redirect loop. Traefik's web entrypoint redirects HTTP→HTTPS, cloudflared forwards the 301 back to the client, client loops. Fix: point cloudflared at traefik:443 and enable "No TLS Verify" on the CF tunnel public hostname (the cert is for whoami.herbylab.dev, not traefik).
• Tunnel != tunnel credentials. A cloudflared instance owns one tunnel. Cloning the existing setup means cloning the compose pattern, not the credentials. New tunnel traefik-vm was created separately; old cloudflared-mcp on HerbyDev continues running untouched.
• Compose label escaping for bcrypt. $ is variable expansion in compose. Every $ in a bcrypt hash must be doubled to $$ when used inline in a label. Caught before deploy.
• Capability-based naming pays off immediately. traefik.herbylab.dev for the dashboard, whoami.herbylab.dev for the canary — no host infix, the names mean what they do.

Follow-ons

• Snapshot VM 102 as phase2-complete
• Phase 3 — MCP migration (file provider, parallel-path cutover, new tunnel hosts mcp.herbylab.dev)
• Decommission cloudflared-mcp on HerbyDev after MCP cutover verified
• Decide whoami fate (keep as permanent diagnostic vs tear down after Phase 3)
• Migrate dashboard auth from basicauth to Authentik forward-auth when Authentik is wired into Traefik (likely as part of MCP work)
• Knot zone file into git
• Ansible role for Knot primary
• Knot secondary on NAS
• Move Traefik VM to Lab VLAN (cosmetic, not blocking)