mcp: session-notes — traefik-phase1-phase2-wildcard-canary

Sources/Homelab/2026-05-11-traefik-phase1-phase2-wildcard-canary.md

@@ -0,0 +1,94 @@
+---
+created: '2026-05-11'
+path: Sources/Homelab
+project: traefik-deployment
+status: active
+tags:
+- homelab
+- traefik
+- tls
+- cloudflare
+- dns
+- docker
+type: session-notes
+updated: '2026-05-11'
+---
+
+# Traefik — Phase 1 close-out + Phase 2 canary complete
+
+## Outcome
+
+Closed Phase 1.6 (wildcard cert) and shipped all of Phase 2 (canary, both
+surfaces, dashboard lockdown) in one session. The full Traefik
+architecture is now running end-to-end. Every service from here forward
+inherits this pattern.
+
+What's live:
+
+- Traefik 3.7 on VM 102 (`10.0.21.181`), Docker + file + ACME providers
+- LE prod wildcard `*.herbylab.dev` issued via Cloudflare DNS-01, in
+  `acme.json`, ~90 day expiry
+- `whoami.herbylab.dev` reachable on both surfaces with real certs
+- `traefik.herbylab.dev` — dashboard routed through Traefik itself
+  (TLS + basic auth), no insecure `:8080` exposed
+- New `cloudflared` tunnel `traefik-vm` on the Traefik VM, parallel to
+  the existing `cloudflared-mcp` on HerbyDev (no migration yet)
+
+## Topics Covered
+
+- Traefik VM vs LXC clarification (it's VM 102, not an LXC — Knot is the
+  LXC at 103)
+- Docker compose layout under `/opt/traefik/`, `/opt/whoami/`,
+  `/opt/cloudflared/` — Traefik infra + services as separate compose
+  projects sharing the external `traefik` Docker network
+- Static + dynamic config split (`traefik.yml` + `dynamic/` file
+  provider directory)
+- ACME staging → prod flip (wipe `acme.json`, remove `caServer`, recreate)
+- Cross-host routing model: same-host services use Docker labels;
+  HerbyDev services will use file provider when migrated
+- cloudflared token method, second tunnel running parallel to the
+  existing one
+- Dashboard exposure tiers (private+basicauth vs Authentik SSO vs
+  disabled) — picked basicauth for now, Authentik comes with MCP migration
+
+## Key Learnings
+
+- **`.env` is read at compose command time, not container runtime.** An
+  empty `CF_DNS_API_TOKEN` produced a Cloudflare `6111: Invalid format
+  for Authorization header` error — the auth header was `Bearer ` with
+  nothing after it. Diagnostic: `docker compose exec traefik printenv
+  CF_DNS_API_TOKEN` shows what the container actually sees.
+- **CF edge cert ≠ origin cert.** Public clients always see Cloudflare's
+  edge cert (Google Trust Services). The LE wildcard lives on Traefik
+  behind cloudflared and is never seen by public traffic. This is
+  expected and correct; tripped on it briefly when reading curl output.
+- **cloudflared → `traefik:80` causes a redirect loop.** Traefik's `web`
+  entrypoint redirects HTTP→HTTPS, cloudflared forwards the 301 back to
+  the client, client loops. Fix: point cloudflared at `traefik:443` and
+  enable "No TLS Verify" on the CF tunnel public hostname (the cert is
+  for `whoami.herbylab.dev`, not `traefik`).
+- **Tunnel != tunnel credentials.** A cloudflared instance owns one
+  tunnel. Cloning the existing setup means cloning the compose pattern,
+  not the credentials. New tunnel `traefik-vm` was created separately;
+  old `cloudflared-mcp` on HerbyDev continues running untouched.
+- **Compose label escaping for bcrypt.** `$` is variable expansion in
+  compose. Every `$` in a bcrypt hash must be doubled to `$$` when used
+  inline in a label. Caught before deploy.
+- **Capability-based naming pays off immediately.** `traefik.herbylab.dev`
+  for the dashboard, `whoami.herbylab.dev` for the canary — no host
+  infix, the names mean what they do.
+
+## Follow-ons
+
+- [ ] Snapshot VM 102 as `phase2-complete`
+- [ ] Phase 3 — MCP migration (file provider, parallel-path cutover, new
+      tunnel hosts `mcp.herbylab.dev`)
+- [ ] Decommission `cloudflared-mcp` on HerbyDev after MCP cutover verified
+- [ ] Decide whoami fate (keep as permanent diagnostic vs tear down
+      after Phase 3)
+- [ ] Migrate dashboard auth from basicauth to Authentik forward-auth
+      when Authentik is wired into Traefik (likely as part of MCP work)
+- [ ] Knot zone file into git
+- [ ] Ansible role for Knot primary
+- [ ] Knot secondary on NAS
+- [ ] Move Traefik VM to Lab VLAN (cosmetic, not blocking)