diff --git a/Sources/Dev/2026-06-24-petal-ingress-lxc-build-session-notes-2026-06-24.md b/Sources/Dev/2026-06-24-petal-ingress-lxc-build-session-notes-2026-06-24.md new file mode 100644 index 0000000..2d33c5e --- /dev/null +++ b/Sources/Dev/2026-06-24-petal-ingress-lxc-build-session-notes-2026-06-24.md @@ -0,0 +1,145 @@ +--- +created: '2026-06-24' +path: Sources/Dev +project: petal-dispatch-phase-4-2-ingress-split +tags: +- petal-dispatch +- lovebug +- petal-ingress +- lxc +- wireguard +- postgres +- proxmox +- ansible +- terraform +- session-notes +- p4 +type: session-notes +--- + +# petal-ingress LXC Build — Session Notes (2026-06-24) + +Parent plan: `petal-dispatch-phase-4-2-ingress-split` + +## Outcome + +Provisioned and fully networked the `petal-ingress` LXC — the low-privilege, +internet-facing half of the Phase 4.2 ingress/executor split. Completed Task 1 +(provision the ingress LXC) **and** its riskiest dependency: cross-VLAN reach to +the shared pgvector Postgres, which the plan flagged as the most likely place to +need firewall / `pg_hba` work. It came up clean over WireGuard — DB never +exposed on the LAN. Infrastructure foundation for the split is done; remaining +work is app-layer code extraction (Lovebug-heavy, needs the SSE-seam +investigation first). + +## Container facts (for later Ansible/Terraform codification) + +- **CT ID:** 230 (PVE host `10.0.21.188`) +- **Hostname:** petal-ingress +- **OS template:** Debian 12.12 (Bookworm) — chosen over 13.1 deliberately; + 13.x gave folder-mapping/namespace grief on the Knot LXC. Bookworm matches the + other working LXCs. +- **Unprivileged:** yes +- **Nesting:** OFF (minimal surface; not needed for a plain Python/async app) +- **Resources:** 2 cores, 1024 MB RAM, 512 MB swap, 12 GB rootfs on `local-lvm` +- **onboot:** 1 +- **Snapshot:** `after-install` exists as a rollback point +- **Network (LAN):** + - bridge `vmbr0`, VLAN tag **11**, firewall on + - IPv4 `10.0.11.30/24`, gateway `10.0.11.1` + - hwaddr `BC:24:11:6A:77:55` + - Sits on the same segment as Knot (`10.0.11.10`) and Traefik (`10.0.11.20`) +- **Addressing convention:** CT 230 / LAN .30 / WG .30 all echo each other. + +## Host-side prerequisites applied (PVE host `10.0.21.188`) + +These are container-config and host-module changes needed for WireGuard in an +unprivileged LXC — must be reproduced in any IaC version: + +- **tun passthrough** added to `/etc/pve/lxc/230.conf` (live config block, not + the snapshot block): + ``` + lxc.cgroup2.devices.allow: c 10:200 rwm + lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file + ``` +- **WireGuard kernel module** loaded on the host (`modprobe wireguard`) and made + persistent: + ``` + echo wireguard > /etc/modules-load.d/wireguard.conf + ``` + +## In-container setup + +- `apt update && apt upgrade` +- Installed: `fish`, `wireguard` / `wireguard-tools`, `postgresql-client`, + `netcat-openbsd` +- **Tailscale: deliberately NOT installed** — all of this box's paths are + LAN-local; tailnet adds surface + the `--accept-routes` footgun for no benefit. +- **systemd-logind masked** (`systemctl mask systemd-logind`) — fixed a ~25s + post-auth SSH login delay caused by `logind` failing with `226/NAMESPACE` in + the unprivileged container (dbus kept retrying activation every 25s = the + delay). logind has no purpose on a headless single-app box; masking is the + clean fix and keeps nesting off. **Also recommend `UseDNS no` in sshd_config** + as general hygiene (not the cause here, but standard). + +## WireGuard spoke (the cross-VLAN Postgres path) + +The shared pgvector Postgres is published on the WG hub address +(`10.99.0.1:5432`), not on the LAN — so the ingress reaches it as a WG spoke, +exactly like the tower does. The LAN-direct path was a dead end by design (the +`postgres:16-alpine` container — Authentik's DB — is unrelated; the +`pgvector/pgvector:pg17` container is the shared instance and is the target). + +- **Hub:** herbys-dev, interface **`wg-lan`**, pubkey + `nbLpIXOmbaIXeJySVLZ+8sCQkR9Qz2uQzAfcIkTACmE=`, ListenPort 51820, + endpoint `10.0.21.207:51820` +- **Spoke (petal-ingress):** + - interface `wg-lan`, tunnel IP `10.99.0.30/24` + - pubkey `gB/2fe2xTK7egwDSIt6XCQ3ZXBoJw0ZQu0bDqlPrlFI=` + - `/etc/wireguard/wg-lan.conf` `[Interface]` Address `10.99.0.30/24`; + `[Peer]` = hub pubkey, Endpoint `10.0.21.207:51820`, + **AllowedIPs `10.99.0.0/24`** (WG subnet only — NOT `0.0.0.0/0`, so the + container's normal LAN/default route is untouched), `PersistentKeepalive = 25` +- **Hub peer block added** for petal-ingress: `AllowedIPs 10.99.0.30/32`, + applied live with `wg syncconf wg-lan <(wg-quick strip wg-lan)` so the tower's + existing connection was not disturbed. +- **Persistence:** `systemctl enable wg-quick@wg-lan` on the spoke (so the tunnel + survives container reboot). + +## Verification (all passed) + +- LAN ping `10.0.11.30 → 10.0.21.207`: OK (cross-VLAN routing fine) +- LAN `nc 10.0.21.207:5432`: **refused** — expected; pgvector isn't published on + the LAN IP (only `10.99.0.1:5432` and `127.0.0.1:5433`) +- WG handshake hub↔spoke: OK (`latest handshake`, bytes both directions) +- WG `nc 10.99.0.1:5432`: **open** +- `psql -h 10.99.0.1` connected and queried; authenticated as + `dispatch_trellis_read` against PostgreSQL 17.9 — confirms `pg_hba` already + accepts `10.99.0.0/24` (covers `.30` for free; **no `pg_hba` edit needed**) and + confirms the target instance is pgvector:pg17. + +## IaC notes (for the eventual Ansible/Terraform build) + +- Terraform (proxmox provider) creates CT 230 with the resource/network spec + above; the **tun passthrough lines** must be injected into the LXC config + (provider `features` / raw config), and the **host must have the wireguard + module persistent** before the container starts. +- Ansible role for in-container: apt upgrade; install fish + wireguard + + postgresql-client; `mask systemd-logind`; `UseDNS no`; template + `/etc/wireguard/wg-lan.conf` (private key from vault/secret store, never + committed); `enable wg-quick@wg-lan`. +- WG keypair is a secret — generate per-host, store private key in + KeePassXC/secret store, only the pubkey goes in config/peer blocks. +- The hub-side peer addition is a separate play against herbys-dev (idempotent + `[Peer]` block + `wg syncconf`). + +## Next (Phase 4.2 continuation — fresh head / Lovebug) + +- **First:** the SSE-seam investigation (is the stream DB-backed or in-memory + coupled?) — gates the whole effort estimate. See plan `Needs Lovebug + Investigation`. +- Extract the HTTP layer (static + submit + SSE) into the ingress app on this box +- Scope a dedicated **ingress** Postgres role (write tasks / read events only) — + note `dispatch_trellis_read` already exists, so role separation is established +- Stand up `split.herbylab.dev` → Traefik → petal-ingress for the parallel run +- Make the executor headless on herbys-dev (`ss -tlnp` clean = success check) \ No newline at end of file