--- created: '2026-06-24' path: Sources/Dev project: petal-dispatch-phase-4-2-ingress-split tags: - petal-dispatch - lovebug - petal-ingress - lxc - wireguard - postgres - proxmox - ansible - terraform - session-notes - p4 type: session-notes --- # petal-ingress LXC Build — Session Notes (2026-06-24) Parent plan: `petal-dispatch-phase-4-2-ingress-split` ## Outcome Provisioned and fully networked the `petal-ingress` LXC — the low-privilege, internet-facing half of the Phase 4.2 ingress/executor split. Completed Task 1 (provision the ingress LXC) **and** its riskiest dependency: cross-VLAN reach to the shared pgvector Postgres, which the plan flagged as the most likely place to need firewall / `pg_hba` work. It came up clean over WireGuard — DB never exposed on the LAN. Infrastructure foundation for the split is done; remaining work is app-layer code extraction (Lovebug-heavy, needs the SSE-seam investigation first). ## Container facts (for later Ansible/Terraform codification) - **CT ID:** 230 (PVE host `10.0.21.188`) - **Hostname:** petal-ingress - **OS template:** Debian 12.12 (Bookworm) — chosen over 13.1 deliberately; 13.x gave folder-mapping/namespace grief on the Knot LXC. Bookworm matches the other working LXCs. - **Unprivileged:** yes - **Nesting:** OFF (minimal surface; not needed for a plain Python/async app) - **Resources:** 2 cores, 1024 MB RAM, 512 MB swap, 12 GB rootfs on `local-lvm` - **onboot:** 1 - **Snapshot:** `after-install` exists as a rollback point - **Network (LAN):** - bridge `vmbr0`, VLAN tag **11**, firewall on - IPv4 `10.0.11.30/24`, gateway `10.0.11.1` - hwaddr `BC:24:11:6A:77:55` - Sits on the same segment as Knot (`10.0.11.10`) and Traefik (`10.0.11.20`) - **Addressing convention:** CT 230 / LAN .30 / WG .30 all echo each other. ## Host-side prerequisites applied (PVE host `10.0.21.188`) These are container-config and host-module changes needed for WireGuard in an unprivileged LXC — must be reproduced in any IaC version: - **tun passthrough** added to `/etc/pve/lxc/230.conf` (live config block, not the snapshot block): ``` lxc.cgroup2.devices.allow: c 10:200 rwm lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file ``` - **WireGuard kernel module** loaded on the host (`modprobe wireguard`) and made persistent: ``` echo wireguard > /etc/modules-load.d/wireguard.conf ``` ## In-container setup - `apt update && apt upgrade` - Installed: `fish`, `wireguard` / `wireguard-tools`, `postgresql-client`, `netcat-openbsd` - **Tailscale: deliberately NOT installed** — all of this box's paths are LAN-local; tailnet adds surface + the `--accept-routes` footgun for no benefit. - **systemd-logind masked** (`systemctl mask systemd-logind`) — fixed a ~25s post-auth SSH login delay caused by `logind` failing with `226/NAMESPACE` in the unprivileged container (dbus kept retrying activation every 25s = the delay). logind has no purpose on a headless single-app box; masking is the clean fix and keeps nesting off. **Also recommend `UseDNS no` in sshd_config** as general hygiene (not the cause here, but standard). ## WireGuard spoke (the cross-VLAN Postgres path) The shared pgvector Postgres is published on the WG hub address (`10.99.0.1:5432`), not on the LAN — so the ingress reaches it as a WG spoke, exactly like the tower does. The LAN-direct path was a dead end by design (the `postgres:16-alpine` container — Authentik's DB — is unrelated; the `pgvector/pgvector:pg17` container is the shared instance and is the target). - **Hub:** herbys-dev, interface **`wg-lan`**, pubkey `nbLpIXOmbaIXeJySVLZ+8sCQkR9Qz2uQzAfcIkTACmE=`, ListenPort 51820, endpoint `10.0.21.207:51820` - **Spoke (petal-ingress):** - interface `wg-lan`, tunnel IP `10.99.0.30/24` - pubkey `gB/2fe2xTK7egwDSIt6XCQ3ZXBoJw0ZQu0bDqlPrlFI=` - `/etc/wireguard/wg-lan.conf` `[Interface]` Address `10.99.0.30/24`; `[Peer]` = hub pubkey, Endpoint `10.0.21.207:51820`, **AllowedIPs `10.99.0.0/24`** (WG subnet only — NOT `0.0.0.0/0`, so the container's normal LAN/default route is untouched), `PersistentKeepalive = 25` - **Hub peer block added** for petal-ingress: `AllowedIPs 10.99.0.30/32`, applied live with `wg syncconf wg-lan <(wg-quick strip wg-lan)` so the tower's existing connection was not disturbed. - **Persistence:** `systemctl enable wg-quick@wg-lan` on the spoke (so the tunnel survives container reboot). ## Verification (all passed) - LAN ping `10.0.11.30 → 10.0.21.207`: OK (cross-VLAN routing fine) - LAN `nc 10.0.21.207:5432`: **refused** — expected; pgvector isn't published on the LAN IP (only `10.99.0.1:5432` and `127.0.0.1:5433`) - WG handshake hub↔spoke: OK (`latest handshake`, bytes both directions) - WG `nc 10.99.0.1:5432`: **open** - `psql -h 10.99.0.1` connected and queried; authenticated as `dispatch_trellis_read` against PostgreSQL 17.9 — confirms `pg_hba` already accepts `10.99.0.0/24` (covers `.30` for free; **no `pg_hba` edit needed**) and confirms the target instance is pgvector:pg17. ## IaC notes (for the eventual Ansible/Terraform build) - Terraform (proxmox provider) creates CT 230 with the resource/network spec above; the **tun passthrough lines** must be injected into the LXC config (provider `features` / raw config), and the **host must have the wireguard module persistent** before the container starts. - Ansible role for in-container: apt upgrade; install fish + wireguard + postgresql-client; `mask systemd-logind`; `UseDNS no`; template `/etc/wireguard/wg-lan.conf` (private key from vault/secret store, never committed); `enable wg-quick@wg-lan`. - WG keypair is a secret — generate per-host, store private key in KeePassXC/secret store, only the pubkey goes in config/peer blocks. - The hub-side peer addition is a separate play against herbys-dev (idempotent `[Peer]` block + `wg syncconf`). ## Next (Phase 4.2 continuation — fresh head / Lovebug) - **First:** the SSE-seam investigation (is the stream DB-backed or in-memory coupled?) — gates the whole effort estimate. See plan `Needs Lovebug Investigation`. - Extract the HTTP layer (static + submit + SSE) into the ingress app on this box - Scope a dedicated **ingress** Postgres role (write tasks / read events only) — note `dispatch_trellis_read` already exists, so role separation is established - Stand up `split.herbylab.dev` → Traefik → petal-ingress for the parallel run - Make the executor headless on herbys-dev (`ss -tlnp` clean = success check)