petal-ingress LXC Build — Session Notes (2026-06-24)

Parent plan: petal-dispatch-phase-4-2-ingress-split

Outcome

Provisioned and fully networked the petal-ingress LXC — the low-privilege, internet-facing half of the Phase 4.2 ingress/executor split. Completed Task 1 (provision the ingress LXC) and its riskiest dependency: cross-VLAN reach to the shared pgvector Postgres, which the plan flagged as the most likely place to need firewall / pg_hba work. It came up clean over WireGuard — DB never exposed on the LAN. Infrastructure foundation for the split is done; remaining work is app-layer code extraction (Lovebug-heavy, needs the SSE-seam investigation first).

Container facts (for later Ansible/Terraform codification)

CT ID: 230 (PVE host 10.0.21.188)
Hostname: petal-ingress
OS template: Debian 12.12 (Bookworm) — chosen over 13.1 deliberately; 13.x gave folder-mapping/namespace grief on the Knot LXC. Bookworm matches the other working LXCs.
Unprivileged: yes
Nesting: OFF (minimal surface; not needed for a plain Python/async app)
Resources: 2 cores, 1024 MB RAM, 512 MB swap, 12 GB rootfs on local-lvm
onboot: 1
Snapshot: after-install exists as a rollback point
Network (LAN):
- bridge vmbr0, VLAN tag 11, firewall on
- IPv4 10.0.11.30/24, gateway 10.0.11.1
- hwaddr BC:24:11:6A:77:55
- Sits on the same segment as Knot (10.0.11.10) and Traefik (10.0.11.20)
Addressing convention: CT 230 / LAN .30 / WG .30 all echo each other.

Host-side prerequisites applied (PVE host `10.0.21.188`)

These are container-config and host-module changes needed for WireGuard in an unprivileged LXC — must be reproduced in any IaC version:

tun passthrough added to /etc/pve/lxc/230.conf (live config block, not the snapshot block):

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

WireGuard kernel module loaded on the host (modprobe wireguard) and made persistent:
```
echo wireguard > /etc/modules-load.d/wireguard.conf
```

In-container setup

apt update && apt upgrade
Installed: fish, wireguard / wireguard-tools, postgresql-client, netcat-openbsd
Tailscale: deliberately NOT installed — all of this box's paths are LAN-local; tailnet adds surface + the --accept-routes footgun for no benefit.
systemd-logind masked (systemctl mask systemd-logind) — fixed a ~25s post-auth SSH login delay caused by logind failing with 226/NAMESPACE in the unprivileged container (dbus kept retrying activation every 25s = the delay). logind has no purpose on a headless single-app box; masking is the clean fix and keeps nesting off. Also recommend UseDNS no in sshd_config as general hygiene (not the cause here, but standard).

WireGuard spoke (the cross-VLAN Postgres path)

The shared pgvector Postgres is published on the WG hub address (10.99.0.1:5432), not on the LAN — so the ingress reaches it as a WG spoke, exactly like the tower does. The LAN-direct path was a dead end by design (the postgres:16-alpine container — Authentik's DB — is unrelated; the pgvector/pgvector:pg17 container is the shared instance and is the target).

Hub: herbys-dev, interface wg-lan, pubkey nbLpIXOmbaIXeJySVLZ+8sCQkR9Qz2uQzAfcIkTACmE=, ListenPort 51820, endpoint 10.0.21.207:51820
Spoke (petal-ingress):
- interface wg-lan, tunnel IP 10.99.0.30/24
- pubkey gB/2fe2xTK7egwDSIt6XCQ3ZXBoJw0ZQu0bDqlPrlFI=
- /etc/wireguard/wg-lan.conf [Interface] Address 10.99.0.30/24; [Peer] = hub pubkey, Endpoint 10.0.21.207:51820, AllowedIPs 10.99.0.0/24 (WG subnet only — NOT 0.0.0.0/0, so the container's normal LAN/default route is untouched), PersistentKeepalive = 25
Hub peer block added for petal-ingress: AllowedIPs 10.99.0.30/32, applied live with wg syncconf wg-lan <(wg-quick strip wg-lan) so the tower's existing connection was not disturbed.
Persistence: systemctl enable wg-quick@wg-lan on the spoke (so the tunnel survives container reboot).

Verification (all passed)

LAN ping 10.0.11.30 → 10.0.21.207: OK (cross-VLAN routing fine)
LAN nc 10.0.21.207:5432: refused — expected; pgvector isn't published on the LAN IP (only 10.99.0.1:5432 and 127.0.0.1:5433)
WG handshake hub↔spoke: OK (latest handshake, bytes both directions)
WG nc 10.99.0.1:5432: open
psql -h 10.99.0.1 connected and queried; authenticated as dispatch_trellis_read against PostgreSQL 17.9 — confirms pg_hba already accepts 10.99.0.0/24 (covers .30 for free; no pg_hba edit needed) and confirms the target instance is pgvector:pg17.

IaC notes (for the eventual Ansible/Terraform build)

Terraform (proxmox provider) creates CT 230 with the resource/network spec above; the tun passthrough lines must be injected into the LXC config (provider features / raw config), and the host must have the wireguard module persistent before the container starts.
Ansible role for in-container: apt upgrade; install fish + wireguard + postgresql-client; mask systemd-logind; UseDNS no; template /etc/wireguard/wg-lan.conf (private key from vault/secret store, never committed); enable wg-quick@wg-lan.
WG keypair is a secret — generate per-host, store private key in KeePassXC/secret store, only the pubkey goes in config/peer blocks.
The hub-side peer addition is a separate play against herbys-dev (idempotent [Peer] block + wg syncconf).

Next (Phase 4.2 continuation — fresh head / Lovebug)

First: the SSE-seam investigation (is the stream DB-backed or in-memory coupled?) — gates the whole effort estimate. See plan Needs Lovebug Investigation.
Extract the HTTP layer (static + submit + SSE) into the ingress app on this box
Scope a dedicated ingress Postgres role (write tasks / read events only) — note dispatch_trellis_read already exists, so role separation is established
Stand up split.herbylab.dev → Traefik → petal-ingress for the parallel run
Make the executor headless on herbys-dev (ss -tlnp clean = success check)

6.4 KiB Raw Blame History