--- created: '2026-06-24' path: Sources/Dev project: petal-dispatch-phase-4-2-ingress-split tags: - petal-dispatch - lovebug - ingress - executor - lxc - postgres - sse - security - isolation - p4 - phase-4 type: project-plan --- # petal-dispatch Phase 4.2 — Ingress / Executor Split ## Goal Split the internet-facing HTTP layer out of the privileged executor so that a compromise of the exposed surface cannot reach host execution. Today petal-dispatch is a single host service that **both** accepts inbound requests **and** spawns Claude/Lovebug processes with host access. That fusion means an RCE in the web layer is effectively host access. This phase separates the two roles using the DB as the boundary, activating the blast-radius floor the architecture already has the plumbing for. ## Why now The auth work (4.1, Managed OAuth via CF Access) is done and the app + web client are connected and working. But the security model isn't real yet: the queue/event-log boundary exists as a *data pattern*, not as a *privilege boundary*, because both ends live in one privileged process. This phase makes the boundary load-bearing. Nothing built in 3.x/4.x is discarded — auth, the queue, the event log, and SSE fan-out all stay; only the *hosting process* for the HTTP layer changes. Related prior art: vault `dev-srv-split` (Dev/Srv Deployment Split) is the conceptual parent of this work. ## Architecture ### Current (fused) ``` [ petal-dispatch (host service on herbys-dev) ] = static pages + submit endpoint + SSE stream + queue consumer + Claude spawn = internet-facing AND host-privileged ← the problem ``` ### Target (split) ``` [ Apache/static? NO — not needed ] Traefik already fronts + terminates TLS [ ingress app — NEW, in its own LXC ] ├─ serves static pages ├─ POST submit (auth'd) → WRITE task row to Postgres └─ SSE /events/stream → READ event log from Postgres, fan out to all clients • NO host exec, NO process spawning • Postgres role: write tasks + read events ONLY • the only internet-facing surface │ (Postgres is the air gap — no direct ingress→executor connection) ▼ [ Postgres ] = task queue + event log (LISTEN/NOTIFY) [ executor (petal-dispatch) — stays on herbys-dev, goes HEADLESS ] ├─ reads task queue (LISTEN/NOTIFY) ├─ spawns Claude / Lovebug (host access retained) └─ writes events to log • NO listening socket — outbound DB connections only • NOT internet-facing [ watcher (petal-dispatch-worker) ] unchanged — reads events, does its thing ``` ### Settled decisions - **DB is the boundary.** Ingress writes tasks; executor reads them. No direct network path ingress→executor. Consistent with the locked "DB is truth / stream is display" principle (see vault `execution-journal`, `synthesis-layer`). - **Ingress owns BOTH submit and SSE.** The executor must end up with *zero* inbound sockets. Moving only the submit endpoint while leaving SSE on the executor would leave it internet-facing and defeat the purpose. - **Target host = LXC, small Debian.** Lighter than a VM, good-enough isolation for the low-privilege half (it can't exec; an escape escapes the harmless box). - **No local Apache/nginx.** Traefik already fronts everything for TLS. The ingress app (extracted async stack) serves static + API + SSE directly; Traefik routes to its port. - **Executor stays on herbys-dev for now.** It keeps host-exec (its job). Its own VM-isolation is a later, optional defense-in-depth step — out of scope here. - **Lovebug-trust boundary is explicitly NOT addressed.** Malicious *task content* coming through the queue is a separate concern (the turtle ends here). This phase closes the *connection-level* surface, not content trust. - **Parallel-run on a temporary hostname.** The current fused service stays ACTIVE and untouched throughout the build. The ingress LXC is brought up under a throwaway name — `split.herbylab.dev` — and all routing/executor/role work is validated against that name first. Only once the split is proven clean end-to-end do `dispatch.herbylab.dev` / `dispatch-internal.herbylab.dev` get repointed to the ingress LXC and the fused service retired. No cutover happens on the live names until the parallel path is fully working. ## Needs Lovebug Investigation (do FIRST — sizes the whole job) These are code-reality questions that determine clean-lift vs. refactor: - [ ] **Is the SSE stream truly DB-backed, or does it share in-process state with the executor?** Grep the SSE handler: does it query Postgres (event log + `from_seq`) or subscribe to an in-memory emitter the executor feeds? The locked design says DB-backed (LISTEN/NOTIFY, doorbell-not-carrier), but verify implementation hasn't drifted. **Clean lift if DB-backed; refactor if in-memory coupled.** - [ ] **Are submit + SSE cleanly separable from the executor process?** Confirmed verbally that "everything is served by the executor process." Determine whether the HTTP layer is a distinct module (FastAPI app) or entangled with executor internals (shared objects, direct calls). - [ ] **What is the executor's true required reach?** To confirm it can go fully headless: does anything *require* the executor to receive inbound connections, or are all its inputs the task queue + all outputs the event log? If anything else dials into it, surface it now. ## Tasks ### 1. Provision the ingress LXC - [ ] Create small Debian LXC (sibling pattern to Knot LXC 103, Gitea-to-LXC plan) - [ ] Network: reachable by Traefik; able to reach Postgres (WireGuard/bridge as appropriate); NO need to reach the executor directly - [ ] Base runtime for the extracted async app (uv/Python per existing stack) ### 2. Extract the HTTP layer - [ ] Pull static-serving + submit endpoint + SSE handler out of the executor into the ingress app - [ ] Submit endpoint: validate auth'd request → INSERT task row - [ ] SSE endpoint: read event log (`from_seq` backfill) → fan out to clients - [ ] Carry over 4.1 auth: CF Access Managed OAuth assertion validation lives here now (ingress is the internet-facing surface) ### 3. Scope the ingress Postgres role - [ ] New DB role: `INSERT` on tasks table, `SELECT` on event log — nothing else - [ ] No access to other services' tables in the shared Postgres - [ ] Verify the role cannot read/write outside its two needs ### 4. Make the executor headless - [ ] Remove the HTTP/listening layer from the executor - [ ] Executor consumes task queue via LISTEN/NOTIFY (already the pattern) - [ ] Executor writes events to log (unchanged) - [ ] **Verify zero listening sockets:** `ss -tlnp` on herbys-dev shows the executor binding nothing client-facing. This is the success check. ### 5. Parallel-run on temporary hostname - [ ] Stand up `split.herbylab.dev` → Traefik → ingress LXC (Knot record + Traefik router; mirror the dispatch entrypoint pattern) - [ ] Keep the current fused `dispatch.herbylab.dev` service ACTIVE and untouched - [ ] Validate the full path against `split.herbylab.dev`: submit from app + web → task lands in DB → executor picks up → events stream back to ALL clients via ingress SSE - [ ] Confirm CF Access (public) + tailnet bypass behavior on the temporary name before trusting it for the real names ### 6. Cutover (only after parallel run is clean) - [ ] Repoint `dispatch.herbylab.dev` (public) and `dispatch-internal.herbylab.dev` (tailnet) from the fused service to the ingress LXC - [ ] Confirm CF Access + tailnet bypass still behave per 4.1 on the live names - [ ] Retire the fused service's HTTP layer; executor goes headless - [ ] Decommission `split.herbylab.dev` once the live names are proven ## Success Criteria - The full path works against `split.herbylab.dev` with the fused service still live and untouched — proven before any cutover. - Executor has no inbound listening socket (`ss -tlnp` clean). - Ingress app is the only internet-facing component; it can write tasks + read events and do nothing else (no exec, scoped DB role). - App + web client function identically to pre-split (auth, submit, live SSE). - An RCE in the ingress app yields "can enqueue malformed tasks," not host access. ## Open Items - [ ] Confirm SSE seam (clean-lift vs. refactor) — gates effort estimate. - [ ] Executor VM-isolation as later defense-in-depth — deferred, not this phase. - [ ] Decide whether the watcher (`petal-dispatch-worker`) needs any DB-role scoping in the same pass, or stays as-is. ## Notes / Tradeoffs - **LXC vs VM:** LXC chosen for the ingress (low-privilege half). Slightly softer boundary than a VM, but acceptable because the contained process can't exec — an escape escapes the harmless box. VM-grade isolation matters more for the executor, which is deferred. - **The floor was already plumbed, not built.** This phase doesn't invent the queue/event-log boundary — it relocates one side of it to a separate, unprivileged process so the boundary finally separates privileges, not just data. - **Async, not request/response.** Submit is fire-and-forget into the queue; results return via SSE off the event log. This matches the existing model and the fact that most submitted work "requires thought" (no synchronous reply needed).