wiki-vault/Sources/Dev/petal-dispatch-phase-4-2-ingress-split.md

---
created: '2026-06-24'
path: Sources/Dev
project: petal-dispatch-phase-4-2-ingress-split
tags:
- petal-dispatch
- lovebug
- ingress
- executor
- lxc
- postgres
- sse
- security
- isolation
- p4
- phase-4
type: project-plan
---

# petal-dispatch Phase 4.2 — Ingress / Executor Split

## Goal

Split the internet-facing HTTP layer out of the privileged executor so that a
compromise of the exposed surface cannot reach host execution. Today
petal-dispatch is a single host service that **both** accepts inbound requests
**and** spawns Claude/Lovebug processes with host access. That fusion means an
RCE in the web layer is effectively host access. This phase separates the two
roles using the DB as the boundary, activating the blast-radius floor the
architecture already has the plumbing for.

## Why now

The auth work (4.1, Managed OAuth via CF Access) is done and the app + web
client are connected and working. But the security model isn't real yet: the
queue/event-log boundary exists as a *data pattern*, not as a *privilege
boundary*, because both ends live in one privileged process. This phase makes
the boundary load-bearing. Nothing built in 3.x/4.x is discarded — auth, the
queue, the event log, and SSE fan-out all stay; only the *hosting process* for
the HTTP layer changes.

Related prior art: vault `dev-srv-split` (Dev/Srv Deployment Split) is the
conceptual parent of this work.

## Architecture

### Current (fused)

```
[ petal-dispatch (host service on herbys-dev) ]
   = static pages + submit endpoint + SSE stream + queue consumer + Claude spawn
   = internet-facing AND host-privileged  ← the problem
```

### Target (split)

```
[ Apache/static? NO — not needed ]        Traefik already fronts + terminates TLS

[ ingress app  — NEW, in its own LXC ]
   ├─ serves static pages
   ├─ POST submit (auth'd)  → WRITE task row to Postgres
   └─ SSE /events/stream    → READ event log from Postgres, fan out to all clients
   • NO host exec, NO process spawning
   • Postgres role: write tasks + read events ONLY
   • the only internet-facing surface

        │  (Postgres is the air gap — no direct ingress→executor connection)
        ▼
[ Postgres ]  = task queue + event log (LISTEN/NOTIFY)

[ executor (petal-dispatch) — stays on herbys-dev, goes HEADLESS ]
   ├─ reads task queue (LISTEN/NOTIFY)
   ├─ spawns Claude / Lovebug (host access retained)
   └─ writes events to log
   • NO listening socket — outbound DB connections only
   • NOT internet-facing

[ watcher (petal-dispatch-worker) ]  unchanged — reads events, does its thing
```

### Settled decisions

- **DB is the boundary.** Ingress writes tasks; executor reads them. No direct
  network path ingress→executor. Consistent with the locked "DB is truth /
  stream is display" principle (see vault `execution-journal`, `synthesis-layer`).
- **Ingress owns BOTH submit and SSE.** The executor must end up with *zero*
  inbound sockets. Moving only the submit endpoint while leaving SSE on the
  executor would leave it internet-facing and defeat the purpose.
- **Target host = LXC, small Debian.** Lighter than a VM, good-enough isolation
  for the low-privilege half (it can't exec; an escape escapes the harmless box).
- **No local Apache/nginx.** Traefik already fronts everything for TLS. The
  ingress app (extracted async stack) serves static + API + SSE directly;
  Traefik routes to its port.
- **Executor stays on herbys-dev for now.** It keeps host-exec (its job). Its
  own VM-isolation is a later, optional defense-in-depth step — out of scope here.
- **Lovebug-trust boundary is explicitly NOT addressed.** Malicious *task
  content* coming through the queue is a separate concern (the turtle ends here).
  This phase closes the *connection-level* surface, not content trust.
- **Parallel-run on a temporary hostname.** The current fused service stays
  ACTIVE and untouched throughout the build. The ingress LXC is brought up under
  a throwaway name — `split.herbylab.dev` — and all routing/executor/role work is
  validated against that name first. Only once the split is proven clean
  end-to-end do `dispatch.herbylab.dev` / `dispatch-internal.herbylab.dev` get
  repointed to the ingress LXC and the fused service retired. No cutover happens
  on the live names until the parallel path is fully working.

## Needs Lovebug Investigation (do FIRST — sizes the whole job)

These are code-reality questions that determine clean-lift vs. refactor:

- [ ] **Is the SSE stream truly DB-backed, or does it share in-process state
      with the executor?** Grep the SSE handler: does it query Postgres
      (event log + `from_seq`) or subscribe to an in-memory emitter the executor
      feeds? The locked design says DB-backed (LISTEN/NOTIFY, doorbell-not-carrier),
      but verify implementation hasn't drifted. **Clean lift if DB-backed;
      refactor if in-memory coupled.**
- [ ] **Are submit + SSE cleanly separable from the executor process?** Confirmed
      verbally that "everything is served by the executor process." Determine
      whether the HTTP layer is a distinct module (FastAPI app) or entangled with
      executor internals (shared objects, direct calls).
- [ ] **What is the executor's true required reach?** To confirm it can go fully
      headless: does anything *require* the executor to receive inbound
      connections, or are all its inputs the task queue + all outputs the event
      log? If anything else dials into it, surface it now.

## Tasks

### 1. Provision the ingress LXC

- [ ] Create small Debian LXC (sibling pattern to Knot LXC 103, Gitea-to-LXC plan)
- [ ] Network: reachable by Traefik; able to reach Postgres (WireGuard/bridge as
      appropriate); NO need to reach the executor directly
- [ ] Base runtime for the extracted async app (uv/Python per existing stack)

### 2. Extract the HTTP layer

- [ ] Pull static-serving + submit endpoint + SSE handler out of the executor
      into the ingress app
- [ ] Submit endpoint: validate auth'd request → INSERT task row
- [ ] SSE endpoint: read event log (`from_seq` backfill) → fan out to clients
- [ ] Carry over 4.1 auth: CF Access Managed OAuth assertion validation lives
      here now (ingress is the internet-facing surface)

### 3. Scope the ingress Postgres role

- [ ] New DB role: `INSERT` on tasks table, `SELECT` on event log — nothing else
- [ ] No access to other services' tables in the shared Postgres
- [ ] Verify the role cannot read/write outside its two needs

### 4. Make the executor headless

- [ ] Remove the HTTP/listening layer from the executor
- [ ] Executor consumes task queue via LISTEN/NOTIFY (already the pattern)
- [ ] Executor writes events to log (unchanged)
- [ ] **Verify zero listening sockets:** `ss -tlnp` on herbys-dev shows the
      executor binding nothing client-facing. This is the success check.

### 5. Parallel-run on temporary hostname

- [ ] Stand up `split.herbylab.dev` → Traefik → ingress LXC (Knot record +
      Traefik router; mirror the dispatch entrypoint pattern)
- [ ] Keep the current fused `dispatch.herbylab.dev` service ACTIVE and untouched
- [ ] Validate the full path against `split.herbylab.dev`: submit from app + web →
      task lands in DB → executor picks up → events stream back to ALL clients via
      ingress SSE
- [ ] Confirm CF Access (public) + tailnet bypass behavior on the temporary name
      before trusting it for the real names

### 6. Cutover (only after parallel run is clean)

- [ ] Repoint `dispatch.herbylab.dev` (public) and
      `dispatch-internal.herbylab.dev` (tailnet) from the fused service to the
      ingress LXC
- [ ] Confirm CF Access + tailnet bypass still behave per 4.1 on the live names
- [ ] Retire the fused service's HTTP layer; executor goes headless
- [ ] Decommission `split.herbylab.dev` once the live names are proven

## Success Criteria

- The full path works against `split.herbylab.dev` with the fused service still
  live and untouched — proven before any cutover.
- Executor has no inbound listening socket (`ss -tlnp` clean).
- Ingress app is the only internet-facing component; it can write tasks + read
  events and do nothing else (no exec, scoped DB role).
- App + web client function identically to pre-split (auth, submit, live SSE).
- An RCE in the ingress app yields "can enqueue malformed tasks," not host access.

## Open Items

- [ ] Confirm SSE seam (clean-lift vs. refactor) — gates effort estimate.
- [ ] Executor VM-isolation as later defense-in-depth — deferred, not this phase.
- [ ] Decide whether the watcher (`petal-dispatch-worker`) needs any DB-role
      scoping in the same pass, or stays as-is.

## Notes / Tradeoffs

- **LXC vs VM:** LXC chosen for the ingress (low-privilege half). Slightly softer
  boundary than a VM, but acceptable because the contained process can't exec —
  an escape escapes the harmless box. VM-grade isolation matters more for the
  executor, which is deferred.
- **The floor was already plumbed, not built.** This phase doesn't invent the
  queue/event-log boundary — it relocates one side of it to a separate,
  unprivileged process so the boundary finally separates privileges, not just data.
- **Async, not request/response.** Submit is fire-and-forget into the queue;
  results return via SSE off the event log. This matches the existing model and
  the fact that most submitted work "requires thought" (no synchronous reply
  needed).