Phase 4.2 design correction — executor as full headless backend

Outcome

After the parallel-run shipped 2026-06-24 (split.herbylab.dev live, submit_queue path proven end-to-end), the chat-surface discovery exposed that the literal "submit = spawn" reading of plan 300 doesn't match how petal-dispatch actually works in production. We resolved the design tension by treating plan 300 as a goal / boundary document (split the privileged half from the internet-facing half for security; preserve all functionality), not a mechanism specification. This artifact is the operative design reference for Phase 4.2 going forward; plan 300 itself stays immutable but should be read alongside this doc.

Topics Covered

What the original split implementation got wrong

The 2026-06-24 worker built POST /api/chat → INSERT into dispatch_harness.submit_queue → executor spawns a fresh claude task per submit. That's the contract for start_task, not for chat. Real dispatch chat is a write to a long-lived claude subprocess (the Lovebug-on-server pid, plus any per-thread procs spawned by Conductor). The stranger reply Travis got to his first split-chat test ("is this working?" → "Yep, working. What're we getting into?") came from a fresh contextless claude that ran for ~3s and died at task.idle. It pretended to be Lovebug but had none of the shared history.

Abstraction error was Lovebug's, not the worker's — the worker built exactly what the brief said. The brief conflated chat and task-spawn.

Goal restated

Plan 300's actual goal: security blast-radius reduction. Split the internet-facing HTTP layer out of the privileged executor so an RCE on the public surface can't reach host execution. The DB is the air gap. User-facing functionality must remain identical to the fused service. Anything less is a partial split that defeats the parity success criterion.

Architecture (corrected)

dispatch.herbylab.dev → fused petal-dispatch :8001 (unchanged)
                          └─ owns Lovebug (thread_id=null) + thread procs (int ids)

split.herbylab.dev    → ingress LXC :8001 (10.0.11.30)
                          │
                          └─ Postgres dispatch_harness (the air gap)
                                │
                                └─ petal-executor :8002 (the new full headless backend)
                                      ├─ owns split-Lovebug persistent claude (thread_id=-1)
                                      ├─ owns split per-thread procs (Phase 2+)
                                      ├─ owns task workers (submit_queue path, already live)
                                      └─ mirrors all snapshot state to Postgres tables

The ingress is a thin proxy: static UI + write endpoints (insert into queue tables) + SSE/queries against Postgres state tables. No claude procs on the ingress. No inbound sockets from executor back to ingress — the air gap is one-way (ingress writes; executor reads + writes; executor never opens a listening port for the ingress).

Data path

Live updates (event-shaped): executor publishes to dispatch_harness.events. Ingress SSE forwards. UI renders. (Already wired today for task.* and dispatch.*; no new path needed for new event types.)
Snapshots (query-shaped): executor mirrors state to Postgres tables. Ingress queries those tables directly. Tables to add: chat_queue (write), stop_queue (write, Phase 3), thread_state (snapshot, Phase 2), possibly chat_history (Phase 4 if executor's journal needs cross-air-gap reads rather than just streamable updates).
Writes (UI actions): ingress writes to a queue table → AFTER INSERT trigger fires NOTIFY → executor LISTENs → consumer claims and acts.

Temp ID for split-Lovebug

To allow parallel-run coexistence (two pools, fused-Lovebug + split-Lovebug, neither can share a claude subprocess by construction), split-Lovebug uses internal thread_id = -1 in the chat_queue and the executor's proc holder. The UI keeps using the standard "lovebug" / "none" / "" conductor tokens that map to null via _expectFromToken. The ingress's POST handler does the symbol translation: token → -1 for chat_queue write; response echoes target_thread_id: null to satisfy the UI's version check.

The UI doesn't need to know -1 exists. It's an executor-side label so split-Lovebug stays distinct from fused-Lovebug at the data layer.

At cutover (out of scope this phase), -1 retires: split-Lovebug gets claude --resume <real-session-id> to inherit the fused-Lovebug's session, fused service goes down, executor's now-renamed Lovebug becomes "the" Lovebug.

Rejected alternative

Considered extracting a dispatch_core library (refactor petal_dispatch.main into core proc/state machinery + thin FastAPI wrapper, both fused and executor import core). Rejected as over-engineering for this phase. Same end-state is reachable by lifting the specific code paths petal_executor needs (claude spawn, send_user_message, pump_claude_to_bus, proc lifecycle) into the executor's package without library factoring. Library extraction stays available as a refactor option if duplication becomes painful.

Watcher integration

A second watcher process for the split is explicitly acceptable per Travis. Do NOT over-engineer the existing watcher to be dual-routing. The current watcher continues firing into fused-Lovebug; a split-side watcher (Phase 3) will fire into split-Lovebug. Cheaper than smart routing.

Key Learnings

Plans are goal/boundary documents, not mechanism specs. Plan 300 said "submit task row → executor spawns Claude," which Lovebug and the worker read literally. The right reading was: "the executor owns claude spawning; the ingress writes requests via Postgres; everything that was a single-process flow becomes a queue-mediated flow." Boundary choices (who owns what) are locked by the plan; mechanism choices (queue shape, message shape) are implementation-level.
Chat in dispatch is process-coupled, not task-spawn shaped. Worth flagging anywhere a future "minimal split" temptation arises: the persistent dispatch claude procs are core to the UX. Cannot be swapped for on-demand spawns without behavior loss.
Parallel-run forces dual-proc-pool ownership. During parallel-run, fused and executor each own their own claude procs. There's no clever way to share — a subprocess's stdin/stdout has exactly one parent. The temp ID -1 is the coexistence labeling, not architecture work.
DB-is-the-air-gap principle scales to all surfaces. Chat, stop, threads, history — every cross-process action becomes "write to a queue table, executor consumes, mirrors live state to events table for live UI, mirrors snapshot state to flat tables for cold loads." Single mental model.

Follow-ons

Phase 1 (this session): persistent split-Lovebug proc on executor (thread_id=-1) + chat_queue table + chat_consumer on executor + ingress /api/chat rewrite + target_thread_id echo. End-to-end chat round-trip with context preservation across messages.
Phase 2: per-thread proc support on executor — proc_map, thread_state table mirrored to Postgres, Conductor + Trellis integration on executor side.
Phase 3: stop button (stop_queue) + split-side watcher daemon. Second watcher process for the split is acceptable.
Phase 4: chat/journal history on split — executor writes journal rows to Postgres for ingress to query on chat-history initial load.
Submit queue endpoint relocation: move off POST /api/chat to POST /api/tasks (or similar) so the chat surface and the task-spawn surface don't share a URL. Lower priority than the chat fix itself.
Cutover (eventual): stop fused service, spawn one Lovebug on executor with --resume <real-session-id> to inherit production history, retire the -1 sentinel, decommission the fused service's HTTP layer. Out of scope until phases 1-4 land and parity is proven.

Notes

The fused petal-dispatch.service MainPID 691925 has been running since 2026-06-24 09:20:01 EDT. That process holds this Lovebug conversation. Don't restart it across any phase work — its survival is the parallel-run constraint.
Plan 300's "verify zero listening sockets" success criterion is for the cutover endpoint, not the parallel-run state. During parallel-run the new petal-executor binds 8002 (its own MCP mounts + healthz); the strict zero-listen state arrives only at cutover when the fused service retires and the executor takes over the public name.

8.6 KiB Raw Blame History