From 0e1d5c5ecc772e063f53dab92f63addc2770ab12 Mon Sep 17 00:00:00 2001 From: Lovebug MCP Date: Thu, 25 Jun 2026 06:50:27 +0000 Subject: [PATCH] =?UTF-8?q?mcp:=20session-notes=20=E2=80=94=20Phase=204.2?= =?UTF-8?q?=20design=20correction=20=E2=80=94=20executor=20as=20full=20hea?= =?UTF-8?q?dless=20backend?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...ction-executor-as-full-headless-backend.md | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 Sources/Dev/2026-06-25-phase-42-design-correction-executor-as-full-headless-backend.md diff --git a/Sources/Dev/2026-06-25-phase-42-design-correction-executor-as-full-headless-backend.md b/Sources/Dev/2026-06-25-phase-42-design-correction-executor-as-full-headless-backend.md new file mode 100644 index 0000000..6bef93f --- /dev/null +++ b/Sources/Dev/2026-06-25-phase-42-design-correction-executor-as-full-headless-backend.md @@ -0,0 +1,92 @@ +--- +created: '2026-06-25' +path: Sources/Dev +project: petal-dispatch-phase-4-2-ingress-split +tags: +- petal-dispatch +- lovebug +- dispatch +- phase-4 +- ingress +type: session-notes +--- + +# Phase 4.2 design correction — executor as full headless backend + +## Outcome + +After the parallel-run shipped 2026-06-24 (split.herbylab.dev live, submit_queue path proven end-to-end), the chat-surface discovery exposed that the literal "submit = spawn" reading of plan 300 doesn't match how `petal-dispatch` actually works in production. We resolved the design tension by treating plan 300 as a **goal / boundary document** (split the privileged half from the internet-facing half for security; preserve all functionality), **not a mechanism specification**. This artifact is the operative design reference for Phase 4.2 going forward; plan 300 itself stays immutable but should be read alongside this doc. + +## Topics Covered + +### What the original split implementation got wrong + +The 2026-06-24 worker built `POST /api/chat` → INSERT into `dispatch_harness.submit_queue` → executor spawns a fresh `claude` task per submit. That's the contract for `start_task`, not for chat. Real dispatch chat is a **write to a long-lived `claude` subprocess** (the Lovebug-on-server pid, plus any per-thread procs spawned by Conductor). The stranger reply Travis got to his first split-chat test ("is this working?" → "Yep, working. What're we getting into?") came from a fresh contextless claude that ran for ~3s and died at task.idle. It pretended to be Lovebug but had none of the shared history. + +Abstraction error was Lovebug's, not the worker's — the worker built exactly what the brief said. The brief conflated chat and task-spawn. + +### Goal restated + +Plan 300's actual goal: **security blast-radius reduction**. Split the internet-facing HTTP layer out of the privileged executor so an RCE on the public surface can't reach host execution. The DB is the air gap. User-facing functionality must remain identical to the fused service. Anything less is a partial split that defeats the parity success criterion. + +### Architecture (corrected) + +``` +dispatch.herbylab.dev → fused petal-dispatch :8001 (unchanged) + └─ owns Lovebug (thread_id=null) + thread procs (int ids) + +split.herbylab.dev → ingress LXC :8001 (10.0.11.30) + │ + └─ Postgres dispatch_harness (the air gap) + │ + └─ petal-executor :8002 (the new full headless backend) + ├─ owns split-Lovebug persistent claude (thread_id=-1) + ├─ owns split per-thread procs (Phase 2+) + ├─ owns task workers (submit_queue path, already live) + └─ mirrors all snapshot state to Postgres tables +``` + +The ingress is a thin proxy: static UI + write endpoints (insert into queue tables) + SSE/queries against Postgres state tables. No claude procs on the ingress. No inbound sockets from executor back to ingress — the air gap is one-way (ingress writes; executor reads + writes; executor never opens a listening port for the ingress). + +### Data path + +- **Live updates (event-shaped):** executor publishes to `dispatch_harness.events`. Ingress SSE forwards. UI renders. (Already wired today for `task.*` and `dispatch.*`; no new path needed for new event types.) +- **Snapshots (query-shaped):** executor mirrors state to Postgres tables. Ingress queries those tables directly. Tables to add: `chat_queue` (write), `stop_queue` (write, Phase 3), `thread_state` (snapshot, Phase 2), possibly `chat_history` (Phase 4 if executor's journal needs cross-air-gap reads rather than just streamable updates). +- **Writes (UI actions):** ingress writes to a queue table → AFTER INSERT trigger fires NOTIFY → executor LISTENs → consumer claims and acts. + +### Temp ID for split-Lovebug + +To allow parallel-run coexistence (two pools, fused-Lovebug + split-Lovebug, neither can share a claude subprocess by construction), split-Lovebug uses **internal `thread_id = -1`** in the `chat_queue` and the executor's proc holder. The UI keeps using the standard "lovebug" / "none" / "" conductor tokens that map to `null` via `_expectFromToken`. The ingress's POST handler does the symbol translation: token → -1 for chat_queue write; response echoes `target_thread_id: null` to satisfy the UI's version check. + +The UI doesn't need to know -1 exists. It's an executor-side label so split-Lovebug stays distinct from fused-Lovebug at the data layer. + +At cutover (out of scope this phase), -1 retires: split-Lovebug gets `claude --resume ` to inherit the fused-Lovebug's session, fused service goes down, executor's now-renamed Lovebug becomes "the" Lovebug. + +### Rejected alternative + +Considered extracting a `dispatch_core` library (refactor `petal_dispatch.main` into core proc/state machinery + thin FastAPI wrapper, both fused and executor import core). Rejected as over-engineering for this phase. Same end-state is reachable by lifting the *specific code paths* petal_executor needs (claude spawn, send_user_message, pump_claude_to_bus, proc lifecycle) into the executor's package without library factoring. Library extraction stays available as a refactor option if duplication becomes painful. + +### Watcher integration + +A second watcher process for the split is explicitly acceptable per Travis. Do NOT over-engineer the existing watcher to be dual-routing. The current watcher continues firing into fused-Lovebug; a split-side watcher (Phase 3) will fire into split-Lovebug. Cheaper than smart routing. + +## Key Learnings + +- **Plans are goal/boundary documents, not mechanism specs.** Plan 300 said "submit task row → executor spawns Claude," which Lovebug and the worker read literally. The right reading was: "the executor owns claude spawning; the ingress writes requests via Postgres; everything that was a single-process flow becomes a queue-mediated flow." Boundary choices (who owns what) are locked by the plan; mechanism choices (queue shape, message shape) are implementation-level. +- **Chat in dispatch is process-coupled, not task-spawn shaped.** Worth flagging anywhere a future "minimal split" temptation arises: the persistent dispatch claude procs are core to the UX. Cannot be swapped for on-demand spawns without behavior loss. +- **Parallel-run forces dual-proc-pool ownership.** During parallel-run, fused and executor each own their own claude procs. There's no clever way to share — a subprocess's stdin/stdout has exactly one parent. The temp ID -1 is the coexistence labeling, not architecture work. +- **DB-is-the-air-gap principle scales to all surfaces.** Chat, stop, threads, history — every cross-process action becomes "write to a queue table, executor consumes, mirrors live state to events table for live UI, mirrors snapshot state to flat tables for cold loads." Single mental model. + +## Follow-ons + +- [ ] **Phase 1 (this session):** persistent split-Lovebug proc on executor (thread_id=-1) + `chat_queue` table + chat_consumer on executor + ingress `/api/chat` rewrite + `target_thread_id` echo. End-to-end chat round-trip with context preservation across messages. +- [ ] **Phase 2:** per-thread proc support on executor — `proc_map`, `thread_state` table mirrored to Postgres, Conductor + Trellis integration on executor side. +- [ ] **Phase 3:** stop button (`stop_queue`) + split-side watcher daemon. Second watcher process for the split is acceptable. +- [ ] **Phase 4:** chat/journal history on split — executor writes journal rows to Postgres for ingress to query on chat-history initial load. +- [ ] **Submit queue endpoint relocation:** move off `POST /api/chat` to `POST /api/tasks` (or similar) so the chat surface and the task-spawn surface don't share a URL. Lower priority than the chat fix itself. +- [ ] **Cutover (eventual):** stop fused service, spawn one Lovebug on executor with `--resume ` to inherit production history, retire the -1 sentinel, decommission the fused service's HTTP layer. Out of scope until phases 1-4 land and parity is proven. + +## Notes + +- The fused `petal-dispatch.service` MainPID 691925 has been running since 2026-06-24 09:20:01 EDT. That process holds this Lovebug conversation. Don't restart it across any phase work — its survival is the parallel-run constraint. +- Plan 300's "verify zero listening sockets" success criterion is for the cutover endpoint, not the parallel-run state. During parallel-run the new petal-executor binds 8002 (its own MCP mounts + healthz); the strict zero-listen state arrives only at cutover when the fused service retires and the executor takes over the public name. \ No newline at end of file