mcp: session-notes — Phase 4.2 design correction — executor as full headless backend
This commit is contained in:
Lovebug MCP
parent
85643eb62a
commit
0e1d5c5ecc
@ -0,0 +1,92 @@
|
|||||||
|
---
|
||||||
|
created: '2026-06-25'
|
||||||
|
path: Sources/Dev
|
||||||
|
project: petal-dispatch-phase-4-2-ingress-split
|
||||||
|
tags:
|
||||||
|
- petal-dispatch
|
||||||
|
- lovebug
|
||||||
|
- dispatch
|
||||||
|
- phase-4
|
||||||
|
- ingress
|
||||||
|
type: session-notes
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 4.2 design correction — executor as full headless backend
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
After the parallel-run shipped 2026-06-24 (split.herbylab.dev live, submit_queue path proven end-to-end), the chat-surface discovery exposed that the literal "submit = spawn" reading of plan 300 doesn't match how `petal-dispatch` actually works in production. We resolved the design tension by treating plan 300 as a **goal / boundary document** (split the privileged half from the internet-facing half for security; preserve all functionality), **not a mechanism specification**. This artifact is the operative design reference for Phase 4.2 going forward; plan 300 itself stays immutable but should be read alongside this doc.
|
||||||
|
|
||||||
|
## Topics Covered
|
||||||
|
|
||||||
|
### What the original split implementation got wrong
|
||||||
|
|
||||||
|
The 2026-06-24 worker built `POST /api/chat` → INSERT into `dispatch_harness.submit_queue` → executor spawns a fresh `claude` task per submit. That's the contract for `start_task`, not for chat. Real dispatch chat is a **write to a long-lived `claude` subprocess** (the Lovebug-on-server pid, plus any per-thread procs spawned by Conductor). The stranger reply Travis got to his first split-chat test ("is this working?" → "Yep, working. What're we getting into?") came from a fresh contextless claude that ran for ~3s and died at task.idle. It pretended to be Lovebug but had none of the shared history.
|
||||||
|
|
||||||
|
Abstraction error was Lovebug's, not the worker's — the worker built exactly what the brief said. The brief conflated chat and task-spawn.
|
||||||
|
|
||||||
|
### Goal restated
|
||||||
|
|
||||||
|
Plan 300's actual goal: **security blast-radius reduction**. Split the internet-facing HTTP layer out of the privileged executor so an RCE on the public surface can't reach host execution. The DB is the air gap. User-facing functionality must remain identical to the fused service. Anything less is a partial split that defeats the parity success criterion.
|
||||||
|
|
||||||
|
### Architecture (corrected)
|
||||||
|
|
||||||
|
```
|
||||||
|
dispatch.herbylab.dev → fused petal-dispatch :8001 (unchanged)
|
||||||
|
└─ owns Lovebug (thread_id=null) + thread procs (int ids)
|
||||||
|
|
||||||
|
split.herbylab.dev → ingress LXC :8001 (10.0.11.30)
|
||||||
|
│
|
||||||
|
└─ Postgres dispatch_harness (the air gap)
|
||||||
|
│
|
||||||
|
└─ petal-executor :8002 (the new full headless backend)
|
||||||
|
├─ owns split-Lovebug persistent claude (thread_id=-1)
|
||||||
|
├─ owns split per-thread procs (Phase 2+)
|
||||||
|
├─ owns task workers (submit_queue path, already live)
|
||||||
|
└─ mirrors all snapshot state to Postgres tables
|
||||||
|
```
|
||||||
|
|
||||||
|
The ingress is a thin proxy: static UI + write endpoints (insert into queue tables) + SSE/queries against Postgres state tables. No claude procs on the ingress. No inbound sockets from executor back to ingress — the air gap is one-way (ingress writes; executor reads + writes; executor never opens a listening port for the ingress).
|
||||||
|
|
||||||
|
### Data path
|
||||||
|
|
||||||
|
- **Live updates (event-shaped):** executor publishes to `dispatch_harness.events`. Ingress SSE forwards. UI renders. (Already wired today for `task.*` and `dispatch.*`; no new path needed for new event types.)
|
||||||
|
- **Snapshots (query-shaped):** executor mirrors state to Postgres tables. Ingress queries those tables directly. Tables to add: `chat_queue` (write), `stop_queue` (write, Phase 3), `thread_state` (snapshot, Phase 2), possibly `chat_history` (Phase 4 if executor's journal needs cross-air-gap reads rather than just streamable updates).
|
||||||
|
- **Writes (UI actions):** ingress writes to a queue table → AFTER INSERT trigger fires NOTIFY → executor LISTENs → consumer claims and acts.
|
||||||
|
|
||||||
|
### Temp ID for split-Lovebug
|
||||||
|
|
||||||
|
To allow parallel-run coexistence (two pools, fused-Lovebug + split-Lovebug, neither can share a claude subprocess by construction), split-Lovebug uses **internal `thread_id = -1`** in the `chat_queue` and the executor's proc holder. The UI keeps using the standard "lovebug" / "none" / "" conductor tokens that map to `null` via `_expectFromToken`. The ingress's POST handler does the symbol translation: token → -1 for chat_queue write; response echoes `target_thread_id: null` to satisfy the UI's version check.
|
||||||
|
|
||||||
|
The UI doesn't need to know -1 exists. It's an executor-side label so split-Lovebug stays distinct from fused-Lovebug at the data layer.
|
||||||
|
|
||||||
|
At cutover (out of scope this phase), -1 retires: split-Lovebug gets `claude --resume <real-session-id>` to inherit the fused-Lovebug's session, fused service goes down, executor's now-renamed Lovebug becomes "the" Lovebug.
|
||||||
|
|
||||||
|
### Rejected alternative
|
||||||
|
|
||||||
|
Considered extracting a `dispatch_core` library (refactor `petal_dispatch.main` into core proc/state machinery + thin FastAPI wrapper, both fused and executor import core). Rejected as over-engineering for this phase. Same end-state is reachable by lifting the *specific code paths* petal_executor needs (claude spawn, send_user_message, pump_claude_to_bus, proc lifecycle) into the executor's package without library factoring. Library extraction stays available as a refactor option if duplication becomes painful.
|
||||||
|
|
||||||
|
### Watcher integration
|
||||||
|
|
||||||
|
A second watcher process for the split is explicitly acceptable per Travis. Do NOT over-engineer the existing watcher to be dual-routing. The current watcher continues firing into fused-Lovebug; a split-side watcher (Phase 3) will fire into split-Lovebug. Cheaper than smart routing.
|
||||||
|
|
||||||
|
## Key Learnings
|
||||||
|
|
||||||
|
- **Plans are goal/boundary documents, not mechanism specs.** Plan 300 said "submit task row → executor spawns Claude," which Lovebug and the worker read literally. The right reading was: "the executor owns claude spawning; the ingress writes requests via Postgres; everything that was a single-process flow becomes a queue-mediated flow." Boundary choices (who owns what) are locked by the plan; mechanism choices (queue shape, message shape) are implementation-level.
|
||||||
|
- **Chat in dispatch is process-coupled, not task-spawn shaped.** Worth flagging anywhere a future "minimal split" temptation arises: the persistent dispatch claude procs are core to the UX. Cannot be swapped for on-demand spawns without behavior loss.
|
||||||
|
- **Parallel-run forces dual-proc-pool ownership.** During parallel-run, fused and executor each own their own claude procs. There's no clever way to share — a subprocess's stdin/stdout has exactly one parent. The temp ID -1 is the coexistence labeling, not architecture work.
|
||||||
|
- **DB-is-the-air-gap principle scales to all surfaces.** Chat, stop, threads, history — every cross-process action becomes "write to a queue table, executor consumes, mirrors live state to events table for live UI, mirrors snapshot state to flat tables for cold loads." Single mental model.
|
||||||
|
|
||||||
|
## Follow-ons
|
||||||
|
|
||||||
|
- [ ] **Phase 1 (this session):** persistent split-Lovebug proc on executor (thread_id=-1) + `chat_queue` table + chat_consumer on executor + ingress `/api/chat` rewrite + `target_thread_id` echo. End-to-end chat round-trip with context preservation across messages.
|
||||||
|
- [ ] **Phase 2:** per-thread proc support on executor — `proc_map`, `thread_state` table mirrored to Postgres, Conductor + Trellis integration on executor side.
|
||||||
|
- [ ] **Phase 3:** stop button (`stop_queue`) + split-side watcher daemon. Second watcher process for the split is acceptable.
|
||||||
|
- [ ] **Phase 4:** chat/journal history on split — executor writes journal rows to Postgres for ingress to query on chat-history initial load.
|
||||||
|
- [ ] **Submit queue endpoint relocation:** move off `POST /api/chat` to `POST /api/tasks` (or similar) so the chat surface and the task-spawn surface don't share a URL. Lower priority than the chat fix itself.
|
||||||
|
- [ ] **Cutover (eventual):** stop fused service, spawn one Lovebug on executor with `--resume <real-session-id>` to inherit production history, retire the -1 sentinel, decommission the fused service's HTTP layer. Out of scope until phases 1-4 land and parity is proven.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The fused `petal-dispatch.service` MainPID 691925 has been running since 2026-06-24 09:20:01 EDT. That process holds this Lovebug conversation. Don't restart it across any phase work — its survival is the parallel-run constraint.
|
||||||
|
- Plan 300's "verify zero listening sockets" success criterion is for the cutover endpoint, not the parallel-run state. During parallel-run the new petal-executor binds 8002 (its own MCP mounts + healthz); the strict zero-listen state arrives only at cutover when the fused service retires and the executor takes over the public name.
|
||||||
Loading…
Reference in New Issue
Block a user