From c4eaf69908baf8aeef68586a2040b5a521238c35 Mon Sep 17 00:00:00 2001 From: Lovebug MCP Date: Tue, 9 Jun 2026 02:10:41 +0000 Subject: [PATCH] =?UTF-8?q?mcp:=20project-plan=20=E2=80=94=20petal-dispatc?= =?UTF-8?q?h=20worker=E2=86=92conductor=20wake=20(watcher)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Sources/Dev/petal-dispatch-worker.md | 80 ++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 Sources/Dev/petal-dispatch-worker.md diff --git a/Sources/Dev/petal-dispatch-worker.md b/Sources/Dev/petal-dispatch-worker.md new file mode 100644 index 0000000..0f3cfac --- /dev/null +++ b/Sources/Dev/petal-dispatch-worker.md @@ -0,0 +1,80 @@ +--- +created: '2026-06-09' +path: Sources/Dev +project: petal-dispatch-worker +tags: +- dispatch +- claude-code +- mcp +- automation +- lovebug +- python +- docker +type: project-plan +--- + +## Goal + +Let the conductor (Dispatch claude / Lovebug) spawn a worker and immediately **release the turn** instead of blocking until the worker finishes. Today the conductor blocks because blocking is the only reliable way not to miss a worker's completion — there is no worker→conductor push path. The model already narrates the intent ("I'll ping you when it completes") on every spawn, but has no actuator to make it true. + +This spec adds that actuator: a standalone **watcher** that observes worker terminal events and pings the conductor, so the conductor can go idle after spawning and be woken when there's something to look at. + +This is **net-new architecture that p4 deferred** — the inbound half of the orchestrator/harness split and the control-surface "headline open question." It is distinct from p4's pull model (FR-010, `get_handoffs`) and from FR-013's journal→Trellis context bridge (which is outbound, journal→Trellis, for the human to read). A future p5 should fold this in; this note is the interim spec. + +## Locked Decisions + +**D-1 — The watcher is an always-on daemon, not an invoked tool.** An invoked watcher would defeat the purpose: the conductor would have to stay alive to manage it, which is just blocking again. A released conductor is idle and can't invoke anything. So the watcher is a separate, always-running, non-Claude process that outlives any single conductor turn. + +**D-2 — Event source is the existing bus.** The watcher subscribes to worker terminal events (`task.done` / `task.crashed`) via the bus that already exists (Postgres `LISTEN/NOTIFY` → broker → SSE, D-6 in p4). The worker terminal path already emits these with full data (`tasks.py:544-575` `_emit_terminal_event`); they currently have no consumer on the orchestrator side. The watcher is that consumer. + +**D-3 — Routing is automatic via parent lookup.** On a terminal event, the watcher looks up the spawning parent (`parent_id` / `thread_id` in `petaldispatch.journal`) and resolves the target thread/session. No registration, no conductor involvement. Even under the current single-spawner assumption (D-7) where the lookup always resolves to the active conductor, the watcher **still performs the parent_id lookup** — so multi-spawner support later is "the lookup returns a different target," a config reality, not a rewrite. + +**D-4 — Delivery is the existing intake call, not injection.** The wake is delivered via the existing `POST /api/chat` endpoint (`main.py:898`) as an authenticated caller with its own service credential — the same door the human client uses, just a new caller. This is not stdin injection; it collapses the trust boundary to "authenticate one more client on an endpoint that already needs authn." + +**D-5 — The wake is an idempotent pointer, not a payload.** The wake message says, in effect, "a worker finished, check the log for completed tasks" — it never carries the result. The conductor reacts by **sweeping the journal for ALL terminal tasks**, not by handling the one task the wake named. This makes the wake stateless and idempotent: N coalesced wakes and 1 wake produce the identical correct outcome, and a redundant wake for an already-handled worker is a harmless no-op. + +**D-6 — Watcher-sourced messages get a distinct source/role marker.** A third source (alongside user/assistant) in the data model so the wake is attributed to the watcher — distinct UI bubble, honest provenance (not fabricated as a user or conductor turn, consistent with the Phase 9.5 interrupt-marker integrity concern). + +**D-7 — Single-spawner assumption (current scope).** The conductor is the only thing spawning workers, operating one active thread, so the wake target always resolves to the active conductor. Multi-thread spawners (e.g. harnesses spawning workers into other threads) are explicitly out of scope for now. The non-active-thread routing question is deferred under this assumption (but see D-3 forward-compat). + +**D-8 — Companion instruction flip is required, not optional.** The tool alone does not change behavior; the conductor blocks because that's the pattern it reaches for. A conductor operating-instruction edit ships **with** the watcher: "after spawning a worker, end your turn — do not poll, you'll be woken when it completes." Rule: **release by default; block inline only when the result is needed within the current turn** to answer what Travis just asked. ("Spawn this and tell me now" → block via `read_transcript`. Everything else → release.) Tool enables, instruction directs. + +**D-9 — Do NOT build `wait_for_any_task`.** The watcher subsumes its only justification (fan-in across a batch of workers — which is exactly the spawn-and-release case). The residual inline-single-worker block is already covered by the existing `read_transcript(session, max_wait)` long-poll. The only thing a dedicated tool would uniquely serve is an inline *batch* block ("spawn three, all results this turn"), which is vanishingly rare and handled by looping `read_transcript`. + +## Open Items + +- **Schema migration:** add the watcher source/role marker (new `event_type` value or a payload/source field) for watcher-originated messages. +- **Watcher service identity:** credential for the watcher to call `POST /api/chat` (the `X-Authentik-Username` header path, or `PETAL_DISPATCH_AUTH_DISABLED` semantics for an internal caller). +- **Restart safety:** a last-processed-`seq` cursor so a watcher restart does not re-wake for completions it already handled (the journal `seq` is a table-wide `BIGSERIAL`; the cursor must pin per the events it consumes). +- **Loop-safety:** bound the wake→sweep→spawn→complete→wake cycle. Intended convergence, but guard against runaway. +- **Deployment:** runs as a compose service alongside petal-dispatch (compose.yml; never bare `docker run`). Lifecycle/restart policy TBD. +- **Dedup (polish, optional):** redundant wakes are harmless under D-5, so dedup is an optimization, not a correctness requirement. Defer. +- **White-box boundary-race probe (optional):** the black-box harness can't reach the CLI-internal microsecond window between writing `result` to stdout and reading the next stdin line; a tighter probe would need a hook at `ws.py:82` (`pump_claude_to_bus`). Not blocking. + +## Build Phases + +**Phase 1 — Schema + provenance.** Add the watcher source/role marker and migration. Define the wake message shape (pointer text + source attribution). + +**Phase 2 — Watcher daemon.** Standalone process: subscribe to `task.done`/`task.crashed` on the bus; on event, look up `parent_id`/`thread_id`, resolve target, `POST /api/chat` the pointer-wake with its service credential. Implement the last-processed-`seq` cursor for restart safety. Package as a compose service. + +**Phase 3 — Conductor instruction flip.** Edit the conductor's operating instructions per D-8 (release-by-default, inline-block exception). Ship with Phase 2. + +**Phase 4 — Verification.** Reuse `scripts/turn_boundary_repro.py` (now supports `primary` / `two-mid-turn` / `boundary-race` flows) pointed at the watcher path. Acceptance: +1. Conductor spawns a worker and ends its turn (no poll) under the default instruction. +2. On worker terminal, the watcher posts a wake to the parent within N seconds. +3. The conductor's next turn sweeps the journal and reports/acts on the completed worker(s) without Travis prompting. +4. Two workers finishing near-simultaneously are both handled despite turn coalescing (the sweep catches both). +5. A watcher restart does not replay old completions. + +## Notes + +**Verified CLI / queue behavior (empirical, 2026-06-08, harness `scripts/turn_boundary_repro.py`, throwaway threads 593/594/595):** + +- **No service-side queue.** A mid-turn `POST /api/chat` writes the message to the conductor's CLI stdin *immediately* — `claude_proc.py` `_stdin_lock` (`:118`) is byte-level serialization only, no in-flight guard, no boundary watcher. Confirmed by grepping the whole `src/` tree. +- **The claude CLI internally defers** a mid-turn user message's *generation* to the current turn's boundary. Verified: B posted mid-turn, B's row landed immediately, but B's reply only generated after A's `result.success`; a fresh `system.init` marks the CLI picking up the next envelope. So the "land in the queue, picked up at the boundary" behavior is real — but **owned by the claude CLI, not petal-dispatch.** If a future CLI changes input handling, this guarantee moves; the eventual argument for petal-dispatch growing its own in-flight guard is to make the contract *ours*. +- **The CLI COALESCES** multiple queued mid-turn messages into ONE turn with one assistant reply (thread 595: B and C queued → single combined turn, only 2 `result.success` rows for 3 user messages; model followed the most recent under adversarial "reply only with X" prompts). **Harmless under D-5's pointer design** — N "go look" wakes collapse to one "go look," and the sweep catches everything. All `user.message` rows still land independently, so the audit trail stays complete. The only residual is cosmetic: a UI rendering one reply bubble per user message would mis-render a coalesced turn (client detail, not correctness). +- **Post-boundary arrivals attach cleanly** as fresh turns — no drops, duplicates, or reorder across 3 boundary-race iterations (B posted ~28-55µs after A's result SSE). Caveat: that timing is measured downstream of the CLI (SSE propagation is ~5-50ms after the CLI's actual emit), so this proves "right after A finishes is safe" at petal-dispatch's I/O layer, not the CLI-internal microsecond window. + +**Conflict resolution under coalescing** is the model's call. Adversarial contradictory prompts collapsed to "most recent wins," but a normal human question plus a "check the log" wake aren't in conflict — the model does both in one turn. Recency-wins was a test artifact, not a general rule. + +**Cross-references:** build state lives in trellis thread 588 (`petal-dispatch-v3-build`). The immutable parent spec is the p4 plan (`petal-dispatch-p4`). Related deferred work: Phase 9.5-interrupt-marker (confabulation fix — same provenance/integrity family as D-6). \ No newline at end of file