wiki-vault/Sources/Dev/petal-dispatch-worker.md

created	path	project	tags	type
2026-06-09	Sources/Dev	petal-dispatch-worker	dispatch claude-code mcp automation lovebug python docker	project-plan

Goal

Let the conductor (Dispatch claude / Lovebug) spawn a worker and immediately release the turn instead of blocking until the worker finishes. Today the conductor blocks because blocking is the only reliable way not to miss a worker's completion — there is no worker→conductor push path. The model already narrates the intent ("I'll ping you when it completes") on every spawn, but has no actuator to make it true.

This spec adds that actuator: a standalone watcher that observes worker terminal events and pings the conductor, so the conductor can go idle after spawning and be woken when there's something to look at.

This is net-new architecture that p4 deferred — the inbound half of the orchestrator/harness split and the control-surface "headline open question." It is distinct from p4's pull model (FR-010, get_handoffs) and from FR-013's journal→Trellis context bridge (which is outbound, journal→Trellis, for the human to read). A future p5 should fold this in; this note is the interim spec.

Locked Decisions

D-1 — The watcher is an always-on daemon, not an invoked tool. An invoked watcher would defeat the purpose: the conductor would have to stay alive to manage it, which is just blocking again. A released conductor is idle and can't invoke anything. So the watcher is a separate, always-running, non-Claude process that outlives any single conductor turn.

D-2 — Event source is the existing bus. The watcher subscribes to worker terminal events (task.done / task.crashed) via the bus that already exists (Postgres LISTEN/NOTIFY → broker → SSE, D-6 in p4). The worker terminal path already emits these with full data (tasks.py:544-575 _emit_terminal_event); they currently have no consumer on the orchestrator side. The watcher is that consumer.

D-3 — Routing is automatic via parent lookup. On a terminal event, the watcher looks up the spawning parent (parent_id / thread_id in petaldispatch.journal) and resolves the target thread/session. No registration, no conductor involvement. Even under the current single-spawner assumption (D-7) where the lookup always resolves to the active conductor, the watcher still performs the parent_id lookup — so multi-spawner support later is "the lookup returns a different target," a config reality, not a rewrite.

D-4 — Delivery is the existing intake call, not injection. The wake is delivered via the existing POST /api/chat endpoint (main.py:898) as an authenticated caller with its own service credential — the same door the human client uses, just a new caller. This is not stdin injection; it collapses the trust boundary to "authenticate one more client on an endpoint that already needs authn."

D-5 — The wake is an idempotent pointer, not a payload. The wake message says, in effect, "a worker finished, check the log for completed tasks" — it never carries the result. The conductor reacts by sweeping the journal for ALL terminal tasks, not by handling the one task the wake named. This makes the wake stateless and idempotent: N coalesced wakes and 1 wake produce the identical correct outcome, and a redundant wake for an already-handled worker is a harmless no-op.

D-6 — Watcher-sourced messages get a distinct source/role marker. A third source (alongside user/assistant) in the data model so the wake is attributed to the watcher — distinct UI bubble, honest provenance (not fabricated as a user or conductor turn, consistent with the Phase 9.5 interrupt-marker integrity concern).

D-7 — Single-spawner assumption (current scope). The conductor is the only thing spawning workers, operating one active thread, so the wake target always resolves to the active conductor. Multi-thread spawners (e.g. harnesses spawning workers into other threads) are explicitly out of scope for now. The non-active-thread routing question is deferred under this assumption (but see D-3 forward-compat).

D-8 — Companion instruction flip is required, not optional. The tool alone does not change behavior; the conductor blocks because that's the pattern it reaches for. A conductor operating-instruction edit ships with the watcher: "after spawning a worker, end your turn — do not poll, you'll be woken when it completes." Rule: release by default; block inline only when the result is needed within the current turn to answer what Travis just asked. ("Spawn this and tell me now" → block via read_transcript. Everything else → release.) Tool enables, instruction directs.

D-9 — Do NOT build wait_for_any_task. The watcher subsumes its only justification (fan-in across a batch of workers — which is exactly the spawn-and-release case). The residual inline-single-worker block is already covered by the existing read_transcript(session, max_wait) long-poll. The only thing a dedicated tool would uniquely serve is an inline batch block ("spawn three, all results this turn"), which is vanishingly rare and handled by looping read_transcript.

Open Items

• Schema migration: add the watcher source/role marker (new event_type value or a payload/source field) for watcher-originated messages.
• Watcher service identity: credential for the watcher to call POST /api/chat (the X-Authentik-Username header path, or PETAL_DISPATCH_AUTH_DISABLED semantics for an internal caller).
• Restart safety: a last-processed-seq cursor so a watcher restart does not re-wake for completions it already handled (the journal seq is a table-wide BIGSERIAL; the cursor must pin per the events it consumes).
• Loop-safety: bound the wake→sweep→spawn→complete→wake cycle. Intended convergence, but guard against runaway.
• Deployment: runs as a compose service alongside petal-dispatch (compose.yml; never bare docker run). Lifecycle/restart policy TBD.
• Dedup (polish, optional): redundant wakes are harmless under D-5, so dedup is an optimization, not a correctness requirement. Defer.
• White-box boundary-race probe (optional): the black-box harness can't reach the CLI-internal microsecond window between writing result to stdout and reading the next stdin line; a tighter probe would need a hook at ws.py:82 (pump_claude_to_bus). Not blocking.

Build Phases

Phase 1 — Schema + provenance. Add the watcher source/role marker and migration. Define the wake message shape (pointer text + source attribution).

Phase 2 — Watcher daemon. Standalone process: subscribe to task.done/task.crashed on the bus; on event, look up parent_id/thread_id, resolve target, POST /api/chat the pointer-wake with its service credential. Implement the last-processed-seq cursor for restart safety. Package as a compose service.

Phase 3 — Conductor instruction flip. Edit the conductor's operating instructions per D-8 (release-by-default, inline-block exception). Ship with Phase 2.

Phase 4 — Verification. Reuse scripts/turn_boundary_repro.py (now supports primary / two-mid-turn / boundary-race flows) pointed at the watcher path. Acceptance:

1. Conductor spawns a worker and ends its turn (no poll) under the default instruction.
2. On worker terminal, the watcher posts a wake to the parent within N seconds.
3. The conductor's next turn sweeps the journal and reports/acts on the completed worker(s) without Travis prompting.
4. Two workers finishing near-simultaneously are both handled despite turn coalescing (the sweep catches both).
5. A watcher restart does not replay old completions.

Notes

Verified CLI / queue behavior (empirical, 2026-06-08, harness scripts/turn_boundary_repro.py, throwaway threads 593/594/595):

• No service-side queue. A mid-turn POST /api/chat writes the message to the conductor's CLI stdin immediately — claude_proc.py _stdin_lock (:118) is byte-level serialization only, no in-flight guard, no boundary watcher. Confirmed by grepping the whole src/ tree.
• The claude CLI internally defers a mid-turn user message's generation to the current turn's boundary. Verified: B posted mid-turn, B's row landed immediately, but B's reply only generated after A's result.success; a fresh system.init marks the CLI picking up the next envelope. So the "land in the queue, picked up at the boundary" behavior is real — but owned by the claude CLI, not petal-dispatch. If a future CLI changes input handling, this guarantee moves; the eventual argument for petal-dispatch growing its own in-flight guard is to make the contract ours.
• The CLI COALESCES multiple queued mid-turn messages into ONE turn with one assistant reply (thread 595: B and C queued → single combined turn, only 2 result.success rows for 3 user messages; model followed the most recent under adversarial "reply only with X" prompts). Harmless under D-5's pointer design — N "go look" wakes collapse to one "go look," and the sweep catches everything. All user.message rows still land independently, so the audit trail stays complete. The only residual is cosmetic: a UI rendering one reply bubble per user message would mis-render a coalesced turn (client detail, not correctness).
• Post-boundary arrivals attach cleanly as fresh turns — no drops, duplicates, or reorder across 3 boundary-race iterations (B posted ~28-55µs after A's result SSE). Caveat: that timing is measured downstream of the CLI (SSE propagation is ~5-50ms after the CLI's actual emit), so this proves "right after A finishes is safe" at petal-dispatch's I/O layer, not the CLI-internal microsecond window.

Conflict resolution under coalescing is the model's call. Adversarial contradictory prompts collapsed to "most recent wins," but a normal human question plus a "check the log" wake aren't in conflict — the model does both in one turn. Recency-wins was a test artifact, not a general rule.

Cross-references: build state lives in trellis thread 588 (petal-dispatch-v3-build). The immutable parent spec is the p4 plan (petal-dispatch-p4). Related deferred work: Phase 9.5-interrupt-marker (confabulation fix — same provenance/integrity family as D-6).