wiki-vault/Sources/Dev/2026-06-09-petal-dispatch-watcher-deploy-fix.md

created	path	project	tags	type
2026-06-09	Sources/Dev	petal-dispatch-worker	dispatch claude-code automation lovebug python	session-notes

Outcome

Built, deployed, and root-cause-debugged the worker→conductor watcher daemon (petal-dispatch-worker plan) end to end in one session. The daemon is implemented, tested (22 watcher tests + full suite green, zero-check 5/5), deployed via systemd, and verified working live after a root-cause fix. The branch watcher-wake-path carries three commits (03a180d daemon + provenance, eeb0d62 diagnostic instrumentation, d521c9f the terminal-kinds fix) off main, awaiting Travis's merge + a petal-dispatch-watcher restart to persist, plus a separate petal-dispatch restart to activate the conductor instruction flip.

The headline correction this note exists to record: the spec's D-2 names the wrong terminal events. See Key Learnings.

Topics Covered

• Watcher daemon build: asyncpg LISTEN/NOTIFY on dispatch_event, fsync-stable seq cursor, sliding-window rate limiter, coalescing nap, httpx POST to /api/chat with an X-Watcher-Token shared secret. Compose service + Dockerfile + systemd unit (systemd chosen — matches how petal-dispatch itself runs).
• Live deploy steps: migration 004_watcher_event_type.sql (idempotent INSERT into petaldispatch.event_type), shared secret in ~/.config/petal-dispatch/secrets.env (read by both services), dispatch restart, watcher systemd enable.
• Cold-start backfill bug: a fresh watcher with no cursor file seeds from seq 0 and replays the entire history as one coalesced wake. Harmless under the idempotent-pointer design (D-5) but noisy; should seed cursor to current-max on cold start. Not yet fixed — deferred.
• Live failure diagnosis driven to root cause through the build code session (orchestrator is sandboxed off the herbydev host, so all docker exec / journalctl / psql host work routes through an on-host code session).

Key Learnings

• task.idle is the real worker terminal signal, NOT task.done/task.crashed. The spec (petal-dispatch-worker plan, D-2) and the original build brief both named task.done/task.crashed. That matched design language but not operational reality. Workers run as claude --print --input-format stream-json --resume <uuid> — in stream-json input mode the CLI does not exit after a turn, it stays alive on stdin. So the subprocess-exit codepath in tasks.py:_emit_terminal_event (:544-575) that fires task.done/task.crashed almost never triggers. Ground truth: across the system's entire history, dispatch_harness.events holds 36 task.crashed, 18 task.idle, 2 task.deleted, and 0 task.done. The fix (d521c9f) expands TERMINAL_KINDS to task.done, task.crashed, task.stopped, task.deleted, task.idle (excluding task.paused, which is reversible). Trust this note over D-2 in the frozen plan body.
• Worker terminal events live in dispatch_harness.events, not petaldispatch.journal. Two independent BIGSERIAL sequences in two different tables. The watcher's cursor value (e.g. 2208, 2810) is a dispatch_harness.events.seq, unrelated to journal seq. A query of petaldispatch.journal WHERE seq > N for worker terminals will always return zero rows — wrong table. petaldispatch.journal registers only task.spawned / task.exit among task events.
• The single NOTIFY channel is dispatch_event, fired by an AFTER-INSERT trigger on dispatch_harness.events (db/migrations/001_dispatch_harness.sql:32-42). Publish path: Worker._emit_terminal_event → _make_publisher → broker.publish → harness.py:104-115 INSERT. The watcher's channel/table were correct all along; only the kind filter was wrong. The LISTEN socket was healthy (notifies_received=17 during the synthetic test) — the bug looked like a dead socket but was a too-narrow filter silently dropping every real event.
• Semantic shift from the fix: the wake now means "worker hit a turn boundary / has output," not strictly "worker exited." For the one-shot worker pattern (spawn → one turn → idle) that equals "done" and is exactly what the conductor needs. But for long-lived/interactive workers that idle repeatedly mid-task (p4's deferred v4 stage), the wake would fire on each idle — revisit then.
• Live testing earns its keep. Unit tests mocked the event source, so a wrong-kind filter sailed through 22 green tests. Only a real worker on the real bus exposed it. Same lesson as the turn-boundary harness work earlier in the session: verify against ground truth, don't trust the claim.

Follow-ons

• Merge watcher-wake-path to main (git merge --no-ff …) and sudo systemctl restart petal-dispatch-watcher to persist the fix (currently running in-memory only).
• sudo systemctl restart petal-dispatch to activate the conductor instruction flip (release-turn-after-spawn) — the last unverified acceptance criterion (a).
• Re-run the spawn-a-worker live test after both restarts; confirm the full loop (spawn → release turn → watcher wakes on idle → conductor sweeps unprompted).
• Fix the cold-start cursor seeding: on missing cursor file, seed to current MAX(seq) instead of 0, so a fresh deploy doesn't replay history.
• Reap orphan worker process PID 1088451 (stream-json worker alive since the prior day) — Travis-gated.
• Consider folding the corrected terminal-kinds reality into a p5 revision of the plan, since the project-plan body is frozen.