diff --git a/Sources/Dev/2026-06-09-petal-dispatch-watcher-deploy-fix.md b/Sources/Dev/2026-06-09-petal-dispatch-watcher-deploy-fix.md new file mode 100644 index 0000000..458a3fb --- /dev/null +++ b/Sources/Dev/2026-06-09-petal-dispatch-watcher-deploy-fix.md @@ -0,0 +1,42 @@ +--- +created: '2026-06-09' +path: Sources/Dev +project: petal-dispatch-worker +tags: +- dispatch +- claude-code +- automation +- lovebug +- python +type: session-notes +--- + +## Outcome + +Built, deployed, and root-cause-debugged the worker→conductor watcher daemon (`petal-dispatch-worker` plan) end to end in one session. The daemon is implemented, tested (22 watcher tests + full suite green, zero-check 5/5), deployed via systemd, and **verified working live** after a root-cause fix. The branch `watcher-wake-path` carries three commits (`03a180d` daemon + provenance, `eeb0d62` diagnostic instrumentation, `d521c9f` the terminal-kinds fix) off `main`, awaiting Travis's merge + a `petal-dispatch-watcher` restart to persist, plus a separate `petal-dispatch` restart to activate the conductor instruction flip. + +**The headline correction this note exists to record:** the spec's D-2 names the wrong terminal events. See Key Learnings. + +## Topics Covered + +- Watcher daemon build: asyncpg LISTEN/NOTIFY on `dispatch_event`, fsync-stable seq cursor, sliding-window rate limiter, coalescing nap, httpx POST to `/api/chat` with an `X-Watcher-Token` shared secret. Compose service + Dockerfile + systemd unit (systemd chosen — matches how `petal-dispatch` itself runs). +- Live deploy steps: migration `004_watcher_event_type.sql` (idempotent INSERT into `petaldispatch.event_type`), shared secret in `~/.config/petal-dispatch/secrets.env` (read by both services), dispatch restart, watcher systemd enable. +- Cold-start backfill bug: a fresh watcher with no cursor file seeds from seq 0 and replays the entire history as one coalesced wake. Harmless under the idempotent-pointer design (D-5) but noisy; should seed cursor to current-max on cold start. Not yet fixed — deferred. +- Live failure diagnosis driven to root cause through the build code session (orchestrator is sandboxed off the herbydev host, so all `docker exec` / `journalctl` / `psql` host work routes through an on-host code session). + +## Key Learnings + +- **`task.idle` is the real worker terminal signal, NOT `task.done`/`task.crashed`.** The spec (`petal-dispatch-worker` plan, D-2) and the original build brief both named `task.done`/`task.crashed`. That matched design language but not operational reality. Workers run as `claude --print --input-format stream-json --resume ` — in stream-json input mode the CLI does **not** exit after a turn, it stays alive on stdin. So the subprocess-exit codepath in `tasks.py:_emit_terminal_event` (`:544-575`) that fires `task.done`/`task.crashed` almost never triggers. Ground truth: across the system's entire history, `dispatch_harness.events` holds 36 `task.crashed`, 18 `task.idle`, 2 `task.deleted`, and **0 `task.done`**. The fix (`d521c9f`) expands `TERMINAL_KINDS` to `task.done, task.crashed, task.stopped, task.deleted, task.idle` (excluding `task.paused`, which is reversible). **Trust this note over D-2 in the frozen plan body.** +- **Worker terminal events live in `dispatch_harness.events`, not `petaldispatch.journal`.** Two independent `BIGSERIAL` sequences in two different tables. The watcher's cursor value (e.g. 2208, 2810) is a `dispatch_harness.events.seq`, unrelated to journal seq. A query of `petaldispatch.journal WHERE seq > N` for worker terminals will always return zero rows — wrong table. `petaldispatch.journal` registers only `task.spawned` / `task.exit` among task events. +- **The single NOTIFY channel is `dispatch_event`**, fired by an AFTER-INSERT trigger on `dispatch_harness.events` (`db/migrations/001_dispatch_harness.sql:32-42`). Publish path: `Worker._emit_terminal_event` → `_make_publisher` → `broker.publish` → `harness.py:104-115` INSERT. The watcher's channel/table were correct all along; only the kind filter was wrong. The LISTEN socket was healthy (`notifies_received=17` during the synthetic test) — the bug looked like a dead socket but was a too-narrow filter silently dropping every real event. +- **Semantic shift from the fix:** the wake now means "worker hit a turn boundary / has output," not strictly "worker exited." For the one-shot worker pattern (spawn → one turn → idle) that equals "done" and is exactly what the conductor needs. But for long-lived/interactive workers that idle repeatedly mid-task (p4's deferred v4 stage), the wake would fire on each idle — revisit then. +- **Live testing earns its keep.** Unit tests mocked the event source, so a wrong-kind filter sailed through 22 green tests. Only a real worker on the real bus exposed it. Same lesson as the turn-boundary harness work earlier in the session: verify against ground truth, don't trust the claim. + +## Follow-ons + +- [ ] Merge `watcher-wake-path` to `main` (`git merge --no-ff …`) and `sudo systemctl restart petal-dispatch-watcher` to persist the fix (currently running in-memory only). +- [ ] `sudo systemctl restart petal-dispatch` to activate the conductor instruction flip (release-turn-after-spawn) — the last unverified acceptance criterion (a). +- [ ] Re-run the spawn-a-worker live test after both restarts; confirm the full loop (spawn → release turn → watcher wakes on idle → conductor sweeps unprompted). +- [ ] Fix the cold-start cursor seeding: on missing cursor file, seed to current `MAX(seq)` instead of 0, so a fresh deploy doesn't replay history. +- [ ] Reap orphan worker process PID 1088451 (stream-json worker alive since the prior day) — Travis-gated. +- [ ] Consider folding the corrected terminal-kinds reality into a p5 revision of the plan, since the project-plan body is frozen. \ No newline at end of file