42 lines
5.4 KiB
Markdown
42 lines
5.4 KiB
Markdown
---
|
|
created: '2026-06-09'
|
|
path: Sources/Dev
|
|
project: petal-dispatch-worker
|
|
tags:
|
|
- dispatch
|
|
- claude-code
|
|
- automation
|
|
- lovebug
|
|
- python
|
|
type: session-notes
|
|
---
|
|
|
|
## Outcome
|
|
|
|
Built, deployed, and root-cause-debugged the worker→conductor watcher daemon (`petal-dispatch-worker` plan) end to end in one session. The daemon is implemented, tested (22 watcher tests + full suite green, zero-check 5/5), deployed via systemd, and **verified working live** after a root-cause fix. The branch `watcher-wake-path` carries three commits (`03a180d` daemon + provenance, `eeb0d62` diagnostic instrumentation, `d521c9f` the terminal-kinds fix) off `main`, awaiting Travis's merge + a `petal-dispatch-watcher` restart to persist, plus a separate `petal-dispatch` restart to activate the conductor instruction flip.
|
|
|
|
**The headline correction this note exists to record:** the spec's D-2 names the wrong terminal events. See Key Learnings.
|
|
|
|
## Topics Covered
|
|
|
|
- Watcher daemon build: asyncpg LISTEN/NOTIFY on `dispatch_event`, fsync-stable seq cursor, sliding-window rate limiter, coalescing nap, httpx POST to `/api/chat` with an `X-Watcher-Token` shared secret. Compose service + Dockerfile + systemd unit (systemd chosen — matches how `petal-dispatch` itself runs).
|
|
- Live deploy steps: migration `004_watcher_event_type.sql` (idempotent INSERT into `petaldispatch.event_type`), shared secret in `~/.config/petal-dispatch/secrets.env` (read by both services), dispatch restart, watcher systemd enable.
|
|
- Cold-start backfill bug: a fresh watcher with no cursor file seeds from seq 0 and replays the entire history as one coalesced wake. Harmless under the idempotent-pointer design (D-5) but noisy; should seed cursor to current-max on cold start. Not yet fixed — deferred.
|
|
- Live failure diagnosis driven to root cause through the build code session (orchestrator is sandboxed off the herbydev host, so all `docker exec` / `journalctl` / `psql` host work routes through an on-host code session).
|
|
|
|
## Key Learnings
|
|
|
|
- **`task.idle` is the real worker terminal signal, NOT `task.done`/`task.crashed`.** The spec (`petal-dispatch-worker` plan, D-2) and the original build brief both named `task.done`/`task.crashed`. That matched design language but not operational reality. Workers run as `claude --print --input-format stream-json --resume <uuid>` — in stream-json input mode the CLI does **not** exit after a turn, it stays alive on stdin. So the subprocess-exit codepath in `tasks.py:_emit_terminal_event` (`:544-575`) that fires `task.done`/`task.crashed` almost never triggers. Ground truth: across the system's entire history, `dispatch_harness.events` holds 36 `task.crashed`, 18 `task.idle`, 2 `task.deleted`, and **0 `task.done`**. The fix (`d521c9f`) expands `TERMINAL_KINDS` to `task.done, task.crashed, task.stopped, task.deleted, task.idle` (excluding `task.paused`, which is reversible). **Trust this note over D-2 in the frozen plan body.**
|
|
- **Worker terminal events live in `dispatch_harness.events`, not `petaldispatch.journal`.** Two independent `BIGSERIAL` sequences in two different tables. The watcher's cursor value (e.g. 2208, 2810) is a `dispatch_harness.events.seq`, unrelated to journal seq. A query of `petaldispatch.journal WHERE seq > N` for worker terminals will always return zero rows — wrong table. `petaldispatch.journal` registers only `task.spawned` / `task.exit` among task events.
|
|
- **The single NOTIFY channel is `dispatch_event`**, fired by an AFTER-INSERT trigger on `dispatch_harness.events` (`db/migrations/001_dispatch_harness.sql:32-42`). Publish path: `Worker._emit_terminal_event` → `_make_publisher` → `broker.publish` → `harness.py:104-115` INSERT. The watcher's channel/table were correct all along; only the kind filter was wrong. The LISTEN socket was healthy (`notifies_received=17` during the synthetic test) — the bug looked like a dead socket but was a too-narrow filter silently dropping every real event.
|
|
- **Semantic shift from the fix:** the wake now means "worker hit a turn boundary / has output," not strictly "worker exited." For the one-shot worker pattern (spawn → one turn → idle) that equals "done" and is exactly what the conductor needs. But for long-lived/interactive workers that idle repeatedly mid-task (p4's deferred v4 stage), the wake would fire on each idle — revisit then.
|
|
- **Live testing earns its keep.** Unit tests mocked the event source, so a wrong-kind filter sailed through 22 green tests. Only a real worker on the real bus exposed it. Same lesson as the turn-boundary harness work earlier in the session: verify against ground truth, don't trust the claim.
|
|
|
|
## Follow-ons
|
|
|
|
- [ ] Merge `watcher-wake-path` to `main` (`git merge --no-ff …`) and `sudo systemctl restart petal-dispatch-watcher` to persist the fix (currently running in-memory only).
|
|
- [ ] `sudo systemctl restart petal-dispatch` to activate the conductor instruction flip (release-turn-after-spawn) — the last unverified acceptance criterion (a).
|
|
- [ ] Re-run the spawn-a-worker live test after both restarts; confirm the full loop (spawn → release turn → watcher wakes on idle → conductor sweeps unprompted).
|
|
- [ ] Fix the cold-start cursor seeding: on missing cursor file, seed to current `MAX(seq)` instead of 0, so a fresh deploy doesn't replay history.
|
|
- [ ] Reap orphan worker process PID 1088451 (stream-json worker alive since the prior day) — Travis-gated.
|
|
- [ ] Consider folding the corrected terminal-kinds reality into a p5 revision of the plan, since the project-plan body is frozen. |