wiki-vault/petal-dispatch-v2.md at 9b0b7ceb4e9852c75dbcea353a76763a95dcdd10

petal-power/wiki-vault

Fork 0

Lovebug MCP 5d4db2fcdd mcp: project-plan — petal-dispatch v2

2026-06-04 01:42:53 +00:00

41 KiB

Raw Blame History

created

path

project

Constitution

Six principles. The agent treats violations as critical and resolves design forks toward these.

DB is truth, stream is display. Durable state is the typed event journal. The screen is a lossy view. Displayed/streamed content MUST NOT become a source of truth — every displayed item reconciles to a durable record or is marked orphaned.
Client is a view over the store, never coupled to stdout. Data flows stdout → DB → client via API, never stdout → client. N clients = N subscribers to one store; refresh/reconnect is free.
Active context is visible. The orchestrator's working-set (current task, recent worker handoffs, active mode) MUST be inspectable in the client, not trapped in the session's head.
Session role and lifecycle are runtime properties, never hardcoded categories. The data model and API MUST key on ids and treat role (orchestrator / worker / …) as data and lifecycle (one-shot / persistent / held-open) as observed. Nothing may assume a singleton orchestrator, an ephemeral worker, or a fixed process model. (The anti-corner principle — governs v2/v3/v4 build stages. The schema is where this is most likely violated by accident.)
[Scoped: concurrency only] Completion signal derives from process state (exit), not from the worker electing to call a signal tool. The one-shot worker model makes this literally true: one prompt → one response → exit, so process exit is the turn boundary, fired on OS reap (rc=0). Note: the long-lived/interactive worker (later build stage) decouples idle from exit; that case gets its own turn-boundary signal when built.
Conclusions are derived, never elected. Both the completion signal (Principle #5) and the handoff (the derived per-task conclusion) are produced by deterministic machinery — process exit for completion, the extractor + validation gate for the handoff — never by a language model choosing to emit a well-formed artifact. The journal stays trustworthy because what gets written is enforced in code, not by model goodwill. (Generalizes the write-discipline insight the execution-journal plan applied to write-enforcement; resolves the "who writes the handoff" open item toward derivation.)

Constitution cross-check: No violations. Principle #3's control surface is partially deferred (visibility ships in the v1 build stage; steering levers are the headline open question). Principle #6 is new in v2 and is what the handoff model rests on.

Spec

Goal

A seamless loop — send a request → orchestrator spawns workers / synthesis as needed → results flow back → back to me — with full visibility and a durable record of worker activity, reachable from phone and browser. The orchestrator decides its next move from a bounded, current view (derived handoffs) rather than raw transcripts.

Problem

Workers stall silently. The orchestrator only learns a worker is done if the worker emits the right outbound message, and only acts when it gets a turn. A finished worker can sit unseen. Separately: agent-written work outpaces mental-model formation — the gap is in drift detection, coverage, and decision archaeology, not code quality. The build exists to make the stall structurally impossible to miss and to give every actor a durable, queryable record keyed to the immutable plan.

Out of scope (non-goals)

No document/file download from chat. Control surface, not a file-transfer channel.
No direct plugin/connector integrations (GitHub, Gmail, etc.). The tool talks to the orchestrator; integration work is Lovebug's job, not the UI's.
App build is Android ecosystem only (native Android) — no iOS, no cross-platform mobile framework.
No multi-user input arbitration in the v1 build stage. Single user (Travis), single concurrent writer. Multi-user deferred and may never be built.
No cumulative compression layer in the v1 build stage. The per-step extractor ships; the compression-of-many-handoffs synthesizer is deferred and scale-triggered (Derived Layer, Stage C).

Primary user journey

I open a client (phone or browser) and see current work state.
I send an instruction to the orchestrator.
The orchestrator spawns a worker (or workers), which run and return.
Worker raw output lands in the journal; the extractor derives a handoff; the orchestrator reads the handoff and acts, reporting back.
I see completion/stall state without watching a terminal, and can read the orchestrator's working-set.

Requirements

FR-001 WHEN a worker completes or stalls, THE SYSTEM SHALL surface that state in the client without the user watching a terminal.
FR-002 WHEN the orchestrator or a worker emits an event, THE SYSTEM SHALL persist it as a typed raw-tier record in the journal keyed by id.
FR-003 WHEN a client connects, THE SYSTEM SHALL render state as a view over the store (never coupled to stdout).
FR-004 WHEN multiple clients are connected, THE SYSTEM SHALL present consistent state to all of them (multi-client read concurrency).
FR-005 WHEN the user sends input from a client, THE SYSTEM SHALL route it to the orchestrator session and record it as a USER_MESSAGE event that all clients render (echo via store round-trip).
FR-006 [PROSE] The active orchestrator context SHALL be inspectable in the client. (Visibility specified; control/steering surface is an open question, not yet a requirement.)
FR-007 WHEN the user sends input while a worker is running, THE SYSTEM SHALL queue the input client-side, render it as a visible pending state, and deliver it on the next turn boundary.
FR-008 WHEN a worker task completes (however the completion is signaled), THE SYSTEM SHALL derive a structured handoff from the worker's raw output and persist it as a derived-tier record, validated against the active contract before write.
FR-009 WHEN the extractor's output fails contract validation, THE SYSTEM SHALL NOT write the handoff as valid; it retries or flags the task, so a malformed handoff never silently enters the store.
FR-010 WHEN the orchestrator decides its next move, THE SYSTEM SHALL serve it a bounded view of derived handoffs (get_handoffs(scope_id)), not raw transcripts.
FR-011 [PROSE] A handoff that records a ruled-out approach SHALL carry it in a structured form the harness can check a candidate next-step against, so a prior rule-out is a lookup, not a re-derivation. (The "doomed to repeat history" guard.)

Success criteria

Connect from phone in ≤3 refreshes.
≤5 crashes/day tolerated during initial build (first Android app, explicit grace); trending toward zero is the real target. (Grace allowance, not a shipped bar.)
Context switch costs ≤1 action (at most one tap).
Harness completes 3 sequential actions unattended — one instruction, then the worker→harness loop closes itself twice without user input. (Can only pass if completion-signaling AND handoff derivation work — strongest metric.)
≥1 genuine takeaway from reading the journal (journal is legible, not just present).
1 full project completed in the tool (top-line, end-to-end).
Zero silent stalls over the test project — every worker that finishes or hangs becomes visible in the client. Caveat: partially bounded by upstream Claude Code signaling behavior; measured best-effort, not a hard gate.
The orchestrator never re-proposes an approach a prior handoff recorded as ruled-out, over the test project. (Validates FR-011 and the derived layer's reason for existing.)

Edge cases

Empty state — no active tasks: client shows an empty but live view.
Worker stall / hang — surfaces as a visible stuck state (the founding problem).
Orphaned display (optimistic-render stage) — streamed item with no durable record after timeout → marked orphaned, doubles as a stall detector.
Record disagrees with stream (optimistic-render stage) — final record wins.
Concurrent clients — both reflect the same store; no input collision (single user).
User types during worker run — input queued, pending-state shown, delivered on next turn (FR-007).
Extractor produces malformed handoff — caught by the validation gate; retried or flagged, never stored as valid (FR-009).
Worker exits failed / partial / skipped / gated — status reflects the real outcome; the not-clean-return policy (Cross-cutting) decides retry / retry-modified / escalate / skip.

Plan

Decisions (ADR-style)

D-1 — Native Android client. Native Android (Kotlin/Compose). Rejected served web-on-mobile (caps at browser-tab capabilities). Driver: full UI capabilities. One-way door accepted; the store is client-agnostic, the Kotlin codebase is the real commitment.

D-2 — Journal is in scope (build it, don't consume it). petal-dispatch builds the execution journal (first real instance). Two builds: the durable store and the clients. Journal is Phase 0.

D-3 — Relational journal, not flat. Typed envelope columns + JSONB payload. Task/session dimension + events table. Exact relational shape derived in Phase 0 from a real stream-json dump.

seq invariant: writer guarantees seq reflects causal order within a start_task transaction (await the spawn publish before scheduling user_message — fixes the prototype ordering bug).

D-4 — event_type as reference table. Open set in a lookup table; never PG ENUM or hardcoded constants. Extending is an insert, not a migration. Two-way door. (This principle is reused one level up for the handoff contract — see D-15.)

D-5 — App → API → backend. Clients talk to a backend API on herbys-dev that owns the journal + orchestrator session. Clients never touch stdout.

D-6 — SSE + Postgres LISTEN/NOTIFY for store→client push. Asymmetric traffic (occasional writes in, streamy reads out). Rejected full WebSocket as overkill.

D-7 — Tailscale-only reachability for the v1 build stage. Phone on the tailnet, API over the mesh, nothing public. (Status: already in place — dispatch.herbylab.dev Traefik route is tailnet-only.)

D-8 — Multi-client read concurrency in v1 (Android + browser); input arbitration deferred. v1: concurrent readers of one store, single writer, echo via USER_MESSAGE. Enforcement is convention only in v1, no lock/queue infra; the arbitration mechanism is built when concurrent input becomes real (may never).

D-9 — Sideload v1, self-hosted F-Droid repo as fast-follow. Gate for F-Droid: first stable v1 release on phone with no daily crashes for 3 consecutive days.

D-10 — Render-on-final v1, optimistic render as a later stage. v1 paints on durable-record arrival, keyed by the reconcile key. Optimistic paint + reconcile layers onto the same id-keyed path additively, not as a rewrite.

D-11 — One-shot worker as v1 default. Default worker = one prompt → one response → exit. Process exit = turn boundary = completion, on OS reap. Satisfies Principle #5 with zero added mechanism. A lean, not a commitment (Principle #4 protects it): the schema never assumes one-shot. Long-lived/interactive worker is a later lifecycle stage.

D-12 — Reconcile key is a petal-dispatch-issued UUID, not the Anthropic message ID. PD-issued UUID as row primary key and client reconcile key — stable across all event types (the Anthropic message.id only appears on assistant message events and would leave user/lifecycle events keyless). Anthropic's message.id is captured in payload JSONB as a forensic cross-reference. Keeps Principle #4 (no dependency on substrate's ID shape) while preserving forensic linkage. This UUID convention extends to handoff_id and all derived-tier ids.

D-13 — Two-service split. Service 1 petal-dispatch (Python/FastAPI): journal writer, extractor, API (REST + SSE), orchestrator-host, static web client from /static. Service 2 petal-dispatch-android (Kotlin/Compose): existing repo, talks to Service 1 over HTTPS + SSE. Rejected three services (premature) and one-repo (mixed Python+Kotlin operationally painful).

D-14 — Two-step auth rollout. Backend ships OPEN_MODE=1 (auth bypass; trust the tailnet) so Android can be developed against an unauthenticated backend; flip to Authentik-fronted via Traefik once Android auth is correct. Tailscale stays the network gate throughout.

D-15 — Handoff is derived, not worker-emitted (Option B). The worker stays dumb (one prompt, one response, exit). An extractor reads the worker's raw final output and produces a structured handoff. Rejected Option A (worker emits its own handoff event) as fragile — it makes journal correctness depend on a model electing to emit well-formed output, the exact failure mode Principle #6 forbids. Consequence: there IS a small per-step extractor/synthesizer from the v1 build stage (this overrides v1's old "no synthesizer built" companion note). The cumulative compression layer remains deferred (Derived Layer, Stage C).

D-16 — One store, two tiers (not two stores). Everything lives in the journal. Records carry a tier: raw (machine-emitted: stdout, spawns, exits, deltas — high-volume, never read in the hot path) vs derived (extractor-produced: the handoff — low-volume, what the harness reads). The "report"/read surface is a query filtered to the derived tier. One store so derived records reconcile by construction (own seq, append-only, point back at the raw events they distilled) — avoids the second-source-of-truth / sync problem.

D-17 — Handoff schema is data, in a contract table. Applies D-4 one level up. The fields a handoff can carry are rows in a contract table, not a baked-in struct: each row defines a field's key, required flag, who reads it, semantics, and value shape (flexible JSON; e.g. ruled_out shape is {type: list, item: {approach, reason}}). Adding a handoff field is an insert. The contract is write-side only (governs generation + validation) and is never consulted at read time.

D-18 — Validation gate on the derived write path. The extractor's output is validated against the active contract before it becomes a valid derived record (required fields present, shapes match). Malformed → reject/retry/flag (FR-009). This is what makes the derived handoff trustworthy where the rejected worker-emitted handoff (D-15) was not. The contract has two write-time consumers: the extractor (how to generate) and the validator (how to enforce).

D-19 — Write-side / read-side split → no handoff versioning. The contract is consumed only at write time; handoffs are immutable and self-describing at read time (a reader handles whatever fields are present). Because nobody reads a handoff through the contract, the contract version that produced it is irrelevant at read time — so no version-stamping, no version-capture, no recovery story. Contract rows are immutable; the contract set is append-only and grows as the system learns (drift-by-accretion). Handoff shape drifts over time; old handoffs stay readable. contract.field_key is stable and never reused with a different meaning (supersession is a new row, never an edit). Read-side discipline: readers must be shape-tolerant — read what exists, branch on the fields they care about, tolerate absence and unknowns (same posture as D-4's open event_type set).

D-20 — parent_id nesting tree, orthogonal to the id hierarchy. Records carry a self-referential parent_id for arbitrary-depth execution nesting, in addition to the scope ids (project_id / session_id / task_id). The ids answer "which scope" (addressing/keyed reads); parent_id answers "what nests under what" (the execution tree). This reconciles the execution-journal plan's tree instinct with the synthesis-session id hierarchy — they're not competitors. It also makes the recursive-handoff model literal: a handoff exists at any node, a parent's handoff is a real record (Lovebug's feedback to Travis), not merely an aggregation view. (Supersedes the synthesis-session "handoffs only at task grain; higher levels are views" framing.)

Data model

Journal (one table family, two tiers).

Tier discriminator: every record carries tier ∈ {raw, derived}.
Typed envelope columns: event_id (PD-issued UUID, primary key), project_id, session_id, task_id, parent_id (nullable, self-FK — the nesting tree), message_id (nullable PD-issued UUID, reconcile key for client render), event_type (FK to reference table), tier, created_at, monotonic seq.
event_type — reference table, open set, never PG ENUM or hardcoded.
JSONB payload — variable structured body. Anthropic's message.id (when present) lives inside payload as a forensic field. No display markup stored as truth. (Exception: HTML that is a worker's actual output content is data, held as a JSONB string.)
seq is the ordering authority — consumers order by seq, never arrival time.
Plan linkage: project_id keys to the immutable vault plan; a plan_ref field (task addressing) links a record to a specific plan task. (See Cross-cutting → Task addressing — depends on plans gaining addressable task IDs.)

Handoff (derived tier).

handoff_id (PD-issued UUID), the scope ids it belongs to, parent_id for its place in the tree, tier = derived, seq, created_at.
Payload is self-describing — carries whatever fields the active contract defined at write time. Guaranteed-present and shape-enforced fields come from the contract (D-17/D-18). Candidate field set (all defined as contract rows, not hardcoded): goal, current_state, next_step, ruled_out ({approach, reason} list — the load-bearing field, FR-011), status, artifacts, gate-decision fields.
status ∈ {success, failed, partial, skipped, gated} — adopted from the execution-journal plan's result set (richer than the synthesis session's three values; skipped/gated are real outcomes; gated ties into autonomous-mode gate logic).
artifacts — commits / files / logs / vault refs (adopted from the execution-journal plan).

Contract table.

One row per handoff field. Columns: field_key (stable, never reused), required, read_by (e.g. harness / human), semantics, shape (JSON). Append-only, immutable rows.
Consumed only at write time (extractor + validator). Never read at handoff-read time.

Interfaces / contracts

Outbound (store → client): SSE stream + LISTEN/NOTIFY-driven "new record" notifications. Multiple concurrent readers.
Inbound (client → orchestrator): REST action → backend → orchestrator session stdin. Backend is the single writer to stdin (v1 trivially; arbitration built later if needed).
Echo: user input becomes a USER_MESSAGE event; all clients render it from the store.
Client render: keyed by the reconcile key (PD-issued UUID) so render-on-final and optimistic render share a path.
Derived read surface (the MCP/API for orchestration):
- get_handoffs(scope_id, limit?, order=desc) — ordered handoffs for whatever scope the id resolves to (task / session / project), newest-first. The scope-generalized successor to the execution-journal plan's get_task_context / get_project_state and the synthesis plan's get_project_summaries(slug) (corrected to id-keyed per D-12).
- get_project_state(project_id) — bounded rollup: which plan tasks done / in-flight / failed / blocked. A few hundred tokens; what the orchestrator calls before deciding the next move.
- log_step / handoff write — goes through the validation gate (D-18); single-writer-per-entry, enforced deterministically (Principle #6).

Deployment

Backend (API + journal + extractor + orchestrator session) on herbys-dev. Postgres = petalbrain instance.
Reachability: Tailscale-only for v1 (phone on tailnet); already in place.
Android artifact: sideload v1 (adb install / direct APK); self-hosted F-Droid behind Traefik as fast-follow.
Web client: served from petal-dispatch's /static, same Traefik route as the API.
GitHub pushes: Travis only (standing model); Lovebug works on Gitea / petal-power.

Tests

Rigorous on journal/backend/extractor — the truth + derivation layers; bugs corrupt everything downstream. zero-check applies. The Phase 0 stream-json dump is the canonical fixture set for journal-writer AND extractor tests (feed real raw output, assert valid handoffs).
Validation gate gets real tests — malformed extractor output must be caught (FR-009).
Pragmatic on clients — they're views; the reconcile logic (optimistic-render stage) is the one client-side piece earning real tests.

NFRs

Latency / throughput: n/a as hard budgets for v1 (single user). Render-on-final accepts slight screen lag until the optimistic-render stage.
Threat model: single-user personal tool, Tailscale-only, not publicly exposed in v1. Trust boundary is the tailnet. Public exposure is a deliberate later decision with its own threat review.
Observability: the journal is the observability surface — same event log the tool exposes (three prototype bugs were found by observing the live stream).

Phases

Phase −1 — Viability spike (bounds the OAuth pre-mortem)

Confirm production auth permits the remaining required control flow. Already partially cleared by the prototype: turn-injection works, the event stream is observable, session_id threads through every event. Remaining: confirm which credential the running session uses and whether concurrency-required controls hold under it. Framed as measuring the dependency, not erasing it. Satisfies: de-risks FR-001/002/005.

Phase 0 — The journal

Dump live stream-json → derive event schema from observed structure → build journal (task/session dimension + events table, parent_id self-FK, reference-table event types, JSONB payload, seq ordering, tier discriminator) + the writer that captures session/worker events.

Dump methodology — required scenario set (curated, not happy-path):

One-shot worker, text-only response.
One-shot worker that emits tool_use blocks.
Long-lived worker running multiple turns (forward-compat / Principle #4).
Worker that errors out.
Worker externally SIGTERMed mid-turn.
Worker that hits rate limits or auth refresh during run.

Must-resolve-before-code: (a) reconcile-key structure — does each assistant turn emit a message_start with a stable id (captured into payload, not relied on as the key — D-12); (b) exact relational shape; (c) decide parent_id write semantics (who sets it, when). Satisfies: FR-002.

Phase 0.5 — The extractor + contract + derived tier

Define the contract table; seed it with the v1 field set (goal, current_state, next_step, ruled_out, status, artifacts, gate-decision fields) as rows.
Build the extractor: on worker task completion (trigger-agnostic — fires on whatever the completion event turns out to be), read the worker's raw final output, produce a structured handoff.
Build the validation gate (D-18): validate extractor output against the active contract before the derived write; reject/retry/flag on failure.
Confirm the worker's raw final output carries enough signal to populate the required fields (especially ruled_out); tune the extractor prompt against the Phase 0 dump fixtures. Satisfies: FR-008, FR-009, FR-011.

Phase 1 — Backend API + push

REST for actions, SSE for live read, LISTEN/NOTIFY → SSE fan-out, backend owns the single write path to orchestrator stdin. Derived read surface (get_handoffs, get_project_state) exposed. Backend bootstraps with OPEN_MODE=1 (D-14). Web client served from /static. systemd unit + secrets.env carried from the demo. Traefik snippet matches the demo's pattern. Satisfies: FR-003, FR-005 (mechanism), FR-007 (mechanism), FR-010, foundation for FR-004.

Phase 2 — Orchestrator loop against the journal

Define the loop: get_project_state → pick next task → build worker brief via get_handoffs(task scope) → spawn worker → worker raw output lands → extractor derives handoff → loop. Orchestrator never reads raw worker output; per-cycle context stays roughly constant regardless of project size.
Implement the not-clean-return policy (the one place the orchestrator needs real logic): worker returns failed / partial / gated → retry same brief / retry modified / escalate to Travis / skip. Define upfront, not improvised. Satisfies: success criterion 4, 8; FR-010.

Phase 3 — Web client

Browser client, render-on-final keyed by the reconcile key, Tasks/Chat/Activity surfaces. Proves store→API→client→render end-to-end on the known stack before Android. Home tab: Travis picks before build (current lean Task-first). Satisfies: FR-001, FR-003, FR-006 (visibility), FR-007 (UI surface).

Phase 4 — Native Android client

Extend the existing petal-dispatch-android repo (v0 chat scaffold from the demo). Carry forward: Kotlin/Compose setup, Material 3 theme, OkHttp, the chat screen as the v1 "Chat" tab. New surfaces: Tasks tab (running + recent worker tasks, status, tap-to-drill), Activity tab (chronological event feed, forensic view), Chat tab (existing, repointed to v1 API + SSE). Networking rewrite: drop WebSocket-only, switch to REST (POST actions) + SSE. Same OkHttp transport. Tailscale reachability: assume phone on tailnet; show clear "off-tailnet" state with reconnect, don't crash. Auth: trust the network initially (OPEN_MODE=1); add HTTPS auth + Authentik header when D-14 flips (TODOs at call sites). Sideload: debug APK → dist/, transfer via Tailscale share / Syncthing / USB. Satisfies: FR-004, FR-001 (mobile), success criteria 1 & 3.

Phase 5 — Optimistic render (the "feel" stage)

Optimistic paint + reconcile-in-place onto the existing id-keyed path, both clients. Orphan-timeout handling. Satisfies: live-streaming feel; edge cases (orphaned display, record-disagrees-with-stream).

Iterate-shaped phases

Phase 5 (optimistic render) and the home-tab/UI work are converge-with-Travis loops — done when it feels right, not when a spec criterion is met. Agent treats these as iterate, not finish-and-handoff. The extractor-prompt tuning (Phase 0.5) is similarly iterate-shaped against the dump fixtures.

Human-required handoffs

Coordinated live with Lovebug during execution. Known touch-points: the stream-json dump + schema sign-off (Phase 0), the contract seed + extractor-prompt review (Phase 0.5), GitHub pushes (all phases), Android-toolchain / physical-phone steps (Phase 4), Tailscale/cert wiring, the OPEN_MODE → Authentik flip (D-14).

Cross-cutting

Deferred build stages (the "version stack")

v2 is the canonical plan; these are sequenced build stages after the v1 cut, all forward-compatible with the schema by Principle #4 — "more of what the schema already allows," not rewrites.

v1 — single orchestrator + workers + journal + extractor/derived tier + Android & web clients; render-on-final; one-shot worker; multi-client read concurrency.
v1.1 — optimistic render + reconcile.
Derived Layer Stage C (synthesis compression) — cumulative rolling brief over the handoff list. Trigger: the per-step handoff list grows past the point where reading it all on every harness cycle is comfortable; define "uncomfortable" with real numbers then. Open sub-decision (decide with data): cumulative rolling brief vs. windowed last-N. If cumulative, add drift detection anchored to the journal/plan (immutable references): structural-invariant checks (active rule-outs persist unless a journal entry reverses them), periodic reconciliation (rebuild from journal, diff against brief). Never anchor a drift check to another synthesis pass — that's the regress.
v2 (concurrency / multi-Dispatch) — multiple concurrent Dispatch sessions; simultaneous multi-client input arbitration (if ever); desktop client; other reachability pathways.
v4 (interactive workers) — long-lived / held-open workers, routable for direct input; the turn-boundary signal (task.idle) decoupled from exit gets built here.

Why per-step (not cumulative) for the v1 derived layer

Per-step handoffs are immutable once written; the "history" is the ordered list itself — no fold-forward, no rewriting, no carry-forward discipline. Trades read-time cost (read several handoffs) for write-time safety (no drift surface inside any handoff). Cumulative rolling summaries introduce per-step drift at the synthesis pass and have a circular recovery story ("regenerate from the journal" is itself the hard synthesis problem). At v1 scope (~10–30 steps), reading the full list is fine — token budget is not the constraint. Compression is added later, with real data. (The simple v1 is "simple because we reasoned to it," not "simple because we didn't look.")

Task addressing (dependency, needs a pass)

The execution-journal plan's plan_ref (FK into a plan task like T-042) and the "task IDs feed worker briefs" elegance both depend on vault plans being structured with addressable task IDs — which they mostly are NOT yet (plans use ## Phases checkbox lists, no stable task IDs). Either plans gain task IDs (this "nudges plan structure toward spec-shaped, addressable tasks" — aligned with the plan-interviewer / spec-driven direction, not additional), or plan_ref is looser than an FK for v1. Open item — resolve before relying on brief→journal traceability.

Autonomous-mode gates

The journal pairs with two execution modes: interactive (Travis approves each advance) and autonomous (Lovebug completes + documents, stops at hard gates). Gate categories — operations where cost-of-wrong is high and cost-of-asking is low: outside the project tree, sudo/privilege escalation, network-facing config (firewall/proxy/DNS), secrets touched, destructive data ops, git ops beyond the working branch, dependency/package additions, cross-service changes. A gated status + gate-decision fields in the handoff log the event ("considered touching X outside project, stayed in-scope") as positive evidence the gate logic fired — not just that it never came up. Decision-doc contract for autonomous mode: the handoff must capture the decision point (options on the table), the chosen option, the rule-out for the rest, and what would have triggered a gate — auditable without re-reading transcripts.

Codebase assumptions

Relies on, does not touch: Postgres (petalbrain), Traefik (fronts API + web client), Tailscale (reachability), Gitea/petal-power (dev), existing claude Code sessions and --output-format stream-json behavior.
Spawn mechanism: observe-only. Build observes session/worker events; does not modify how workers spawn (Claude Code native). A launch wrapper is a conditional fallback (only if native completion signaling proves insufficient) — moot under one-shot (exit is the signal), relevant only in the concurrent + long-lived world.
Substrate-dependency containment (pre-mortem mitigation): the harness/adapter is the sole layer touching Claude Code's control surface. Journal, extractor, API, clients view the store. If the substrate changes (CLI version, auth, stream format) or is swapped, rebuild that one adapter, not the tool.
MCP is the information surface, not the control surface. MCP exposes read/write tools agents call; it cannot spawn, reap, or swap workers. Control (lifecycle) lives in the harness. Handoff generation is task logic owned by whatever is executing at that layer — NOT a system-level trigger the journal models. The journal's only jobs: store the typed record, key it by id, serve it by scope. (This is why the derived layer does not depend on petal-dispatch settling the orchestrator turn-boundary task.done vs task.idle question.)

What carries forward from the demo (proven patterns, not code)

The prototype (petal-power/petal-dispatch, to be renamed petal-dispatch-demo) ran a working harness, worker→harness event wiring, Android chat APK, and Authentik+Tailscale integration. Code is a sandbox; lessons carry verbatim:

Postgres LISTEN/NOTIFY → SSE fan-out works at single-user scale.
Event taxonomy starting point: task.spawned, task.message, task.idle, task.user_message, task.crashed (kinds list to start from; schema evolves in Phase 0 from the dump).
X-Authentik-Username header pattern (D-14 flip).
OPEN_MODE env flag pattern (verbatim for v1 bootstrap).
systemd unit + secrets.env petalbrain DSN delivery.
Tailscale + Traefik dynamic config (no changes needed for v1).
Kotlin scaffold structure (extended, not restarted).

Does not carry forward: existing dispatch_harness.events rows (historical, not migrated — Phase 0 dump informs v1 schema from scratch); tasks.json subprocess-manager persistence format; existing FastAPI route layout.

Repo & naming

Existing petal-power/petal-dispatch → renamed petal-power/petal-dispatch-demo (stays accessible for reference + as runtime demo while v1 is built; systemd unit stays pointed at it during transition).
New petal-power/petal-dispatch → v1 backend (Service 1).
Existing petal-power/petal-dispatch-android → continues as v1 client (Service 2), README updated to v1 contract.

Companion vault projects (now folded here)

execution-journal, synthesis-layer — superseded by this artifact. Their slugs should be marked superseded once Lovebug fixes the execution-journal slug collision (see below). operating-mode (status: idea) remains separate — the client is the natural surface to invoke + display the active mode; ties into Principle #3 visibility and the control-surface open question.

Prototype findings (open as repo issues against the new petal-dispatch repo)

task.spawned can arrive after task.user_message (unordered asyncio.create_task). Fix baked into D-3's seq invariant.
task.crashed.last_error populated with the assistant's reply on clean SIGTERM — conflates "last output" with "error cause." Fix: split last_error (real fault only) from last_output, or detect signal-exit (143/130). Undercuts success criterion 5 if unfixed. (This is also why the handoff status enum includes the real-outcome values rather than inferring from last_error.)
task.done unreachable under the long-lived v0.5 worker — dissolves under one-shot (done fires on every exit = every turn); re-emerges only as a v4 question.

Unresolved questions

Must-resolve-before-code: reconcile-key structure (Phase 0 dump); exact journal relational shape (Phase 0); parent_id write semantics (Phase 0); whether worker raw output reliably carries the ruled_out signal the extractor needs (Phase 0.5); remaining production-auth control confirmation (Phase −1).
Needs-a-pass: task addressing / plan_ref dependency on plans gaining addressable task IDs (Cross-cutting → Task addressing).
Can-defer-and-tag: home-tab choice (before Phase 3 layout); the three prototype bug-fixes (repo issues); whether the spawn mechanism ever needs a wrapper (only if concurrent); the cumulative-vs-windowed compression decision (Derived Layer Stage C, decide with data).
Headline open question (defer, do not block foundation): the control surface for context — what levers Travis gets to steer/shape the orchestrator's context. v1 has zero control surface beyond "send a message to Dispatch." Steering levers (operating-mode invocation, context-trim, worker-tool-allowlist) are all later stages. Visibility ships in v1; control is undefined and explored later.

Pre-mortem

Failure mode (Travis's intent): Six weeks out, the OAuth subscription doesn't permit the control flow needed — the architecture is sound and the tool is still dead because the substrate won't allow the control. Structural, not a bug: the foundation is someone else's tool under consumer auth, control surfaces owned by Anthropic and subject to change. Bounded, not eliminated, by (a) the Phase −1 spike measuring the dependency before building on it, and (b) Principle #2 / containment keeping the dependency in one swappable adapter.

Secret suspicions (surfaced)

The worker-model elegance (one-shot → exit = completion) is a lean, not a tested commitment — interactive/long-lived workers were already observed in the prototype, so one-shot may not survive contact with how Travis actually works. Principle #4 is the deliberate hedge.
The extractor (D-15) is itself a language model reading prose and emitting structure — the same kind of actor we refused to trust in Option A. What makes it trustworthy is that extraction is its whole job under a tight prompt, and its output is schema-validated at the gate (D-18) — a malformed handoff is rejected, not silently stored. If the extractor proves unreliable even under those constraints, the fallback is a stricter contract + retry budget, not trusting raw output.

Notes

Why one artifact. The journal (durable store), the derived layer (extractor + contract + handoffs), and the clients are one architecture with shared seams — the tier discriminator, the contract-as-data pattern (D-4 reused as D-17), parent_id, and the PD-issued-UUID id convention (D-12 extended to handoff_id) all cross what used to be three plans. Keeping them separate was creating contradictions (notably the "no synthesizer built" line that Option B overrides). One artifact is the single source of truth; v1/journal/synthesis are superseded.

The blackboard pattern. This is the well-established blackboard / shared-state pattern for multi-agent orchestration, built purpose-fit for the herbylab stack rather than adopted from a heavyweight framework (LangGraph/CrewAI/AutoGen). What is not out there is a purpose-built, plan-keyed execution journal with a derived handoff layer as a first-class orchestration participant for a personal stack.

Why journaling matters for context. Models cannot dump-and-reload their own context — writing to a table does not shrink the window within a session. Journaling is not a within-session workaround; it is what makes ending a session survivable. The agent dies, the journal persists, the next agent hydrates from a small derived slice. Context stays small because the read API does the filtering, not the model. Agents are cheap and replaceable; the journal is durable.

Slug-collision follow-on (Lovebug). execution-journal currently resolves (via get_project) to a UI-patterns session-note (2026-05-28-ui-poc-process-learnings.md) rather than the real "Execution Journal" plan (Sources/Dev/execution-journal.md). Lovebug is fixing the resolver/slug. Until fixed, pulling execution-journal by slug returns the wrong artifact — relevant to any agent hydrating from these plans.

Drift is bounded, not eliminated. 100% accurate history is not the goal and never was. The real asymmetry is correction-loop tempo: agents can compound 50 generations of lossy retelling in an afternoon while the human correction loop is slow. The hedge is mechanical checks anchored to immutable references (journal + plan), not another agent's second opinion — and, for v1, per-step immutable handoffs that have no internal drift surface to begin with.

41 KiB Raw Blame History Unescape Escape