wiki-vault/petal-dispatch-p4.md at 7d3cc87da904d5425368f72b7b10ea207059cf7d

petal-power/wiki-vault

Fork 0

Lovebug MCP 8d10ce23dc mcp: project-plan — petal-dispatch p4

2026-06-05 00:13:55 +00:00

50 KiB

Raw Blame History

created

path

project

Constitution

Six principles. The agent treats violations as critical and resolves design forks toward these.

DB is truth, stream is display. Durable state is the typed event journal. The screen is a lossy view. Displayed/streamed content MUST NOT become a source of truth — every displayed item reconciles to a durable record or is marked orphaned.
Client is a view over the store, never coupled to stdout. Data flows stdout → DB → client via API, never stdout → client. N clients = N subscribers to one store; refresh/reconnect is free.
Active context is visible. The orchestrator's working-set (current task, recent worker handoffs, active mode, live thread) MUST be inspectable in the client, not trapped in the session's head. (This is the operational-visibility spine of the reframed Goal — the human should not reconstruct session state by hand.)
Session role and lifecycle are runtime properties, never hardcoded categories. The data model and API MUST key on ids and treat role (orchestrator / worker / …) as data and lifecycle (one-shot / persistent / held-open) as observed. Nothing may assume a singleton orchestrator, an ephemeral worker, or a fixed process model. (The anti-corner principle — governs the v2/v4 build stages, incl. the deferred orchestrator/harness split. The schema is where this is most likely violated by accident.)
[Scoped: concurrency only] Completion signal derives from process state (exit), not from the worker electing to call a signal tool. The one-shot worker model makes this literally true: one prompt → one response → exit, so process exit is the turn boundary, fired on OS reap (rc=0). Note: the long-lived/interactive worker (later build stage) decouples idle from exit; that case gets its own turn-boundary signal when built.
Conclusions are derived, never elected. Both the completion signal (Principle #5) and the handoff (the derived per-task conclusion) are produced by deterministic machinery — process exit for completion, the extractor + validation gate for the handoff — never by a language model choosing to emit a well-formed artifact. The journal stays trustworthy because what gets written is enforced in code, not by model goodwill.

Constitution cross-check: No violations. Principle #3's control surface is partially deferred (visibility ships in v1; steering levers are the headline open question). Principle #6 is what the handoff model rests on.

Spec

Goal

A working environment that gives me greater control and visibility over tasks, context, and execution: I can see what's live, what the workers are doing, and how long I've been on a task at a glance, and switch focus between them with minimal effort — without holding that state in my own head or reconstructing it by typing questions every time I log in. The seamless execution loop (send a request → harness spawns workers → results flow back → back to me, with a durable record throughout) is the output that serves this; the harness decides its next move from a bounded, current view of derived handoffs rather than raw transcripts.

Problem

Today the interface is a wall of text with no operational surface. Session state — which thread is live, what the workers are doing, how long a task has been running, what context is loaded — is invisible; I reconstruct it every login by asking. Focus changes are manual and high-effort: remember to save notes to the thread, recall the thread by name, clear context, navigate to other screens, all by hand. The human is doing the state-tracking the system should be doing.

Silent worker stalls are one consequence of this: the orchestrator only learns a worker is done if the worker emits the right outbound message and only acts when it gets a turn, so a finished or hung worker can sit unseen — invisible for the same reason everything else is, because nothing surfaces its state. Separately, agent-written work outpaces mental-model formation — the gap is in drift detection, coverage, and decision archaeology, not code quality. The build exists to give the human an operational surface over the work, make the stall structurally impossible to miss, and give every actor a durable, queryable record keyed to the immutable plan.

Out of scope (non-goals)

No document/file download from chat. Control surface, not a file-transfer channel.
No direct plugin/connector integrations (GitHub, Gmail, etc.). The tool talks to the orchestrator; integration work is Lovebug's job, not the UI's. (Trellis is not a "plugin integration" in this sense — it is the thread/context store the client surfaces; see FR-012.)
App build is Android ecosystem only (native Android) — no iOS, no cross-platform mobile framework.
No multi-user input arbitration in the v1 build stage. Single user (Travis), single concurrent writer. Multi-user deferred and may never be built.
No cumulative compression layer in the v1 build stage. The per-step extractor ships; the compression-of-many-handoffs synthesizer is deferred and scale-triggered (Derived Layer, Stage C).
No two-layer orchestrator/harness split in v1. v1 ships the fused conversational-plus-executor entity. The split (conversational orchestrator above a fresh harness) is a deferred build stage, designed after living with v1.

Primary user journey

I open a client (phone or browser) and see current work state at a glance — live thread, running/idle/blocked workers, task durations — without asking.
I send an instruction to the orchestrator, or switch focus to another thread/task in one action.
The orchestrator spawns a worker (or workers), which run and return.
Worker raw output lands in the journal; the extractor derives a handoff; the orchestrator reads the handoff and acts, reporting back.
I see completion/stall state without watching a terminal, and can read the orchestrator's working-set.

Requirements

FR-001 WHEN a worker completes or stalls, THE SYSTEM SHALL surface that state in the client without the user watching a terminal.
FR-002 WHEN the orchestrator or a worker emits an event, THE SYSTEM SHALL persist it as a typed raw-tier record in the journal keyed by id.
FR-003 WHEN a client connects, THE SYSTEM SHALL render state as a view over the store (never coupled to stdout).
FR-004 WHEN multiple clients are connected, THE SYSTEM SHALL present consistent state to all of them (multi-client read concurrency).
FR-005 WHEN the user sends input from a client, THE SYSTEM SHALL route it to the orchestrator session and record it as a USER_MESSAGE event that all clients render (echo via store round-trip).
FR-006 [PROSE] The active orchestrator context SHALL be inspectable in the client. (Visibility specified; control/steering surface is an open question, not yet a requirement.)
FR-007 WHEN the user sends input while a worker is running, THE SYSTEM SHALL queue the input client-side, render it as a visible pending state, and deliver it on the next turn boundary.
FR-008 WHEN a worker task completes (however the completion is signaled), THE SYSTEM SHALL derive a structured handoff from the worker's raw output and persist it as a derived-tier record, validated against the active contract before write.
FR-009 WHEN the extractor's output fails contract validation, THE SYSTEM SHALL persist the task with status = extraction_failed rather than retrying or writing the handoff as valid; downstream routing is decided by the not-clean-return policy (Phase 2).
FR-010 WHEN the orchestrator decides its next move, THE SYSTEM SHALL serve it a bounded view of derived handoffs (get_handoffs(scope_id)), not raw transcripts.
FR-011 [PROSE] A handoff that records a ruled-out approach SHALL carry it in a structured form the harness can check a candidate next-step against, so a prior rule-out is a lookup, not a re-derivation. (The "doomed to repeat history" guard.)
FR-012 WHEN a client connects, THE SYSTEM SHALL surface live thread/context state from Trellis (which thread is live, its identity, recent context) so the user sees and switches focus without reconstructing state by hand. Trellis is the thread/context store; the backend exposes a Trellis read surface (Phase 1), the clients render it (Phase 3 web, Phase 4 Android). (Integration, not construction — Trellis already holds this state.)
FR-013 [PROSE, deferred → Stage C] The synthesis layer SHALL read execution truth from the journal and write a derived working-context picture to Trellis, keyed (via thread_id) to the journal evidence it was derived from. Distinguishes the per-task extractor (journal→journal record) from synthesis (journal→Trellis context). Deferred to Derived Layer Stage C; the thread_id seam (migration 003) exists in v1 so this is non-foreclosed.

Success criteria

Connect from phone in ≤3 refreshes.
≤5 crashes/day tolerated during initial build (first Android app, explicit grace); trending toward zero is the real target. (Grace allowance, not a shipped bar.)
Context switch costs ≤1 action (at most one tap). (Directly encodes the reframed Goal's low-effort focus-switching.)
Harness completes 3 sequential actions unattended — one instruction, then the worker→harness loop closes itself twice without user input. (Can only pass if completion-signaling AND handoff derivation work — strongest metric. An SC-4 failure must be diagnosed as loop-logic vs. extractor reliability, since it now depends on both.)
≥1 genuine takeaway from reading the journal (journal is legible, not just present).
1 full project completed in the tool (top-line, end-to-end).
Zero silent stalls over the test project — every worker that finishes or hangs becomes visible in the client. Caveat: partially bounded by upstream Claude Code signaling behavior; measured best-effort, not a hard gate.
The orchestrator never re-proposes an approach a prior handoff recorded as ruled-out, over the test project. (Validates FR-011 and the derived layer's reason for existing.)

Test project for SC-4 / SC-6 / SC-8 to be selected before Phase 2 closes.

Edge cases

Empty state — no active tasks: client shows an empty but live view.
Worker stall / hang — surfaces as a visible stuck state (a consequence of the founding problem).
Orphaned display (optimistic-render stage) — streamed item with no durable record after timeout → marked orphaned, doubles as a stall detector.
Record disagrees with stream (optimistic-render stage) — final record wins.
Concurrent clients — both reflect the same store; no input collision (single user).
User types during worker run — input queued, pending-state shown, delivered on next turn (FR-007).
Trellis unavailable — client degrades gracefully: worker/journal state still renders (journal is independent of Trellis); thread/context tiles show an unavailable state rather than blocking the view.
Extractor produces malformed handoff — caught by the validation gate; task lands with status = extraction_failed (no retry in v1). The not-clean-return policy (Phase 2) decides downstream routing.
Worker exits failed / crashed / cancelled / rate_limited / partial / skipped / gated — status reflects the real outcome via the expanded enum; the not-clean-return policy decides retry / retry-modified / escalate / skip.

Plan

Decisions (ADR-style)

D-1 — Native Android client. Native Android (Kotlin/Compose). Rejected served web-on-mobile (caps at browser-tab capabilities). Driver: full UI capabilities. One-way door accepted; the store is client-agnostic, the Kotlin codebase is the real commitment.

D-2 — Journal is in scope (build it, don't consume it). petal-dispatch builds the execution journal (first real instance). Two builds: the durable store and the clients. Journal is Phase 0.

D-3 — Relational journal, not flat. Typed envelope columns + JSONB payload. Task/session dimension + events table. Exact relational shape derived in Phase 0 from a real stream-json dump.

seq invariant: writer guarantees seq reflects causal order within a start_task transaction (await the spawn publish before scheduling user_message — fixes the prototype ordering bug).
thread_id (migration 003): nullable BIGINT, no FK, partial index; derived rows inherit parent's thread_id. Forward-compat seam for FR-013 (synthesis → Trellis). Convention-not-FK deliberately, to avoid coupling journal writes to Trellis's thread lifecycle. Sits NULL until synthesis exists.

D-4 — event_type as reference table. Open set in a lookup table; never PG ENUM or hardcoded constants. Extending is an insert, not a migration. Two-way door. (Reused one level up for the handoff contract — see D-17.)

D-5 — App → API → backend. Clients talk to a backend API on herbys-dev that owns the journal + orchestrator session. Clients never touch stdout.

D-6 — SSE + Postgres LISTEN/NOTIFY for store→client push. Asymmetric traffic (occasional writes in, streamy reads out). Rejected full WebSocket as overkill.

D-7 — Tailscale-only reachability for the v1 build stage. Phone on the tailnet, API over the mesh, nothing public. (Status: already in place — dispatch.herbylab.dev Traefik route is tailnet-only.)

D-8 — Multi-client read concurrency in v1 (Android + browser); input arbitration deferred. v1: concurrent readers of one store, single writer, echo via USER_MESSAGE. Enforcement is convention only in v1, no lock/queue infra; the arbitration mechanism is built when concurrent input becomes real (may never).

D-9 — Sideload v1, self-hosted F-Droid repo as fast-follow. Gate for F-Droid: first stable v1 release on phone with no daily crashes for 3 consecutive days.

D-10 — Render-on-final v1, optimistic render as a later stage. v1 paints on durable-record arrival, keyed by the reconcile key. Optimistic paint + reconcile layers onto the same id-keyed path additively, not as a rewrite.

D-11 — One-shot worker as v1 default. Default worker = one prompt → one response → exit. Process exit = turn boundary = completion, on OS reap. Satisfies Principle #5 with zero added mechanism. A lean, not a commitment (Principle #4 protects it): the schema never assumes one-shot. Long-lived/interactive worker is a later lifecycle stage.

OS exit code mapping → handoff status:

0 → success (subject to validation gate; gate failure → extraction_failed instead)
nonzero clean exit → failed
130 (SIGINT) or 143 (SIGTERM) → cancelled (intentional kill from a client)
137 (SIGKILL) → crashed (abnormal exit: OOM, segfault, kernel reap)
rate_limited is not derivable from exit code alone — surfaced by the process-supervision layer observing substrate quota / auth-refresh aborts, and overrides the otherwise-derived value.

D-12 — Reconcile key is a petal-dispatch-issued UUID, not the Anthropic message ID. PD-issued UUID as row primary key and client reconcile key — stable across all event types. Anthropic's message.id captured in payload JSONB as a forensic cross-reference. Keeps Principle #4 (no dependency on substrate's ID shape) while preserving forensic linkage. Extends to handoff_id and all derived-tier ids.

D-13 — Two-service split. Service 1 petal-dispatch (Python/FastAPI): journal writer, extractor, API (REST + SSE), orchestrator-host, static web client from /static. Service 2 petal-dispatch-android (Kotlin/Compose): existing repo, talks to Service 1 over HTTPS + SSE. Rejected three services (premature) and one-repo (mixed Python+Kotlin operationally painful).

D-14 — Two-step auth rollout. Backend ships OPEN_MODE=1 (auth bypass; trust the tailnet) so Android can be developed against an unauthenticated backend; flip to Authentik-fronted via Traefik once Android auth is correct. Tailscale stays the network gate throughout.

D-15 — Handoff is derived, not worker-emitted (Option B). The worker stays dumb (one prompt, one response, exit). An extractor reads the worker's raw final output and produces a structured handoff. Rejected Option A (worker emits its own handoff event) as fragile — it makes journal correctness depend on a model electing to emit well-formed output, the exact failure mode Principle #6 forbids. Consequence: there IS a small per-step extractor from the v1 build stage. The cumulative compression layer remains deferred (Derived Layer, Stage C).

Extractor implementation for v1: see D-21 — runs as a Claude Code spawn, not a direct Anthropic API call.

D-16 — One store, two tiers (not two stores). Everything execution-related lives in the journal. Records carry a tier: raw (machine-emitted — high-volume, never read in the hot path) vs derived (extractor-produced handoff — low-volume, what the harness reads). The read surface is a query filtered to the derived tier. One store so derived records reconcile by construction. (Note the deliberate boundary: the journal holds execution truth; Trellis holds context truth. These are different stores by design — see D-22. "One store" means one store for execution, not one store for everything.)

D-17 — Handoff schema is data, in a contract table. Applies D-4 one level up. Handoff fields are rows in a contract table, not a baked-in struct: each row defines key, required flag, who reads it, semantics, value shape. Adding a field is an insert. Write-side only (governs generation + validation); never consulted at read time.

Drift posture in v1: single-user / single-writer means adding a required field while extractors are in-flight is structurally low-risk. No mechanism added; mitigation is the scope constraint itself (Out of scope #4). Later concurrency stages revisit.

D-18 — Validation gate on the derived write path. Extractor output is validated against the active contract before becoming a valid derived record. On failure, the task lands with status = extraction_failed; no retry in v1 (FR-009). The not-clean-return policy (Phase 2) decides downstream routing. Framed as visibility over reliability: a malformed extractor pass becomes a structurally visible outcome, not a silent stall or an automated retry loop. This is what makes the derived handoff trustworthy where the rejected worker-emitted handoff was not.

D-19 — Write-side / read-side split → no handoff versioning. Contract consumed only at write time; handoffs immutable and self-describing at read time. No version-stamping. Contract rows immutable; the contract set is append-only (drift-by-accretion). contract.field_key stable, never reused with a different meaning (supersession is a new row). Read-side discipline: readers must be shape-tolerant — read what exists, branch on what they care about, tolerate absence and unknowns.

D-20 — parent_id nesting tree, orthogonal to the id hierarchy. Records carry a self-referential parent_id for arbitrary-depth execution nesting, in addition to the scope ids. The ids answer "which scope"; parent_id answers "what nests under what." Makes the recursive-handoff model literal: a handoff exists at any node; a parent's handoff is a real record, not merely an aggregation view.

D-21 — Extractor runs as a Claude Code spawn for v1. The extractor (D-15) is a Claude Code worker, not a direct Anthropic API call. Same substrate posture as every other worker: prompt + tools + exit. Driver: steerable with the same techniques, debuggable with the same observability (the journal), runs under the same auth (Phase −1 OAuth) — no second credential surface, no second control plane. Doubles spawn volume per orchestrator turn (worker + extractor), which is what Phase −1's per-account limit test is for. Cross-refs: D-15, Phase 0.5, Phase −1.

Open sub-question (deferred): does the extractor's own Claude Code run appear in the journal as a regular worker event or is it special-cased / suppressed? Default lean: appears as a worker event. Decide in Phase 0.5.

D-22 — Two stores by kind of truth: journal (execution) and Trellis (context). (p4) The journal owns execution truth (what happened — events, handoffs; immutable, forensic). Trellis owns context truth (what's in play — threads, conversation, working picture). They own different objects and cross-reference by id (thread_id on the journal envelope; Trellis threads reference project_id) — link, don't duplicate. This is the same link-not-duplicate discipline as parent_id and plan_ref, applied across the two systems, and it is what keeps D-16's "no second source of truth" intact across stores: the journal entry is the evidence, the Trellis update is the understanding built from it, never the same fact written twice. The synthesis layer (FR-013) is the one-directional transform between them (journal → synthesis → Trellis); the extractor (D-15) stays journal-internal. Open (Stage C): the precise synthesis↔Trellis write contract — what context objects synthesis writes, at what grain — designed when synthesis is built.

Data model

Journal (one table family, two tiers).

Tier discriminator: every record carries tier ∈ {raw, derived}.
Typed envelope columns: event_id (PD-issued UUID, PK), project_id, session_id, task_id, parent_id (nullable, self-FK — the nesting tree), thread_id (nullable BIGINT — Trellis cross-reference, migration 003), message_id (nullable PD-issued UUID, reconcile key for client render), event_type (FK to reference table), tier, created_at, monotonic seq.
event_type — reference table, open set, never PG ENUM or hardcoded.
JSONB payload — variable structured body. Anthropic's message.id (when present) lives inside payload as a forensic field. No display markup stored as truth. (Exception: HTML that is a worker's actual output content is data, held as a JSONB string.)
seq is the ordering authority — consumers order by seq, never arrival time.
Plan linkage: project_id keys to the immutable vault plan; a plan_ref field links a record to a specific plan task. (See Cross-cutting → Task addressing.)
thread_id semantics: nullable, no FK (convention-not-FK, D-3/D-22); a derived row inherits its parent raw row's thread_id; sits NULL until synthesis writes it. Makes "all journal evidence for thread X" a typed-column query.

Handoff (derived tier).

handoff_id (PD-issued UUID), the scope ids it belongs to, parent_id for its place in the tree, thread_id inherited from parent, tier = derived, seq, created_at.
Payload is self-describing — carries whatever fields the active contract defined at write time. Candidate field set (all defined as contract rows, not hardcoded): goal, current_state, next_step, ruled_out ({approach, reason} list — load-bearing, FR-011), status, artifacts, gate-decision fields.
status ∈ {success, failed, crashed, cancelled, rate_limited, partial, skipped, gated, extraction_failed}. Semantics:
- success — clean exit (0), validation gate passed.
- failed — clean nonzero exit; worker ran, returned a real error.
- crashed — abnormal exit (137 / SIGKILL, OOM, segfault, kernel reap).
- cancelled — intentional kill (130 / SIGINT or 143 / SIGTERM).
- rate_limited — substrate quota or auth refresh aborted the run; signals back off, not retry. Not derivable from exit code; the supervision layer surfaces it.
- partial — clean exit with incomplete work.
- skipped — worker did not run (gate or precondition declined).
- gated — autonomous-mode hard gate fired; logged as positive evidence the gate logic ran.
- extraction_failed — worker completed cleanly, validation gate rejected the derived handoff. Raw events landed but no valid derived record produced; the raw tier remains the forensic record.
artifacts — commits / files / logs / vault refs.

Contract table.

One row per handoff field. Columns: field_key (stable, never reused), required, read_by, semantics, shape (JSON). Append-only, immutable rows.
Consumed only at write time (extractor + validator). Never read at handoff-read time.

Interfaces / contracts

Outbound (store → client): SSE stream + LISTEN/NOTIFY-driven "new record" notifications. Multiple concurrent readers.
Inbound (client → orchestrator): REST action → backend → orchestrator session stdin. Backend is the single writer to stdin (v1 trivially; arbitration built later if needed).
Echo: user input becomes a USER_MESSAGE event; all clients render it from the store.
Client render: keyed by the reconcile key (PD-issued UUID) so render-on-final and optimistic render share a path.
Trellis read surface (p4): backend exposes a read API over Trellis thread/context state (live thread, identity, recent context) for the clients to render (FR-012). Read-only from petal-dispatch's side in v1; petal-dispatch does not write Trellis until synthesis (FR-013, Stage C).
Derived read surface (the MCP/API for orchestration):
- get_handoffs(scope_id, limit?, order=desc) — ordered handoffs for whatever scope the id resolves to (task / session / project), newest-first.
- get_project_state(project_id) — bounded rollup: which plan tasks done / in-flight / failed / blocked. A few hundred tokens; what the orchestrator calls before deciding the next move.
- log_step / handoff write — goes through the validation gate (D-18); enforced deterministically (Principle #6).

Deployment

Backend (API + journal + extractor + orchestrator session) on herbys-dev. Postgres = petalbrain instance.
Reachability: Tailscale-only for v1 (phone on tailnet); already in place.
Android artifact: sideload v1 (adb install / direct APK); self-hosted F-Droid behind Traefik as fast-follow.
Web client: served from petal-dispatch's /static, same Traefik route as the API.
GitHub pushes: Travis only (standing model); Lovebug works on Gitea / petal-power.

Tests

Rigorous on journal/backend/extractor — the truth + derivation layers; bugs corrupt everything downstream. zero-check applies. The Phase 0 stream-json dump is the canonical fixture set for journal-writer AND extractor tests.
Validation gate gets real tests — malformed extractor output must surface as status = extraction_failed, not as a silently-stored handoff (FR-009).
Trellis surface gets a graceful-degradation test — Trellis-unavailable must not block the journal-backed view (Edge cases).
Pragmatic on clients — they're views; the reconcile logic (optimistic-render stage) is the one client-side piece earning real tests.

NFRs

Latency / throughput: n/a as hard budgets for v1 (single user). Render-on-final accepts slight screen lag until the optimistic-render stage.
Threat model: single-user personal tool, Tailscale-only, not publicly exposed in v1. Trust boundary is the tailnet. Public exposure is a deliberate later decision with its own threat review.
Observability: the journal is the observability surface — same event log the tool exposes.

Phases

Phase −1 — Viability spike (bounds the OAuth pre-mortem)

Confirm production auth permits the remaining required control flow. Already partially cleared by the prototype: turn-injection works, the event stream is observable, session_id threads through every event.

Decision: use the Claude subscription OAuth credential, not an API key. Sets the substrate posture for the orchestrator session and D-21's extractor spawns. Remaining items are tests under OAuth, not credential choices.

Numbered test list — one decision rule per item:

MCP loading without re-prompts. The orchestrator session loads the MCP servers it needs (vault, trellis, dispatch) on the OAuth credential without interactive re-prompts. Fails if any MCP requires a user-driven re-auth step the long-lived session cannot service unattended.
Per-account limits under doubled spawn volume. D-21's extractor doubles spawn count per turn. Run a realistic per-cycle load, observe quota headroom. Fails if a single typical cycle bumps the rate limit, or if the per-day envelope cannot absorb a full SC-4 sequential run.
Auth refresh / token expiry on the long-lived orchestrator. Fails if a token expiry interrupts a worker mid-spawn or leaves a stale credential the session cannot recover from without external intervention.

Framed as measuring the dependency, not erasing it. Satisfies: de-risks FR-001/002/005.

Phase 0 — The journal

Dump live stream-json → derive event schema → build journal (envelope incl. parent_id self-FK and thread_id, reference-table event types, JSONB payload, seq ordering, tier discriminator) + the writer.

Dump methodology — required scenario set (curated, not happy-path):

One-shot worker, text-only response.
One-shot worker that emits tool_use blocks.
Long-lived worker running multiple turns (forward-compat / Principle #4).
Worker that errors out.
Worker externally SIGTERMed mid-turn.
Worker that hits rate limits or auth refresh during run.

Must-resolve-before-code: (a) reconcile-key structure — does each assistant turn emit a message_start with a stable id (captured into payload, not relied on as the key — D-12); (b) exact relational shape; (c) parent_id write semantics (who sets it, when). Status note: journal envelope frozen + applied (migration 001); thread_id added via migration 003 (p4 seam). Satisfies: FR-002.

Phase 0.5 — The extractor + contract + derived tier (in progress)

Define the contract table; seed the v1 field set as rows.
Build the extractor as a Claude Code spawn (D-21): on worker task completion, spawn an extractor worker that reads the source worker's raw final output and produces a structured handoff.
Build the validation gate (D-18): validate before the derived write. On failure, no retry in v1 — task lands extraction_failed.
Confirm worker raw output carries enough signal for required fields (especially ruled_out); tune the extractor prompt against Phase 0 fixtures.
Decide whether the extractor's own run appears as a worker event (default lean: yes). Status note: migration 002 (contract table + new event_types) written, apply bundled with migration 003. Satisfies: FR-008, FR-009, FR-011.

Phase 1 — Backend API + push

REST for actions, SSE for live read, LISTEN/NOTIFY → SSE fan-out, backend owns the single write path to orchestrator stdin. Derived read surface (get_handoffs, get_project_state) exposed. Trellis read surface exposed (FR-012) — read-only API over Trellis thread/context state for the clients. Backend bootstraps with OPEN_MODE=1 (D-14). Web client served from /static. systemd unit + secrets.env carried from the demo. Satisfies: FR-003, FR-005 (mechanism), FR-007 (mechanism), FR-010, FR-012 (backend), foundation for FR-004.

Phase 2 — Orchestrator loop against the journal

Define the loop: get_project_state → pick next task → build worker brief via get_handoffs(task scope) → spawn worker → raw output lands → extractor derives handoff → loop. Orchestrator never reads raw worker output; per-cycle context stays roughly constant.
Implement the not-clean-return policy: worker returns failed / crashed / cancelled / rate_limited / partial / gated / extraction_failed → retry same brief / retry modified / escalate / skip / back off. Define upfront. crashed vs failed vs cancelled warrant different responses; extraction_failed means raw work is fine but derivation needs attention; rate_limited means back off.
Select the test project for SC-4 / SC-6 / SC-8 before this phase closes. Satisfies: success criteria 4, 8; FR-010.

Phase 3 — Web client

Browser client, render-on-final keyed by the reconcile key, Tasks/Chat/Activity surfaces. Trellis thread/context surfaced (FR-012) — the glanceable live-thread/context tiles (Lovebug's mockups are the starting point). Proves store→API→client→render end-to-end on the known stack before Android. Home tab: Travis picks before build (current lean Task-first). Satisfies: FR-001, FR-003, FR-006 (visibility), FR-007 (UI surface), FR-012 (web surfacing).

Phase 4 — Native Android client

Extend the existing petal-dispatch-android repo (v0 chat scaffold from the demo). Carry forward Kotlin/Compose setup, Material 3 theme, OkHttp, the chat screen as the v1 "Chat" tab. New surfaces: Tasks tab (running + recent worker tasks, status, tap-to-drill), Activity tab (chronological event feed), Chat tab (existing, repointed to v1 API + SSE), Trellis thread/context surfacing (FR-012) on the relevant tab(s). Networking rewrite: drop WebSocket-only, switch to REST (POST actions) + SSE. Same OkHttp transport. Tailscale reachability: assume phone on tailnet; show clear "off-tailnet" state with reconnect, don't crash. Auth: trust the network initially (OPEN_MODE=1); add HTTPS auth + Authentik header when D-14 flips. Sideload: debug APK → dist/, transfer via Tailscale share / Syncthing / USB. Satisfies: FR-004, FR-001 (mobile), FR-012 (Android surfacing), success criteria 1 & 3.

Phase 5 — Optimistic render (the "feel" stage)

Optimistic paint + reconcile-in-place onto the existing id-keyed path, both clients. Orphan-timeout handling. Satisfies: live-streaming feel; edge cases (orphaned display, record-disagrees-with-stream).

Iterate-shaped phases

Phase 5 (optimistic render), the home-tab/UI work, and the Trellis surfacing UI (FR-012, Phase 3/4) are converge-with-Travis loops — done when it feels right, not when a spec criterion is met. The extractor-prompt tuning (Phase 0.5) is similarly iterate-shaped against the dump fixtures.

Human-required handoffs

Coordinated live with Lovebug during execution. Known touch-points: the stream-json dump + schema sign-off (Phase 0), the contract seed + extractor-prompt review (Phase 0.5), the test-project selection (Phase 2), GitHub pushes (all phases), Android-toolchain / physical-phone steps (Phase 4), Tailscale/cert wiring, the OPEN_MODE → Authentik flip (D-14).

Cross-cutting

Deferred build stages (the "version stack")

p4 is the canonical plan; these are sequenced build stages after the v1 cut, all forward-compatible with the schema by Principle #4 — "more of what the schema already allows," not rewrites.

v1 — single orchestrator + workers + journal + extractor/derived tier + Trellis read surface + Android & web clients; render-on-final; one-shot worker; multi-client read concurrency.
v1.1 — optimistic render + reconcile.
Derived Layer Stage C (synthesis compression + journal→Trellis context bridge) — the synthesis layer (FR-013): reads handoffs from the journal, writes a working-context picture to Trellis keyed by thread_id. And cumulative rolling brief over the handoff list. Trigger: the per-step handoff list grows past comfortable per-cycle reading; define "uncomfortable" with real numbers then. Open sub-decisions (decide with data): cumulative rolling brief vs. windowed last-N; the precise synthesis↔Trellis write contract (D-22). If cumulative, add drift detection anchored to the journal/plan (immutable references) — structural-invariant checks, periodic reconciliation. Never anchor a drift check to another synthesis pass.
Orchestrator/harness split (p4 — deferred) — a conversational, context-rich orchestrator (Lovebug, the layer you chat with) above a fresh, disposable harness (the executor, fed a bounded brief). Different context disciplines by design: the orchestrator accumulates context (and is the natural reader of Trellis context); the harness stays fresh (reads bounded handoffs from the journal). The handoff between them is a compressed brief ("here's the plan, execute as written, report back"). Not foreclosed by Principle #4 + the v2 multi-orchestrator stage — it's "more session-roles," not a rewrite. Designed after living with v1's visibility reveals its requirements. Undecided whether Travis also drives the harness directly; the harness should accept a brief from either the orchestrator or Travis.
v2 (concurrency / multi-Dispatch) — multiple concurrent Dispatch sessions; simultaneous multi-client input arbitration (if ever); desktop client; other reachability pathways. (The single-user / single-writer drift-low-risk posture under D-17 stops holding here; revisit contract-mutation safety then.)
v4 (interactive workers) — long-lived / held-open workers, routable for direct input; the turn-boundary signal (task.idle) decoupled from exit gets built here.

Why per-step (not cumulative) for the v1 derived layer

Per-step handoffs are immutable once written; the "history" is the ordered list itself — no fold-forward, no rewriting. Trades read-time cost for write-time safety (no drift surface inside any handoff). Cumulative rolling summaries introduce per-step drift and a circular recovery story. At v1 scope (~10–30 steps), reading the full list is fine. Compression is added later, with real data. (Simple because we reasoned to it, not because we didn't look.)

Task addressing (dependency, needs a pass)

plan_ref (FK into a plan task like T-042) and the "task IDs feed worker briefs" elegance both depend on vault plans being structured with addressable task IDs — which they mostly are NOT yet. Either plans gain task IDs (nudges plan structure toward spec-shaped, addressable tasks — aligned with plan-interviewer direction), or plan_ref is looser than an FK for v1. Open item — resolve before relying on brief→journal traceability.

Autonomous-mode gates

Two execution modes: interactive (Travis approves each advance) and autonomous (Lovebug completes + documents, stops at hard gates). Gate categories — operations where cost-of-wrong is high and cost-of-asking is low: outside the project tree, sudo/privilege escalation, network-facing config, secrets touched, destructive data ops, git ops beyond the working branch, dependency/package additions, cross-service changes. A gated status + gate-decision fields log the event as positive evidence the gate logic fired. Decision-doc contract for autonomous mode: the handoff must capture the decision point, the chosen option, the rule-out for the rest, and what would have triggered a gate — auditable without re-reading transcripts.

Codebase assumptions

Relies on, does not touch: Postgres (petalbrain), Traefik (fronts API + web client), Tailscale (reachability), Gitea/petal-power (dev), existing claude Code sessions and --output-format stream-json behavior, Trellis (read-only in v1 — petal-dispatch reads thread/context state; does not write it until synthesis).
Spawn mechanism: observe-only. Build observes session/worker events; does not modify how workers spawn (Claude Code native). A launch wrapper is a conditional fallback — moot under one-shot, relevant only in the concurrent + long-lived world.
Substrate-dependency containment (pre-mortem mitigation): the harness/adapter is the sole layer touching Claude Code's control surface. Journal, extractor, API, clients view the store. If the substrate changes or is swapped, rebuild that one adapter, not the tool. (D-21 puts the extractor inside the substrate-dependent surface too.)
MCP is the information surface, not the control surface. MCP exposes read/write tools; it cannot spawn, reap, or swap workers. Control (lifecycle) lives in the harness. The journal's only jobs: store the typed record, key it by id, serve it by scope.

What carries forward from the demo (proven patterns, not code)

The prototype (petal-power/petal-dispatch, to be renamed petal-dispatch-demo) ran a working harness, worker→harness event wiring, Android chat APK, Authentik+Tailscale integration. Code is a sandbox; lessons carry verbatim:

Postgres LISTEN/NOTIFY → SSE fan-out works at single-user scale.
Event taxonomy starting point: task.spawned, task.message, task.idle, task.user_message, task.crashed (schema evolves in Phase 0 from the dump).
X-Authentik-Username header pattern (D-14 flip).
OPEN_MODE env flag pattern (verbatim for v1 bootstrap).
systemd unit + secrets.env petalbrain DSN delivery.
Tailscale + Traefik dynamic config (no changes needed for v1).
Kotlin scaffold structure (extended, not restarted).

Does not carry forward: existing dispatch_harness.events rows; tasks.json subprocess-manager persistence format; existing FastAPI route layout.

Repo & naming

Existing petal-power/petal-dispatch → renamed petal-power/petal-dispatch-demo.
New petal-power/petal-dispatch → v1 backend (Service 1).
Existing petal-power/petal-dispatch-android → continues as v1 client (Service 2), README updated to v1 contract.

Companion vault projects (now folded here)

execution-journal, synthesis-layer — superseded by this artifact. operating-mode (status: idea) remains separate — the client is the natural surface to invoke + display the active mode; ties into Principle #3 visibility and the control-surface open question.

Prototype findings

task.spawned can arrive after task.user_message (unordered asyncio.create_task). Fix baked into D-3's seq invariant.
task.crashed.last_error populated with the assistant's reply on clean SIGTERM. Resolved at the schema level by the expanded status enum — cancelled / crashed / failed / rate_limited carry the real outcome.
task.done unreachable under the long-lived v0.5 worker — dissolves under one-shot; re-emerges only as a v4 (interactive-workers) question.

Unresolved questions

Must-resolve-before-code: reconcile-key structure (Phase 0 dump); exact journal relational shape (Phase 0); parent_id write semantics (Phase 0); whether worker raw output reliably carries the ruled_out signal (Phase 0.5); remaining production-auth control confirmation under OAuth (Phase −1's three tests).
Needs-a-pass: task addressing / plan_ref dependency on plans gaining addressable task IDs; the synthesis↔Trellis write contract (D-22, Stage C).
Can-defer-and-tag: home-tab choice (before Phase 3 layout); whether the spawn mechanism ever needs a wrapper (only if concurrent); cumulative-vs-windowed compression (Stage C); whether the extractor's own run appears as a worker event (Phase 0.5, default lean yes); test-project selection for SC-4 / SC-6 / SC-8 (Phase 2).
Headline open question (defer, do not block foundation): the control surface for context — what levers Travis gets to steer/shape the orchestrator's context. v1 has zero control surface beyond "send a message to Dispatch." Steering levers (operating-mode invocation, context-trim, worker-tool-allowlist) are later stages. Visibility ships in v1; control is undefined and explored later. (The deferred orchestrator/harness split is the eventual home for much of this control surface — the conversational layer is where steering naturally lives.)

Pre-mortem

Failure mode (Travis's intent): Six weeks out, the OAuth subscription doesn't permit the control flow needed — the architecture is sound and the tool is still dead because the substrate won't allow the control. Structural, not a bug: the foundation is someone else's tool under consumer auth, control surfaces owned by Anthropic and subject to change. Bounded, not eliminated, by (a) the Phase −1 spike measuring the dependency before building on it (OAuth named directly, three numbered tests with decision rules), and (b) Principle #2 / containment keeping the dependency in one swappable adapter. (D-21 puts the extractor on the same OAuth substrate, doubling per-turn spawn volume — Phase −1 test #2 is the early-warning instrument.)

Secret suspicions (surfaced)

The worker-model elegance (one-shot → exit = completion) is a lean, not a tested commitment — interactive/long-lived workers were already observed in the prototype, so one-shot may not survive contact with how Travis actually works. Principle #4 is the deliberate hedge.
The extractor (D-15 / D-21) is itself a language model reading prose and emitting structure — the same kind of actor we refused to trust in Option A. What makes it trustworthy: extraction is its whole job under a tight prompt, and its output is schema-validated at the gate (D-18) — a malformed handoff becomes extraction_failed, not a silently-stored record. If it proves unreliable even so, the fallback is a stricter contract + retry budget (deferred in v1), not trusting raw output.

Notes

Why one artifact. The journal, the derived layer, the Trellis seam, and the clients are one architecture with shared seams — the tier discriminator, contract-as-data (D-4 reused as D-17), parent_id, the PD-issued-UUID convention (D-12 → handoff_id), and the thread_id cross-reference (D-22). Keeping them in separate plans created contradictions (notably the "no synthesizer built" line Option B overrides). One artifact is the single source of truth.

Why two stores (journal + Trellis) is not a contradiction of "one store." D-16's "one store, two tiers" is about execution truth — don't split the journal into raw-store + derived-store and then have to sync them. D-22's "two stores" is about kinds of truth — execution (journal) vs. context (Trellis) — which own different objects and link by id. Same link-not-duplicate discipline, two different scopes. The synthesis layer is the one-way bridge; the extractor never crosses it.

The blackboard pattern. This is the well-established blackboard / shared-state pattern for multi-agent orchestration, built purpose-fit for the herbylab stack rather than adopted from a heavyweight framework. What's not out there is a purpose-built, plan-keyed execution journal with a derived handoff layer as a first-class orchestration participant for a personal stack.

Why journaling matters for context. Models cannot dump-and-reload their own context — writing to a table does not shrink the window within a session. Journaling is what makes ending a session survivable: the agent dies, the journal persists, the next agent hydrates from a small derived slice. Context stays small because the read API does the filtering, not the model. Agents are cheap and replaceable; the journal is durable.

Drift is bounded, not eliminated. 100% accurate history is not the goal. The real asymmetry is correction-loop tempo: agents compound lossy retelling fast while the human correction loop is slow. The hedge is mechanical checks anchored to immutable references (journal + plan), not another agent's second opinion — and, for v1, per-step immutable handoffs with no internal drift surface.

Visibility over reliability, intentionally. The no-retry-on-extraction-failure posture (D-18, FR-009) deliberately leaves more surface visible rather than papering over it with retries. The v1 read should be "I can see exactly what went wrong" before "the system handled it for me." Reliability layers in later; it goes on top of visibility, not in place of it.

Slug-collision resolved (p4). The execution-journal slug previously resolved to a UI-patterns session-note rather than the real plan; Lovebug fixed the resolver/slug as of 2026-06-04. Folded plans should be marked superseded.

Supersession (p4 ← p3). p4 supersedes p3: reframed Goal/Problem around operational control & visibility (the seamless loop is now the output, not the goal; silent stalls are one consequence); added Trellis as a client surface (FR-012) and the synthesis→Trellis context bridge (FR-013 / D-22, deferred to Stage C); added the thread_id journal seam (migration 003); recorded the orchestrator/harness split as a deferred build stage. Rationale for each lives inline at the relevant section.

50 KiB Raw Blame History Unescape Escape