--- created: '2026-05-15' path: Sources/Dev project: zero-check-refactor status: active tags: - claude-code - automation - python - go type: session-notes updated: '2026-05-15' --- ## Outcome Reviewed the GitHub Engineering blog post on validating agentic behavior in non-deterministic environments. Concluded the specific technique (dominator analysis on UI execution graphs) is not directly applicable to the zero-check gauntlet today, but the underlying principle — that agents cannot reliably grade their own homework and need an independent structural validator — is exactly the design premise of zero-check itself. Parking the article's deeper ideas for a future "semantic validation layer" if and when the harness/Lovebug starts delegating substantive code work that needs intent-level checks. Reference: ## Topics Covered ### What the article proposes GitHub's Copilot Coding Agent does "computer use" — driving a real VS Code in a containerized environment. Validating this is hard because the agent's path is non-deterministic: loading screens come and go, hotkey vs menu produces the same outcome, timing varies. Traditional record-and-replay or assertion-based tests fail with false negatives. Their approach, summarized: 1. Record 2–10 successful runs as a Prefix Tree Acceptor (a directed graph of observed states + transitions) 2. Merge them into a unified graph using a three-tier equivalence check (perceptual hash → SSIM → multimodal LLM for semantic equivalence) 3. Apply **dominator analysis** (compiler-theory concept: state A dominates state B if every path from start to B passes through A) to extract the *essential* states — the milestones every successful run must hit 4. Validate new runs by checking they hit the essential states in the right relative order; extra/incidental states are tolerated Reported result: 100% accuracy on their test suite vs the agent's self-assessment at 82.2%. Recall jumped from 60% to 100%. ### What's genuinely useful - **Essential vs incidental framing.** Don't validate the exact path; validate whether the agent crossed the checkpoints that matter. Generalizable mental model beyond UI work. - **"Agents can't grade their own homework."** Their data: agent self-assessment got 0% F1 on distinguishing real bugs from environmental noise. Zero. Independent structural check got 52%. This is the explicit design premise of zero-check — the agent writes the code, an external tool decides whether it's valid. - **Learning correctness from examples.** Instead of writing assertions, record successful runs and extract the contract automatically. Cheaper in principle, when applicable. ### What's overhyped or doesn't apply - **The setup is narrow.** Their technique works because UI flows have real graph structure (discrete states, observable transitions). Code generation, refactoring, architectural decisions don't have that structure — the technique doesn't generalize as cleanly as the article implies. - **"100% accuracy" is single-test-suite marketing.** Controlled experiment on one task type. Not a generalizable claim. - **The LLM-in-the-loop is soft-pedaled.** They say "no black-box ML judging," but step 2 of the merge calls a multimodal LLM for semantic state equivalence. Scope-limited, but it's there. - **Bootstrap dependency.** Requires 2–10 *consistently successful* runs to extract dominators. For flaky tasks, getting that bootstrap is itself the hard problem. ### Relevance to zero-check today Honest read: the article is not actionable for the current gauntlet. Zero-check validates deterministic checks (lint, tests, SAST, secrets, deps) where there's a single right answer per tool. Dominator analysis is overkill for "did ruff pass." Where the article *would* matter is a hypothetical second layer — validating that the agent's *overall work* met the stated intent, not just that the code passes mechanical checks. Examples: - "Add JWT auth to the API" — did the agent actually wire JWT into middleware, or just add a TODO comment? - "Refactor to dependency injection" — did the result actually use DI, or just rename files? For those questions, lint-pass is necessary but wildly insufficient. That's the territory where structural intent validation gets interesting. ### If/when that second layer becomes real The article's specific technique (dominators on execution graphs) is the wrong starting point for code-level intent validation — it's the right tool for UI flows, not for diffs. Simpler approaches that fit the existing stack better: - **Spec-first.** Agent writes a one-paragraph "this is what I'm going to do" before the change. A separate validator LLM reads the diff and judges whether it matches the spec. - **Test-as-oracle.** Agent writes a test capturing the intent *before* writing the code; the gauntlet runs the test. Essentially harness-enforced TDD. - **Behavioral fixture.** For tasks with clear input/output shape, record what success looks like and check against it. All three are in the same family as the article — *external structural check on intent* — but cheaper and better matched to code rather than UI. ## Key Learnings - The article's core insight (agents need independent structural validators because self-assessment is unreliable) is *the same principle* zero-check operates on. The refactor we did this session is the deterministic-layer version of what the GitHub post does at the UI layer. - Dominator analysis is a real technique with a clean theoretical basis (compiler control-flow theory) — worth remembering even if not applied today. - The two-layer mental model is useful: deterministic gauntlet (lint/test/SAST) + semantic intent validator. Today only layer 1 is built. Layer 2 is a future concern, not a current gap. - Press-release numbers ("100% accuracy") in research blog posts should be read as "worked on our test set" not "general capability." ## Follow-ons - [ ] Revisit this article if and when harness/Lovebug starts taking on substantive code-generation tasks where mechanical checks aren't enough to know if the work was correct - [ ] If a semantic validation layer is ever built, evaluate spec-first and test-as-oracle approaches before reaching for graph-based techniques - [ ] No action on zero-check itself from this review — the in-flight refactor (manifest-driven routing, inconclusive guard) is the right shape for the deterministic layer