6.4 KiB
| created | path | project | status | tags | type | updated | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| 2026-05-15 | Sources/Dev | zero-check-refactor | active |
|
session-notes | 2026-05-15 |
Outcome
Reviewed the GitHub Engineering blog post on validating agentic behavior in non-deterministic environments. Concluded the specific technique (dominator analysis on UI execution graphs) is not directly applicable to the zero-check gauntlet today, but the underlying principle — that agents cannot reliably grade their own homework and need an independent structural validator — is exactly the design premise of zero-check itself. Parking the article's deeper ideas for a future "semantic validation layer" if and when the harness/Lovebug starts delegating substantive code work that needs intent-level checks.
Topics Covered
What the article proposes
GitHub's Copilot Coding Agent does "computer use" — driving a real VS Code in a containerized environment. Validating this is hard because the agent's path is non-deterministic: loading screens come and go, hotkey vs menu produces the same outcome, timing varies. Traditional record-and-replay or assertion-based tests fail with false negatives.
Their approach, summarized:
- Record 2–10 successful runs as a Prefix Tree Acceptor (a directed graph of observed states + transitions)
- Merge them into a unified graph using a three-tier equivalence check (perceptual hash → SSIM → multimodal LLM for semantic equivalence)
- Apply dominator analysis (compiler-theory concept: state A dominates state B if every path from start to B passes through A) to extract the essential states — the milestones every successful run must hit
- Validate new runs by checking they hit the essential states in the right relative order; extra/incidental states are tolerated
Reported result: 100% accuracy on their test suite vs the agent's self-assessment at 82.2%. Recall jumped from 60% to 100%.
What's genuinely useful
- Essential vs incidental framing. Don't validate the exact path; validate whether the agent crossed the checkpoints that matter. Generalizable mental model beyond UI work.
- "Agents can't grade their own homework." Their data: agent self-assessment got 0% F1 on distinguishing real bugs from environmental noise. Zero. Independent structural check got 52%. This is the explicit design premise of zero-check — the agent writes the code, an external tool decides whether it's valid.
- Learning correctness from examples. Instead of writing assertions, record successful runs and extract the contract automatically. Cheaper in principle, when applicable.
What's overhyped or doesn't apply
- The setup is narrow. Their technique works because UI flows have real graph structure (discrete states, observable transitions). Code generation, refactoring, architectural decisions don't have that structure — the technique doesn't generalize as cleanly as the article implies.
- "100% accuracy" is single-test-suite marketing. Controlled experiment on one task type. Not a generalizable claim.
- The LLM-in-the-loop is soft-pedaled. They say "no black-box ML judging," but step 2 of the merge calls a multimodal LLM for semantic state equivalence. Scope-limited, but it's there.
- Bootstrap dependency. Requires 2–10 consistently successful runs to extract dominators. For flaky tasks, getting that bootstrap is itself the hard problem.
Relevance to zero-check today
Honest read: the article is not actionable for the current gauntlet. Zero-check validates deterministic checks (lint, tests, SAST, secrets, deps) where there's a single right answer per tool. Dominator analysis is overkill for "did ruff pass."
Where the article would matter is a hypothetical second layer — validating that the agent's overall work met the stated intent, not just that the code passes mechanical checks. Examples:
- "Add JWT auth to the API" — did the agent actually wire JWT into middleware, or just add a TODO comment?
- "Refactor to dependency injection" — did the result actually use DI, or just rename files?
For those questions, lint-pass is necessary but wildly insufficient. That's the territory where structural intent validation gets interesting.
If/when that second layer becomes real
The article's specific technique (dominators on execution graphs) is the wrong starting point for code-level intent validation — it's the right tool for UI flows, not for diffs. Simpler approaches that fit the existing stack better:
- Spec-first. Agent writes a one-paragraph "this is what I'm going to do" before the change. A separate validator LLM reads the diff and judges whether it matches the spec.
- Test-as-oracle. Agent writes a test capturing the intent before writing the code; the gauntlet runs the test. Essentially harness-enforced TDD.
- Behavioral fixture. For tasks with clear input/output shape, record what success looks like and check against it.
All three are in the same family as the article — external structural check on intent — but cheaper and better matched to code rather than UI.
Key Learnings
- The article's core insight (agents need independent structural validators because self-assessment is unreliable) is the same principle zero-check operates on. The refactor we did this session is the deterministic-layer version of what the GitHub post does at the UI layer.
- Dominator analysis is a real technique with a clean theoretical basis (compiler control-flow theory) — worth remembering even if not applied today.
- The two-layer mental model is useful: deterministic gauntlet (lint/test/SAST) + semantic intent validator. Today only layer 1 is built. Layer 2 is a future concern, not a current gap.
- Press-release numbers ("100% accuracy") in research blog posts should be read as "worked on our test set" not "general capability."
Follow-ons
- Revisit this article if and when harness/Lovebug starts taking on substantive code-generation tasks where mechanical checks aren't enough to know if the work was correct
- If a semantic validation layer is ever built, evaluate spec-first and test-as-oracle approaches before reaching for graph-based techniques
- No action on zero-check itself from this review — the in-flight refactor (manifest-driven routing, inconclusive guard) is the right shape for the deterministic layer