mcp: session-notes — agentic-validation-article-review

Sources/Dev/2026-05-15-agentic-validation-article-review.md

@@ -0,0 +1,80 @@
+---
+created: '2026-05-15'
+path: Sources/Dev
+project: zero-check-refactor
+status: active
+tags:
+- claude-code
+- automation
+- python
+- go
+type: session-notes
+updated: '2026-05-15'
+---
+
+## Outcome
+
+Reviewed the GitHub Engineering blog post on validating agentic behavior in non-deterministic environments. Concluded the specific technique (dominator analysis on UI execution graphs) is not directly applicable to the zero-check gauntlet today, but the underlying principle — that agents cannot reliably grade their own homework and need an independent structural validator — is exactly the design premise of zero-check itself. Parking the article's deeper ideas for a future "semantic validation layer" if and when the harness/Lovebug starts delegating substantive code work that needs intent-level checks.
+
+Reference: <https://github.blog/ai-and-ml/generative-ai/validating-agentic-behavior-when-correct-isnt-deterministic/>
+
+## Topics Covered
+
+### What the article proposes
+
+GitHub's Copilot Coding Agent does "computer use" — driving a real VS Code in a containerized environment. Validating this is hard because the agent's path is non-deterministic: loading screens come and go, hotkey vs menu produces the same outcome, timing varies. Traditional record-and-replay or assertion-based tests fail with false negatives.
+
+Their approach, summarized:
+
+1. Record 2–10 successful runs as a Prefix Tree Acceptor (a directed graph of observed states + transitions)
+2. Merge them into a unified graph using a three-tier equivalence check (perceptual hash → SSIM → multimodal LLM for semantic equivalence)
+3. Apply **dominator analysis** (compiler-theory concept: state A dominates state B if every path from start to B passes through A) to extract the *essential* states — the milestones every successful run must hit
+4. Validate new runs by checking they hit the essential states in the right relative order; extra/incidental states are tolerated
+
+Reported result: 100% accuracy on their test suite vs the agent's self-assessment at 82.2%. Recall jumped from 60% to 100%.
+
+### What's genuinely useful
+
+- **Essential vs incidental framing.** Don't validate the exact path; validate whether the agent crossed the checkpoints that matter. Generalizable mental model beyond UI work.
+- **"Agents can't grade their own homework."** Their data: agent self-assessment got 0% F1 on distinguishing real bugs from environmental noise. Zero. Independent structural check got 52%. This is the explicit design premise of zero-check — the agent writes the code, an external tool decides whether it's valid.
+- **Learning correctness from examples.** Instead of writing assertions, record successful runs and extract the contract automatically. Cheaper in principle, when applicable.
+
+### What's overhyped or doesn't apply
+
+- **The setup is narrow.** Their technique works because UI flows have real graph structure (discrete states, observable transitions). Code generation, refactoring, architectural decisions don't have that structure — the technique doesn't generalize as cleanly as the article implies.
+- **"100% accuracy" is single-test-suite marketing.** Controlled experiment on one task type. Not a generalizable claim.
+- **The LLM-in-the-loop is soft-pedaled.** They say "no black-box ML judging," but step 2 of the merge calls a multimodal LLM for semantic state equivalence. Scope-limited, but it's there.
+- **Bootstrap dependency.** Requires 2–10 *consistently successful* runs to extract dominators. For flaky tasks, getting that bootstrap is itself the hard problem.
+
+### Relevance to zero-check today
+
+Honest read: the article is not actionable for the current gauntlet. Zero-check validates deterministic checks (lint, tests, SAST, secrets, deps) where there's a single right answer per tool. Dominator analysis is overkill for "did ruff pass."
+
+Where the article *would* matter is a hypothetical second layer — validating that the agent's *overall work* met the stated intent, not just that the code passes mechanical checks. Examples:
+- "Add JWT auth to the API" — did the agent actually wire JWT into middleware, or just add a TODO comment?
+- "Refactor to dependency injection" — did the result actually use DI, or just rename files?
+
+For those questions, lint-pass is necessary but wildly insufficient. That's the territory where structural intent validation gets interesting.
+
+### If/when that second layer becomes real
+
+The article's specific technique (dominators on execution graphs) is the wrong starting point for code-level intent validation — it's the right tool for UI flows, not for diffs. Simpler approaches that fit the existing stack better:
+
+- **Spec-first.** Agent writes a one-paragraph "this is what I'm going to do" before the change. A separate validator LLM reads the diff and judges whether it matches the spec.
+- **Test-as-oracle.** Agent writes a test capturing the intent *before* writing the code; the gauntlet runs the test. Essentially harness-enforced TDD.
+- **Behavioral fixture.** For tasks with clear input/output shape, record what success looks like and check against it.
+
+All three are in the same family as the article — *external structural check on intent* — but cheaper and better matched to code rather than UI.
+
+## Key Learnings
+
+- The article's core insight (agents need independent structural validators because self-assessment is unreliable) is *the same principle* zero-check operates on. The refactor we did this session is the deterministic-layer version of what the GitHub post does at the UI layer.
+- Dominator analysis is a real technique with a clean theoretical basis (compiler control-flow theory) — worth remembering even if not applied today.
+- The two-layer mental model is useful: deterministic gauntlet (lint/test/SAST) + semantic intent validator. Today only layer 1 is built. Layer 2 is a future concern, not a current gap.
+- Press-release numbers ("100% accuracy") in research blog posts should be read as "worked on our test set" not "general capability."
+
+## Follow-ons
+
+- [ ] Revisit this article if and when harness/Lovebug starts taking on substantive code-generation tasks where mechanical checks aren't enough to know if the work was correct
+- [ ] If a semantic validation layer is ever built, evaluate spec-first and test-as-oracle approaches before reaching for graph-based techniques
+- [ ] No action on zero-check itself from this review — the in-flight refactor (manifest-driven routing, inconclusive guard) is the right shape for the deterministic layer