wiki-vault/Sources/Dev/zero-check-refactor.md

created	path	project	status	tags	type	updated
2026-05-15	Sources/Dev	zero-check-refactor	active	python go automation claude-code	project-plan	2026-05-15

Goal

Refactor the zero-check agentic skill to eliminate silent-fail modes and reduce bash complexity. Move source-file discovery from custom bash find invocations to the native discovery engines of each underlying tool (pytest, ruff, semgrep). Add an "inconclusive" guard so an all-skipped run no longer reports green. Update the global Python project scaffold to carry the project-side boundary rules that the gauntlet stops enforcing.

Locked Decisions

• Stay on bash. The orchestrator is dumb glue calling other CLIs — exactly the job bash is built for. No rewrite to Python/Go/just/pre-commit at this stage; revisit only if the gauntlet grows into non-deterministic agent-output validation.
• Manifest-driven routing. Bash detects ecosystem via pyproject.toml, requirements*.txt, uv.lock, go.mod. Bash does not enumerate source files.
• Native discovery. ruff, semgrep, pytest, go test all walk their own trees and respect project-level exclude config.
• Pytest exit code 5 → skipped (exit 2). "No tests collected" is an honest skip, not a vacuous pass.
• Inconclusive guard. A run where every check skipped is overall: failed with inconclusive: true in results.json.
• Boundary rules live in the project. Non-standard venv names (env/, app_env/) and other project-specific exclusions belong in pyproject.toml [tool.pytest.ini_options] norecursedirs, not in the gauntlet.
• Three-state exit code contract preserved. 0 = passed, 1 = failed, 2 = skipped. Sub-scripts continue to honor this; orchestrator continues to route on it.

Open Items

• Decide whether to add a JSON-schema validator for results.json (defensive, but probably overkill at current scope)
• Decide whether to add a --strict flag to run-all.sh that fails on any skip, not just all-skip (probably not — defeats the point of skipped-as-a-state)
• Smoke-test the refactor against at least one Python project, one Go project, and one polyglot/mixed project before adopting as default

Phases

Phase 1 — Drop in the refactored scripts

Replace the six scripts in ~/zero-check-pipeline/validate/ with the refactored versions drafted in chat. Files:

• run-all.sh — adds inconclusive guard, adds inconclusive field to results.json
• secrets.sh — unchanged (gitleaks already does its own tree walk)
• lint.sh — manifest-routed, source-file probe removed
• sast.sh — manifest-routed, source-file probe removed
• deps.sh — minor normalization (find pattern → -print -quit for consistency)
• tests.sh — manifest-routed, pytest exit 5 mapped to skipped, no source-file probe

Tasks:

• Back up current scripts: cp -r ~/zero-check-pipeline/validate ~/zero-check-pipeline/validate.bak
• Drop in the six new scripts from the chat artifacts
• chmod +x ~/zero-check-pipeline/validate/*.sh
• bash -n syntax-check each (already verified clean in draft)

Phase 2 — Smoke tests

Validate the refactor against real projects before declaring it the default:

• Run against herbylab MCP server (Python + uv): expect lint/sast/deps/tests all pass, secrets clean
• Run against a Go project (pick one from gitea): expect Go branches exercise
• Run against an empty scratch dir: expect inconclusive: true, overall failed
• Run against a Python project with no tests yet: expect tests → skipped, not green
• Confirm results.json shape is unchanged for downstream consumers (just gains the inconclusive field)

Phase 3 — Global Python scaffold update

Move the project-side boundary rules into the default pyproject.toml template so every new Python project carries them.

• Locate the current Python scaffold (uv init template / cookiecutter / personal snippets — wherever new projects start from)

• Add this block to the canonical scaffold:

  [tool.pytest.ini_options]
  norecursedirs = [
      "env",
      "app_env",
      ".venv",
      "venv",
      "node_modules",
      "build",
      "dist",
      ".git",
  ]
  

• Document the rationale inline (one-line comment: "Excludes for tools that walk the tree; keep gauntlet agnostic")

• If multiple scaffolds exist (e.g., one for FastMCP servers, one for plain libraries), update all of them

• Backfill the snippet into existing active Python projects on the tower as opportunity arises — not a forced migration

Phase 4 — SKILL.md hygiene

The skill file at ~/.claude/skills/zero-check/SKILL.md does not need to change — the contract (run-all.sh → results.json → retry loop) is preserved. But it's worth a once-over:

• Verify the retry-loop guidance still makes sense with the inconclusive state (an inconclusive failure should not trigger a fix loop — there's nothing to fix; it should escalate to the human)
• Consider adding: "If inconclusive: true in results.json, STOP and ask the user — do not retry." Inconclusive means the gauntlet did not validate; retrying won't change that.

Notes

What changed and why

The original gauntlet had a class of silent failures rooted in bash trying to mimic the discovery engines of the tools it orchestrates. Two specific failure modes:

1. SIGPIPE under pipefail. find ... | head -1 on dense trees could die with exit 141, killing the script before any output. The -print -quit workaround was already in place but only on some scripts.
2. Vacuous passes. A project with no .py files exited 0 from lint/sast/tests, contributing to a green overall result even though nothing was checked.

The refactor addresses both by removing the bash-side file walking entirely (mode 1 cannot occur on a manifest lookup) and by adding the inconclusive guard (mode 2 is now flagged).

Why not rewrite in Python

Considered and rejected for now. The gauntlet's current job is shelling out to other CLI tools and aggregating exit codes — bash is genuinely correct for this. A Python rewrite would buy real value only if the gauntlet evolves into something more sophisticated (e.g., LLM-driven semantic validation of agent output, per the GitHub article on agentic behavior validation). Revisit then.

Reference: refactored scripts

The six refactored scripts were drafted in conversation and produced as downloadable artifacts. Drop-in replacements for the files in ~/zero-check-pipeline/validate/. The diff against the originals is:

• All five sub-scripts: source-file find calls replaced with manifest detection
• tests.sh: pytest exit code 5 explicitly mapped to skipped state
• lint.sh, tests.sh (Go branch): cd $gomod_dir && <tool> ./... pattern adopted
• run-all.sh: inconclusive guard added after the check loop; inconclusive field added to results.json schema

Reference: GitHub article (parking lot)

Pending review: https://github.blog/ai-and-ml/generative-ai/validating-agentic-behavior-when-correct-isnt-deterministic/. Relevant to the question of whether the gauntlet should grow beyond deterministic checks. Decision deferred to a separate session.