mcp: project-plan — zero-check-refactor

Sources/Dev/zero-check-refactor.md

@@ -0,0 +1,122 @@
+---
+created: '2026-05-15'
+path: Sources/Dev
+project: zero-check-refactor
+status: active
+tags:
+- python
+- go
+- automation
+- claude-code
+type: project-plan
+updated: '2026-05-15'
+---
+
+## Goal
+
+Refactor the `zero-check` agentic skill to eliminate silent-fail modes and reduce bash complexity. Move source-file discovery from custom bash `find` invocations to the native discovery engines of each underlying tool (pytest, ruff, semgrep). Add an "inconclusive" guard so an all-skipped run no longer reports green. Update the global Python project scaffold to carry the project-side boundary rules that the gauntlet stops enforcing.
+
+## Locked Decisions
+
+- **Stay on bash.** The orchestrator is dumb glue calling other CLIs — exactly the job bash is built for. No rewrite to Python/Go/just/pre-commit at this stage; revisit only if the gauntlet grows into non-deterministic agent-output validation.
+- **Manifest-driven routing.** Bash detects ecosystem via `pyproject.toml`, `requirements*.txt`, `uv.lock`, `go.mod`. Bash does not enumerate source files.
+- **Native discovery.** ruff, semgrep, pytest, go test all walk their own trees and respect project-level exclude config.
+- **Pytest exit code 5 → skipped (exit 2).** "No tests collected" is an honest skip, not a vacuous pass.
+- **Inconclusive guard.** A run where every check skipped is `overall: failed` with `inconclusive: true` in `results.json`.
+- **Boundary rules live in the project.** Non-standard venv names (`env/`, `app_env/`) and other project-specific exclusions belong in `pyproject.toml [tool.pytest.ini_options] norecursedirs`, not in the gauntlet.
+- **Three-state exit code contract preserved.** `0 = passed`, `1 = failed`, `2 = skipped`. Sub-scripts continue to honor this; orchestrator continues to route on it.
+
+## Open Items
+
+- [ ] Decide whether to add a JSON-schema validator for `results.json` (defensive, but probably overkill at current scope)
+- [ ] Decide whether to add a `--strict` flag to `run-all.sh` that fails on any skip, not just all-skip (probably not — defeats the point of skipped-as-a-state)
+- [ ] Smoke-test the refactor against at least one Python project, one Go project, and one polyglot/mixed project before adopting as default
+
+## Phases
+
+### Phase 1 — Drop in the refactored scripts
+
+Replace the six scripts in `~/zero-check-pipeline/validate/` with the refactored versions drafted in chat. Files:
+
+- `run-all.sh` — adds inconclusive guard, adds `inconclusive` field to `results.json`
+- `secrets.sh` — unchanged (gitleaks already does its own tree walk)
+- `lint.sh` — manifest-routed, source-file probe removed
+- `sast.sh` — manifest-routed, source-file probe removed
+- `deps.sh` — minor normalization (find pattern → `-print -quit` for consistency)
+- `tests.sh` — manifest-routed, pytest exit 5 mapped to skipped, no source-file probe
+
+Tasks:
+- [ ] Back up current scripts: `cp -r ~/zero-check-pipeline/validate ~/zero-check-pipeline/validate.bak`
+- [ ] Drop in the six new scripts from the chat artifacts
+- [ ] `chmod +x ~/zero-check-pipeline/validate/*.sh`
+- [ ] `bash -n` syntax-check each (already verified clean in draft)
+
+### Phase 2 — Smoke tests
+
+Validate the refactor against real projects before declaring it the default:
+
+- [ ] Run against herbylab MCP server (Python + uv): expect lint/sast/deps/tests all pass, secrets clean
+- [ ] Run against a Go project (pick one from gitea): expect Go branches exercise
+- [ ] Run against an empty scratch dir: expect `inconclusive: true`, overall failed
+- [ ] Run against a Python project with no tests yet: expect `tests` → skipped, not green
+- [ ] Confirm `results.json` shape is unchanged for downstream consumers (just gains the `inconclusive` field)
+
+### Phase 3 — Global Python scaffold update
+
+Move the project-side boundary rules into the default `pyproject.toml` template so every new Python project carries them.
+
+- [ ] Locate the current Python scaffold (uv init template / cookiecutter / personal snippets — wherever new projects start from)
+- [ ] Add this block to the canonical scaffold:
+
+  ```toml
+  [tool.pytest.ini_options]
+  norecursedirs = [
+      "env",
+      "app_env",
+      ".venv",
+      "venv",
+      "node_modules",
+      "build",
+      "dist",
+      ".git",
+  ]
+  ```
+
+- [ ] Document the rationale inline (one-line comment: "Excludes for tools that walk the tree; keep gauntlet agnostic")
+- [ ] If multiple scaffolds exist (e.g., one for FastMCP servers, one for plain libraries), update all of them
+- [ ] Backfill the snippet into existing active Python projects on the tower as opportunity arises — not a forced migration
+
+### Phase 4 — SKILL.md hygiene
+
+The skill file at `~/.claude/skills/zero-check/SKILL.md` does not need to change — the contract (run-all.sh → results.json → retry loop) is preserved. But it's worth a once-over:
+
+- [ ] Verify the retry-loop guidance still makes sense with the inconclusive state (an inconclusive failure should not trigger a fix loop — there's nothing to fix; it should escalate to the human)
+- [ ] Consider adding: "If `inconclusive: true` in results.json, STOP and ask the user — do not retry." Inconclusive means the gauntlet did not validate; retrying won't change that.
+
+## Notes
+
+### What changed and why
+
+The original gauntlet had a class of silent failures rooted in bash trying to mimic the discovery engines of the tools it orchestrates. Two specific failure modes:
+
+1. **SIGPIPE under `pipefail`.** `find ... | head -1` on dense trees could die with exit 141, killing the script before any output. The `-print -quit` workaround was already in place but only on some scripts.
+2. **Vacuous passes.** A project with no `.py` files exited 0 from lint/sast/tests, contributing to a green overall result even though nothing was checked.
+
+The refactor addresses both by removing the bash-side file walking entirely (mode 1 cannot occur on a manifest lookup) and by adding the inconclusive guard (mode 2 is now flagged).
+
+### Why not rewrite in Python
+
+Considered and rejected for now. The gauntlet's current job is shelling out to other CLI tools and aggregating exit codes — bash is genuinely correct for this. A Python rewrite would buy real value only if the gauntlet evolves into something more sophisticated (e.g., LLM-driven semantic validation of agent output, per the GitHub article on agentic behavior validation). Revisit then.
+
+### Reference: refactored scripts
+
+The six refactored scripts were drafted in conversation and produced as downloadable artifacts. Drop-in replacements for the files in `~/zero-check-pipeline/validate/`. The diff against the originals is:
+
+- All five sub-scripts: source-file `find` calls replaced with manifest detection
+- `tests.sh`: pytest exit code 5 explicitly mapped to skipped state
+- `lint.sh`, `tests.sh` (Go branch): `cd $gomod_dir && <tool> ./...` pattern adopted
+- `run-all.sh`: inconclusive guard added after the check loop; `inconclusive` field added to `results.json` schema
+
+### Reference: GitHub article (parking lot)
+
+Pending review: <https://github.blog/ai-and-ml/generative-ai/validating-agentic-behavior-when-correct-isnt-deterministic/>. Relevant to the question of whether the gauntlet should grow beyond deterministic checks. Decision deferred to a separate session.