mcp: project-plan — ML/AI Drills Curriculum

Sources/Dev/ml-drills-curriculum.md

@@ -0,0 +1,134 @@
+---
+created: '2026-05-21'
+path: Sources/Dev
+project: ml-drills-curriculum
+tags:
+- python
+- lovebug
+type: project-plan
+---
+
+## Goal
+
+Internalize the ML/AI workflow decision tree through repetition, not project delivery.
+
+Travis has strong front-end framing (decision analysis, value-focused thinking) and strong back-end interpretation skills, but lacks the middle-of-workflow muscle memory: the inner loop of baseline → split → model → hypothesis → iterate → honest comparison. The analogy is git — after a single course, the operations are known but the workflow isn't yet automatic. The fix is reps on the motion, not more projects.
+
+Concretely, "done" looks like:
+- Baseline-first thinking is automatic (no skipping straight to a "real" model)
+- Hypothesis-before-code is a habit (written hypothesis precedes each iteration)
+- Train/val/test discipline is reflexive (never peek at test, never tune on test)
+- Pattern recognition for common failure modes (leakage, overfitting, bad splits, suspicious metrics)
+- Honest comparison and uncertainty (not over-claiming on noise)
+
+These are habits of mind, not deliverables. The curriculum's success is measured by whether the inner loop runs without conscious scaffolding, not by what gets built.
+
+## Locked Decisions
+
+- **Stack:** Python, Jupyter on the tower (RTX 3080, fast iteration). Possible R drills deferred to later phase.
+- **Repo:** `ml-drills` on Gitea (`gitea.plantbasedsoutherner.com`). One notebook per drill plus the walkthrough markdown alongside.
+- **Format:** Tutorial-style walkthroughs. Agent narrates decision points explicitly, names the forks, names the alternatives, names why each choice. Travis types the code (not copy-paste — the typing is part of the rep).
+- **Calibration mechanism:** Each drill anchors to a canonical public solution. Reasoning is explained alongside the motion. Phase 1 is option-1-heavy (reference points provided up front) per Travis's preference: "reps first, intuition comes later."
+- **Order:** Classical ML drills first. LLM/RAG work transitions in as a later phase, once the inner-loop habits transfer.
+- **Drill 1:** Titanic (binary classification). Drill 2: Ames Housing (regression). Different problem shapes early to broaden pattern exposure.
+- **Drill count:** Open-ended. Drills 1-5 defined below to start; more added as the curriculum reveals gaps in Travis's decision tree.
+- **Cadence:** As-time-allows. No schedule. Each drill is a self-contained artifact Travis can return to when ready.
+- **Phase transitions:** By feel, not by drill count. Travis (with agent input) decides when scaffolding can come off.
+
+## Open Items
+
+- [ ] Define drill 3-5 datasets and decision focus (will emerge from drill 1-2 execution)
+- [ ] Decide criteria for Phase 1 → Phase 2 transition (likely: when the agent's walkthrough narration starts feeling redundant)
+- [ ] Decide when LLM/RAG drills enter the curriculum (Phase 2 or Phase 3)
+- [ ] Decide whether R drills get added later, and at what point
+- [ ] Decide artifact strategy for the repo: notebook commits as they're built, or post-completion only
+
+## Phases
+
+### Phase 1 — Tutorial walkthroughs (heavy scaffolding)
+
+Agent writes a full walkthrough document for each drill. Travis works through it in his notebook, narration is explicit at every decision point, canonical solution is referenced up front with reasoning. Each drill ends with a "Now You" extension where Travis applies the same motion to a small variation, completed async.
+
+**Drill catalog (drills 1-5 to start; more added as curriculum evolves):**
+
+**Drill 1 — The Full Loop, Narrated**
+- Dataset: Titanic (binary classification)
+- Decision being internalized: the entire inner loop end-to-end, narrated out loud
+- Walkthrough covers: framing the question, stupid baseline, train/val/test split, first real model, hypothesis-driven iteration (one cycle), honest comparison
+- Now You: rerun the loop with a different baseline or model choice on the same dataset; write hypotheses before each change
+
+**Drill 2 — Same Loop, Different Shape**
+- Dataset: Ames Housing (regression)
+- Decision being internalized: the loop transfers across problem types; regression vs. classification doesn't change the workflow
+- Walkthrough covers: same five steps, but with regression metrics, regression baselines (predict the mean), and feature engineering that's richer than Titanic
+- Now You: pick a different baseline strategy and justify it; iterate one cycle
+
+**Drill 3 — Decision Points Isolated**
+- Dataset: TBD (something with a known leakage trap or class imbalance issue)
+- Decision being internalized: pattern recognition at the forks — when do you trust a result, when do you suspect leakage, when is a gain real
+- Walkthrough covers: the same loop, but with deliberate slow-downs at each decision point; the agent explicitly says "this is a fork, here are the candidates, here's why we go left"
+
+**Drill 4 — Diagnose a Broken Notebook**
+- Dataset: TBD; agent provides a notebook with a deliberate failure (leakage, overfitting, bad split, broken metric, distribution mismatch)
+- Decision being internalized: pattern recognition for "this looks wrong"
+- Walkthrough covers: the failure modes one at a time, with diagnostics; Travis finds the bug rather than building from scratch
+
+**Drill 5 — Honest Comparison**
+- Dataset: TBD (likely revisit a prior drill's dataset)
+- Decision being internalized: judging two approaches without over-claiming on noise
+- Walkthrough covers: two model families or two feature sets on the same problem; how to compare, how to express uncertainty, what would change your mind
+
+### Phase 2 — Lighter scaffolding
+
+Travis frames the problem and drives the workflow. Agent reviews choices and asks senior-level questions ("why that baseline? what would make you reject this approach?") rather than narrating up front. Likely transitions classical ML → LLM-based work here, or in Phase 3.
+
+### Phase 3 — Self-directed
+
+Travis owns a real problem end-to-end (homelab, PBS, or portfolio). Agent is a sounding board, not a guide. Stakes are real; user is real (could be Travis, could be Jenny, could be a portfolio audience). Messy front-end framing and back-end deployment are explicitly in scope at this phase.
+
+## Notes
+
+### Agent execution protocol
+
+When conducting a drill session:
+
+1. **Confirm context.** Which drill? Which dataset? Where in the arc is Travis? Has the prior drill's Now You been completed? If unclear, ask before proceeding.
+
+2. **Deliver the walkthrough as a markdown artifact.** Either inline (if short) or as a `.md` file dropped into the `ml-drills` repo alongside the notebook. The walkthrough is the durable artifact — Travis should be able to return to it months later and re-run the motion.
+
+3. **Narrate decision points explicitly.** At each fork in the loop:
+   - Name the fork ("we now need to choose a baseline")
+   - Name the candidates ("predict the majority class, predict random, predict by a single feature")
+   - Name the choice and the reason ("we'll predict majority class because it's the most conservative starting point and the dataset is moderately imbalanced")
+   - Name what could change the choice ("if the dataset were highly imbalanced, we'd want a stratified baseline instead")
+
+4. **Anchor to a canonical solution.** Reference a specific public Kaggle notebook or tutorial up front. Explain the choices that solution made and why. Travis is option-1-heavy at Phase 1 — give him the reference point, don't make him search.
+
+5. **Show code with explanation, Travis types it.** Code blocks in the walkthrough are reference, not copy-paste targets. Typing is part of the rep.
+
+6. **End with a "Now You" extension.** Small variation on the drill that Travis completes async. Not a different problem — same motion, slightly different surface details. This is where the rep without scaffolding happens.
+
+7. **Review the Now You when Travis returns.** Compare his choices to common practice in 2-3 different public solutions. The review is comparative, not corrective: "here's where your decisions matched common practice, here's where you diverged, here's whether the divergence was reasonable." The gap between Travis's decision tree and the references is the curriculum.
+
+8. **Capture follow-ons.** If a drill surfaces a gap that should become its own drill, note it. The curriculum is open-ended on purpose.
+
+### Conceptual framing
+
+- **This is rep-and-set on decision trees, not project delivery.** The artifacts (notebooks, walkthroughs) are byproducts. The internalization is the product. Don't optimize for portfolio polish during Phase 1.
+- **Familiarity first, speed later.** No time pressure during Phase 1. Slowness is appropriate while motions are being formed. Speed emerges once the choices stop requiring conscious effort.
+- **The git analogy holds.** Travis didn't need a thousand repos to internalize git's workflow; he needed focused reps on the operations. Same here.
+- **Classical ML first is deliberate.** Loops are faster, signal is cleaner, failure modes are legible (leakage, overfitting, distribution shift all have names). LLM iteration has the same shape but noisier feedback; harder to learn the motion on. Classical first builds the substrate for LLM work later.
+- **"Models are search, not selection."** This is the key transfer from classical ML to LLM/RAG work. Both are iterative searches through a space, not one-shot choices defended on theoretical grounds. Building this intuition on classical ML makes prompt/RAG iteration feel like the same activity later.
+
+### Why this curriculum exists (context for future Travis or any new agent)
+
+Travis has taken AI courses (theory and business-applied) but never worked an AI project. The consequence is that he can talk about AI projects fluently but doesn't have the muscle memory for the inner loop. His decision-analysis background gives him strong front-end framing intuition; the middle-of-workflow rhythm is what's missing. The Kaggle moment: data prepped, "now what?" — that's the gap this curriculum closes.
+
+The framing of "drills, not projects" came out of a conversation where Travis pushed back on a three-phase project-scope progression and reframed it explicitly as workflow internalization. The git analogy was his.
+
+### References (to be populated as drills are authored)
+
+- Drill 1 canonical solution: TBD (will be selected and linked when Drill 1 walkthrough is authored)
+- Drill 2 canonical solution: TBD
+- Future Proof Data Science Jan 2026 job market breakdown (originating context for the curriculum)
+- KDnuggets "Hidden Skill Gap" article by Nate Rosidi, May 2026 (stepping-off point)