Establish eval.yaml lifecycle and CI integration for skill validation

[PureWeen]

Summary

Based on a comprehensive 7-round evaluation of the try-fix skill (PR #34807), we've established best practices for how eval.yaml files should evolve over time. This issue tracks the work to formalize and implement these practices across all 15 skills.

Current State

• 3 of 15 skills have eval.yaml files (try-fix being the 3rd via PR Improve try-fix skill: add eval.yaml and fix prompt issues #34807)
• No CI integration for skill validation
• No documented lifecycle for eval maintenance

Proposed Eval Lifecycle

CI Strategy (Tiered)

Trigger	Tool	Time	Purpose
Every PR touching .github/skills/	skill-validator check	0.1s	Static analysis gate
Weekly scheduled	evaluate --runs 1	~22 min	Behavioral drift detection
Before skill merge	evaluate --runs 3-5	~90 min	Statistical confidence

Eval Update Triggers

• SKILL.md changes → mandatory eval review
• Production failure → add regression scenario from real prompt
• New anti-pattern discovered → add output_not_contains assertion
• Quarterly → audit all evals for drift
• Model upgrade → recalibrate baselines

Eval Writing Best Practices (from 7-round try-fix cycle)

1. Start with 0 output_contains -- vocabulary assertions overfit immediately (scored 0.52)
2. Use output_not_contains for anti-patterns -- more stable than expected vocabulary
3. Use rubric items for quality -- LLM-judge criteria measure improvement, not compliance
4. Vary prompt structure -- uniform formats train eval to test template-following
5. Front-load critical constraints -- prompt order = execution order
6. Negative triggers must be unambiguously off-topic -- "review this fix" is too close to "fix this"
7. Version with changelog header -- lightweight tracking of eval evolution

Streamlined Evaluation Protocol (max 4 rounds)

Round 1: All evaluators assess fresh → identify issues
Round 2: Fixes applied → delta review only
Round 3: Final consensus → KEEP/IMPROVE/REMOVE
Round 4: (if needed) Address IMPROVE items → final verdict

Work Items

• Add skill-validator check to CI for PRs touching .github/skills/
• Create eval.yaml for pr-review skill (highest priority -- orchestrator)
• Create eval.yaml for write-ui-tests skill
• Create eval.yaml for pr-finalize skill
• Document eval lifecycle in .github/skills/README.md
• Request --internal-only flag from dotnet/skills team for orchestrator skills
• Set up weekly scheduled eval run

Tooling Improvements (for dotnet/skills team)

Based on running skill-validator 9+ times during the try-fix evaluation:

1. --internal-only flag -- internal skills get penalized for plugin non-activation (-3.7% overall vs +51.7% isolated)
2. Static eval.yaml validation in check -- catch spec-conformance issues early (e.g., skill name in prompts)
3. Timeout progress indicators -- distinguish "still working" from "stuck in loop"
4. Overfitting context for infrastructure skills -- technique items ARE the intended behavior
5. Results directory auto-naming -- --results-dir auto with --latest flag

Empirical Data (try-fix evaluation)

Metric	Value
Isolated improvement	+51.7%
Plugin improvement	+1.3% (not activated -- by design)
Overfitting	0.40 (Moderate)
Scenarios	6
Evaluation rounds	7 (target: 4 max)

Related

• PR Improve try-fix skill: add eval.yaml and fix prompt issues #34807 -- try-fix skill improvements + eval.yaml
• Epic [Epic] Improve on the agent story inside the .NET MAUI Repository #32454 -- Improve on the agent story inside the .NET MAUI repo

[PureWeen]

Eval Improvement Workflow (Established from try-fix + verify-tests-fail-without-fix cycles)

The Steady-State Loop

Production failure → regression scenario → SKILL.md fix → ship (no multi-round review)
SKILL.md change → eval review in same PR → ship  
Monthly → mine production PRs via labels → add scenarios
Quarterly → re-run all evals, check overfitting drift

CI Integration (Ready Today)

Every PR touching .github/skills/:

- run: |
    gh release download skill-validator-nightly --repo dotnet/skills --pattern "skill-validator-linux-x64.tar.gz"
    for skill in .github/skills/*/; do /tmp/skill-validator check --skills "$skill" --verbose; done

Cost: 0.1s per skill, no LLM tokens, no auth. AOT-compiled 20MB binary.

Weekly scheduled: skill-validator evaluate --runs 1 on all skills (~22 min total)
On-demand: evaluate --runs 3-5 for statistical confidence before merges

New Skill Eval Template (5 scenarios minimum)

1. Happy path (positive trigger)
2. Negative trigger (expect_activation: false)
3. Semantic regression (#1 mistake the skill prevents)
4. Edge case (missing inputs, wrong mode)
5. Anti-pattern guard (banned tool/command usage)
--- after production mining ---
6+ Production-derived regression scenarios

Assertion ratio target: 1:3 (positive:negative)

Production Data Mining

Best method per skill type:

Skill Type	Method
Skills with PR labels	gh pr list --label "<label>" --state all (fastest, most reliable)
Skills with output dirs	find CustomAgentLogsTmp -path "*<skill>*"
Skills without labels	PR comments + session store SQL

Mining cadence: Monthly for skills with ≥5 production runs

Eval Quality Metrics

Metric	Target	Check
Scenario count	≥6 per skill	On PR
Overfitting	0.20-0.45	Weekly
Assertion/rubric conflicts	0	On PR
Negative test coverage	≥1 expect_activation: false	On PR
Rubric:assertion ratio	≥1:1	Quarterly

Scaling Plan (13 remaining skills)

Tier	Skills	Approach
✅ Done	try-fix, verify-tests-fail-without-fix	Shipped (PRs #34807, #34815)
Existing	code-review, evaluate-pr-tests	Review existing eval.yaml
Tier 1	pr-review, write-ui-tests	Full 4-round protocol
Tier 2	pr-finalize, azdo-build-investigator, write-xaml-tests	2-3 rounds
Tier 3	run-device-tests, run-helix-tests, run-integration-tests, find-reviewable-pr, issue-triage	Batch parallel, 1 round each
Tier 4	learn-from-pr	1 round

Tool Wishlist (for dotnet/skills team)

Priority	Tool	Saves
🔴 P0	--eval-lint (assertion/rubric conflict detection)	2-3 review rounds per skill
🟡 P1	--production-mine <label>	Hours of manual data mining
🟡 P1	Overfitting per-scenario breakdown	Pinpoints problematic scenarios
🟡 P1	--eval-diff (run only changed scenarios)	Faster incremental iteration
🟢 P2	--init-eval <skill> (template generator)	Faster initial eval creation

Protocol for Eval Cycles

Initial (new skill): Max 4 rounds

Round 1: 3 workers evaluate fresh → flag everything
Round 2: Fixes applied → delta review only  
Round 3: Final consensus → KEEP/IMPROVE
Round 4: (only if IMPROVE) → last fixes

Steady-state (incremental): Single-scenario additions ship without multi-round review.
Quarterly: Re-run all evals, audit metrics, retire stale scenarios.