57 Evaluation Field Guides & Technical Standards
A comprehensive SEO knowledge repository documenting empirical statistical methods, reliability curves, split discipline, and sample-size governance for foundation models and multi-agent harnesses.
⚡ Visual Metric Decoder (Hyperlinked Index)
Every metric connects directly to its live evaluation leaderboard and telemetry tool:
01 // DYNO SCORE (0-100)
Empirical Harness Standard
6 ground-truth axes; 74/100 is clean pass →
02 // SWE-BENCH VERIFIED
GitHub Issue Resolution
Deterministic bug fixes passing unit tests →
03 // ORTHOGONAL POROSITY (V)
Wingman Error Rescue
Models that fail on different bugs cover each other →
04 // OPERATIONAL SCARS
Failure Recovery SOPs
FENCE, MOUTH_FORM, REASON_400 mitigation →
Statistical Calibration & Curves (9)
- A Brier Score Needs A Reliability Diagram →
- A Calibration Plot Needs A Binning Scheme →
- A Confusion Matrix Needs A Class Balance →
- A Log Loss Needs A Calibration Curve →
- A Roc Curve Needs An Operating Point →
- An Ece Score Needs A Reliability Bin →
- Calibration Notes Should Name The Exception →
- Calibration Starts With Shared Examples →
- Confidence Without Calibration →
Evaluation Foundations & Splits (10)
- A Baseline Needs A Version Label →
- A Benchmark Needs A Baseline Window →
- A Benchmark Needs A Held Out Split →
- A Dataset Card Needs A License Field →
- A Kappa Score Needs A Chance Baseline →
- A Leaderboard Needs A Sample Size Note →
- A Prompt Suite Needs A Version Pin →
- Baseline Before Benchmark →
- Sample Size Before You Brag →
- Version Pin Your Evals →
Leaderboard Metrics & Scores (7)
Rubrics & Benchmark Standards (31)
- A Good Example Makes The Rule Portable →
- A Good Rubric Names The Boundary Case →
- A Metric Card Needs A Confidence Interval →
- A Precision Recall Curve Needs A Threshold →
- A Rating Needs An Evidence Window →
- A Regression Set Needs A Freeze Date →
- A Rubric Change Needs A Changelog →
- A Rubric Needs A Tie Break Rule →
- A Rubric Needs Examples →
- A Score Needs A Disconfirming Example →
- A Scorecard Needs A Confidence Band →
- A Scorecard Needs An Abstain Rule →
- Agreement Needs A Disagreement Log →
- An Ablation Needs A Control Arm →
- An Auc Score Needs A Prevalence Note →
- An Eval Harness Needs A Seed Lock →
- Buyer Eval Checklist →
- Contamination Checks Beat Averages →
- Edge Cases Belong In The Rubric →
- Evaluation Notes Are Data →
- Examples Make A Rubric Operational →
- Examples Should Explain The Boundary →
- Holdout Sets Are Sacred →
- Reviewer Agreement Needs A Rule →
- Rubrics Need Examples →
- Run A Blind Sample First →
- Score The Failure Modes →
- Soft Lander Home →
- The Example Should Mark The Edge →
- The Rubric Should Show What Changes →
- The Rubric Should Survive The Surprise →