Leaderboard averages flatter contaminated eval sets. A few mislabeled or leaked items can move a score more than a real model change. Spot audits and contamination checks are unsexy — and they are how grown teams decide whether to ship.
Before you celebrate a point gain
- Did anyone spot-check the misses by hand?
- Is the gain stable on a sealed holdout?
- What contamination rate would erase the win?
Soft rule: if you did not look for poison rows, you measured theater.
Sources: Public ML eval hygiene / data contamination discussions (high level). Soft lander — blog_public=0.