1/ CS majors are drilled to think about "worst-case" performance of algorithms. By contrast, much of the discourse on AI evals focuses on average-case or best-case (e.g. LLM X can solve IMO problems). Maybe one key to "reliability" is certifying the 1st quantile of outputs too, not just the mean.
2/ Model A may beat model B on average, but model A can still lose to model B if judged by the min. over several tasks.
I wrote a brief blog post on this (good time to announce I started a substack!).
shuvom.substack.com/p/revenge-of...
Shuvom Sadhuka
Shuvom Sadhuka
Or maybe, revenge of the 1st quantile. What common AI benchmarking discourse misses.
