What are AI Benchmarks?

AI benchmarks are set lists of questions that every model is asked, so the scores can be compared like for like. They are where the numbers in every launch announcement come from. A model can top the table and still disappoint you, which is why who ran the test matters as much as the score.

You could just try a model yourself, but you would be testing whatever questions occurred to you that afternoon, with no way to tell whether it beat a rival or simply had an easy run. A benchmark is one specific list of questions with known answers, and every model is asked that identical list, which is the only reason scores can be compared at all. Most such lists were made public so anyone could check the results, and that is where the trouble starts. Models are trained on the open internet, so a model may well have read both the questions and the answers before it ever sat the test. The field calls this contamination, and it means a high score can simply mean the model recognised the paper. A second problem follows from competition itself, because once everyone chases one number, teams tune for the number rather than the skill beneath it. So the more trusted benchmarks are now held by independent organisations that keep the list private, put each model through it themselves, and release only the marks. Every model still faces identical questions, and nobody can revise in advance.

A restaurant guide's star rating. It is a real assessment, and no kitchen collects three stars by accident, but it reflects one set of judges applying one set of standards on the nights they happened to visit. Kitchens also work out what those judges reward and cook toward it, so the stars climb while the food stays exactly where it was. The rating is honest, and it still cannot tell you whether you will enjoy dinner there on a wet Tuesday.