The Benchmarking Illusion: How AI Labs Are Writing Their Own Rules of Victory
AI-generated image · US National Wire
Opinion: As neutral evaluation standards collapse, the shift toward proprietary and embedded evaluators transforms technical benchmarks into a corporate power play.
For years, the AI industry has operated under the guise of a meritocracy, using model benchmarking as the supposedly neutral arbiter of progress. These evaluations were meant to be the objective yardstick by which we determine if one product is superior to another. But as the technical reality sets in, it is becoming clear that these benchmarks are not a science—they are a battlefield.
In my view, we are witnessing the collapse of neutral benchmarking. When the metrics used to define 'victory' are unstable, the labs themselves effectively become the rule-makers. If you can define the test, you can define the winner.
Irene Solaiman, the chief policy officer at Hugging Face, highlighted the fragility of this system during a discussion at TechCrunch Disrupt. As TechCrunch first reported, Solaiman noted that the methods behind these evaluations are still maturing and complicated. The core issue is reproducibility. As Solaiman pointed out, the ability to trust results depends heavily on whether an evaluation can be reproduced and whether there is sufficient access to the model—a point where open weights become critical.
Without a consistent science, benchmarking becomes a tool for narrative rather than a tool for truth. Solaiman warned that these tests evolve quickly and often become ineffective, yet the industry lacks a thriving ecosystem to meaningfully compare different evaluations.
Instead of moving toward a transparent, industry-wide standard, we are seeing a pivot toward 'embedded evaluators.' This concept involves placing supposedly neutral third-party observers inside AI organizations. While labs promote this as a solution to safety concerns and a way to avoid active government regulation, the reality looks more like a proprietary power play.
Take, for example, Anthropic’s decision to hire the consulting firm Accenture as its first evaluator. As TechCrunch reports, this move sparked significant backlash from critics who questioned whether a consulting firm possesses the necessary qualifications for such a technical task. When a lab chooses its own 'neutral' observer, the observer is no longer an independent auditor; they are a curated partner.
Furthermore, the goalposts are shifting. Evaluations are moving beyond the model itself to encompass the 'harness'—the tool used to access the model—and the 'agent.' Solaiman noted that a weaker model paired with a superior harness could actually outperform a frontier model. Yet, while we have many evaluations for models, we lack them for harnesses.
This creates a dangerous vacuum. If the industry cannot agree on a uniform approach, the 'winner' of the AI race will not be the company with the most capable technology, but the company most adept at designing the harness and the evaluation criteria that favor its own architecture.
We are currently in a 'free-for-all,' as described by the context of Solaiman's remarks. Unless the industry adopts a standardized, reproducible science of evaluation, benchmarking will cease to be a technical standard and will instead become a corporate marketing tool. When the labs write the rules, the results are a foregone conclusion.