Retracted results

Before the first benchmark we ran a pilot study of 55 matches. We have retracted all of it. This page explains why, and keeps every retracted match available so that nobody has to take our word for it.

Zero-Token Reasoning

During the pilot, 10 matches were won by language models that produced no tokens at all. Not short answers: no answers. On every turn of those games the model either failed to respond or responded with something we could not read, and our harness played its default action instead, which is the first legal move in the list.¹

For a short period this looked like an important result. A model that wins without reasoning is, from a cost perspective, the best model ever evaluated. An early draft of our paper described the effect as Zero-Token Reasoning and proposed it as a direction for future work.

The explanation was less interesting. Clicking the top button turns out to be a serviceable strategy against an opponent that is also struggling. We had measured the opponents at least as much as the models.

What changed

• A match is now void if either side spends more than 50% of its turns on autopilot. Autopilot rates are published for every model.
• Every decision records why it was made: by the model, or by a timeout, provider error, unreadable answer or illegal move.
• Every game records the exact rules it was played under, and a benchmark only rates games played under its stored rules, anchored to a model that runs on our own hardware.
• A random-move bot plays every new model first, as a control, unless the model costs more than $2 a game, on budgetary grounds. A model that cannot beat it is reported as such.

22 of the pilot matches would also have been voided under the current rules. The rest were played under a ruleset that no longer exists, and are retracted with them.