Methodology changes

Changelog

A change to a benchmark’s rules starts a new version of it, and games of different versions are never rated together without saying so. Everything else is listed here too, so that nobody can say we changed the rules quietly.

Gen 9 OU · own teams, version s3 · October 2026

• Drafting in two steps. Models first pick six Pokémon from the legal list, then get the abilities and moves those six can legally have and build their sets. In the first version, most rejected teams were Pokémon given moves they cannot learn or abilities they cannot have (117 of 168 problems), which measured memory of movepools more than teambuilding. The legal options are rules, like the legal moves listed in a battle.
• Every version’s rules are stored and every game is checked against them, so a benchmark only ever rates games played under the same rules. The earlier one-step version stays on the site as “own teams (v1)” and its games count in Overall.
• Faster pairs. Both games of a pair run at once, each model’s requests are answered together, and the anchor’s games come first so every model is rated as early as possible.

Gen 9 OU · own teams, version s2 (v1, retired) · October 2026

• Teams. Every model builds its own team from the rules of the format: the clauses, the bans and the list of legal Pokémon, read from Pokémon Showdown. No sample teams, usage statistics or advice. Showdown’s validator is the referee, and an illegal team goes back to the model with its exact messages, up to five times.
• Repairs. A model that never builds a legal team still plays, with its last team repaired: only what the validator rejected is replaced, by unremarkable choices. This was judged fairer than both disqualification and being handed a good team for failing.
• Schedule. Every language model plays every other one, once from each side, in balanced rounds. The bots sit this one out.
• Rankings. Games are played once; rankings are views over them. Overall pools every benchmark, and each benchmark also has its own board. Each version’s exact rules are stored, and games are checked against them.
• Gen 9 OU · fixed teams is paused partway through placement and will be finished later. Its games count.

Gen 9 OU · fixed teams (version s1) · October 2026

• Ratings. Bradley-Terry ratings with bootstrap intervals replace the pilot’s Elo, anchored to a local open-weights model at 1500. The anchor runs on our own hardware and is therefore not at risk of deprecation.
• Placement. New models play a fixed placement schedule: a random-move control, both anchors, a sparring partner, and the closest-rated model. Expensive models skip the control, for reasons we describe as methodological.
• Lineup. The first lineup is limited to open-weights models that run on one consumer GPU, plus two free sparring partners. This keeps the benchmark accessible, and also cheap.
• Leads. Models now choose their own lead at team preview. Previously the first Pokémon on the team led, whatever the model thought of it.
• Reasoning. Models reason at medium effort with a fixed output budget per turn. The previous turn timer and time bank were designed for live commentary and have been removed; the only clock left is a hang guard.
• Transparency. Every decision records whether the model made it, and if not, why. The autopilot rate is published for every model, and matches with too much autopilot are void.
• Bots. Fixed a bug in how the reference bots evaluated switching. We would like the old behaviour to be remembered as a bug rather than a strategy.

Pilot (version v0) · August–September 2026

• Ran exploratory matches between commercial models, with a tournament mode, and live text-to-speech commentary in six languages.
• Discovered Zero-Token Reasoning. Retracted.
• Retired the tournament, which had never been run, and the commentary, which had.