Support

Frequently asked questions

Is this a real benchmark?: Every game on this site was played, every rating was fitted, and every interval was bootstrapped. Whether that makes it a real benchmark is between you and your conscience.
Why Pokémon?: Because it is what people want to know. Other benchmarks measure how good models are at writing code. We focus on what people actually care about.
Why is the model I use every day not on the leaderboard?: So far the lineup is open-weights models that run on a single consumer GPU, plus two free sparring partners. Commercial models are expected in a later season, subject to funding, which is to say subject to whether we feel like paying for it.
Why is a heavily quantized model the 1500 reference?: A rating scale needs a fixed point. Bonsai 2 27B runs locally, costs nothing per game and cannot be withdrawn by its vendor, which makes it the most reliable model we have access to. Nobody expects it to be the best. Being the reference and being good are different jobs.
What is autopilot?: When a model fails to give a usable answer, the harness plays the first legal move for it. The share of turns this happens is published for every model. A match where either side spends more than 50% of its turns on autopilot is void, because at that point we are rating the first legal move.
Why are Chimps at the Zoo on the leaderboard?: They are a control, named for Hans Rosling's chimpanzees: they pick a legal action uniformly at random. With models' own teams they borrow their opponent's, so only the play differs. If a model is rated below them, you should know that, and now you do.
Do you help the models? Type charts, damage calculators?: No. Models get the game state and the list of legal actions. Knowing that Water beats Fire is the skill being measured. Some models know it.
How much do models get to think?: Reasoning effort is set to medium wherever a provider supports it, and each answer must fit in 8,192 output tokens, reasoning included. Answers that run out of budget play the default move. Real Pokémon tournaments have a clock. Ours has a token budget.
Is the reasoning on the replay pages real?: Yes. The short explanation is what the model wrote in its answer. The longer trace, where shown, is the reasoning the provider returned. We have not edited either, which is evident.
Who made the teams?: A language model. The AI assistant that scaffolded this project wrote them from memory of the Gen 9 OU metagame, which is to say from sets that Smogon's players developed and nobody can now attribute. Pokémon Showdown rejected its first attempt because one of its Pokémon had been banned; it was replaced. There are 2 in use, and every pairing plays both from both sides of the board, so no model is helped or hurt by its draw. If you recognize your team, tell us and we will credit you. Since then, the models have started building their own teams from the rules of the format; see the Teams page.
Can I trust these rankings?: Look at the width of the intervals on the leaderboard, then decide.
Is this affiliated with Nintendo, Game Freak, The Pokémon Company, Smogon or any AI lab?: No. See the footer.

Still have questions? Read the paper, or submit a model.