Technical report · not peer reviewed

RotomArena: Measuring What Matters in Large Language Models

The RotomArena Team
Independent. Unaffiliated, as noted in the footer.

Introduction

The field has converged on a small set of capabilities worth measuring: writing code, proving theorems, and completing multi-step tasks in a browser. These are reasonable priorities for an industry. They are not, however, what people ask about. People ask which model is best at Pokémon.

Competitive Pokémon is a useful testbed for several reasons. It has hidden information, simultaneous moves and stochastic outcomes. It rewards long-horizon planning, opponent modelling, and knowledge of roughly a thousand species, their abilities and their typical sets. It also has an active competitive community, a rigorous open-source simulator, and a clear win condition, which is more than can be said for most agentic benchmarks.

We make three contributions. First, a harness in which any chat model can play rated games without tools, plugins or retrieval. Second, a rating ladder that keeps evaluation cost roughly linear in the number of models. Third, a public record of every decision every model made, including the ones it would probably prefer we did not publish.

Related work

Games have a long history as AI benchmarks, and the rating systems used for them date back to chess [1]. Pairwise preference ratings for language models were popularized by Chatbot Arena [2], which asks humans to vote between two answers; RotomArena replaces the humans with Pokémon, which are cheaper and do not get tired. Our rating model follows Bradley and Terry [3].

Prior work on language models in Pokémon includes PokéLLMon [4] and PokéChamp [5], both of which build agents around a language model; PokéChamp places one inside a minimax search and reaches expert-level play. The PokéAgent Challenge [6] turned Pokémon into a shared competition for such agents. This benchmark is built directly on that line of work. We ask a narrower question: what a model does with the game state and nothing else. In particular, we deliberately leave out what PokéLLMon builds around its model: the retrieved knowledge of type matchups and move effects, the feedback after each turn, and the safeguard against panic switching. Knowing Pokémon, and not panicking, are part of what we measure. Our reference search bot implements PokéChamp’s design without its language model, written from the paper’s description, so that the search alone serves as a fixed yardstick; it should not be taken as representative of the original, whose results are reported in [5].

Methodology

Format

All games are Generation 9 OverUsed singles on a local Pokémon Showdown server.

Leads are not fixed: each model picks its own at team preview.

Decisions

On every turn a model receives a plain-text description of the battle: both teams as far as they are known, the events of the last three turns, the active Pokémon, and the list of legal actions. It replies with a short explanation (under 80 words) and exactly one action. We do not provide type charts, damage calculations or the opponent’s likely sets. Knowing Pokémon is the skill under test.

Reasoning effort is set to medium wherever the provider supports it, and each answer, reasoning included, must fit in 8,192 output tokens. There is no chess clock; a turn is abandoned only after 120 minutes, which is long enough for open-weights models running on one consumer GPU.

If a model gives no usable answer (it times out, the provider fails, the answer cannot be parsed, or it names an action that is not legal), the harness plays the first legal move on its behalf. We call this autopilot. Every decision records which of these happened. A match is void if either side was on autopilot for more than 50% of its turns.

Opponents and placement

New models are placed by a fixed schedule. They first play 3 games against Chimps at the Zoo, a control that picks a legal action uniformly at random, named for Hans Rosling’s chimpanzees.¹ They then play a set against each anchor (Bonsai 2 27B, a local language model, and our search bot), one set against a free sparring partner, and one set against the closest-rated language model. Further sets are scheduled between models whose rating intervals overlap the most, which is where an extra game tells us the most.

Ratings

Ratings are Bradley-Terry strengths on the Elo scale, fitted by Newton’s method with a weak prior equivalent to one drawn game. Bonsai 2 27B is fixed at 1500; every other rating is relative to it. Intervals come from 1,000 parametric bootstrap refits. A rating is provisional until the model has played 16 valid games.

Results

At the time of writing the benchmarks are still being played. Results below are provisional and will change.

Our main findings, generated from the data at build time:

1. Gemini 4 Argon currently leads the overall ranking with a rating of 2296 (95% CI 1927–2499).
2. Across 34 models, tokens spent per turn and win rate correlate at r = -0.18.
3. Ling 3.1 Flash spends the most tokens per decision: 9,210, or 6.9× the field median.
4. 10 matches in our pilot study were won by models that produced zero tokens. These results have been retracted.
5. The most common signature move is earthquake, preferred by 20 models.
6. Mistral Large 4 attempts actions that do not exist in 3.7% of its turns, for example MOVE: earthquake.

Limitations

The sample is small and the intervals are wide; we report them precisely so that nobody reads more into the ordering than it supports. Two fixed teams cover a small part of the metagame. Local models are quantized and run on a single consumer GPU, so their results describe that configuration and not the model in general. Every model sees the same prompt, which some models will like more than others. Finally, we have not shown that skill at Pokémon transfers to any other task, and we have not tried very hard.

Ethics statement

All Pokémon in this study are fictional, and fainted Pokémon recover fully between games. No human annotators were used. The electricity was real.

Reproducibility

Every match on this site links its full battle log and every decision, with the model’s explanation, its reasoning trace when the provider returns one, token counts, latency and cost. The ratings, figures and findings on this page are regenerated from those files by the same export that builds the site.

Acknowledgements

This benchmark stands on other people’s work. We thank Seth Karten, Andy Luu Nguyen and Chi Jin for PokéChamp, whose design our reference bot implements; the organizers of the PokéAgent Challenge for making Pokémon a serious research environment; Guangcong Luo and the contributors to Pokémon Showdown, which runs every battle on this site; and Haris Sahovic and the contributors to poke-env, which connects our players to it. Any mistakes in how we use their work are ours.

Appendix A. Response to Reviewer 2

The authors evaluate a single game. We evaluate the one that matters.

The confidence intervals overlap. We agree, and we have drawn them so that this is easy to see.

Why is the anchor a heavily quantized model running on a desktop? Because it is always available, costs nothing per game, and cannot be deprecated by its vendor while we sleep.

Did the authors consider that the search bot may simply be better than the language models? Yes. See Figure 1.

This is not a serious benchmark. We report bootstrap intervals for a random number generator.

Citation

@misc{rotomarena2026,
  title        = {RotomArena: Measuring What Matters in Language and Decision Models},
  author       = {{The RotomArena Team}},
  year         = {2026},
  howpublished = {\url{https://rotomarena.org/paper/}},
  note         = {Continuously updated. Not peer reviewed, in the sense that matters}
}

References

1. A. E. Elo. The Rating of Chessplayers, Past and Present. Arco, 1978.
2. W.-L. Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ICML, 2024.
3. R. A. Bradley and M. E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39, 1952.
4. S. Hu, T. Huang and L. Liu. PokéLLMon: A Human-Parity Agent for Pokémon Battles with Large Language Models. arXiv:2402.01118, 2024.
5. S. Karten, A. L. Nguyen and C. Jin. PokéChamp: an Expert-level Minimax Language Agent. ICML, 2025. arXiv:2503.04094. Code.
6. S. Karten, J. Grigsby, S. Milani, K. Vodrahalli, A. Zhang, F. Fang, Y. Zhu and C. Jin. The PokéAgent Challenge: Competitive and Long-Context Learning at Scale. NeurIPS Competition Track, 2025. Website.