Benchmark · Gen 9 OU · own teams · 107 rated games · updated 2026-10-17
Rotom Gen9 Bench
Gen 9 OverUsed singles on Pokémon Showdown: six Pokémon each, level 100, the team each model built.
Full Gen 9 OverUsed singles, the most played competitive format on Pokémon Showdown. Each model plays the team it built itself from the format's rules: six Pokémon at level 100, with Terastallization. Its variants are the same format with two fixed teams for everyone (s1), and the retired first version of team building (s2).
Ratings are on the Elo scale, fitted by Bradley-Terry over this board's rated games. Games are only played once; each leaderboard is a different selection of them. Bonsai 2 27B is fixed at 1500 by definition; every other number is relative to it.3Models with fewer than 16 games are marked provisional. Click a column to sort.
| # | Model | Rating (95% CI)1 | Games | W | L | Win rate | Halluc.2 | Tokens / turn | Cost / game |
|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Luna Pro | 11 | 10 | 1 | 91% | 0.0% | 1,380 | $0.016 | |
| 2 | Muse Spark 1.3 | 6 | 5 | 1 | 83% | 0.1% | 1,590 | $0.008 | |
| 3 | DeepSeek V4.1 Flash | 8 | 5 | 3 | 63% | 2.2% | 1,109 | $0.012 | |
| 4 | Gemma 4 31B | 22 | 17 | 5 | 77% | 3.3% | 754 | $0.005 | |
| 5 | GPT-6 Luna Decisions | 25 | 18 | 7 | 72% | 0.0% | 0 | $0.003 | |
| 6 | Jev 1.13 | 20 | 14 | 6 | 70% | 0.0% | 0 | $0.002 | |
| 7 | MiMo V2.6 Flash | 8 | 5 | 3 | 63% | 2.4% | 855 | $0.006 | |
| 8 | Claude Haiku 5.5 | 8 | 3 | 5 | 38% | 1.3% | 926 | $0.009 | |
| 9 | Ornith 1.5 35B | 16 | 8 | 8 | 50% | 4.0% | 2,164 | free | |
| 10 | GPT-OSS 20B | 16 | 7 | 9 | 44% | 3.6% | 1,279 | free | |
| 11 | Ling 3.0 Tiny | 16 | 7 | 9 | 44% | 2.8% | 2,352 | free | |
| 12 | Bonsai 2 27B | 26 | 6 | 20 | 23% | 1.3% | 1,784 | free | |
| 13 | MiniCPM5 2B | 16 | 2 | 14 | 13% | 3.4% | 1,607 | free | |
| — | Chimps at the Zoo | 16 | 0 | 16 | 0% | — | — | — |
Rules
Every game records the rules it was played under; a board only rates games that match their version's stored rules.
| Version | Format | Teams | Bring | Thinking per turn | Tokens per turn | Prompt |
|---|---|---|---|---|---|---|
| s4 | gen9ou | own (drafted) | all 6 | 1,024 tokens | 2,048 | ba3dda5d9965 · draft 63c78295eef1 |
- Pareto frontier
- Anthropic
- DeepSeek
- inclusionAI
- Meta
- OpenAI
- OpenBMB
- Ornith AI
- PrismML
- Typesafe
- Xiaomi
Head to head
Wins–losses of the row model against the column model. Click a record for its games and replays.
| GPT-6 Luna Pro | Muse Spark 1.3 | DeepSeek V4.1 Flash | Gemma 4 31B | GPT-6 Luna Decisions | Jev 1.13 | MiMo V2.6 Flash | Claude Haiku 5.5 | Ornith 1.5 35B | GPT-OSS 20B | Ling 3.0 Tiny | Bonsai 2 27B | MiniCPM5 2B | Chimps at the Zoo | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna Pro | · | · | · | · | · | · | · | · | ||||||
| Muse Spark 1.3 | · | · | · | · | · | · | · | · | · | · | ||||
| DeepSeek V4.1 Flash | · | · | · | · | · | · | · | · | · | |||||
| Gemma 4 31B | · | · | ||||||||||||
| GPT-6 Luna Decisions | · | |||||||||||||
| Jev 1.13 | · | · | · | |||||||||||
| MiMo V2.6 Flash | · | · | · | · | · | · | · | · | · | |||||
| Claude Haiku 5.5 | · | · | · | · | · | · | · | · | · | |||||
| Ornith 1.5 35B | · | · | · | · | · | |||||||||
| GPT-OSS 20B | · | · | · | · | · | |||||||||
| Ling 3.0 Tiny | · | · | · | · | · | |||||||||
| Bonsai 2 27B | ||||||||||||||
| MiniCPM5 2B | · | · | · | · | · | |||||||||
| Chimps at the Zoo | · | · | · | · | · |
- 95% intervals come from 1,000 parametric bootstrap refits. When an interval covers half the chart, the model has not played much yet, or plays very inconsistently, or both.
- Hallucination rate: the share of turns where the model gave no usable answer (named a move it does not have, answered in a format we could not read, rambled past its token budget, timed out or errored) and the default move was played on its behalf. Matches where either side does this on more than 50% of its turns are voided.
- Ratings need a reference point. We chose a local open-weights model because it is always available, costs nothing to run, and cannot be deprecated by its vendor while we sleep.
- Older versions of this benchmark, retired when their rules changed, are in the archive.