To avoid disrupting human players, PokéAgent participants will compete on a custom AI-focused Showdown server hosted by the competition. Participants will battle against organizer-hosted baselines and each other on ranked ladders to qualify for tournament brackets.

Track 1 currently has a prize pool of over $6,000 plus $3,000 worth of Google Cloud Platform (GCP) credits. We are still accepting new sponsors, so these totals are subject to change.

The competition is divided into three phases:

• Free Play Period (July 11th - October 12th): Open practice period with ladder exhibitions, tutorial events, and practice tournaments. All participants can test methods and climb practice ladders.
• Qualifying Window (October 13th - October 26th): Formal qualification period where registered teams compete for tournament spots. Ratings are reset and only registered usernames can participate.
• Tournament Stage (October 29th onwards): Qualifying teams face off in bracket tournaments to determine cash prizes.

Competitive Pokémon is extremely complex and creates an interesting game-playing benchmark all by itself. However, we'd add that Pokémon's popularity and Showdown's replay dataset are key features that give this domain a unique place in current AI research:

• Pokémon's widespread internet presence equips LLMs with extensive knowledge of its mechanics and strategies.
• Pokémon Showdown makes millions of competitive battles publicly available, enabling the creation of large, high-quality datasets.

Taken together, Pokémon creates a fun opportunity for areas like LLM-Agents and RL to compete and collaborate on a level playing field. In that spirit, the PokéAgent Challenge was organized by the teams behind PokéChamp and Metamon, which are recent papers that demonstrate strong human-level play in singles formats. Both projects are open-source and have recently been updated to create a more helpful starting point for this competition. We have also been joined by VGC-Bench to extend support to VGC (doubles)!

These projects have their own repositories and publications where you can find more detailed information. The "Resources" section below highlights datasets and baselines from these efforts that may be more broadly useful in the development of new methods. Participants looking for more of a blank slate to get started are encouraged to check out poke-env — the Python interface to Showdown used by most recent academic work.

Datasets

Showdown Replay Logs

Showdown makes battle replays accessible via a public API. The competition organizers maintain curated datasets for convenience and (a little) extra privacy, and host them on Hugging Face to spare Showdown download requests from this competition. We'd encourage you to use them unless you have a good reason not to. Collectively, they cover the entire range of supported rulesets for the competition:

Replays as Agent Training Data

Showdown replays are saved from the point of view of a spectator rather than the point of view of a player, which can make it difficult to use them directly in an imperfect information game like Pokémon. We need to reconstruct (and often predict) the perspective of each player to create a more typical offline RL or imitation learning dataset.

metamon converts its replay dataset into a flexible format that allows customization of observations, actions, and rewards. More than 3.5M full battle trajectories are stored on Hugging Face at metamon-parsed-replays and can be accessed through the metamon repo. Utilities in pokechamp can recreate its LLM-Agent prompts and decisions from replays in a similar fashion. vgc-bench covers the Gen 9 VGC ruleset and does not have to resort to predicting unobserved information (because VGC replays can have "Open Team Sheets" where ground-truth team info is public).

Miscellaneous

teams: All of the Showdown rulesets in the competition require players to pick their own teams. This dataset provides sets of teams gathered from forums, predicted from replays, and/or procedurally generated from Showdown trends. This creates a starting point for anyone less familiar with Competitive Pokémon, and establishes diverse team sets for self-play.

usage-stats: A convenient way to access the Showdown team usage stats for the formats covered by the competition and the timeframe covered by the replay datasets. This dataset also includes a log of all the partially revealed teams in replays. Team prediction is an interesting subproblem that your method may want to address!

Baselines

Organizers will inflate the player pool on the PokéAgent ladder with a rotating cast of existing baselines covering a wide range of skill levels and team choices. At launch, these will include:

1. Simple Heuristics to simulate the low-ELO ladder and let new methods get started. These are mainly sourced from metamon's basic evaluation opponents.
2. The best metamon and pokechamp agents playing with competitive team choices. These baselines have already demonstrated performance comparable to strong human players. If all goes well, they will be at the bottom of the leaderboard by the end of the competition!
3. Mixed metamon and pokechamp agents aimed at increasing variety and forcing participants to battle teams that resemble the real Showdown ladder. For example, varied LLM backends with several prompting and search strategies. Hundreds of checkpoints from metamon policies at various stages of training, sampling from thousands of unique teams.

Launch baselines are already open-source, so you are free to skip the ladder queue by hosting them on your local server with help from their home repo. Any additional (stronger) agents in development by the organizers will be added to the ladder rotation as the competition progresses.

We hope the ladder will also be full of participants trying new ideas; your agents' competition will always be improving!

Support

Competition staff will be active on the community Discord and are committed to answering questions (and fixing any issues that may arise) related to the starter datasets, baselines, competition logistics, and Showdown/Pokémon more broadly. However, due to limited bandwidth, we caution that the technical details involved in improving upon provided methods may be deemed out-of-scope (e.g., RL training details beyond the provided documentation). This is mainly because, given the datasets and baselines, you would have many other viable options that are maintained by larger teams and better suited to a broad audience. Still, please feel free to reach out, and we will help in any way we can.