Case study: Pokémon TCG
From research agents to better Pokémon TCG decisions
Roughly two months of player design, strategic experiments, and a question I’m still pursuing: how can a player keep getting better?
October 30 update: 5,600-game closed-loop scorer gate and ESS = 48.06 audit, Research tool: keep corrections attached to results
A game became a research program
I entered the Pokémon TCG AI Battle Challenge learning the game and asking a larger question about learning systems. Over roughly two months, I worked as a solo developer with the team (AI coding and research agents I directed across multiple vendors). I directed the questions, implementation work, and comparisons.
Two questions shaped the project: how do you teach a player useful behavior, and how do you identify behavior worth teaching? I implemented components of a recurrent learning player and tested a strategic intervention in a separate automated Lucario player.
The counter improved mean game score by 20.42 and 18.75 percentage points against two tested Alakazam implementations; the other five tested opponents averaged −0.47 points. The player components provide an interface for learning decisions. Connecting those discoveries to improvements the player retains is my open objective.
What I built and tested
Player design
Give each choice meaning
I implemented visible-state encoding, recurrent memory, legal-option scoring, and sequential selection using established learning methods.
Inspect the learner excerptsStrategic experiment
How the Lucario counter disrupts the opponent's hand
I changed a card and a targeting rule in an automated Lucario player, then compared the changes separately and together.
See the experimentThe experimental Lucario work builds on makthanithin’s Apache-2.0 community_1084 policy. My competition submission used tetsutani’s unmodified public Grimmsnarl policy. The experiments described here are separate development branches. Contribution and evidence map.
“Option three” is not a strategy
A card-game player faces a changing menu: attach Energy, evolve a Pokémon, select an attack, or choose its target. If the menu changes, the third option can mean something entirely different.
My v2 learner encodes the offered actions and links them to their sources and targets. A 256-unit recurrent network combines the visible game state with memory across decisions, then scores the legal options. The state projection uses the acting player’s view and rejects hidden-information fields.
Multiple selections introduce another design choice. When a prompt allows up to three items, selecting one can be enough. The decoder needs to represent that decision.
Worked interface example
Choose up to three items
- Score the available choices
The player sees legal options A, B, and C in the current game context.
- Select A, then update the context
A cannot be selected again. The decoder scores the remaining choices with A already selected.
- Choose STOP when the minimum is satisfied
The player can finish with A instead of taking every available item.
Run the seven selection checks. These component tests use fixed scores and run with Python and PyTorch on CPU. They demonstrate selection constraints; they do not evaluate game play.
Give practice somewhere to go
Teacher-query tooling reconstructs player-visible positions and requests choices from a credited public teacher. This provides a route to advice on situations the learner reaches itself. Training, checkpoints, and inference are separate components, so the teacher or player can change without redefining the experiment.
An earlier August checkpoint improved held-out imitation loss but won only 23 of 1,200 decided games against the reference policy. That comparison made complete-game evaluation essential alongside prediction metrics. It evaluates the earlier checkpoint; current v2 competitive strength remains unmeasured. Training and evaluation evidence.
A counter that targets two resources
Alakazam’s Powerful Hand attack scales with the cards in its player’s hand. Its next attacker also depends on developing Abra and Kadabra. That suggested pressure on two resources: the hand available now and the evolution line preparing for later.
I replaced one Carmine with Xerosic’s Machinations to reduce a large opposing hand to three cards at resolution. Separately, I modified the targeting rule to prefer visible Abra or Kadabra when the Alakazam line was detected. The tradeoff is concrete: disruption uses a Supporter opportunity, and attacking a developing Pokémon can mean passing up another prize.
This separation tests whether changing the deck, changing the policy, or using both improves whole-game outcomes. The development comparisons supported the combined counter. Full experimental design and intervals.
Look at the opponents behind the average
The later evaluation contains 8,400 game records: seven opponent implementations, with 600 games for the baseline and 600 for the counter against each. Game score is 1 for a win, ½ for a draw, and 0 for a loss.
The two tested Alakazam matchups gained 20.42 and 18.75 percentage points. The other five averaged −0.47 points, with a 95% interval of [−2.64, +1.71]. This is a large gain in those two matchups. For the other five, the average's interval includes zero.
A counter’s value changes with the field. The supporting numerical methods show how estimated benefit varies under assumed opponent mixtures; those scenarios are not a forecast of the competitive metagame.
Closed-loop learning audit. Added October 30, 2026
Why a -0.0031% confidence-interval bound stopped a +5.625 point promotion across 5,600 fresh games
To test whether outcome-fitted action scoring can improve live play, I directed the team in our local PTCG-Learning-Lab workspace to train Ridge regression action scorers on 511 eligible first-search development interventions (512 games total) and wire the learned coefficients back into the live player. Across 5,600 fresh confirmation games (1,002,605 verified action records), the closed-loop scorers changed 266, 812, and 868 live search decisions.
In the first confirmation study (tcg-scorer-v1, 88 features, model digest 52462522..., 800 fresh games), the scorer lifted mean game score by +5.625 percentage points (0.81875 vs 0.76250), with a 95% bootstrap confidence interval of [-0.003125, +11.125] percentage points. Because our pre-registered promotion gate required the lower 95% bound to clear zero, that -0.003125% shortfall blocked promotion. A higher-power 2,400-game replication (tcg-scorer-replication-v1) showed why the gate mattered: on 2,400 fresh games, the gain shrank to +1.00 percentage point ([-2.125, +4.125]).
Two mathematical audits explained the result:
- Feature aliasing (236 of 511 development rows): 58 groups of distinct game states shared identical 88-feature vectors despite different visible hands (45 also differed in boards), such as
dev-0010vsdev-0145when choosing whether to search for card647while already holding a copy in hand. Adding two relational card-copy features (tcg-relational-v1, 90 features, digest389221c9..., 2,400 fresh games) changed 868 choices (99 vs the 88-feature scorer) and scored -1.50 percentage points ([-4.667, +1.667]). - Uniform exploration starvation (ESS = 48.06): Across the 511 eligible development games, the player faced an average menu of 10.865 legal search options. Uniform random exploration matched the exact parent choice in only 58 of 511 games (11.35%), yielding an exact-parent Effective Sample Size of 48.06 (with inverse-propensity weights up to 27.0) against a 100-observation direct-comparison screen.
These studies verify the end-to-end closed-loop mechanism in live games while keeping the efficacy verdict negative: retained competitive improvement in Pokémon TCG remains unproven. Explore the interactive ESS calculator and all 10 learning demonstrations, or inspect the verification table.
Research reader. Added October 12, 2026
Keep a correction attached to the result
A research assistant can retrieve an old result and miss the note withdrawing it. I directed AI-assisted work on a reader that connects selected experiment reports to their methods, findings and caveats, so the next investigator can check what a result still supports.
A review caught a precise failure: the source hash matched, but retrieval and search omitted an introductory retraction. The repair preserves that introduction alongside the report’s sections and makes it searchable. Read the case and its limits.
Synthetic regression example
Before the repair
Result without its warning
4/4 wins
Introductory retraction missing
The retrieved result loses the qualification that changes how it can be used.
After the repair
Warning and result together
Retracted: Wrong comparison arm. Do not reuse.
4/4 wins original result retained
The introduction remains visible in the record and can be found by search.
For a developer, the check is concrete: test whether a warning survives extraction, not just whether the stored file matches its hash.
Inspect the implementation and its limits
The reader uses one committed Git revision, giving each lookup a specific source snapshot. Uncommitted edits alone do not update that snapshot. The published review describes the repair and remaining gaps; the implementation excerpt shows how the original introduction is kept and displayed. These are selected source excerpts, not a complete runnable reader or a fresh test run.