From AlphaZero to Dota 2: Three Bottlenecks in Reinforcement Learning and How They Were Broken

Core Takeaway (Top of the Pyramid)

AlphaZero and OpenAI Five (Dota 2) surpassed humans not because they invented brand-new reinforcement-learning algorithms, but because each used highly targeted engineering and design to overcome three common bottlenecks that RL faces in complex games:

1. Where does the data come from? — Without human labels, how do you generate training data?
2. How do you keep training stable? — The non-stationarity and catastrophic forgetting caused by self-play.
3. How do you scale up? — The exponential growth of state space and computation.

In each section below, I will first clarify what problem was encountered, then explain the method used to solve it, and finally give verifiable factual evidence (paper / official-blog data).

📌 All key numbers and mechanisms in this article are cited from the original papers and official materials; a complete reference list is appended at the end.

First Bottleneck: Where Does the Data Come From? — Self-Play

Problem: Without Human Labels, How Does RL Obtain a Training Signal?

Supervised learning needs paired (input, label) data. But in board games and complex real-time strategy games, there is no objective standard for the “correct next move” — a game has no standard answer, only the delayed signal of “did you eventually win or lose?”

The deeper problem is that even if you have human expert game records, those are only human local optima. Before AlphaZero, AlphaGo first used human game records for supervised pre-training and then turned to self-play. But DeepMind wanted to ask a more radical question:

Is it possible to learn a strategy that surpasses humans from scratch (tabula rasa), without relying on any human knowledge at all?

Solution: Self-Play + Neural-Network-Guided Search

AlphaZero’s solution — Self-play policy iteration:

1. Use the current neural network + Monte Carlo Tree Search (MCTS) to play games against itself, generating game data.
2. The neural network has two output heads: a policy head (probability of the next move) and a value head (win-rate evaluation of the position).
3. MCTS does not brute-force the entire tree; it uses the network’s predictions to search directionally, then feeds the search results back as “better labels” to train the network.
4. The network improves → self-play quality rises → training data improves → the network improves further… forming a positive feedback loop.

The key insight is: MCTS acts as a “policy improvement operator” — the network’s raw predictions are “polished” by search and become more accurate, and these polished moves become the targets for the next round of training ¹.

OpenAI Five’s solution — Large-scale self-play:

Dota 2 cannot use MCTS (state space too large, continuous actions, imperfect information), so OpenAI Five took a different path: pure policy-gradient methods + massive self-play. It does not search or plan; it directly outputs actions from a neural network and learns from the final win/loss feedback via the PPO algorithm. Its training intensity is staggering:

OpenAI Five’s daily self-play volume is equivalent to roughly 180 years of human gameplay; the entire training run accumulated about 45,000 years of Dota 2 game experience ^{2 3}.

Factual Evidence

• AlphaZero: Published in Science in December 2017. The paper shows that, starting from scratch, it surpassed the then-strongest chess engine Stockfish after only 4 hours of training (about 300,000 steps); in shogi, only 2 hours ^{4 5}.
• AlphaZero vs. Stockfish 100-game evaluation: 28 wins, 72 draws, 0 losses ⁶.
• OpenAI Five: The official page states explicitly that training used 256 GPUs + 128,000 CPU cores, running the PPO algorithm and processing about 2 million frames every 2 seconds ^{2 7}.

Second Bottleneck: How Do You Keep Training Stable? — Non-Stationarity and Catastrophic Forgetting

Problem: When You Play Against Yourself, the Data Distribution Keeps Changing

This is the most hidden and deadly problem of self-play. In supervised learning, the dataset is fixed; but in self-play, the “opponent” that generates the data — the network itself — changes every round. This has two serious consequences:

Consequence 1: Non-stationarity. Training data is generated by the “current version” of the network; as soon as the network updates, the data distribution changes immediately. In academic terms, this is a non-stationary data stream problem — the network is chasing a constantly moving target ⁸.

Consequence 2: Catastrophic Forgetting. Neural networks store knowledge in distributed representations, and newly learned strategies can overwrite old ones, even strategies that had already been mastered. RL agents often exhibit the phenomenon of “the move it knew yesterday, it suddenly forgets today” ⁹.

In AlphaZero’s early self-play phase, this problem was especially obvious: when network weights were randomly initialized, early games were extremely low quality, and the draw rate quickly climbed from about 17% to about 35% — the network had not yet truly learned “how to win” ¹⁰.

Solution

AlphaZero’s solution — Sliding-Window Replay Buffer:

This is the key design that stabilizes training. AlphaZero keeps the most recent roughly 500,000 games of self-play data (corresponding to a time window), and each training step uniformly samples a mini-batch from the whole window rather than using only the newest data ^{11 8}.

The effect of this mechanism is temporal smoothing:

• Not using only the latest data → avoids being dominated by the current version’s biases.
• Discarding overly old data → avoids polluting training with data generated by ancient weak versions.

The window size is a delicate engineering trade-off: too large and the data becomes stale; too small and the smoothing effect is lost and training becomes unstable.

OpenAI Five’s solution — Multiple Stabilization Mechanisms:

OpenAI Five, facing a Dota 2 training run as long as 45,000 years, faced an even more severe stability challenge. It used several mutually reinforcing techniques:

1. PPO Clipped Surrogate Objective: PPO does not use the raw policy gradient directly; instead it uses a “surrogate loss” that clips the importance-sampling ratio within a range, implicitly constraining the step size of policy updates (similar to a trust region), so that large-scale training does not collapse from taking too large a step ⁷.

2. Zero-Sum Reward Shaping: The paper explicitly states —

   “We ensure all rewards are zero-sum by subtracting the mean of the enemy heroes’ rewards from each hero’s reward.” ⁷

   This step is crucial. It prevents two agents from collectively “farming points” — for example, both sides farming safely in their own zones, with each hero’s gold / experience increasing (positive local reward), which does nothing to help “win the game.” After zero-sum transformation, one side’s gain must be the other side’s loss, locking the optimization target firmly onto “defeat the opponent.”

3. Dense Rewards Replacing Sparse Signals: A Dota game lasts about 30–45 minutes; if the only reward were a final +1 / -1, the signal would be too sparse and credit assignment would be nearly impossible. OpenAI Five designed shaped rewards based on frequent events such as gold, experience, kills, tower pushes, and Roshan, giving the agents dense feedback that greatly accelerated learning ¹².

Factual Evidence

• AlphaZero’s sliding-window replay-buffer mechanism is described in detail in the original paper (Silver et al., 2017) and in ELF OpenGo’s reproduction study ^{11 8}.
• The OpenAI Five paper (arXiv:1912.06680, 66 pages) devotes sections specifically to PPO, reward shaping, and zero-sum design ⁷.
• The theoretical background on catastrophic forgetting and continual learning can be found in the IBM and NeurIPS surveys ^{9 13}.

Third Bottleneck: How Do You Scale Up? — Exponential Challenges of Computation and Complexity

Problem: The State Space Is Too Large to Exhaustively Search

This is the most intuitive challenge for game AI. Consider a few mind-boggling numbers:

Traditional minimax search (Minimax + Alpha-Beta pruning) can barely cope with chess, but Go’s branching factor is too large for brute-force search. Dota 2 is even worse — it has no “discrete state” to speak of: hero positions are continuous coordinates, item combinations are combinatorially explosive, and the fog of war means players cannot see all of the opponent’s information (partial observability).

Solution

AlphaZero’s solution — Use a Neural Network to Compress Value Judgment, and Use MCTS for Intelligent Search:

AlphaZero abandoned the traditional idea of “exhaustively search to the end, then evaluate,” and instead used a neural network to directly evaluate positions — given a board, the network immediately outputs “how good is this position” and “which move should be played next.” MCTS performs only limited-depth search (typically a few dozen simulations), using the network’s predictions to guide the search direction.

The result is astonishing: Stockfish searches about 70 million positions per second, while AlphaZero searches only about 80,000 positions per second — yet AlphaZero won. It did not rely on raw computational superiority, but on an intuition-like evaluation that “looks once and knows good from bad” ¹⁴.

OpenAI Five’s solution — No Search, Just Brute-Force Learning Through Scale and Compute:

The complexity of Dota 2 makes MCTS entirely infeasible (cannot enumerate, imperfect information). OpenAI Five’s choice was very “simple and blunt”: completely abandon search and rely purely on the scale of self-play to build capability.

Its training configuration was one of the largest engineering efforts in RL history ^{2 15}:

• 256 NVIDIA Tesla P100 GPUs (deployed on Google Cloud)
• 128,000 CPU cores (used to run a large number of parallel Dota instances)
• Approximately 10 months of continuous training
• The distributed system Rapid framework, scheduling massive rollouts

But “brute force” does not mean “brainless” — OpenAI Five also made extensive simplifications to make the problem learnable, and these simplifications are precisely where engineering wisdom shows:

• Limited hero pool: Only about 17 heroes were trained, with 5 randomly drawn from the pool each game. OpenAI estimated that supporting all heroes would require only about 20% more training, but the skill/item interactions among heroes create combinatorial explosion that was unrealistic on their timeline ⁷.
• Early mirror matches: Both sides used the same lineup to exploit symmetry; independent drafting was introduced only later.
• “Surrender” mechanism: Games that were clearly one-sided ended early, avoiding wasted compute on already-decided positions (with a curriculum-like effect, though not a hand-designed curriculum).

⚠️ An important honest statement: The OpenAI Five paper explicitly notes that it did not use a hand-designed curriculum. The original text is clear — “these mechanisms were not introduced to construct a perfect curriculum.” The difficulty gradient mainly came from sampling opponents from self-play history (the agent faces different historical versions of itself), naturally forming an adaptive difficulty curve ⁷.

Factual Evidence

• AlphaZero inference required only 1 machine + 4 TPUs; during the match, Stockfish used 64 threads but searched nearly a thousand times as many positions as AlphaZero and still lost ¹⁶.
• OpenAI Five’s hardware configuration (256 GPUs + 128k CPUs) and 10-month training duration are listed on the official page and in the paper ^{2 7}.
• The hero-pool limitations and simplification conditions are discussed in a dedicated section of paper arXiv:1912.06680.

Extension: AlphaStar Pushes “Partial Observability + Multi-Agent” to the Extreme

If you want to understand the “ultimate challenge” of RL in complex games, DeepMind’s AlphaStar (StarCraft II) is worth mentioning. The difficulties it faces are even trickier than Dota’s ^{17 18}:

• Partial observability: The fog of war hides the opponent.
• Randomness: The game contains random elements.
• Multi-agent: Each side controls a large number of units.
• Non-transitive strategy cycles: Rock-paper-scissors-style counter relationships (strategy A beats B, B beats C, C beats A), which can trap self-play in local optima.

AlphaStar’s core innovation is “League Training”: instead of a single agent playing against itself, it maintains a diverse population of agents — main agents, league exploiters, and main exploiters — three classes that specialize in finding holes in each other’s strategies, thereby actively generating strategic diversity and breaking the deadlock caused by non-transitivity ¹⁷. In the end, AlphaStar reached Grandmaster level, placing it in the top 0.2% of human players.

Pyramid Convergence: A Side-by-Side Comparison of the Three Projects

Returning to the top of the pyramid, the following table condenses the three bottlenecks and their solutions:

Summary and Takeaways

The breakthroughs of AlphaZero and OpenAI Five in reinforcement learning were essentially not a single stroke of algorithmic genius, but a series of interlocking engineering and algorithmic decisions:

1. No data? Generate it yourself with self-play, turning the delayed win/loss signal into an optimizable objective.
2. Training unstable? Use a replay buffer to smooth non-stationarity, PPO to constrain update step size, and zero-sum rewards to lock onto the true optimization target.
3. Scale exploding? Use a neural network to compress evaluation (AlphaZero), use compute and engineering simplifications (OpenAI Five), and use population diversity to counter strategy cycles (AlphaStar).

These lessons have long since left the gaming world and permeated fields such as robot control, recommender systems, and large-model alignment (RLHF). Understanding how they “hit the wall” and how they “broke through” gives you the central thread of modern reinforcement learning.

In the next post, I will continue writing about the details of policy gradients and PPO, unpacking the “clipped surrogate objective” behind OpenAI Five.

Happy learning! 🚀

References and Citations