[{"content":"Today I did some infrastructure housekeeping for this site. The changes are small, but all relate to day-to-day experience: I can now understand more clearly how articles are read, and the filing information and resource loading are now more standardized.\nAnalytics: Covering Both Domestic and Overseas Visitors The site now uses two analytics services at the same time:\nBaidu Analytics: better suited for observing domestic search, referral channels, and page visits; Google Analytics 4: supplements data on overseas visits and international search sources. The numbers from the two will not be exactly the same—browser privacy settings, network environments, and script loading conditions can all cause differences. The focus here is on long-term trends rather than treating a single day\u0026rsquo;s PV as an absolute answer.\nThis data will be used to judge which topics are worth continuing to write about and which articles need supplementation or revision; it will not affect readers\u0026rsquo; normal reading of content.\nFooter: The Filing Information Finally Has a Proper Place The footer is now arranged in three layers:\nsite copyright; ICP filing and public-security network filing; technical credits for Hugo and PaperMod. A corresponding icon has been added next to the public-security filing number, and both filing numbers link directly to their official query pages. On mobile, they wrap automatically when space is insufficient, preventing long filing numbers from breaking the layout.\nCaching: Make the Right Assets Faster and Keep Updates Timely This time the caching rules for static assets have also been reorganized:\nHugo-generated content-fingerprinted assets (such as CSS and JavaScript) can be cached for one year. When file content changes, the URL also changes, so long-term caching is safe; Fixed-URL assets such as icons and images are cached for 30 days. This reduces repeated downloads while also preventing situations where a replaced file with the same name is not visible for a long time; HTML pages are not long-term cached; they are still validated via ETag and Last-Modified time, so new posts or page updates are visible promptly. The server also continues to provide gzip and zstd compression to reduce the transfer size of text assets.\nWhat\u0026rsquo;s Next The value of infrastructure lies not in stacking tools, but in keeping things simple and maintainable. Going forward, I will continue to optimize content and performance based on actual visit patterns, and gradually supplement the site\u0026rsquo;s data and privacy notices.\nThanks for every visit.🙂\n","permalink":"https://styleofwong.cn/en/posts/site-infrastructure-update-20260713/","summary":"Today I added domestic and international traffic analytics, displayed filing information, and reorganized the static asset caching strategy for this site.","title":"Site Infrastructure Update: Analytics, ICP Filing, and Caching"},{"content":"Core Conclusion (Top of the Pyramid) To stably achieve a 99% Ante 8 clear rate on Balatro White Stake, end-to-end deep reinforcement learning (such as PPO) is not viable. The feasible route is \u0026ldquo;hierarchical architecture + hybrid search\u0026rdquo;:\nTactical layer (within a single round): model \u0026ldquo;selecting and playing cards\u0026rdquo; as a combinatorial optimization problem, and use MCTS or exact enumeration to find an approximately optimal solution—but note that this layer cannot be solved independently of the strategic layer (see §3.3: accumulating jokers give tactical actions long-term consequences) Strategic layer (across rounds + shop): this is where RL should really focus—building (joker selection), resource management (gold/draws/discards), risk control, and providing long-term value estimates $V(s\u0026rsquo;)$ to the tactical layer Reward shaping: the sparse \u0026ldquo;clear\u0026rdquo; signal must be densified, but we must also avoid the agent learning to \u0026ldquo;farm score\u0026rdquo; instead of \u0026ldquo;clear\u0026rdquo; We break it down layer by layer below. First, clarify why this goal is both hard-core and feasible.\nI. Calibrating the Goal First: What Does a 99% White Stake Clear Rate Mean? 1.1 Difficulty Baseline (Human Level) White Stake is Balatro\u0026rsquo;s lowest difficulty (no stake modifiers). But even so, clearing Ante 8 is by no means easy. According to community statistics 1:\nPlayer Tier White Stake Clear Rate Casual players ~20-30% Experienced players ~50% Top players ~80% (some decks such as Checkered can be higher) 📌 Key Judgment: The 99% target is significantly above human top performance. It means the agent cannot rely on \u0026ldquo;lucky builds\u0026rdquo; and must systematically avoid death risks and maximize the expected clear probability of every decision. This is precisely RL\u0026rsquo;s natural advantage over humans—RL can optimize long-term expectation, while humans are limited by the emotion and visible horizon of a single run.\n1.2 The Score Wall: The Hard Threshold of Ante 8 The score required by Ante 8 Boss Blind on White Stake (Red Deck baseline) 2:\nAnte Small Blind Big Blind Boss Blind 1 300 450 600 4 8,000 12,000 16,000 8 ~100,000 ~150,000 ~200,000+ Some bosses (e.g., The Wall) can push the Ante 8 score wall to 300,000 3. This means: entering Ante 7-8, the agent must have exponential score growth ability (xMult joker + hand upgrades). The build path chosen in the early-to-mid game already determines whether the late game can break through this wall.\nII. Modeling the Game as an MDP: State, Action, Reward This is the foundation of all RL work. The difficulty of Balatro lies in its decisions being hybrid, sparse, and highly build-dependent.\n2.1 State Space $S$ A complete state needs to encode:\n状态 = ( 手牌(花色/点数/增强), # 当前可打出的牌 牌堆/弃牌堆(残差信息), # 推断剩余牌分布 当前 blind 目标分数 + 已得分, 剩余出牌次数 / 弃牌次数, # 资源 金币, # 商店购买力 当前 joker 列表 + 每张效果, # ★ 核心构筑,150+ 种 # ★★ 每张 joker 的运行时累积状态(如 Green Joker 当前 mult、Ride the Bus 计数、Supernova 永久加成) # 这一项绝不能省——见 §3.3,它是跨回合耦合的根源 牌组(deck)构成, # 已增删/增强的牌 消耗品(塔罗/星球/光谱), 当前 ante / blind 类型 / Boss 效果, 商店内容(若在商店阶段) ) Encoding Challenge: Joker effects are extremely heterogeneous (some add chips, some multiply mult, some trigger conditionally, some rewrite scoring rules such as Burglar / Eternal types). They cannot be simply one-hot encoded and require structured/embedded encoding—this is the focus of subsequent network design. More critically, every joker must also carry its runtime accumulated state (current mult/chips bonuses, trigger counts, etc.); otherwise the model cannot understand long-range investments such as \u0026ldquo;discard to stack buffs\u0026rdquo; (see §3.3 for details).\n2.2 Action Space $A$ (There Is a Big Pitfall Here) Balatro\u0026rsquo;s actions are phased and heterogeneous:\nPhase Action Action Count Play Select 1–5 cards and play them $\\binom{8}{1..5} + \\text{ordering variants}$, but we only care about hand-type combinations, roughly $10^2$ Discard Select 1–5 cards to discard Same as above Shop Buy joker / consumable / reroll shop / sell joker / skip ~20-30 Ordering Reorder hand cards Affects some jokers (e.g., left-to-right triggers) ⚠️ Key Insight: The \u0026ldquo;combinatorial explosion\u0026rdquo; of playing cards is a false proposition. Selecting 1–5 cards from 8 gives $\\sum_{k=1}^{5}\\binom{8}{k} = 218$ combinations, and after excluding invalid hand types the actually valid plays are often only a few dozen. This subproblem can be solved by enumeration + exact scoring; no RL needed!\n2.3 Reward $R$: The Design That Needs the Most Care The direct reward is \u0026ldquo;clear = +1, fail = 0\u0026rdquo;. But this signal is too sparse—a run lasts 15–40 minutes with hundreds of decisions, yet there is only one terminal signal, making credit assignment nearly impossible.\nDrawing on OpenAI Five\u0026rsquo;s experience in Dota 4, we must do reward shaping, but we must watch the direction:\nShaping Item Recommended? Rationale Clear +1 / Fail 0 ✅ Terminal main signal Must keep; defines \u0026ldquo;win\u0026rdquo; Gold reward for defeating Blind ✅ Weak Aligns with in-game economy Excess score over target ⚠️ Use with caution Easily causes the agent to farm score instead of surviving Survive to next ante ✅ Recommended Densifies progress signal \u0026ldquo;Power increase\u0026rdquo; after buying a joker ✅ Recommended Guides build quality Remaining plays/discards ⚠️ Weak Avoid over-spending resources Core Principle (borrowing the zero-sum idea from OpenAI Five 4): The net effect of shaping rewards must not deviate from the \u0026ldquo;clear\u0026rdquo; objective. A practical approach is potential-based reward shaping $F = \\gamma\\Phi(s\u0026rsquo;) - \\Phi(s)$, where $\\Phi$ is a handcrafted or learned \u0026ldquo;clearance potential function\u0026rdquo; that theoretically does not change the optimal policy and only accelerates learning 5.\nIII. Algorithm Selection: Why Pure PPO Is Not Enough and Why an MCTS+RL Hybrid Architecture Is Needed 3.1 Why Pure End-to-End PPO Fails Intuitively, Dota 2 succeeded with PPO, so Balatro should too? No. There are three reasons:\nSample efficiency: Dota was built on 256 GPUs + 45,000 years of game experience 6. Although Balatro community simulators (such as Jackdaw) are fast, a single run is long and decisions are dense; stacking up to Dota\u0026rsquo;s scale is unrealistic. Precision of combinatorial actions: Playing cards is a discrete combinatorial problem; PPO samples from a stochastic policy and struggles to stably converge to \u0026ldquo;the optimal subset of the current hand\u0026rdquo;—which is exactly solvable exactly. Long-range dependency of builds: Which joker to buy in Ante 3 directly determines whether Ante 8 can be cleared. PPO\u0026rsquo;s credit assignment span is too large. 3.2 Recommended Architecture: Exact Tactical Layer + Learning Strategic Layer Borrowing the core idea of AlphaZero—use search as a \u0026ldquo;policy improvement operator\u0026rdquo; and use the network to guide search 7 8—but hierarchical:\n┌─────────────────────────────────────────────┐ │ 战略层(跨回合,RL 的主战场) │ │ - 商店决策:买/卖/跳过/重置 │ │ - 资源管理:金币、抽牌、弃牌的分配 │ │ - 风险控制:何时安全过 blind、何时贪分 │ │ 算法:PPO 或 AlphaZero-style 策略价值网络 │ └──────────────────┬──────────────────────────┘ │ (提供当前状态 + 战略意图) ▼ ┌─────────────────────────────────────────────┐ │ 战术层(单回合内,组合优化) │ │ - 出牌选牌:从手牌中选最优子集 │ │ - 精确计分:含 joker 链触发顺序 │ │ 算法:精确枚举 / MCTS / 分支限界 │ └─────────────────────────────────────────────┘ Why does this division (preliminary idea) seem effective?\nThe tactical layer seems \u0026ldquo;exactly solvable\u0026rdquo;—there are only 8 cards in hand, and the number of valid play combinations is only a few dozen; we can directly enumerate each combination\u0026rsquo;s actual score (considering all joker triggers) and pick the best. The strategic layer is where RL belongs—shop building and resource scheduling are the real long-term planning problems; RL learns \u0026ldquo;future-oriented\u0026rdquo; policies here. This seems to greatly compress RL\u0026rsquo;s action space and decision frequency. ⚠️ But this \u0026ldquo;division\u0026rdquo; has a fatal flaw; see the next section §3.3. The preliminary idea of \u0026ldquo;independently exactly solving the tactical layer\u0026rdquo; does not hold under Balatro\u0026rsquo;s real mechanics.\n3.3 Key Correction: Cross-Round State Coupling Means the Tactical Layer Cannot Be Solved Independently The above judgment that \u0026ldquo;the tactical layer is exactly solvable\u0026rdquo; only holds when pure scoring jokers (e.g., +chips, +mult) are present. But Balatro has a large class of jokers whose internal state changes with tactical actions and persists across rounds—this directly breaks the assumption of \u0026ldquo;single-round closure\u0026rdquo;.\nTypical Example: Accumulating Jokers\nJoker Mechanism Cross-Round Impact Green Joker +1 mult per discard, -1 mult per play The mult bonus permanently accumulates, affecting all subsequent rounds Ride the Bus +0.2 mult for each hand played with no face cards Accumulates mult until playing a hand with a face card resets it Banner +30 chips per remaining discard Discard count itself is a cross-round resource Supernova +mult permanently for the hand type scored this time Permanent memory for the whole run Mystic Summit First few discards trigger +mult One-time consumption, cross-round budget A concrete deduction (using Green Joker as an example): there is a sure-win pair in hand; playing it directly clears the blind. But if we discard twice first and then play the pair, this round permanently gains +2 mult, which persists to all subsequent rounds. The real decision is:\n方案 A:直接出对子 → 过关,mult 不变 方案 B:先弃牌 2 次 → 攒永久 mult → 再出对子 → 过关,mult +2(全周期有效) 方案 C:弃牌 5 次 → 攒更多 mult → 但可能把好牌弃了,过不了关 \u0026ldquo;Whether to discard to stack buffs\u0026rdquo; is a long-term investment decision, not a single-round optimum. The assumption of \u0026ldquo;exactly solving the tactical layer\u0026rdquo; directly collapses when such jokers exist.\nConclusion: On the surface Balatro is a card game, but underneath it is actually a resource-management + long-term-investment game disguised as a card game. Many of its mechanics (accumulating mult, deck evolution, consumable investment) are essentially trade-offs of \u0026ldquo;sacrifice now for future returns\u0026rdquo;. RL is better than MCTS at this kind of problem—because MCTS excels at discrete adversarial search (Go), while RL excels at long-horizon investment planning and credit assignment.\n3.4 Corrected Architecture: Tactical Layer Depends on the Strategic Layer\u0026rsquo;s Long-Term Value In the original architecture the tactical layer was an \u0026ldquo;independent black-box solver\u0026rdquo;; after the correction it must be guided by the strategic-layer network, and the two layers are jointly trained end-to-end:\n┌─────────────────────────────────────────────────┐ │ 战略层(RL:学习长期价值 V(s)) │ │ - 输入:完整状态(★ 含 joker 运行时累积状态) │ │ - 输出:V(s) = 期望通关率 │ │ - 用途:① 商店决策 ② 给战术层提供长期价值 │ └──────────────────┬──────────────────────────────┘ │ 提供 V(s\u0026#39;) —— 操作后状态的价值 ▼ ┌─────────────────────────────────────────────────┐ │ 战术层(组合优化 + 长期价值引导) │ │ - 枚举出牌/弃牌组合 c │ │ - 对每个组合 c,精确计算综合价值: │ │ 单回合得分 + γ · V(操作后状态 s\u0026#39;) │ │ - 选综合价值最高的组合 │ └─────────────────────────────────────────────────┘ Key Change: The tactical layer no longer pursues \u0026ldquo;highest single-round score\u0026rdquo;, but the combined optimum of \u0026ldquo;single-round score + discounted long-term value\u0026rdquo;. The long-term value of Green Joker\u0026rsquo;s \u0026ldquo;discard to accumulate mult\u0026rdquo; operation will be recognized and rewarded by the strategic-layer network $V(s\u0026rsquo;)$.\nImpact on State Encoding: The input to the Set Transformer is no longer just \u0026ldquo;joker type embeddings\u0026rdquo;, but a joint embedding of \u0026ldquo;joker type + runtime accumulated state (current mult/chips bonuses, trigger counts, etc.)\u0026rdquo;.\nImpact on Training: The two layers must be jointly optimized with gradient flow, approaching an AlphaZero-style \u0026ldquo;network guides search, search improves network\u0026rdquo; architecture—rather than \u0026ldquo;tactical layer independent, strategic layer independent\u0026rdquo;.\n3.5 Concrete Network Design for the Strategic Layer State encoding (the key innovation) uses Set Transformer / GNN to process the joker set, because joker order and combinations both matter:\njoker 编码 = Transformer/SetEncoder(joker_embeddings) 全状态向量 = concat(标量状态, 手牌编码, joker编码, deck编码, 盲注信息) ↓ 策略网络 π(a|s) → 商店动作分布 价值网络 V(s) → \u0026#34;从当前状态出发的期望通关率\u0026#34; The value network $V(s)$ outputs the expected clear probability, which is exactly the quantity we ultimately want to maximize—letting the agent directly learn to evaluate \u0026ldquo;given my current build and situation, how confident am I of clearing Ante 8\u0026rdquo;.\nIV. Training Pipeline: Curriculum Learning + Self-Play 4.1 Curriculum Learning Don\u0026rsquo;t train on Ante 8 right away. Referring to general RL engineering experience 9:\nPhase 1: First train on Ante 1-3, letting the agent learn basic scoring and shop rhythm Phase 2: Ante 4-6, introducing build synergy and resource management Phase 3: Ante 7-8, focusing on breaking the score wall and Boss handling Phase 4: Full-ante random seeds, complete-run training ⚠️ Note: OpenAI Five did not use a hand-designed curriculum in Dota 4. But Balatro is different—its ante is a naturally increasing difficulty ladder, so exploiting this structure for curriculum learning is reasonable and efficient.\n2.2 Self-Play + Opponent Pool Because Balatro is a single-player versus stochastic environment (non-adversarial), no opponent pool is needed. But we can maintain a seed pool / difficulty pool: weightedly sample random seeds on which the agent tends to lose (rare boss combinations, bad shops) for prioritized experience—this is equivalent to transferring AlphaStar\u0026rsquo;s \u0026ldquo;exploiter\u0026rdquo; idea 10 to the single-agent setting.\nV. Key Technical Risks and Mitigations Risk Description Mitigation Simulator fidelity Third-party simulators such as Jackdaw may differ from the real game Do A/B validation with the real-game API (e.g., BalatroBot); align key scoring logic with unit tests against the official wiki 2 Reward hacking The agent may find simulator bugs or edge behaviors to farm score Use clear/fail as the terminal signal; make shaping rewards zero-sum / potential-based; periodically audit agent behavior manually Joker combinatorial space explosion The strategic space of 150+ joker combinations is huge Set Transformer encoding + transfer learning (first train on a small joker subset) Insufficient handling of rare bosses Extreme Ante 8 bosses such as The Wall (300k score) Adversarial seed sampling, specifically hardening extreme scenarios Overfitting to fixed randomness RL may \u0026ldquo;memorize answers\u0026rdquo; on fixed seeds Use a large number of random seeds during training, and a held-out seed set for evaluation VI. Is 99% Really Achievable? My Assessment Achievable, but conditional:\nThe randomness of White Stake is limited: shop items are random, but it is not adversarial—there are no \u0026ldquo;unwinnable seeds\u0026rdquo;. Theoretically almost every random seed has a solution; only the difficulty differs. Human top performance reaches ~80% 1, showing that the remaining ~19% of failures mostly come from suboptimal decisions rather than \u0026ldquo;dead seeds\u0026rdquo;. RL systematically optimizes long-term expectation, leaving room to eliminate that 19%. The bottleneck lies in long-term planning at the build layer, and this is exactly the strength of an AlphaZero-style \u0026ldquo;search + learning\u0026rdquo; architecture—MCTS can look ahead several rounds during shop decisions. 🎯 Realistic Expectation: With Jackdaw + exact tactical layer + AlphaZero-style strategic-layer hybrid architecture, after training for several weeks on moderate compute (single-machine multi-GPU), raising the White Stake Ante 8 clear rate from near 0% random baseline to 90%+ is supported by community precedent; pushing to 99% through adversarial sampling and architecture refinement is an aggressive but not fantastical goal.\nThere are already reports of RL projects achieving their first win in the real game 11, proving that this path works.\nVII. Recommended Production-Ready Tech Stack Component Recommended Solution Notes Simulator Jackdaw (Python, Gymnasium) First choice; native Python with no Lua dependency Real-game interface BalatroBot (JSON-RPC API) Validate simulator fidelity Tactical-layer solver Exact enumeration + scoring simulation Choosing 5 out of 8 is fully enumerable Strategic-layer algorithm PPO (Stable Baselines3) or AlphaZero-style SB3 is quick to start; AlphaZero has higher ceiling Network architecture Set Transformer + MLP Encode joker set Training framework Ray RLlib / CleanRL Distributed training Evaluation Held-out 10k-seed clear rate Prevent overfitting VIII. Summary: Three Sentences to Remember This Design The tactical layer cannot be decoupled from the strategic layer: accumulating jokers make single-round actions have long-term consequences (§3.3), so the tactical layer must be guided by the strategic layer\u0026rsquo;s long-term value $V(s\u0026rsquo;)$, and the two layers are jointly trained end-to-end. Be careful with reward shaping: densifying progress signals is fine, but the terminal signal must lock onto \u0026ldquo;clear\u0026rdquo; to avoid reward hacking. 99% on White Stake is achievable: because randomness is limited and human ~80% shows the remainder is caused by suboptimal decisions, RL\u0026rsquo;s optimization of long-term expectation has room to close the gap. The next post will implement a minimal runnable Balatro RL demo (based on Jackdaw), write out the exact tactical-layer solver, and get the first baseline running. Happy hacking! 🃏\nReferences Jackdaw — Balatro Simulator for RL BalatroBot — JSON-RPC API Balatro Wiki — Stakes Reddit r/balatro, What is an overall good winrate on Balatro, https://www.reddit.com/r/balatro/comments/1ctabl2/what_is_an_overall_good_winrate_on_balatro/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBalatro Wiki, Blinds and Antes, https://balatrowiki.org/w/Blinds_and_Antes\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nReddit r/balatro, Is the Ante 8 Boss Blind always 300,000?, https://www.reddit.com/r/balatro/comments/1bkyqtg/is_the_ante_8_boss_blind_always_300000_or_did_i/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBerner, C. et al., Dota 2 with Large Scale Deep Reinforcement Learning, arXiv:1912.06680. https://arxiv.org/abs/1912.06680\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nNg, A. et al., Policy Invariance Under Reward Transformations, ICML 1999. (theoretical foundation of potential-based reward shaping)\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOpenAI, OpenAI Five, https://openai.com/index/openai-five/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSilver, D. et al., Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, arXiv:1712.01815. https://arxiv.org/abs/1712.01815\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSurag Nair, A Simple AlphaZero Tutorial, https://suragnair.github.io/posts/alphazero.html\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nNarvekar, S. et al., Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey, arXiv:2103.04794.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nDeepMind, AlphaStar: Grandmaster Level in StarCraft II, https://deepmind.google/blog/alphastar-grandmaster-level-in-starcraft-ii-using-multi-agent-reinforcement-learning/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nReddit r/reinforcementlearning, My Balatro RL project just won its first run, https://www.reddit.com/r/reinforcementlearning/comments/1m0te9n/my_balatro_rl_project_just_won_its_first_run_in/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://styleofwong.cn/en/posts/balatro-rl-design/","summary":"If we want to achieve a 99% Ante 8 clear rate on Balatro White Stake, how should the RL algorithm be designed? This article presents a concrete technical blueprint from game-mechanics modeling, state/action spaces, reward shaping to algorithm selection, and explains why pure PPO won\u0026rsquo;t work and why a hybrid MCTS+RL architecture is necessary.","title":"Taming Balatro: Pushing White Stake Clear Rate to 99% with Reinforcement Learning"},{"content":"Core Takeaway (Top of the Pyramid) AlphaZero and OpenAI Five (Dota 2) surpassed humans not because they invented brand-new reinforcement-learning algorithms, but because each used highly targeted engineering and design to overcome three common bottlenecks that RL faces in complex games:\nWhere does the data come from? — Without human labels, how do you generate training data? How do you keep training stable? — The non-stationarity and catastrophic forgetting caused by self-play. How do you scale up? — The exponential growth of state space and computation. In each section below, I will first clarify what problem was encountered, then explain the method used to solve it, and finally give verifiable factual evidence (paper / official-blog data).\n📌 All key numbers and mechanisms in this article are cited from the original papers and official materials; a complete reference list is appended at the end.\nFirst Bottleneck: Where Does the Data Come From? — Self-Play Problem: Without Human Labels, How Does RL Obtain a Training Signal? Supervised learning needs paired (input, label) data. But in board games and complex real-time strategy games, there is no objective standard for the \u0026ldquo;correct next move\u0026rdquo; — a game has no standard answer, only the delayed signal of \u0026ldquo;did you eventually win or lose?\u0026rdquo;\nThe deeper problem is that even if you have human expert game records, those are only human local optima. Before AlphaZero, AlphaGo first used human game records for supervised pre-training and then turned to self-play. But DeepMind wanted to ask a more radical question:\nIs it possible to learn a strategy that surpasses humans from scratch (tabula rasa), without relying on any human knowledge at all?\nSolution: Self-Play + Neural-Network-Guided Search AlphaZero\u0026rsquo;s solution — Self-play policy iteration:\nUse the current neural network + Monte Carlo Tree Search (MCTS) to play games against itself, generating game data. The neural network has two output heads: a policy head (probability of the next move) and a value head (win-rate evaluation of the position). MCTS does not brute-force the entire tree; it uses the network\u0026rsquo;s predictions to search directionally, then feeds the search results back as \u0026ldquo;better labels\u0026rdquo; to train the network. The network improves → self-play quality rises → training data improves → the network improves further\u0026hellip; forming a positive feedback loop. The key insight is: MCTS acts as a \u0026ldquo;policy improvement operator\u0026rdquo; — the network\u0026rsquo;s raw predictions are \u0026ldquo;polished\u0026rdquo; by search and become more accurate, and these polished moves become the targets for the next round of training 1.\nOpenAI Five\u0026rsquo;s solution — Large-scale self-play:\nDota 2 cannot use MCTS (state space too large, continuous actions, imperfect information), so OpenAI Five took a different path: pure policy-gradient methods + massive self-play. It does not search or plan; it directly outputs actions from a neural network and learns from the final win/loss feedback via the PPO algorithm. Its training intensity is staggering:\nOpenAI Five\u0026rsquo;s daily self-play volume is equivalent to roughly 180 years of human gameplay; the entire training run accumulated about 45,000 years of Dota 2 game experience 2 3.\nFactual Evidence AlphaZero: Published in Science in December 2017. The paper shows that, starting from scratch, it surpassed the then-strongest chess engine Stockfish after only 4 hours of training (about 300,000 steps); in shogi, only 2 hours 4 5. AlphaZero vs. Stockfish 100-game evaluation: 28 wins, 72 draws, 0 losses 6. OpenAI Five: The official page states explicitly that training used 256 GPUs + 128,000 CPU cores, running the PPO algorithm and processing about 2 million frames every 2 seconds 2 7. Second Bottleneck: How Do You Keep Training Stable? — Non-Stationarity and Catastrophic Forgetting Problem: When You Play Against Yourself, the Data Distribution Keeps Changing This is the most hidden and deadly problem of self-play. In supervised learning, the dataset is fixed; but in self-play, the \u0026ldquo;opponent\u0026rdquo; that generates the data — the network itself — changes every round. This has two serious consequences:\nConsequence 1: Non-stationarity. Training data is generated by the \u0026ldquo;current version\u0026rdquo; of the network; as soon as the network updates, the data distribution changes immediately. In academic terms, this is a non-stationary data stream problem — the network is chasing a constantly moving target 8.\nConsequence 2: Catastrophic Forgetting. Neural networks store knowledge in distributed representations, and newly learned strategies can overwrite old ones, even strategies that had already been mastered. RL agents often exhibit the phenomenon of \u0026ldquo;the move it knew yesterday, it suddenly forgets today\u0026rdquo; 9.\nIn AlphaZero\u0026rsquo;s early self-play phase, this problem was especially obvious: when network weights were randomly initialized, early games were extremely low quality, and the draw rate quickly climbed from about 17% to about 35% — the network had not yet truly learned \u0026ldquo;how to win\u0026rdquo; 10.\nSolution AlphaZero\u0026rsquo;s solution — Sliding-Window Replay Buffer:\nThis is the key design that stabilizes training. AlphaZero keeps the most recent roughly 500,000 games of self-play data (corresponding to a time window), and each training step uniformly samples a mini-batch from the whole window rather than using only the newest data 11 8.\nThe effect of this mechanism is temporal smoothing:\nNot using only the latest data → avoids being dominated by the current version\u0026rsquo;s biases. Discarding overly old data → avoids polluting training with data generated by ancient weak versions. The window size is a delicate engineering trade-off: too large and the data becomes stale; too small and the smoothing effect is lost and training becomes unstable.\nOpenAI Five\u0026rsquo;s solution — Multiple Stabilization Mechanisms:\nOpenAI Five, facing a Dota 2 training run as long as 45,000 years, faced an even more severe stability challenge. It used several mutually reinforcing techniques:\nPPO Clipped Surrogate Objective: PPO does not use the raw policy gradient directly; instead it uses a \u0026ldquo;surrogate loss\u0026rdquo; that clips the importance-sampling ratio within a range, implicitly constraining the step size of policy updates (similar to a trust region), so that large-scale training does not collapse from taking too large a step 7.\nZero-Sum Reward Shaping: The paper explicitly states —\n\u0026ldquo;We ensure all rewards are zero-sum by subtracting the mean of the enemy heroes\u0026rsquo; rewards from each hero\u0026rsquo;s reward.\u0026rdquo; 7\nThis step is crucial. It prevents two agents from collectively \u0026ldquo;farming points\u0026rdquo; — for example, both sides farming safely in their own zones, with each hero\u0026rsquo;s gold / experience increasing (positive local reward), which does nothing to help \u0026ldquo;win the game.\u0026rdquo; After zero-sum transformation, one side\u0026rsquo;s gain must be the other side\u0026rsquo;s loss, locking the optimization target firmly onto \u0026ldquo;defeat the opponent.\u0026rdquo;\nDense Rewards Replacing Sparse Signals: A Dota game lasts about 30–45 minutes; if the only reward were a final +1 / -1, the signal would be too sparse and credit assignment would be nearly impossible. OpenAI Five designed shaped rewards based on frequent events such as gold, experience, kills, tower pushes, and Roshan, giving the agents dense feedback that greatly accelerated learning 12.\nFactual Evidence AlphaZero\u0026rsquo;s sliding-window replay-buffer mechanism is described in detail in the original paper (Silver et al., 2017) and in ELF OpenGo\u0026rsquo;s reproduction study 11 8. The OpenAI Five paper (arXiv:1912.06680, 66 pages) devotes sections specifically to PPO, reward shaping, and zero-sum design 7. The theoretical background on catastrophic forgetting and continual learning can be found in the IBM and NeurIPS surveys 9 13. Third Bottleneck: How Do You Scale Up? — Exponential Challenges of Computation and Complexity Problem: The State Space Is Too Large to Exhaustively Search This is the most intuitive challenge for game AI. Consider a few mind-boggling numbers:\nGame State-Space Size (Order of Magnitude) Chess $\\approx 10^{47}$ Go $\\approx 10^{170}$ Dota 2 Continuous state + continuous actions + partial observability, impossible to measure by counting Traditional minimax search (Minimax + Alpha-Beta pruning) can barely cope with chess, but Go\u0026rsquo;s branching factor is too large for brute-force search. Dota 2 is even worse — it has no \u0026ldquo;discrete state\u0026rdquo; to speak of: hero positions are continuous coordinates, item combinations are combinatorially explosive, and the fog of war means players cannot see all of the opponent\u0026rsquo;s information (partial observability).\nSolution AlphaZero\u0026rsquo;s solution — Use a Neural Network to Compress Value Judgment, and Use MCTS for Intelligent Search:\nAlphaZero abandoned the traditional idea of \u0026ldquo;exhaustively search to the end, then evaluate,\u0026rdquo; and instead used a neural network to directly evaluate positions — given a board, the network immediately outputs \u0026ldquo;how good is this position\u0026rdquo; and \u0026ldquo;which move should be played next.\u0026rdquo; MCTS performs only limited-depth search (typically a few dozen simulations), using the network\u0026rsquo;s predictions to guide the search direction.\nThe result is astonishing: Stockfish searches about 70 million positions per second, while AlphaZero searches only about 80,000 positions per second — yet AlphaZero won. It did not rely on raw computational superiority, but on an intuition-like evaluation that \u0026ldquo;looks once and knows good from bad\u0026rdquo; 14.\nOpenAI Five\u0026rsquo;s solution — No Search, Just Brute-Force Learning Through Scale and Compute:\nThe complexity of Dota 2 makes MCTS entirely infeasible (cannot enumerate, imperfect information). OpenAI Five\u0026rsquo;s choice was very \u0026ldquo;simple and blunt\u0026rdquo;: completely abandon search and rely purely on the scale of self-play to build capability.\nIts training configuration was one of the largest engineering efforts in RL history 2 15:\n256 NVIDIA Tesla P100 GPUs (deployed on Google Cloud) 128,000 CPU cores (used to run a large number of parallel Dota instances) Approximately 10 months of continuous training The distributed system Rapid framework, scheduling massive rollouts But \u0026ldquo;brute force\u0026rdquo; does not mean \u0026ldquo;brainless\u0026rdquo; — OpenAI Five also made extensive simplifications to make the problem learnable, and these simplifications are precisely where engineering wisdom shows:\nLimited hero pool: Only about 17 heroes were trained, with 5 randomly drawn from the pool each game. OpenAI estimated that supporting all heroes would require only about 20% more training, but the skill/item interactions among heroes create combinatorial explosion that was unrealistic on their timeline 7. Early mirror matches: Both sides used the same lineup to exploit symmetry; independent drafting was introduced only later. \u0026ldquo;Surrender\u0026rdquo; mechanism: Games that were clearly one-sided ended early, avoiding wasted compute on already-decided positions (with a curriculum-like effect, though not a hand-designed curriculum). ⚠️ An important honest statement: The OpenAI Five paper explicitly notes that it did not use a hand-designed curriculum. The original text is clear — \u0026ldquo;these mechanisms were not introduced to construct a perfect curriculum.\u0026rdquo; The difficulty gradient mainly came from sampling opponents from self-play history (the agent faces different historical versions of itself), naturally forming an adaptive difficulty curve 7.\nFactual Evidence AlphaZero inference required only 1 machine + 4 TPUs; during the match, Stockfish used 64 threads but searched nearly a thousand times as many positions as AlphaZero and still lost 16. OpenAI Five\u0026rsquo;s hardware configuration (256 GPUs + 128k CPUs) and 10-month training duration are listed on the official page and in the paper 2 7. The hero-pool limitations and simplification conditions are discussed in a dedicated section of paper arXiv:1912.06680. Extension: AlphaStar Pushes \u0026ldquo;Partial Observability + Multi-Agent\u0026rdquo; to the Extreme If you want to understand the \u0026ldquo;ultimate challenge\u0026rdquo; of RL in complex games, DeepMind\u0026rsquo;s AlphaStar (StarCraft II) is worth mentioning. The difficulties it faces are even trickier than Dota\u0026rsquo;s 17 18:\nPartial observability: The fog of war hides the opponent. Randomness: The game contains random elements. Multi-agent: Each side controls a large number of units. Non-transitive strategy cycles: Rock-paper-scissors-style counter relationships (strategy A beats B, B beats C, C beats A), which can trap self-play in local optima. AlphaStar\u0026rsquo;s core innovation is \u0026ldquo;League Training\u0026rdquo;: instead of a single agent playing against itself, it maintains a diverse population of agents — main agents, league exploiters, and main exploiters — three classes that specialize in finding holes in each other\u0026rsquo;s strategies, thereby actively generating strategic diversity and breaking the deadlock caused by non-transitivity 17. In the end, AlphaStar reached Grandmaster level, placing it in the top 0.2% of human players.\nPyramid Convergence: A Side-by-Side Comparison of the Three Projects Returning to the top of the pyramid, the following table condenses the three bottlenecks and their solutions:\nDimension AlphaZero (Board Games) OpenAI Five (Dota 2) AlphaStar (StarCraft II) Data Source Self-play + MCTS Large-scale pure self-play League self-play (multi-agent population) Core Algorithm MCTS + policy/value network PPO (policy gradient) Off-policy multi-agent RL Stability Solution Sliding-window replay buffer PPO clipping + zero-sum reward shaping + dense rewards League exploiters break cycles Scale Solution Neural network compresses evaluation, 4-TPU inference 256 GPUs + 128k CPUs, 10 months Large-scale distributed + policy diversity Representative Result 4 hours to surpass Stockfish (28-0-72) Beat professional players at TI8 Grandmaster (top 0.2%) Summary and Takeaways The breakthroughs of AlphaZero and OpenAI Five in reinforcement learning were essentially not a single stroke of algorithmic genius, but a series of interlocking engineering and algorithmic decisions:\nNo data? Generate it yourself with self-play, turning the delayed win/loss signal into an optimizable objective. Training unstable? Use a replay buffer to smooth non-stationarity, PPO to constrain update step size, and zero-sum rewards to lock onto the true optimization target. Scale exploding? Use a neural network to compress evaluation (AlphaZero), use compute and engineering simplifications (OpenAI Five), and use population diversity to counter strategy cycles (AlphaStar). These lessons have long since left the gaming world and permeated fields such as robot control, recommender systems, and large-model alignment (RLHF). Understanding how they \u0026ldquo;hit the wall\u0026rdquo; and how they \u0026ldquo;broke through\u0026rdquo; gives you the central thread of modern reinforcement learning.\nIn the next post, I will continue writing about the details of policy gradients and PPO, unpacking the \u0026ldquo;clipped surrogate objective\u0026rdquo; behind OpenAI Five.\nHappy learning! 🚀\nReferences and Citations Surag Nair, A Simple AlphaZero Tutorial, https://suragnair.github.io/posts/alphazero.html\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOpenAI, OpenAI Five, https://openai.com/index/openai-five/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWikipedia, OpenAI Five, https://en.wikipedia.org/wiki/OpenAI_Five\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSilver, D. et al., A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play, Science (2018).\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nDeepMind, AlphaZero: Shedding new light on the chess, shogi, and Go, https://deepmind.google/blog/alphazero-shedding-new-light-on-chess-shogi-and-go/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nChess.com, Google\u0026rsquo;s AlphaZero Destroys Stockfish In 100-Game Match, https://www.chess.com/news/view/google-s-alphazero-destroys-stockfish-in-100-game-match\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBerner, C. et al., Dota 2 with Large Scale Deep Reinforcement Learning, arXiv:1912.06680 (2019). https://arxiv.org/abs/1912.06680\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nTian, Y. et al., ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero, ICML 2019. https://yuandong-tian.com/reproducibility.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIBM, What is Catastrophic Forgetting?, https://www.ibm.com/think/topics/catastrophic-forgetting\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAI StackExchange, AlphaZero chess: high portion of draws during first rounds of self-play, https://ai.stackexchange.com/questions/43517\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSilver, D. et al., Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, arXiv:1712.01815 (2017). https://arxiv.org/abs/1712.01815\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOpenAI Blog, OpenAI Five (reward shaping discussion), https://openai.com/index/openai-five/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nExperience Replay for Continual Learning, NeurIPS. http://papers.neurips.cc/paper/8327-experience-replay-for-continual-learning.pdf\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nReberLab, AlphaZero Beats Chess In 4 Hours, https://www.reberlab.psych.northwestern.edu/2017/12/12/alphazero-beats-chess-in-4-hours/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAlignment Forum, How OpenAI Five Distributed Their Training Computation, https://www.alignmentforum.org/posts/6tikKda9LBzrkLfBJ/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nChess StackExchange, Hardware used in AlphaZero vs Stockfish match, https://chess.stackexchange.com/questions/19366\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nDeepMind, AlphaStar: Grandmaster Level in StarCraft II Using Multi-agent Reinforcement Learning, https://deepmind.google/blog/alphastar-grandmaster-level-in-starcraft-ii-using-multi-agent-reinforcement-learning/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAlphaStar Unplugged: Large-Scale Offline Reinforcement Learning, arXiv:2308.03526. https://arxiv.org/abs/2308.03526\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://styleofwong.cn/en/posts/rl-breakthroughs-alphazero-dota/","summary":"AlphaZero and OpenAI Five are two milestones in reinforcement learning. What walls did each of them hit, and what engineering and algorithmic innovations knocked those walls down? This article dissects them with a pyramid structure: three shared bottlenecks — data, stability, and scale — and how each project broke through them.","title":"From AlphaZero to Dota 2: Three Bottlenecks in Reinforcement Learning and How They Were Broken"},{"content":"Why Do We Need Reinforcement Learning? You are already familiar with supervised learning: given a pile of (input, label) pairs, a model learns to predict the label from the input. But in reality many problems have no labels—\nPlaying chess: there is no standard answer, only the delayed feedback of \u0026ldquo;won / lost\u0026rdquo; Autonomous driving: there are no annotations for the \u0026ldquo;correct action\u0026rdquo;; the goal is \u0026ldquo;arrive safely\u0026rdquo; Game AI: the goal is \u0026ldquo;clear the game\u0026rdquo;, but there is no ready-made answer for which move to take at each step The common feature of these problems is \u0026ldquo;sequential decision-making + delayed rewards\u0026rdquo;: an agent makes consecutive decisions in an environment, affects future states, and finally receives a reward. This is the problem that reinforcement learning (RL) aims to solve.\n💡 One-sentence intuition: reinforcement learning is letting AI learn to act through trial and error, discovering the optimal policy on its own through reward signals.\n1. Core Concepts: The MDP Five-Tuple RL problems are usually modeled as a Markov Decision Process (MDP). An MDP is defined by five elements:\nSymbol Name Meaning $S$ State Space the set of all possible environment states $A$ Action Space all actions the agent can take $P(s\u0026rsquo; \\mid s,a)$ Transition Probability the probability of transitioning to $s\u0026rsquo;$ after taking action $a$ in state $s$ $R(s,a)$ Reward Function the immediate reward obtained by taking action $a$ in state $s$ $\\gamma$ Discount Factor the discount rate for future rewards, in $[0,1]$ The Markov property means: the next state $s\u0026rsquo;$ depends only on the current state $s$ and action $a$, and is independent of history. That is:\n$$P(s_{t+1} \\mid s_t, a_t, s_{t-1}, a_{t-1}, \\dots) = P(s_{t+1} \\mid s_t, a_t)$$\nThis assumption greatly simplifies the problem—we do not need to remember the entire history; we only need to focus on the \u0026ldquo;current state\u0026rdquo; to make decisions.\n2. Objective: Maximize Cumulative Reward The goal of RL is to find a policy $\\pi$—a mapping from states to actions $\\pi: S \\to A$—that maximizes the expected cumulative reward.\nBecause future rewards are not as \u0026ldquo;valuable\u0026rdquo; as immediate rewards (100 dollars today is worth more than 100 dollars tomorrow), we introduce the discount factor $\\gamma \\in [0,1]$ and define the return $G_t$:\n$$G_t = R_{t+1} + \\gamma R_{t+2} + \\gamma^2 R_{t+3} + \\cdots = \\sum_{k=0}^{\\infty} \\gamma^k R_{t+k+1}$$\n$\\gamma = 0$: only looks at the present; short-sighted $\\gamma = 1$: treats the future the same as the present; may not converge In practice $\\gamma$ is usually set to $0.9 \\sim 0.99$ Our ultimate goal is to find the optimal policy $\\pi^*$ that maximizes expected return:\n$$\\pi^* = \\arg\\max_\\pi \\mathbb{E}\\left[\\sum_{k=0}^{\\infty} \\gamma^k R_{t+k+1} \\right]$$\n3. Value Functions: Measuring \u0026ldquo;How Good a State Is\u0026rdquo; Directly solving for $\\pi^*$ is too abstract; we need a more concrete tool—the value function.\nState-Value Function $V^\\pi(s)$ Under policy $\\pi$, the expected return starting from state $s$:\n$$V^\\pi(s) = \\mathbb{E}_\\pi \\left[ G_t \\mid S_t = s \\right]$$\nAction-Value Function $Q^\\pi(s,a)$ Under policy $\\pi$, the expected return after taking action $a$ in state $s$:\n$$Q^\\pi(s,a) = \\mathbb{E}_\\pi \\left[ G_t \\mid S_t = s, A_t = a \\right]$$\nThe relationship between the two is intuitive—the state value is the policy-weighted average of all action values in that state:\n$$V^\\pi(s) = \\sum_a \\pi(a \\mid s) , Q^\\pi(s,a)$$\n💡 Intuition: $V(s)$ tells you \u0026ldquo;how good it is to stand here,\u0026rdquo; while $Q(s,a)$ tells you \u0026ldquo;how good it is to stand here and take a particular action.\u0026rdquo; The latter is richer in information and is at the core of most RL algorithms.\n4. Bellman Equation: The Cornerstone of RL Value functions satisfy an elegant recursive relationship—the Bellman equation. It connects the \u0026ldquo;value of the current state\u0026rdquo; with the \u0026ldquo;value of the next state\u0026rdquo;.\nBellman Equation for $V$ $$V^\\pi(s) = \\sum_a \\pi(a \\mid s) \\sum_{s\u0026rsquo;} P(s\u0026rsquo; \\mid s,a) \\left[ R(s,a) + \\gamma V^\\pi(s\u0026rsquo;) \\right]$$\nIntuition: the value of the current state = the expected value, over all actions, of (immediate reward + discounted value of the next state).\nBellman Equation for $Q$ $$Q^\\pi(s,a) = \\sum_{s\u0026rsquo;} P(s\u0026rsquo; \\mid s,a) \\left[ R(s,a) + \\gamma \\sum_{a\u0026rsquo;} \\pi(a\u0026rsquo; \\mid s\u0026rsquo;) Q^\\pi(s\u0026rsquo;,a\u0026rsquo;) \\right]$$\nBellman Optimality Equation For the optimal value functions $V^$ and $Q^$, action selection is no longer \u0026ldquo;weighted by the policy\u0026rdquo; but \u0026ldquo;directly takes the maximum\u0026rdquo;:\n$$Q^(s,a) = \\sum_{s\u0026rsquo;} P(s\u0026rsquo; \\mid s,a) \\left[ R(s,a) + \\gamma \\max_{a\u0026rsquo;} Q^(s\u0026rsquo;,a\u0026rsquo;) \\right]$$\nThis is the core of all RL—as long as we can solve for $Q^*$, the optimal policy is to choose the action with the largest $Q$ value in each state:\n$$\\pi^(s) = \\arg\\max_a Q^(s,a)$$\n5. Q-Learning: The Classic Model-Free Learning Algorithm In reality, the transition probability $P$ and reward function $R$ are often unknown (this is called the model-free setting), and we can only learn by interacting with the environment. Q-Learning is the most classic solution.\nCore Idea Maintain a Q-table (a tabular $Q(s,a)$), continuously update it through interaction with the environment, and finally converge to $Q^*$.\nUpdate Rule $$Q(s,a) \\leftarrow Q(s,a) + \\alpha \\left[ r + \\gamma \\max_{a\u0026rsquo;} Q(s\u0026rsquo;,a\u0026rsquo;) - Q(s,a) \\right]$$\nwhere:\n$\\alpha \\in (0,1]$ is the learning rate $r + \\gamma \\max_{a\u0026rsquo;} Q(s\u0026rsquo;,a\u0026rsquo;)$ is the TD target (temporal-difference target) $\\delta = r + \\gamma \\max_{a\u0026rsquo;} Q(s\u0026rsquo;,a\u0026rsquo;) - Q(s,a)$ is the TD error Intuition: use the new estimate computed from \u0026ldquo;actual experience\u0026rdquo; $(r, s\u0026rsquo;)$ to correct the old $Q(s,a)$, moving a little closer to the target each time.\nExploration vs. Exploitation Q-Learning uses an $\\varepsilon$-greedy policy to balance exploration and exploitation:\nWith probability $\\varepsilon$: choose an action randomly (exploration, to avoid getting stuck in local optima) With probability $1-\\varepsilon$: choose the action with the largest $Q$ value (exploitation of known information) Usually $\\varepsilon$ decays from 1.0 to 0.1: more exploration early on and more exploitation later.\n6. Code Example: Q-Learning in a Maze Below we use the most classic example—GridWorld maze navigation—to show the complete Q-Learning workflow. The agent moves from the start to the goal while avoiding traps.\nimport numpy as np import random # 网格世界:4x4 # S = 起点, G = 终点(+1), X = 陷阱(-1) # . = 普通格子(0) grid = [ [\u0026#39;S\u0026#39;, \u0026#39;.\u0026#39;, \u0026#39;.\u0026#39;, \u0026#39;X\u0026#39;], [\u0026#39;.\u0026#39;, \u0026#39;X\u0026#39;, \u0026#39;.\u0026#39;, \u0026#39;.\u0026#39;], [\u0026#39;.\u0026#39;, \u0026#39;.\u0026#39;, \u0026#39;.\u0026#39;, \u0026#39;X\u0026#39;], [\u0026#39;X\u0026#39;, \u0026#39;.\u0026#39;, \u0026#39;.\u0026#39;, \u0026#39;G\u0026#39;], ] N = 4 ACTIONS = [\u0026#39;up\u0026#39;, \u0026#39;down\u0026#39;, \u0026#39;left\u0026#39;, \u0026#39;right\u0026#39;] def get_reward(cell): if cell == \u0026#39;G\u0026#39;: return 1.0 # 到达终点,正向奖励 if cell == \u0026#39;X\u0026#39;: return -1.0 # 踩到陷阱,负向奖励 return -0.01 # 普通格子,小惩罚(鼓励尽快到达) def is_terminal(cell): return cell in (\u0026#39;G\u0026#39;, \u0026#39;X\u0026#39;) def step(state, action): \u0026#34;\u0026#34;\u0026#34;执行动作,返回 (next_state, reward, done)\u0026#34;\u0026#34;\u0026#34; r, c = state if action == \u0026#39;up\u0026#39;: r = max(0, r - 1) elif action == \u0026#39;down\u0026#39;: r = min(N - 1, r + 1) elif action == \u0026#39;left\u0026#39;: c = max(0, c - 1) elif action == \u0026#39;right\u0026#39;: c = min(N - 1, c + 1) next_state = (r, c) cell = grid[r][c] return next_state, get_reward(cell), is_terminal(cell) # Q 表:state -\u0026gt; action -\u0026gt; value Q = {} def q_value(state, action): return Q.setdefault(state, {}).setdefault(action, 0.0) # === Q-Learning 训练 === alpha, gamma, epsilon = 0.1, 0.95, 1.0 EPISODES = 2000 for ep in range(EPISODES): state = (0, 0) # 起点 S done = False while not done: # ε-greedy 选动作 if random.random() \u0026lt; epsilon: action = random.choice(ACTIONS) # 探索 else: action = max(ACTIONS, key=lambda a: q_value(state, a)) # 利用 next_state, reward, done = step(state, action) # Q-Learning 更新 td_target = reward + gamma * max( (q_value(next_state, a) for a in ACTIONS), default=0.0 ) Q.setdefault(state, {})[action] += alpha * (td_target - q_value(state, action)) state = next_state # epsilon 衰减:从探索逐渐转向利用 epsilon = max(0.1, epsilon * 0.995) # === 提取学到的最优策略 === print(\u0026#34;学到的最优策略:\u0026#34;) arrows = {\u0026#39;up\u0026#39;: \u0026#39;↑\u0026#39;, \u0026#39;down\u0026#39;: \u0026#39;↓\u0026#39;, \u0026#39;left\u0026#39;: \u0026#39;←\u0026#39;, \u0026#39;right\u0026#39;: \u0026#39;→\u0026#39;} for r in range(N): row = [] for c in range(N): cell = grid[r][c] if cell in (\u0026#39;G\u0026#39;, \u0026#39;X\u0026#39;, \u0026#39;S\u0026#39;): row.append(f\u0026#39; {cell} \u0026#39;) else: best = max(ACTIONS, key=lambda a: q_value((r, c), a)) row.append(f\u0026#39; {arrows[best]} \u0026#39;) print(\u0026#39;\u0026#39;.join(row)) After running it, you will see the path the agent learned—it will bypass traps and head toward the goal, even though we never told it \u0026ldquo;how to get there\u0026rdquo;.\n7. From Q-Learning to Deep Reinforcement Learning The fatal limitation of Q-Learning: a Q-table cannot handle large state spaces. Chess has about $10^{47}$ states, Go about $10^{170}$—a table simply cannot store them.\nThe natural solution: use a neural network to approximate the $Q$ function $Q_\\theta(s,a)$. This is DQN (Deep Q-Network); DeepMind used it to make AI reach human-level performance on Atari games for the first time.\nKey innovations of DQN:\nExperience replay: store interaction data in a buffer, sample randomly for training, and break data correlation Target network: use a separate network to compute the TD target, stabilizing training Beyond that, the RL family continues to evolve:\nCategory Representative Algorithms Characteristics Value-based DQN, Rainbow learn the $Q$ function Policy gradient REINFORCE, A2C, A3C directly optimize the policy $\\pi_\\theta$ Actor-Critic PPO, SAC combine value and policy; current mainstream AlphaGo/Zero MCTS + RL Monte Carlo tree search + self-play 8. Final Words The beauty of reinforcement learning is that it uses a unified mathematical framework (Bellman equation + value functions) to describe \u0026ldquo;how to learn to act through trial and error without a teacher\u0026rdquo;. This may be the learning paradigm closest to the \u0026ldquo;essence of intelligence\u0026rdquo;.\nThis article walks through the core thread from intuition to Q-Learning:\nMDP models the problem Value functions measure goodness Bellman equation gives the recursive relationship Q-Learning achieves model-free learning DQN scales to large state spaces with deep networks Suggested next steps:\n📖 Sutton \u0026amp; Barto, Reinforcement Learning: An Introduction (the RL bible, free PDF) 🎮 OpenAI Gym / Gymnasium — the standard tool for running various RL environments 🧪 Stable Baselines3 — out-of-the-box implementations of RL algorithms (PPO/SAC, etc.) 🏆 Start from simple environments (CartPole) and gradually tackle more complex tasks The next post will cover policy gradient methods, discussing why \u0026ldquo;directly optimizing the policy\u0026rdquo; is more powerful than value-based methods in some scenarios.\nHappy learning! 🚀\n","permalink":"https://styleofwong.cn/en/posts/reinforcement-learning-intro/","summary":"Reinforcement learning is the paradigm that lets AI learn to \u0026ldquo;grow through trial and error\u0026rdquo;. Starting from the most fundamental intuition, this article clearly explains MDP, the Bellman equation, and Q-Learning, and includes a runnable Python example.","title":"Introduction to Reinforcement Learning: From Intuition to Q-Learning"},{"content":"👋 Hello! Welcome to Huijiu Wang\u0026rsquo;s Little Site! This is my personal space for sharing knowledge.\n🧑‍💻 About Me Profession: Programmer Interests: Algorithm research, gaming Current Focus: Concentrating on the AI field 📝 What You\u0026rsquo;ll Find Here This site will mainly share the following:\nAI Tech Notes —— Practices and thoughts on large models, machine learning, and deep learning Algorithm Research —— Problem-solving insights, algorithm principles, data structures Programming Experience —— Development tips, pitfall records, tool recommendations Gaming Experiences —— Some games worth talking about 💻 A Sample Code Snippet This site itself is also built with technology (Hugo + Nginx):\npackage main import \u0026#34;fmt\u0026#34; func main() { fmt.Println(\u0026#34;Hello, World! 博客上线 🎉\u0026#34;) } 🚀 Next Steps Writing slowly, accumulating gradually. Thanks for visiting!\n","permalink":"https://styleofwong.cn/en/posts/hello-world/","summary":"This is the first post on Huijiu Wang\u0026rsquo;s personal blog, marking the birth of this little site.","title":"Hello, World — The Blog Is Live"}]