← All notes

RESEARCH & FIELD NOTES/

2048 Agent Release Report: 80,000 Games and Three Evaluation Sets

The 80,000-game model reached 184/200, 459/500, and 917/1000 across fixed, validation, and first final evaluations. This report records the training method, checkpoints, failed experiments, controlled feature expansion, and evaluation charts.

The current research checkpoint has played 80,000 self-play games along its active lineage. It reached the 2048 tile in 184/200 fixed evaluation games (92.0%), 459/500 validation games (91.8%), and 917/1,000 games (91.7%) on a final seed set run for the first time after the first two sets passed. All three measured rates clear the predeclared 90% threshold under these simulated game rules.

The playable 2048 website is a separate deployment. At this report's cutoff it still served the earlier 24,000-game model; the 80,000-game checkpoint and all results below are from the local training environment.

What was trained

The browser game and training program use the same 4 × 4 Python rules. A valid move slides and merges tiles, scores the values created by merging, then spawns a 2 with 90% probability or a 4 with 10% probability. Games continue after reaching 2048 and end only when no legal move remains. Accordingly, the metric in this report is the fraction of simulated games that ever reached 2048, not the fraction that ended at that tile.

The agent scores each legal move using the board after sliding and merging but before the random tile arrives. This is an afterstate. Its decision rule is:

move value = merge score now + predicted future score of the afterstate

The predicted future score is a sum of learned N-tuple lookup values. The first model used 17 overlapping patterns: four rows, four columns, and nine 2 × 2 regions. The later experiment added six 2 × 3 and six 3 × 2 regions, giving 29 patterns. There is no neural network and no human game dataset.

Each self-play game starts from a new board. The agent chooses a legal move, receives the merge score, observes the next board, and updates the previous afterstate toward the next reward plus the new afterstate estimate. At game over, the final afterstate is updated toward zero future reward. The objective is cumulative score; there is no explicit bonus for first reaching 2048. Early training used a 0.1 learning rate and 10% random exploration for the first 17,000 games. The 17,000-to-24,000 stage set exploration to zero. Later runs continued from saved weights; the current learning rate is 0.05, discount 1.0, exploration 0, and stage length 7,000 games.

new game → legal moves → afterstates → choose reward + estimated value
         → random tile → next move → temporal-difference update → game over

How results were measured

Every checkpoint below uses the same 200-game evaluation seed set. From 31,000 games onward, a separate 500-game seed set was also evaluated. Both sets have been inspected repeatedly to compare models, so neither is an untouched final holdout. The reserved 1,000-game seed set was run once, only after the 80k checkpoint passed both monitored sets. Its result is now reported and that seed set is no longer unused. The three thresholds were 180/200, 450/500, and 900/1,000.

A training-log window is a moving observation during learning. It is not a formal checkpoint result: the weights change throughout that window, and those games were part of training. For example, the last 500 training games in the 80,000-game stage reached 2048 at 92.2%, while the saved checkpoint scored 92.0% on the fixed set, 91.8% on validation, and 91.7% on the final set.

Chart 1 — fixed 200-game reach rate

Each block represents about 3.3 percentage points; the counts and percentages are exact. The 3k, 10k, and 17k figures come from the original published training note. Results from 24k onward are preserved in checkpoint reports.

3k         █                                4/200  2.0%
10k        ██                              11/200  5.5%
17k        ██                              11/200  5.5%
24k        ███████████████                101/200  50.5%
31k        ██████████████████████         144/200  72.0%
38k        ██████████████████████         145/200  72.5%
45k        ███████████████████████        150/200  75.0%
52k        ███████████████████████        152/200  76.0%
59k        █████████████████████████      169/200  84.5%
66k base   ███████████████████████        156/200  78.0%
66k 2x3    ███████████████████████████    180/200  90.0%
73k 2x3    ███████████████████████████    179/200  89.5%
80k 2x3    ████████████████████████████   184/200  92.0%

The long early plateau ended with a jump from 11/200 at 17k to 101/200 at 24k. That stage changed two things at once: it added 7,000 games and turned random exploration off. It does not isolate how much of the gain came from either change.

Chart 2 — second-seed 500-game validation

This set begins at the 31k checkpoint. It was reused for model comparison and is distinct from the untouched final set.

31k        ████████████████████           341/500  68.2%
38k        █████████████████████          349/500  69.8%
45k        ███████████████████████        380/500  76.0%
52k        ███████████████████████        389/500  77.8%
59k        █████████████████████████      422/500  84.4%
66k base   ████████████████████████       402/500  80.4%
66k 2x3    ███████████████████████████    444/500  88.8%
73k 2x3    ████████████████████████████   461/500  92.2%
80k 2x3    ████████████████████████████   459/500  91.8%

The 73k checkpoint improved validation from 444/500 to 461/500, while the fixed set slipped from 180/200 to 179/200. At 80k, fixed evaluation recovered to 184/200 while validation remained above threshold at 459/500. This was the first checkpoint to pass both monitored gates.

Chart 3 — final release gates at 80k

These are three distinct seed sets. The final 1,000 games were evaluated only after the fixed and validation results met their thresholds.

Fixed 200       ████████████████████████████   184/200   92.0%  pass
Validation 500  ████████████████████████████   459/500   91.8%  pass
Final 1,000     ████████████████████████████   917/1000  91.7%  pass

The problems that changed the plan

More games were not always better. From the 59k checkpoint, 7,000 additional games with the original 17 patterns reduced the fixed result from 169/200 to 156/200 and validation from 422/500 to 402/500. We kept both checkpoints and stopped treating episode count as a proxy for quality. The cause of the regression was not isolated; it is an observed result on these seed sets.

A shortcut with symmetry failed. At 52k, averaging the saved model's values over eight board rotations and reflections after training reached only 9/200 (4.5%) on the fixed set, versus 152/200 (76.0%) for the unmodified model. We did not adopt that post-processing. A possible explanation is that the existing lookup weights had been learned in their original orientations; rotation averaging changed their meaning without training them for shared symmetry. That explanation remains a hypothesis, not a measured diagnosis.

We compared a larger board representation from the same starting point. We copied the 59k checkpoint into a separate branch, retained all 17 learned pattern weights, and initialized the 12 new 2 × 3 / 3 × 2 pattern tables to zero. Before training, its fixed evaluation was still exactly 169/200. Both branches then trained for 7,000 games with the same game seeds and learning rate. The feature set was the intended difference.

Chart 4 — controlled 66k branch comparison

F is the fixed 200-game set; V is the 500-game validation set. The two 66k branches descend from the same 59k model.

66k base F    ███████████████████████        156/200  78.0%
66k 2x3 F     ███████████████████████████    180/200  90.0%
66k base V    ████████████████████████       402/500  80.4%
66k 2x3 V     ███████████████████████████    444/500  88.8%

Adding the larger patterns moved the fixed result from 156/200 to 180/200 and validation from 402/500 to 444/500 at the same 66k training depth. This supports the feature expansion under this comparison, but it does not prove the larger architecture will outperform on all unseen games.

Score and 2048 reach rate can move differently. From the expanded 66k model to 73k, fixed-set mean score rose from 51,384.50 to 53,215.08, but the fixed 2048 count fell by one game. The current TD reward optimizes score, while the release gate measures reaching a tile. Reward changes are a candidate future experiment if this mismatch persists; none has been introduced yet.

Chart 5 — fixed-set mean score

Bars use a 60,000-point scale. Exact values are in the saved reports; whole numbers are shown here for readability.

24k        ████████████                   24,827
31k        ████████████████               31,673
38k        █████████████████              33,521
45k        ██████████████████             36,867
52k        ████████████████████           39,025
59k        █████████████████████          41,990
66k base   ███████████████████            37,751
66k 2x3    ██████████████████████████     51,385
73k 2x3    ███████████████████████████    53,215
80k 2x3    █████████████████████████████  57,951

The larger representation also costs storage. Compressed model size grew from about 3.5 MB at 59k with 17 patterns to 30.1 MB at expanded 66k, 39.2 MB at 73k, and 46.1 MB at 80k with 29 patterns. We have not measured a production inference-latency comparison.

59k / 17   ██                         3.5 MB
66k / 29   ████████████████████       30.1 MB
73k / 29   █████████████████████████  39.2 MB
80k / 29   █████████████████████████████  46.1 MB

Release status and reproducibility

The active research model follows the 80k expanded-feature branch. Across all branches, the trainer has executed about 87,000 self-play games: 80,000 along the active lineage plus a separate 7,000-game ordinary-feature comparison from 59k. Those are training attempts, not 87,000 games in one model.

The current local artifact passed all three predeclared 90% evaluation gates: fixed 184/200, validation 459/500, and first final evaluation 917/1000. Automated training was stopped at this checkpoint. This is evidence for the specified simulated environment and seed sets, not a guarantee for every game or a claim that the public website already serves this model.

The underlying local records are the checkpoint reports in artifacts/checkpoints, the eval-*-validation.json and eval-80000-2x3-final.json files, the stage logs, and config.json. The first three early points above are retained from the previous published version of this article. These measurements describe a simulated 2048 agent under specified seeds; they do not claim that a public user will see the same rate in any single game.

RESEARCH & FIELD NOTES/

2048 强化学习模型发布报告:80,000 局与三组评估

80,000 局模型在固定评估、验证和首次最终评估中分别达到 184/200、459/500、917/1000。本文记录训练流程、检查点、失败实验、特征扩展对照与完整评估图表。

当前研究检查点沿一条训练分支自玩了 80,000 局。固定 200 局中 184 局合成 2048(92.0%),另一组 500 局验证中 459 局(91.8%),此前未参与调参、首次启用的最终 1,000 局中 917 局(91.7%)。三组模拟评估均超过预定的 90% 门槛。

2048 网页游戏是单独部署的演示。截至本报告截点,线上仍使用较早的 24,000 局模型;本文的 80,000 局检查点和结果来自本地训练环境,不能当成线上 AI 的实测能力。

训练的到底是什么

网页游戏和训练程序共用一套 4 × 4 Python 规则。合法移动先滑动、合并并计分,再随机生成新方块:90% 为 2,10% 为 4。合成 2048 后游戏不会自动结束,只有无合法移动才结束。因此本文的指标是模拟对局中曾经合成 2048 的比例,不是结束时停在 2048 的比例。

模型先试算每个合法方向,得到“合并滑动之后、随机新方块出现之前”的棋盘,称为动作后状态。它选择:

方向价值 = 本步合并得分 + 动作后棋盘的预计未来得分

预计未来得分来自 N-tuple 查表模型:把多个重叠棋盘区域对应的权重相加。最初使用 17 组特征,包括 4 行、4 列和 9 个 2 × 2 区域。后来的实验增加 6 个 2 × 3、6 个 3 × 2 区域,总共 29 组。这里没有神经网络,也没有人类棋谱训练集。

每局自玩都从新棋盘开始。模型选方向、拿到合并分数、观察下一步,并用时序差分方法把上一个动作后状态的估值向“下一步奖励+新状态估值”修正;游戏结束时,最后一个动作后状态的未来价值修正为零。当前奖励是累积分数,没有“第一次合成 2048”的额外奖励。前 17,000 局的学习率为 0.1,并有 10% 的随机探索;17,000 至 24,000 局将探索设为零。之后各轮保留权重继续训练。当前学习率 0.05、折扣系数 1.0、探索率 0,每段新增 7,000 局。

新棋盘 → 列出合法移动 → 计算动作后状态 → 选“得分+估值”
       → 生成随机方块 → 下一步 → 时序差分更新 → 本局结束

怎样判断模型有没有变好

下方各检查点都使用同一组固定种子评估 200 局。31,000 局起,还用另一组种子验证 500 局。两组都被反复查看、用于比较模型,因此都不算全新保留集。另有 1,000 局此前未使用的最终评估,在 80k 检查点通过前两关之后首次运行;现在这组种子已经使用,不能再称为全新保留集。三组预定达标线分别是 180/200、450/500、900/1,000。

训练日志中“最近 500 局”的数值只能观察训练过程:这 500 局里权重一直在变化,而且本身参与了训练。例如,80,000 局阶段最后 500 局训练窗口为 92.2%,落盘模型的固定评估为 92.0%,另一组验证为 91.8%,最终评估为 91.7%。不能用训练窗口冒充正式通过率。

图 1:固定 200 局评估的变化

每个方块约表示 3.3 个百分点,右侧局数和比例是精确值。3k、10k、17k 的数字来自此前已发布的训练文章;24k 起有保存的检查点报告。

3k         █                                4/200  2.0%
10k        ██                              11/200  5.5%
17k        ██                              11/200  5.5%
24k        ███████████████                101/200  50.5%
31k        ██████████████████████         144/200  72.0%
38k        ██████████████████████         145/200  72.5%
45k        ███████████████████████        150/200  75.0%
52k        ███████████████████████        152/200  76.0%
59k        █████████████████████████      169/200  84.5%
66k base   ███████████████████████        156/200  78.0%
66k 2x3    ███████████████████████████    180/200  90.0%
73k 2x3    ███████████████████████████    179/200  89.5%
80k 2x3    ████████████████████████████   184/200  92.0%

早期 3,000 到 17,000 局进展很慢,之后从 11/200 跳到 24,000 局时的 101/200。但这一段同时增加了 7,000 局训练,并把随机探索设为零,现有数据不能把增益单独归给其中一个变化。

图 2:另一组种子的 500 局验证

该验证从 31k 检查点开始,反复用于模型比较;它与仍未使用的最终保留集不同。

31k        ████████████████████           341/500  68.2%
38k        █████████████████████          349/500  69.8%
45k        ███████████████████████        380/500  76.0%
52k        ███████████████████████        389/500  77.8%
59k        █████████████████████████      422/500  84.4%
66k base   ████████████████████████       402/500  80.4%
66k 2x3    ███████████████████████████    444/500  88.8%
73k 2x3    ████████████████████████████   461/500  92.2%
80k 2x3    ████████████████████████████   459/500  91.8%

扩展特征模型从 66k 训练到 73k,验证由 444/500 提升至 461/500,但固定评估从 180/200 回落到 179/200。继续训练到 80k 后,固定评估回升至 184/200,验证保持在门槛之上,为 459/500。这是两组监控指标首次在同一检查点同时达标。

图 3:80k 检查点的三道发布门槛

三行使用不同种子。只有前两组达标后,才首次运行最终 1,000 局。

固定 200 局   ████████████████████████████   184/200   92.0%  通过
验证 500 局   ████████████████████████████   459/500   91.8%  通过
最终 1000 局  ████████████████████████████   917/1000  91.7%  通过

训练时碰到的问题,以及实际做了什么

只增加局数并不总是更好。 从 59k 的模型继续按原来 17 组特征训练 7,000 局,固定评估由 169/200 跌至 156/200,验证由 422/500 跌至 402/500。我们保留了两个检查点,不再把累计局数当成质量保证。回落的原因还没有被单独识别;这是两组既定种子上的观察结果。

直接在推理阶段套旋转镜像平均失败。 对 52k 模型在训练完成后平均八种旋转、镜像棋盘的估值,固定评估只有 9/200(4.5%),而原模型为 152/200(76.0%),因此没有采用这种后处理。一个可能解释是:原查表权重按原始方向学习,事后平均改变了其含义;这只是待验证的解释,不是已证实的故障原因。

从同一起点对照扩大特征。 我们复制 59k 检查点,保留原有 17 组权重,把新增 12 组 2 × 3、3 × 2 特征初始化为零。训练前重测固定 200 局,结果仍是 169/200。随后,两条分支使用相同训练种子、学习率和新增 7,000 局;主要差别就是棋盘特征集合。

图 4:同为 66k 的两条分支

F 表示固定 200 局,V 表示另一组 500 局。两条分支都从同一个 59k 模型开始。

66k base F    ███████████████████████        156/200  78.0%
66k 2x3 F     ███████████████████████████    180/200  90.0%
66k base V    ████████████████████████       402/500  80.4%
66k 2x3 V     ███████████████████████████    444/500  88.8%

扩展特征让同为 66k 的固定结果由 156/200 提升到 180/200,验证由 402/500 提升到 444/500。这支持在本次对照条件下采用更大的棋盘表示,但不能证明它在所有未见过的随机对局中都一定更好。

得分与合成 2048 的比例不一定同步。 扩展模型从 66k 到 73k,固定评估平均得分由 51,384.50 增至 53,215.08,合成 2048 的局数却少了 1 局。当前时序差分训练优化的是得分,而验收标准是能否合成目标方块。如果这种错位持续,调整奖励值得做新的对照实验;目前尚未加入新奖励。

图 5:固定评估的平均得分

长度按 60,000 分绘制;图中四舍五入为整数,保存的报告里有精确值。

24k        ████████████                   24,827
31k        ████████████████               31,673
38k        █████████████████              33,521
45k        ██████████████████             36,867
52k        ████████████████████           39,025
59k        █████████████████████          41,990
66k base   ███████████████████            37,751
66k 2x3    ██████████████████████████     51,385
73k 2x3    ███████████████████████████    53,215
80k 2x3    █████████████████████████████  57,951

更大的棋盘特征也增加模型体积:17 组特征的 59k 压缩模型约 3.5 MB,扩展为 29 组后,66k 约 30.1 MB、73k 约 39.2 MB、80k 约 46.1 MB。我们尚未测量部署环境的推理延迟差异。

59k / 17    ██                         3.5 MB
66k / 29    ████████████████████       30.1 MB
73k / 29    █████████████████████████  39.2 MB
80k / 29    █████████████████████████████  46.1 MB

当前状态和下一步

当前研究模型是 80k 的扩展特征分支。若把分叉出去的原特征对照也算上,训练器实际运行了约 87,000 局自玩:当前模型的传承路径为 80,000 局,另有从 59k 分叉的 7,000 局普通特征训练。不能把这 87,000 局说成一个模型连续训练的局数。

本地 80k 模型通过了三组预定的 90% 门槛:固定 184/200、验证 459/500、首次最终评估 917/1000。 自动训练已在该检查点停止。这说明指定模拟规则与种子集合下的评估达标,不保证每一局都成功,也不代表公网游戏已换用这个模型。

数据依据是本地 artifacts/checkpoints 下的报告、eval-*-validation.json、eval-80000-2x3-final.json、各阶段训练日志与 config.json。最早的三个节点来自本文原版。所有比例都只描述指定随机种子下的模拟评估,不保证任何一次公网游戏也会得到同样结果。

END OF NOTEContinue reading ↗