<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
    <channel>
        <title><![CDATA[Niina's Blog]]></title>
        <description><![CDATA[Niina's Blog]]></description>
        <link>https://blog.niina.fun</link>
        <generator>RSS for Node</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:18:44 GMT</lastBuildDate>
        <atom:link href="https://blog.niina.fun/rss.xml" rel="self" type="application/rss+xml"/>
        <item>
            <title><![CDATA[怎么让网络变得干净]]></title>
            <description><![CDATA[一个干净的网络，让人省心的使用服务。]]></description>
            <link>https://blog.niina.fun/posts/network-clean</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/network-clean</guid>
            <pubDate>Sun, 11 Oct 2026 10:40:02 GMT</pubDate>
            <content:encoded><![CDATA[<p>简单说，就是如何配置正确的 DNS、代理分流。</p><p>目前（2026年）的网络代理软件非常成熟，正确配置之后不管是基于 TCP 或 UDP 的流量都能正确代理流量。关键在于还有哪些需要做的。</p><h2 id="dns">DNS</h2><p>检测 DNS 的工具</p><p>推荐 <a href="https://ip.sb/" rel="nofollow">ip.sb</a> ，以及 <a href="https://browserleaks.com/" rel="nofollow">broswerleaks.com</a>。 主要看 Location 是否一致，ISP 是否如分流策略一致。记得同时检测 WebRTC，指纹等等。</p><h2 id="分流对象">分流对象</h2><p>需要注意 <a href="https://www.datadoghq.com/" rel="nofollow">datadoghq.com </a>一定需要经过代理，一些应用会通过其探测您的网络环境。</p><p>对特定应用使用前，可以先少量使用确认其网络连接是否遵循分流规则，是否存在漏网之鱼等等。避免网络环境没有按照预料的进行。</p><note><p>订阅的机场提供的配置，不一定覆盖所使用的服务，有些甚至 DNS 配置都很有问题。需要自己通过拓展脚本等的方式，确保配置正确。</p></note><h2 id="实际案例">实际案例</h2><ol><li>Google 转区<br />一开始在TUN+规则模式下发的转区请求（从cn转到ja），都没过（基本两小时以内回复）。最后改成TUN+全局就转成功了。</li><li>Claude 使用<br />把 Claude 相关域名以及 datadog 都 <code>DOMAIN-SUFFIX</code> 写进代理配置后，应该可以在 TUN+rules 下随意注册使用 Claude free 服务。<br />新账号付费的时候会弹 KYC，准备好相关材料，如果是在没有可以用的资料，可以 AI Gen 用来过 KYC。Claude本身是不知道 KYC 的具体资料的。</li></ol>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[搬运] 不敢投入的时代]]></title>
            <description><![CDATA[为什么我们现在越来越不敢投入了。很多时候我们以为的选择，比如你有两个工作offer，现在选择好像是我到底应该考公还是考编。明明这两个选项都不是真正的选项，因为你还没有开始，也不知道最后能不能成。这个时候我们就已经开始犹豫了。]]></description>
            <link>https://blog.niina.fun/posts/the-age-of-hesitation</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/the-age-of-hesitation</guid>
            <pubDate>Sat, 10 Oct 2026 02:44:12 GMT</pubDate>
            <content:encoded><![CDATA[<note><p>本文来自视频<a href="https://www.bilibili.com/video/BV1vnVq6hEFA" rel="nofollow">真正毁掉一个人的，不是不努力，而是不肯狼狈</a>的字幕转稿。</p></note><h2 id="为什么人们越来越不敢冒险投入了">为什么人们越来越不敢冒险/投入了</h2><p>朋友们，今天我想讨论一个话题：为什么我们现在越来越不敢投入了。很多时候我们以为的选择，比如你有两个工作 offer，现在选择好像是我到底应该考公还是考编。明明这两个选项都不是真正的选项，因为你还没有开始，也不知道最后能不能成。这个时候我们就已经开始犹豫了。</p><h3 id="人们在害怕失败后的狼狈">人们在害怕失败后的狼狈</h3><p>我觉得，比起这个表面上看到的“有可能是我还没有想好哪个更适合我”，比起这样一个表面问题，更多的是我们不敢去投入，因为我们害怕自己如果投入了，最后没有结果，是不是看起来就很狼狈，不那么漂亮光鲜。说到底，很多人追求的不是成功本身，而是成功的社会符号。我们拥有的不只是金钱资本，还有文化、社会、象征资本。比如我们知道，房子、头衔、学历、民企经历都属于象征资本的一部分，它可以快速向周围人传递一个信号：这个人过得还不错。而问题在于，信号和真实的现实能力之间并不是同步增长的。</p><p>从进化心理学角度来看，我们长期生活在一个小规模群体里面，在一个几十人的部落里面，谁拥有资源、地位，谁就更受到尊重。这些信息都直接关系到生存和繁衍，因此我们的大脑天然就会关注这个社会排序。这是因为我们是 social animal，这是非常重要的一点。猫或者猴子，它们都有上下等级，这是群居动物的一个特点。当互联网出现之后，事情就发生了变化。现在一个普通人可以和什么比？他可以和这些 Twitter 或全球范围内最顶尖的人进行比较。</p><p>人会通过和他人的比较来评价自己。这是为什么说 Facebook 发展的过程当中，伴随着青少年自杀率的上升。因为你突然发现，你都还是个孩子，你就把自己和全球最厉害、过得最好那一批人进行对比，你是不是突然一下觉得，这个生活还有什么意思。</p><h3 id="他人走过而自己未走过的路让人混乱">他人走过而自己未走过的路让人混乱</h3><p>另一个问题，就是我们对于损失极为敏感。当一个人看到同龄人买房、升职、加薪的时候，大脑就会把它理解成我正在失去什么机会。往往真实的情况只是他人在走他自己的路而已。别人得到，不是我们所失去的。</p><p>另外，有一个概念叫 delayed gratification，延迟回报。这个长期目标的实现，往往需要承受很大程度上的短期不适。做任何事情几乎都是这样的，因为一个复利系统都具有一个共同特点：你长期的增长很缓慢，后期增长就是陡峭的。而芒格也说过，伟大成果往往来自于长时间积累后的一个非线性增长。而从数学上看，复利曲线最容易让人产生幻觉，因为你前半场几乎看不到任何变化。大多数人离场时间都是发生在复利开始加速之前。因为很多人都想要说，网上有这么多年轻漂亮、名校毕业、一毕业就年薪百万的 winner，为什么我不可以试？为什么我要去受这份苦？为什么我就不能 have it all？</p><h3 id="身份认同陷阱">身份认同陷阱</h3><p>另外一件，是关于身份认同陷阱，Identity trap。很多人会逐渐把外部标签和自我价值做一个绑定。比如我喜欢说我是什么什么校人，我是某场的谁谁谁，我是管理者，我拥有房产。当这些标签都变成自我认同的核心的时候，我们的决策方法、方式就会发生变化，关注点就开始从成长转向为维护形象。因此就会产生大量的高成本行为，比如你继续待在一个不喜欢的单位，继续不合适自己的生活方式，拒绝从头学习，回避会暴露短板的环境。那么这一句决策能够维护当时当前的身份，却会削弱你未来的成长。</p><h2 id="价值错位">价值错位</h2><p>那么更多的人越来越强调一个词叫 OPTIALITY，选择权。复杂世界里面最大的优势之一，就是来自于拥有更多的选择。选择权来自于什么？技能、健康、人际网络、现金储备、持续学习的能力、跨领域牵引的能力。这些东西，你看看它和我们能够迅速识别的标签，有一些差距，或者说是它是两套系统。它没有这个房子、职位和你的 title 那么直观，但它却决定了未来 20 年的行动空间。发现了吗？这些东西它和我们所以为的选择不是同一套东西。</p><p>我们以为的选择，是我应该考这个考研，还是应该考公？我是应该出国，还是应该留在父母身边？它不是这种选择，而是你能够是不是给自己带来足够的冗余，在各个方向上让自己有更多的选择空间、有更大自由度。从这个角度来看，就会发现一个很有趣的转变：年轻的时候，我们可能会问，我现在看起来成功吗？随着你的阅历增加，问题逐渐变成，我是否拥有持续创造价值的能力。</p><p>前一个问题，关注的是展示；后一个问题，关注的是生存。前一个问题、后一个问题，它依赖的是自身能力。因此，人生当中最重要的区别，不在于你体面与否，关键在于一个人到底是在积累资产，还是在维护形象。</p><p>这里的资产，不光是我们理解物质上的资产，而是知识、技能、关系、创造力、健康以及信任。这些资产的增长速度往往都很慢，因此会让人觉得有些笨拙，有些迟缓，有些不像成功人士，不像那些看起来一开始就赢在起跑线上的人。</p><p>但是，很多长期卓越的人，他的发展轨迹都呈现类似特征：他们把大量时间投入到了外界难以评价的积累之中，也就是所说的这些部分。而多年以后，当人们看到那一刻，他突然说，我就知道这个人一定会成功的。你不知道，你只是现在知道了。</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[为什么模型上线后，体感像是“降智”了？]]></title>
            <description><![CDATA[权重与 KV cache 的量化都能降低 serving 成本，也可能改变输出；但体感变弱不足以证明服务后来切换了精度。]]></description>
            <link>https://blog.niina.fun/posts/llm</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/llm</guid>
            <pubDate>Tue, 29 Sep 2026 12:44:08 GMT</pubDate>
            <content:encoded><![CDATA[<p>如果你觉得某个模型刚上线时表现更好，但过一段时间后，同一个模型名下的回答却像变迟钝了，serving 侧的量化值得排查。量化能降低显存占用和推理成本，也可能改变模型输出。</p><h2 id="checkpoint-与-serving-配置">Checkpoint 与 serving 配置</h2><p>仅知道训练得到的 checkpoint，并不能判断线上服务是否直接按 checkpoint 保存时的精度运行。Serving 配置涉及模型计算所用的 <code>dtype</code>、权重量化方案和 KV cache 的存储 <code>dtype</code> 等方面；这些配置能否组合使用，以及量化是否同时覆盖权重与激活，取决于量化方案、模型和硬件。</p><p>以 vLLM 为例，在线量化可以在加载 BF16/FP16 模型时把 Linear、MoE 权重转换为 FP8；KV cache 也有独立的 <code>--kv-cache-dtype</code> 设置。<a href="https://docs.vllm.ai/en/latest/features/quantization/online/" rel="nofollow">vLLM 在线量化文档</a>、<a href="https://docs.vllm.ai/en/latest/cli/serve/" rel="nofollow">serving 参数文档</a> 展示了这条技术路径，但没有说明任何特定厂商怎样部署。</p><p>因此，量化不一定以替换 checkpoint 文件的方式发生；服务可以从上线之初就使用量化配置，也可以之后调整配置。对外模型名本身无法告诉用户后端使用的精确 revision、路由或数值格式；“上线时直接跑训练 checkpoint，后来才换成量化 checkpoint”只是众多可能部署过程中的一种。<a href="https://docs.vllm.ai/en/latest/api/vllm/config/model/" rel="nofollow">vLLM 模型配置文档</a> 也分别配置了对外服务名与模型 revision。</p><h2 id="权重量化与-kv-cache-量化">权重量化与 KV cache 量化</h2><p>权重量化用较少位元表示参数，能压缩权重显存占用，并减少解码时读取权重的内存流量；端到端延迟是否下降，取决于量化内核和硬件能否高效执行。数值近似会改变前向计算得到的 logits。如果最高分 token 与其他候选 token 接近，细微误差就可能改变下一个 token，自回归解码随后会沿着新的文本继续。输出因此可能变化，任务质量却不必然明显下降。</p><p><a href="https://arxiv.org/html/2210.17323v2" rel="nofollow">GPTQ（GPT 模型后训练量化方法）论文</a> 在 OPT（Open Pre-trained Transformer）、BLOOM（BigScience Large Open-science Open-access Multilingual Language Model）等模型及 perplexity、zero-shot 基准上评测了离线权重量化。</p><p>例如，OPT-175B 在 WikiText2 上的 perplexity，FP16 基线为 8.34，4-bit GPTQ 为 8.37。该结果表明这一方案在这个指标上保留了质量，但不能保证其他量化方法、模型或线上对话也有相同表现。论文报告的速度收益依赖定制 GPU kernel 和减少内存搬运的实现。</p><p>权重之外，推理还会保存已处理 token 的 key/value，供后续 token 生成时计算 attention。活动序列越长、并发请求越多，KV cache 通常占用越多显存，因此压缩它能腾出并发空间。KV cache 精度与权重精度是不同的配置维度，调整 KV cache 精度也可能影响回答。</p><p><a href="https://arxiv.org/html/2402.02750v2" rel="nofollow">KIVI（KV cache 非对称量化方法）论文</a> 给出了一个 2-bit KV cache 方案：key 按 channel 量化，value 按 token 量化，并为近期 token 保留全精度残余区。它在 Llama、Mistral 的若干基准上报告较小的质量下降，也评测了 LongBench 长上下文任务；论文同时指出，Falcon 的多查询注意力让原始 KV cache 较小，2-bit 方案在部分任务上可能明显退化。</p><p>论文摘要报告，Llama-2-7B 使用 KIVI 时的峰值总显存约为 FP16 基线的 1/2.6（统计口径包含模型权重）。</p><p>第 4.2.4 节另报告一组吞吐实验：研究者在单张 80GB A100 上使用 ShareGPT 衍生负载，逐步增大 batch 至显存耗尽。以 FP16 基线为比较对象、显存上限相近时，KIVI 可容纳最高 4 倍 batch size，吞吐为基线的 2.35 到 3.47 倍。结果受论文所用模型、硬件和负载限制，不能直接外推为生产服务的速度保证。量化效果取决于方案、模型、任务和运行条件；长上下文质量也不能只凭位宽判断。</p><h2 id="怎样判断服务是否改过精度">怎样判断服务是否改过精度</h2><p>运营者采用低精度配置有成本动机：较小的权重与 cache 能释放显存，有机会容纳更多并发请求。但研究论文和开源引擎文档证明的是技术可行性，不是某家服务实际做过哪次变更。同样的体感还可能来自模型 revision 或路由、系统提示词、采样参数和推理预算的变化。</p><p>用户侧要判断服务是否变过，需要保留上线初期的同题输出作基线，固定可见的模型版本、输入、采样参数和工具设置，再重复测试短、长上下文任务并盲评。没有历史基线，只能测量当前表现；如果系统提示词、路由或模型 revision 不公开，前后结果的变化也无法定位到量化。</p><p>能访问 serving 配置时，可以固定 checkpoint <code>revision</code>、代码与量化内核、硬件、tokenizer、模板和采样配置，分别比较权重量化与 <code>kv_cache_dtype</code>。两者可能相互作用，因此还可以补齐四种配置：都不量化、仅量化权重、仅量化 cache、两者都量化。若权重量化方案同时改变激活精度，测到的是整套方案的效果；要单独归因权重，还需固定激活精度或选用 weight-only 方案。</p><p>因此，量化是“体感变弱”的一种可检验解释。要断定某项服务上线后才切换了精度，还需要部署变更记录或能够隔离变量的服务端对照。</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[译] 深度强化学习入门（三）：策略优化入门]]></title>
            <description><![CDATA[OpenAI Spinning Up 教程第三部分的全文翻译：推导最简单的策略梯度、log-derivative trick、EGLP 引理、reward-to-go 与基线，配 PyTorch 示例代码。]]></description>
            <link>https://blog.niina.fun/posts/spinning-up-policy-optimization</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/spinning-up-policy-optimization</guid>
            <pubDate>Mon, 28 Sep 2026 08:49:25 GMT</pubDate>
            <content:encoded><![CDATA[<note title="译者说明"><p>本文是 OpenAI <a href="https://spinningup.openai.com/en/latest/" rel="nofollow">Spinning Up in Deep RL</a> 教程第三部分 <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html" rel="nofollow">Part 3: Intro to Policy Optimization</a> 的全文翻译，原仓库采用 <a href="https://github.com/openai/spinningup/blob/master/LICENSE" rel="nofollow">MIT 协议</a>。术语约定与<a href="/posts/spinning-up-rl-intro">第一篇</a>、<a href="/posts/spinning-up-rl-intro-2">第二篇</a>一致；文中提到的行号指 GitHub 上对应源文件的行号；选读证明与旧版 Tensorflow 内容链接到官方站点。</p></note><p>本节讨论策略优化算法的数学基础，并把内容对接到示例代码上。我们会讲<strong>策略梯度</strong>（policy gradient）理论里的三个关键结果：</p><ul><li>描述「策略性能对策略参数的梯度」的<a href="#%E6%8E%A8%E5%AF%BC%E6%9C%80%E7%AE%80%E5%8D%95%E7%9A%84%E7%AD%96%E7%95%A5%E6%A2%AF%E5%BA%A6">最简单的方程</a>；</li><li>一条允许我们从该表达式里<a href="#%E5%88%AB%E8%AE%A9%E8%BF%87%E5%8E%BB%E5%B9%B2%E6%89%B0%E4%BD%A0">去掉无用项</a>的规则；</li><li>一条允许我们<a href="#%E7%AD%96%E7%95%A5%E6%A2%AF%E5%BA%A6%E4%B8%AD%E7%9A%84%E5%9F%BA%E7%BA%BF">加入有用项</a>的规则。</li></ul><p>最后把这三个结果拼在一起，给出策略梯度的优势函数形式，也就是我们的 <a href="https://spinningup.openai.com/en/latest/algorithms/vpg.html" rel="nofollow">Vanilla Policy Gradient</a>（VPG）实现里用的那个版本。</p><h2 id="推导最简单的策略梯度">推导最简单的策略梯度</h2><p>这里考虑随机型参数化策略 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_{\theta}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>，目标是最大化期望回报 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></munder></mi><mo stretchy="false">[</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">J(\pi_{\theta}) = \underset{\tau \sim \pi_{\theta}}{\mathbb{E}}[R(\tau)]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.5559em;vertical-align:-0.8059em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8059em;"><span></span></span></span></span></span></span><span className="mopen">[</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)]</span></span></span></span>。为推导方便，取 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">R(\tau)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span></span></span></span> 为<a href="/posts/spinning-up-rl-intro#%E5%A5%96%E5%8A%B1%E4%B8%8E%E5%9B%9E%E6%8A%A5">有限视界不折扣回报</a>；换成无限视界折扣回报的情形，推导几乎完全一样。</p><p>我们想用梯度上升来优化策略，比如：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>θ</mi><mrow><mi>k</mi><mo>+</mo><mn>1</mn></mrow></msub><mo>=</mo><msub><mi>θ</mi><mi>k</mi></msub><mo>+</mo><mi>α</mi><msub><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo><mo fence="true">∣</mo></mrow><msub><mi>θ</mi><mi>k</mi></msub></msub></mrow><annotation encoding="application/x-tex">\theta_{k+1} = \theta_k + \alpha \left. \nabla_{\theta} J(\pi_{\theta}) \right|_{\theta_k}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.9028em;vertical-align:-0.2083em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1.1556em;vertical-align:-0.4056em;"></span><span className="mord,mathnormal" style="margin-right:0.0037em;">α</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="minner"><span className="mopen,nulldelimiter"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">∣</span></span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1864em;"><span style="top:-2.4003em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0278em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.4056em;"><span></span></span></span></span></span></span></span></span></span></span><p>策略性能的梯度 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\nabla_{\theta} J(\pi_{\theta})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 叫作<strong>策略梯度</strong>，按这个方式优化策略的算法叫<strong>策略梯度算法</strong>（例子有 Vanilla Policy Gradient 和 TRPO。PPO 常被叫作策略梯度算法，不过严格说稍有不准）。</p><p>要真正用上这个算法，得先有一个能数值计算的策略梯度表达式。这分两步：1）推导策略性能的解析梯度，它最后会化成一个期望的形式；2）给这个期望做样本估计，用有限步「智能体-环境交互」的数据就能算出来。</p><p>本小节先找出这个表达式最简单的形式；后面几小节再展示如何在最简单的形式上改进，得到标准策略梯度实现里真正使用的版本。</p><p>先摆出几个对推导解析梯度有用的事实。</p><p><strong>1. 轨迹的概率</strong>。动作来自 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_{\theta}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 时，轨迹 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>τ</mi><mo>=</mo><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><msub><mi>a</mi><mn>0</mn></msub><mo separator="true">,</mo><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mo separator="true">,</mo><msub><mi>s</mi><mrow><mi>T</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\tau = (s_0, a_0, ..., s_{T+1})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord">...</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 的概率是</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi>ρ</mi><mn>0</mn></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo stretchy="false">)</mo><munderover><mo>∏</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><mi>P</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">P(\tau|\theta) = \rho_0 (s_0) \prod_{t=0}^{T} P(s_{t+1}|s_t, a_t) \pi_{\theta}(a_t |s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.0954em;vertical-align:-1.2671em;"></span><span className="mord"><span className="mord,mathnormal">ρ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∏</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p><strong>2. 对数求导技巧（log-derivative trick）</strong>。它基于微积分里一条简单的规则：<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>log</mi><mo>⁡</mo><mi>x</mi></mrow><annotation encoding="application/x-tex">\log x</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">x</span></span></span></span> 对 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">x</span></span></span></span> 的导数是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi>x</mi></mrow><annotation encoding="application/x-tex">1/x</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">1/</span><span className="mord,mathnormal">x</span></span></span></span>。稍作变形、再与链式法则结合，得到：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\nabla_{\theta} P(\tau | \theta) = P(\tau | \theta) \nabla_{\theta} \log P(\tau | \theta)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span></span></span></span></span><p><strong>3. 轨迹的对数概率</strong>。轨迹的 log 概率就是</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>log</mi><mo>⁡</mo><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><mi>log</mi><mo>⁡</mo><msub><mi>ρ</mi><mn>0</mn></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo stretchy="false">)</mo><mo>+</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><mo fence="false" stretchy="true" minsize="2.4em" maxsize="2.4em">(</mo><mi>log</mi><mo>⁡</mo><mi>P</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo>+</mo><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo fence="false" stretchy="true" minsize="2.4em" maxsize="2.4em">)</mo></mrow><annotation encoding="application/x-tex">\log P(\tau|\theta) = \log \rho_0 (s_0) + \sum_{t=0}^{T} \bigg( \log P(s_{t+1}|s_t, a_t)  + \log \pi_{\theta}(a_t |s_t)\bigg)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">ρ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:3.0954em;vertical-align:-1.2671em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="delimsizing,size3">(</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:2.4em;vertical-align:-0.95em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord"><span className="delimsizing,size3">)</span></span></span></span></span></span><p><strong>4. 环境函数的梯度</strong>。环境与 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi></mrow><annotation encoding="application/x-tex">\theta</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span></span></span></span> 无关，所以 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ρ</mi><mn>0</mn></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\rho_0(s_0)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">ρ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>、<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">P(s_{t+1}|s_t, a_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 和 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">R(\tau)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span></span></span></span> 的梯度都是零。</p><p><strong>5. 轨迹的 grad-log-prob</strong>。于是轨迹对数概率的梯度为</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable rowspacing="0.25em" columnalign="right left" columnspacing="0em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><menclose notation="updiagonalstrike"><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>ρ</mi><mn>0</mn></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo stretchy="false">)</mo></mrow></menclose><mo>+</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><mo fence="false" stretchy="true" minsize="2.4em" maxsize="2.4em">(</mo><menclose notation="updiagonalstrike"><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><mi>P</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow></menclose><mo>+</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo fence="false" stretchy="true" minsize="2.4em" maxsize="2.4em">)</mo></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow></mstyle></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{aligned}
\nabla_{\theta} \log P(\tau | \theta) &amp;= \cancel{\nabla_{\theta} \log \rho_0 (s_0)} + \sum_{t=0}^{T} \bigg( \cancel{\nabla_{\theta} \log P(s_{t+1}|s_t, a_t)}  + \nabla_{\theta} \log \pi_{\theta}(a_t |s_t)\bigg) \\
&amp;= \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t |s_t)
\end{aligned}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:6.4909em;vertical-align:-2.9954em;"></span><span className="mord"><span className="mtable"><span className="col-align-r"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:3.4954em;"><span style="top:-5.4954em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span></span></span><span style="top:-2.1em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:2.9954em;"><span></span></span></span></span></span><span className="col-align-l"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:3.4954em;"><span style="top:-5.4954em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord,cancel-lap"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.75em;"><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="mord,cancel-pad"><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">ρ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span><span className="svg-align" style="top:-2.75em;"><span className="pstrut" style="height:3em;"></span><span style="height:1em;"><svg xmlns="http://www.w3.org/2000/svg" width="100%" height="1em"><line x1="0" y1="100%" x2="100%" y2="0" strokeWidth="0.046em"></line></svg></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.25em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="delimsizing,size3">(</span></span><span className="mord,cancel-lap"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.75em;"><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="mord,cancel-pad"><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span><span className="svg-align" style="top:-2.75em;"><span className="pstrut" style="height:3em;"></span><span style="height:1em;"><svg xmlns="http://www.w3.org/2000/svg" width="100%" height="1em"><line x1="0" y1="100%" x2="100%" y2="0" strokeWidth="0.046em"></line></svg></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.25em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord"><span className="delimsizing,size3">)</span></span></span></span><span style="top:-2.1em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:2.9954em;"><span></span></span></span></span></span></span></span></span></span></span></span><p>把这些拼起来，推出：</p><note title="基本策略梯度的推导"><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable rowspacing="0.25em" columnalign="right left right left" columnspacing="0em 1em 0em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></munder></mi><mo stretchy="false">[</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><msub><mo>∫</mo><mi>τ</mi></msub><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mtext>展开期望</mtext></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><msub><mo>∫</mo><mi>τ</mi></msub><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mtext>把梯度移入积分号内</mtext></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><msub><mo>∫</mo><mi>τ</mi></msub><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mtext>对数求导技巧</mtext></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>θ</mi><mo stretchy="false">)</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mtext>回到期望形式</mtext></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mo>∴</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mtext>代入轨迹的 grad-log-prob</mtext></mrow></mstyle></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{align*}
\nabla_{\theta} J(\pi_{\theta}) &amp;= \nabla_{\theta} \underset{\tau \sim \pi_{\theta}}{\mathbb{E}}[R(\tau)] &amp;&amp; \\
&amp;= \nabla_{\theta} \int_{\tau} P(\tau|\theta) R(\tau) &amp;&amp; \text{展开期望} \\
&amp;= \int_{\tau} \nabla_{\theta} P(\tau|\theta) R(\tau) &amp;&amp; \text{把梯度移入积分号内} \\
&amp;= \int_{\tau} P(\tau|\theta) \nabla_{\theta} \log P(\tau|\theta) R(\tau) &amp;&amp; \text{对数求导技巧} \\
&amp;= \underset{\tau \sim \pi_{\theta}}{\mathbb{E}}\left[\nabla_{\theta} \log P(\tau|\theta) R(\tau)\right] &amp;&amp; \text{回到期望形式} \\
\therefore \nabla_{\theta} J(\pi_{\theta}) &amp;= \underset{\tau \sim \pi_{\theta}}{\mathbb{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t |s_t) R(\tau)\right] &amp;&amp; \text{代入轨迹的 grad-log-prob}
\end{align*}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:14.703em;vertical-align:-7.1015em;"></span><span className="mord"><span className="mtable"><span className="col-align-r"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:7.6015em;"><span style="top:-10.5898em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span><span style="top:-8.124em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span><span style="top:-5.552em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span><span style="top:-2.9801em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span><span style="top:-0.9281em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span><span style="top:2.0061em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mrel,amsrm">∴</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:7.1015em;"><span></span></span></span></span></span><span className="col-align-l"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:7.6015em;"><span style="top:-10.5898em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8059em;"><span></span></span></span></span></span></span><span className="mopen">[</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)]</span></span></span><span style="top:-8.124em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop"><span className="mop,op-symbol,large-op" style="margin-right:0.4445em;position:relative;top:-0.0011em;">∫</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:-0.6106em;"><span style="top:-1.7881em;margin-left:-0.4445em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9119em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span></span></span><span style="top:-5.552em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mop"><span className="mop,op-symbol,large-op" style="margin-right:0.4445em;position:relative;top:-0.0011em;">∫</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:-0.6106em;"><span style="top:-1.7881em;margin-left:-0.4445em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9119em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span></span></span><span style="top:-2.9801em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mop"><span className="mop,op-symbol,large-op" style="margin-right:0.4445em;position:relative;top:-0.0011em;">∫</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:-0.6106em;"><span style="top:-1.7881em;margin-left:-0.4445em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9119em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span></span></span><span style="top:-0.9281em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8059em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">]</span></span></span></span><span style="top:2.0061em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8059em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size4">[</span></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size4">]</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:7.1015em;"><span></span></span></span></span></span><span className="arraycolsep" style="width:1em;"></span><span className="col-align-r"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:7.6015em;"><span style="top:-10.5898em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span><span style="top:-8.124em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span><span style="top:-5.552em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span><span style="top:-2.9801em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span><span style="top:-0.9281em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span><span style="top:2.0061em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:7.1015em;"><span></span></span></span></span></span><span className="col-align-l"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:7.6015em;"><span style="top:-10.5898em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span></span></span><span style="top:-8.124em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mord,text"><span className="mord,cjk_fallback">展开期望</span></span></span></span><span style="top:-5.552em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mord,text"><span className="mord,cjk_fallback">把梯度移入积分号内</span></span></span></span><span style="top:-2.9801em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mord,text"><span className="mord,cjk_fallback">对数求导技巧</span></span></span></span><span style="top:-0.9281em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mord,text"><span className="mord,cjk_fallback">回到期望形式</span></span></span></span><span style="top:2.0061em;"><span className="pstrut" style="height:3.8283em;"></span><span className="mord"><span className="mord"></span><span className="mord,text"><span className="mord,cjk_fallback">代入轨迹的</span><span className="mord"> grad-log-prob</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:7.1015em;"><span></span></span></span></span></span></span></span></span></span></span></span></note><p>这是一个期望，意味着可以用样本均值来估计它。收集一组轨迹 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="script">D</mi><mo>=</mo><mo stretchy="false">{</mo><msub><mi>τ</mi><mi>i</mi></msub><msub><mo stretchy="false">}</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mo separator="true">,</mo><mi>N</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\mathcal{D} = \{\tau_i\}_{i=1,...,N}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathcal" style="margin-right:0.0278em;">D</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.0361em;vertical-align:-0.2861em;"></span><span className="mopen">{</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose"><span className="mclose">}</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">i</span><span className="mrel,mtight">=</span><span className="mord,mtight">1</span><span className="mpunct,mtight">,</span><span className="mord,mtight">...</span><span className="mpunct,mtight">,</span><span className="mord,mathnormal,mtight" style="margin-right:0.109em;">N</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span></span></span></span>，每条轨迹都让智能体按策略 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_{\theta}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 在环境中行动得到，策略梯度就可以这样估计：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mover accent="true"><mi>g</mi><mo>^</mo></mover><mo>=</mo><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><mi mathvariant="script">D</mi><mi mathvariant="normal">∣</mi></mrow></mfrac><munder><mo>∑</mo><mrow><mi>τ</mi><mo>∈</mo><mi mathvariant="script">D</mi></mrow></munder><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\hat{g} = \frac{1}{|\mathcal{D}|} \sum_{\tau \in \mathcal{D}} \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t |s_t) R(\tau)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord,accent"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6944em;"><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="accent-body" style="left:-0.2222em;"><span className="mord">^</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1944em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.15em;vertical-align:-1.3217em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3214em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">∣</span><span className="mord,mathcal" style="margin-right:0.0278em;">D</span><span className="mord">∣</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">1</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.936em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.05em;"><span style="top:-1.8557em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∈</span><span className="mord,mathcal,mtight" style="margin-right:0.0278em;">D</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.3217em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span></span></span></span></span><p>其中 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∣</mi><mi mathvariant="script">D</mi><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">|\mathcal{D}|</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">∣</span><span className="mord,mathcal" style="margin-right:0.0278em;">D</span><span className="mord">∣</span></span></span></span> 是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="script">D</mi></mrow><annotation encoding="application/x-tex">\mathcal{D}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathcal" style="margin-right:0.0278em;">D</span></span></span></span> 里的轨迹条数（这里是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span>）。</p><p>最后这个式子就是我们想要的「可计算表达式」的最简单的版本。只要策略的表示方式让我们能算出 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>a</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\nabla_{\theta} \log \pi_{\theta}(a|s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">a</span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>，并且能把策略放进环境里跑、收集轨迹数据集，我们就能算出策略梯度、走一步更新。</p><h2 id="实现最简单的策略梯度">实现最简单的策略梯度</h2><p>这个最简单版本策略梯度算法的 PyTorch 短实现放在 <code>spinup/examples/pytorch/pg_math/1_simple_pg.py</code>（也可以<a href="https://github.com/openai/spinningup/blob/master/spinup/examples/pytorch/pg_math/1_simple_pg.py" rel="nofollow">在 GitHub 上查看</a>）。它只有 128 行，我们强烈建议通读一遍。这里不逐行讲整个文件，只挑几块重要的部分解释。</p><note title="你应该知道"><p>本节最初写的是 Tensorflow 例子，旧版 Tensorflow 的内容见<a href="https://spinningup.openai.com/en/latest/spinningup/extra_tf_pg_implementation.html#implementing-the-simplest-policy-gradient" rel="nofollow">这里</a>。</p></note><p><strong>1. 搭建策略网络。</strong></p><pre className="language-python3 shiki shiki-themes github-light github-dark" code="# 搭建策略网络的核心
logits_net = mlp(sizes=[obs_dim]+hidden_sizes+[n_acts])

# 构造计算动作分布的函数
def get_policy(obs):
    logits = logits_net(obs)
    return Categorical(logits=logits)

# 构造选动作的函数（输出整数动作，从策略中采样）
def get_action(obs):
    return get_policy(obs).sample().item()
" language="python3" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sJ8bj"># 搭建策略网络的核心
</span></span><span class="line" line="2"><span class="sVt8B">logits_net </span><span class="szBVR">=</span><span class="sVt8B"> mlp(</span><span class="s4XuR">sizes</span><span class="szBVR">=</span><span class="sVt8B">[obs_dim]</span><span class="szBVR">+</span><span class="sVt8B">hidden_sizes</span><span class="szBVR">+</span><span class="sVt8B">[n_acts])
</span></span><span class="line" line="3"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="4"><span class="sJ8bj"># 构造计算动作分布的函数
</span></span><span class="line" line="5"><span class="szBVR">def</span><span class="sScJk"> get_policy</span><span class="sVt8B">(obs):
</span></span><span class="line" line="6"><span class="sVt8B">    logits </span><span class="szBVR">=</span><span class="sVt8B"> logits_net(obs)
</span></span><span class="line" line="7"><span class="szBVR">    return</span><span class="sVt8B"> Categorical(</span><span class="s4XuR">logits</span><span class="szBVR">=</span><span class="sVt8B">logits)
</span></span><span class="line" line="8"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="9"><span class="sJ8bj"># 构造选动作的函数（输出整数动作，从策略中采样）
</span></span><span class="line" line="10"><span class="szBVR">def</span><span class="sScJk"> get_action</span><span class="sVt8B">(obs):
</span></span><span class="line" line="11"><span class="szBVR">    return</span><span class="sVt8B"> get_policy(obs).sample().item()
</span></span></code></pre><p>这一块搭建了前馈神经网络分类策略所需的模块和函数（想复习可以看<a href="/posts/spinning-up-rl-intro#%E9%9A%8F%E6%9C%BA%E5%9E%8B%E7%AD%96%E7%95%A5">第一篇的「随机型策略」一节</a>）。<code>logits_net</code> 模块的输出可以用来构造动作的对数概率和概率，<code>get_action</code> 函数根据 logits 算出的概率采样动作。（注意：这个 <code>get_action</code> 假定每次只传入一个 <code>obs</code>，因此也只输出一个整数动作。这就是它用 <code>.item()</code> 的原因，这个方法用来<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.item" rel="nofollow">取出只含单个元素的 Tensor 的内容</a>。）</p><p>这个例子里有大量工作是由源文件第 36 行的 <code>Categorical</code> 对象完成的。它是 PyTorch 的 <code>Distribution</code> 对象，封装了与概率分布相关的一些数学函数：既有从分布采样的方法（第 40 行用到了），也有计算给定样本对数概率的方法（后面会用到）。PyTorch 的 distributions 对 RL 非常有用，建议翻一翻<a href="https://pytorch.org/docs/stable/distributions.html" rel="nofollow">它们的文档</a>，感受一下怎么用。</p><note title="你应该知道"><p>友好提醒！说一个分类分布有「logits」，意思是各结果的概率由 logits 的 softmax 函数给出。也就是说，logits 为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mi>j</mi></msub></mrow><annotation encoding="application/x-tex">x_j</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7167em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0572em;">j</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span></span></span></span> 的分类分布下，动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi></mrow><annotation encoding="application/x-tex">j</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.854em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0572em;">j</span></span></span></span> 的概率是：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>p</mi><mi>j</mi></msub><mo>=</mo><mfrac><mrow><mi>exp</mi><mo>⁡</mo><mo stretchy="false">(</mo><msub><mi>x</mi><mi>j</mi></msub><mo stretchy="false">)</mo></mrow><mrow><munder><mo>∑</mo><mi>i</mi></munder><mi>exp</mi><mo>⁡</mo><mo stretchy="false">(</mo><msub><mi>x</mi><mi>i</mi></msub><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">p_j = \frac{\exp(x_j)}{\sum_{i} \exp(x_i)}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7167em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal">p</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0572em;">j</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.4127em;vertical-align:-0.9857em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.427em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mop"><span className="mop,op-symbol,small-op" style="position:relative;top:0em;">∑</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.162em;"><span style="top:-2.4003em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2997em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">exp</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mop">exp</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0572em;">j</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9857em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span></span></note><p><strong>2. 构造损失函数。</strong></p><pre className="language-python3 shiki shiki-themes github-light github-dark" code="# 构造损失函数：喂进合适的数据时，它的梯度就是策略梯度
def compute_loss(obs, act, weights):
    logp = get_policy(obs).log_prob(act)
    return -(logp * weights).mean()
" language="python3" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sJ8bj"># 构造损失函数：喂进合适的数据时，它的梯度就是策略梯度
</span></span><span class="line" line="2"><span class="szBVR">def</span><span class="sScJk"> compute_loss</span><span class="sVt8B">(obs, act, weights):
</span></span><span class="line" line="3"><span class="sVt8B">    logp </span><span class="szBVR">=</span><span class="sVt8B"> get_policy(obs).log_prob(act)
</span></span><span class="line" line="4"><span class="szBVR">    return</span><span class="szBVR"> -</span><span class="sVt8B">(logp </span><span class="szBVR">*</span><span class="sVt8B"> weights).mean()
</span></span></code></pre><p>这一块为策略梯度算法构造一个「损失」函数。喂进正确的数据时，这个损失的梯度就等于策略梯度。所谓正确的数据，是一组按当前策略行动时收集的（状态、动作、权重）三元组，其中状态-动作对的权重是它所属回合的回报。（后面的小节会说明，权重还可以换成别的值，同样正确。）</p><note title="你应该知道"><p>虽然我们把它叫作损失函数，但它<strong>不是</strong>监督学习里通常意义上的那种损失函数。它与标准损失函数有两点主要区别。</p><p><strong>1. 数据分布依赖参数</strong>。损失函数通常定义在一个与待优化参数无关的固定数据分布上。这里不是这样：数据必须从最新的策略里采样。</p><p><strong>2. 它不度量性能</strong>。损失函数通常度量我们关心的性能指标。这里我们关心的是期望回报 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">J(\pi_{\theta})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>，但这个「损失」函数连在期望意义上都不近似它。这个「损失」函数之所以有用，只是因为：在当前参数处、用当前参数生成的数据来评估时，它的梯度是性能的负梯度。</p><p>但走出梯度下降的第一步之后，它就与性能没有任何联系了。这意味着：对给定的一批数据，最小化这个「损失」函数<em>完全</em>不保证提升期望回报。你可以把这个损失压到 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>−</mo><mi mathvariant="normal">∞</mi></mrow><annotation encoding="application/x-tex">-\infty</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6667em;vertical-align:-0.0833em;"></span><span className="mord">−</span><span className="mord">∞</span></span></span></span>，而策略性能照样会崩；实际上通常就会崩。有的深度 RL 研究者会把这种结果描述成策略对这批数据「过拟合」了。这个说法有描述作用，但别按字面理解，因为它与泛化误差无关。</p><p>我们特意提这一点，是因为机器学习从业者习惯把损失函数当作训练中的有用信号：「损失降了，一切安好。」在策略梯度里，这个直觉是错的，你只该关心平均回报。损失函数没有任何意义。</p></note><note title="你应该知道"><p>这里构造 <code>logp</code> 张量的方式（调用 PyTorch <code>Categorical</code> 对象的 <code>log_prob</code> 方法）换成其他分布对象时，可能需要一些修改。</p><p>比如，如果你用的是 <a href="https://pytorch.org/docs/stable/distributions.html#normal" rel="nofollow">Normal 分布</a>（对应对角高斯策略），调用 <code>policy.log_prob(act)</code> 返回的张量会给每个向量型动作的每个分量各一个对数概率：输入形状 <code>(batch, act_dim)</code> 的张量，出来的还是 <code>(batch, act_dim)</code>，而构造 RL 损失需要的是 <code>(batch,)</code>。这时要把动作分量的对数概率加总，得到动作的对数概率，也就是计算：</p><pre className="language-python3 shiki shiki-themes github-light github-dark" code="logp = get_policy(obs).log_prob(act).sum(axis=-1)
" language="python3" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sVt8B">logp </span><span class="szBVR">=</span><span class="sVt8B"> get_policy(obs).log_prob(act).sum(</span><span class="s4XuR">axis</span><span class="szBVR">=-</span><span class="sj4cs">1</span><span class="sVt8B">)
</span></span></code></pre></note><p><strong>3. 跑一轮训练。</strong></p><pre className="language-python3 shiki shiki-themes github-light github-dark" code="# 用于训练策略
def train_one_epoch():
    # 建几个空列表，用于记录
    batch_obs = []          # 观测
    batch_acts = []         # 动作
    batch_weights = []      # 策略梯度里的 R(tau) 权重
    batch_rets = []         # 记录回合回报
    batch_lens = []         # 记录回合长度

    # 重置回合内变量
    obs = env.reset()       # 第一个观测来自起始分布
    done = False            # 环境给出的「回合结束」信号
    ep_rews = []            # 本回合累计的奖励列表

    # 每个 epoch 渲染第一个回合
    finished_rendering_this_epoch = False

    # 用当前策略在环境中行动，收集经验
    while True:

        # 渲染
        if (not finished_rendering_this_epoch) and render:
            env.render()

        # 保存观测
        batch_obs.append(obs.copy())

        # 在环境中行动
        act = get_action(torch.as_tensor(obs, dtype=torch.float32))
        obs, rew, done, _ = env.step(act)

        # 保存动作与奖励
        batch_acts.append(act)
        ep_rews.append(rew)

        if done:
            # 回合结束，记录该回合的信息
            ep_ret, ep_len = sum(ep_rews), len(ep_rews)
            batch_rets.append(ep_ret)
            batch_lens.append(ep_len)

            # 每个 logprob(a|s) 的权重是 R(tau)
            batch_weights += [ep_ret] * ep_len

            # 重置回合内变量
            obs, done, ep_rews = env.reset(), False, []

            # 这个 epoch 不再渲染
            finished_rendering_this_epoch = True

            # 经验够了就结束收集循环
            if len(batch_obs) &gt; batch_size:
                break

    # 做一次策略梯度更新
    optimizer.zero_grad()
    batch_loss = compute_loss(obs=torch.as_tensor(batch_obs, dtype=torch.float32),
                              act=torch.as_tensor(batch_acts, dtype=torch.int32),
                              weights=torch.as_tensor(batch_weights, dtype=torch.float32)
                              )
    batch_loss.backward()
    optimizer.step()
    return batch_loss, batch_rets, batch_lens
" language="python3" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sJ8bj"># 用于训练策略
</span></span><span class="line" line="2"><span class="szBVR">def</span><span class="sScJk"> train_one_epoch</span><span class="sVt8B">():
</span></span><span class="line" line="3"><span class="sJ8bj">    # 建几个空列表，用于记录
</span></span><span class="line" line="4"><span class="sVt8B">    batch_obs </span><span class="szBVR">=</span><span class="sVt8B"> []          </span><span class="sJ8bj"># 观测
</span></span><span class="line" line="5"><span class="sVt8B">    batch_acts </span><span class="szBVR">=</span><span class="sVt8B"> []         </span><span class="sJ8bj"># 动作
</span></span><span class="line" line="6"><span class="sVt8B">    batch_weights </span><span class="szBVR">=</span><span class="sVt8B"> []      </span><span class="sJ8bj"># 策略梯度里的 R(tau) 权重
</span></span><span class="line" line="7"><span class="sVt8B">    batch_rets </span><span class="szBVR">=</span><span class="sVt8B"> []         </span><span class="sJ8bj"># 记录回合回报
</span></span><span class="line" line="8"><span class="sVt8B">    batch_lens </span><span class="szBVR">=</span><span class="sVt8B"> []         </span><span class="sJ8bj"># 记录回合长度
</span></span><span class="line" line="9"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="10"><span class="sJ8bj">    # 重置回合内变量
</span></span><span class="line" line="11"><span class="sVt8B">    obs </span><span class="szBVR">=</span><span class="sVt8B"> env.reset()       </span><span class="sJ8bj"># 第一个观测来自起始分布
</span></span><span class="line" line="12"><span class="sVt8B">    done </span><span class="szBVR">=</span><span class="sj4cs"> False</span><span class="sJ8bj">            # 环境给出的「回合结束」信号
</span></span><span class="line" line="13"><span class="sVt8B">    ep_rews </span><span class="szBVR">=</span><span class="sVt8B"> []            </span><span class="sJ8bj"># 本回合累计的奖励列表
</span></span><span class="line" line="14"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="15"><span class="sJ8bj">    # 每个 epoch 渲染第一个回合
</span></span><span class="line" line="16"><span class="sVt8B">    finished_rendering_this_epoch </span><span class="szBVR">=</span><span class="sj4cs"> False
</span></span><span class="line" line="17"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="18"><span class="sJ8bj">    # 用当前策略在环境中行动，收集经验
</span></span><span class="line" line="19"><span class="szBVR">    while</span><span class="sj4cs"> True</span><span class="sVt8B">:
</span></span><span class="line" line="20"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="21"><span class="sJ8bj">        # 渲染
</span></span><span class="line" line="22"><span class="szBVR">        if</span><span class="sVt8B"> (</span><span class="szBVR">not</span><span class="sVt8B"> finished_rendering_this_epoch) </span><span class="szBVR">and</span><span class="sVt8B"> render:
</span></span><span class="line" line="23"><span class="sVt8B">            env.render()
</span></span><span class="line" line="24"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="25"><span class="sJ8bj">        # 保存观测
</span></span><span class="line" line="26"><span class="sVt8B">        batch_obs.append(obs.copy())
</span></span><span class="line" line="27"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="28"><span class="sJ8bj">        # 在环境中行动
</span></span><span class="line" line="29"><span class="sVt8B">        act </span><span class="szBVR">=</span><span class="sVt8B"> get_action(torch.as_tensor(obs, </span><span class="s4XuR">dtype</span><span class="szBVR">=</span><span class="sVt8B">torch.float32))
</span></span><span class="line" line="30"><span class="sVt8B">        obs, rew, done, _ </span><span class="szBVR">=</span><span class="sVt8B"> env.step(act)
</span></span><span class="line" line="31"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="32"><span class="sJ8bj">        # 保存动作与奖励
</span></span><span class="line" line="33"><span class="sVt8B">        batch_acts.append(act)
</span></span><span class="line" line="34"><span class="sVt8B">        ep_rews.append(rew)
</span></span><span class="line" line="35"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="36"><span class="szBVR">        if</span><span class="sVt8B"> done:
</span></span><span class="line" line="37"><span class="sJ8bj">            # 回合结束，记录该回合的信息
</span></span><span class="line" line="38"><span class="sVt8B">            ep_ret, ep_len </span><span class="szBVR">=</span><span class="sj4cs"> sum</span><span class="sVt8B">(ep_rews), </span><span class="sj4cs">len</span><span class="sVt8B">(ep_rews)
</span></span><span class="line" line="39"><span class="sVt8B">            batch_rets.append(ep_ret)
</span></span><span class="line" line="40"><span class="sVt8B">            batch_lens.append(ep_len)
</span></span><span class="line" line="41"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="42"><span class="sJ8bj">            # 每个 logprob(a|s) 的权重是 R(tau)
</span></span><span class="line" line="43"><span class="sVt8B">            batch_weights </span><span class="szBVR">+=</span><span class="sVt8B"> [ep_ret] </span><span class="szBVR">*</span><span class="sVt8B"> ep_len
</span></span><span class="line" line="44"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="45"><span class="sJ8bj">            # 重置回合内变量
</span></span><span class="line" line="46"><span class="sVt8B">            obs, done, ep_rews </span><span class="szBVR">=</span><span class="sVt8B"> env.reset(), </span><span class="sj4cs">False</span><span class="sVt8B">, []
</span></span><span class="line" line="47"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="48"><span class="sJ8bj">            # 这个 epoch 不再渲染
</span></span><span class="line" line="49"><span class="sVt8B">            finished_rendering_this_epoch </span><span class="szBVR">=</span><span class="sj4cs"> True
</span></span><span class="line" line="50"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="51"><span class="sJ8bj">            # 经验够了就结束收集循环
</span></span><span class="line" line="52"><span class="szBVR">            if</span><span class="sj4cs"> len</span><span class="sVt8B">(batch_obs) </span><span class="szBVR">&gt;</span><span class="sVt8B"> batch_size:
</span></span><span class="line" line="53"><span class="szBVR">                break
</span></span><span class="line" line="54"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="55"><span class="sJ8bj">    # 做一次策略梯度更新
</span></span><span class="line" line="56"><span class="sVt8B">    optimizer.zero_grad()
</span></span><span class="line" line="57"><span class="sVt8B">    batch_loss </span><span class="szBVR">=</span><span class="sVt8B"> compute_loss(</span><span class="s4XuR">obs</span><span class="szBVR">=</span><span class="sVt8B">torch.as_tensor(batch_obs, </span><span class="s4XuR">dtype</span><span class="szBVR">=</span><span class="sVt8B">torch.float32),
</span></span><span class="line" line="58"><span class="s4XuR">                              act</span><span class="szBVR">=</span><span class="sVt8B">torch.as_tensor(batch_acts, </span><span class="s4XuR">dtype</span><span class="szBVR">=</span><span class="sVt8B">torch.int32),
</span></span><span class="line" line="59"><span class="s4XuR">                              weights</span><span class="szBVR">=</span><span class="sVt8B">torch.as_tensor(batch_weights, </span><span class="s4XuR">dtype</span><span class="szBVR">=</span><span class="sVt8B">torch.float32)
</span></span><span class="line" line="60"><span class="sVt8B">                              )
</span></span><span class="line" line="61"><span class="sVt8B">    batch_loss.backward()
</span></span><span class="line" line="62"><span class="sVt8B">    optimizer.step()
</span></span><span class="line" line="63"><span class="szBVR">    return</span><span class="sVt8B"> batch_loss, batch_rets, batch_lens
</span></span></code></pre><p><code>train_one_epoch()</code> 函数跑一轮（epoch）策略梯度，我们把它定义为：1）经验收集步（源文件第 67 到 102 行），智能体用最新策略在环境中跑若干个回合；2）紧接一次策略梯度更新步（第 104 到 111 行）。算法的主循环就是反复调用 <code>train_one_epoch()</code>。</p><note title="你应该知道"><p>如果你还不熟悉 PyTorch 的优化流程，注意看第 104 到 111 行做一次梯度下降步的模式：先清空梯度缓存；然后计算损失函数；再对损失做一次反向传播，把新梯度累积进梯度缓存；最后让优化器走一步。</p></note><h2 id="eglp-引理">EGLP 引理</h2><p>本小节推导一个在策略梯度理论里到处会用到的中间结果，我们把它叫作<strong>期望 Grad-Log-Prob（EGLP）引理</strong>。</p><p><strong>EGLP 引理</strong>。设 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">P_{\theta}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 是随机变量 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">x</span></span></span></span> 上一个带参数的概率分布，那么：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>x</mi><mo>∼</mo><msub><mi>P</mi><mi>θ</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\underset{x \sim P_{\theta}}{\mathbb{E}}\left[\nabla_{\theta} \log P_{\theta}(x)\right] = 0</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.6002em;vertical-align:-0.8502em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.3557em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">x</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.1389em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8502em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">x</span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">]</span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">0</span></span></span></span></span><note title="证明"><p>回忆所有概率分布都是<em>归一化</em>的：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mo>∫</mo><mi>x</mi></msub><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\int_x P_{\theta}(x) = 1</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.2719em;vertical-align:-0.9119em;"></span><span className="mop"><span className="mop,op-symbol,large-op" style="margin-right:0.4445em;position:relative;top:-0.0011em;">∫</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:-0.6106em;"><span style="top:-1.7881em;margin-left:-0.4445em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">x</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9119em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">x</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">1</span></span></span></span></span><p>对归一化条件两边取梯度：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><msub><mo>∫</mo><mi>x</mi></msub><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mn>1</mn><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\nabla_{\theta} \int_x P_{\theta}(x) = \nabla_{\theta} 1 = 0</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.2719em;vertical-align:-0.9119em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop"><span className="mop,op-symbol,large-op" style="margin-right:0.4445em;position:relative;top:-0.0011em;">∫</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:-0.6106em;"><span style="top:-1.7881em;margin-left:-0.4445em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">x</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9119em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">x</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">1</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">0</span></span></span></span></span><p>用对数求导技巧得到：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable rowspacing="0.25em" columnalign="right left" columnspacing="0em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mn>0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><msub><mo>∫</mo><mi>x</mi></msub><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><msub><mo>∫</mo><mi>x</mi></msub><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><msub><mo>∫</mo><mi>x</mi></msub><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mo>∴</mo><mn>0</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>x</mi><mo>∼</mo><msub><mi>P</mi><mi>θ</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow></mstyle></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{aligned}
0 &amp;= \nabla_{\theta} \int_x P_{\theta}(x) \\
&amp;= \int_x \nabla_{\theta} P_{\theta}(x) \\
&amp;= \int_x P_{\theta}(x) \nabla_{\theta} \log P_{\theta}(x) \\
\therefore 0 &amp;= \underset{x \sim P_{\theta}}{\mathbb{E}}\left[\nabla_{\theta} \log P_{\theta}(x)\right]
\end{aligned}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:9.406em;vertical-align:-4.453em;"></span><span className="mord"><span className="mtable"><span className="col-align-r"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:4.953em;"><span style="top:-6.953em;"><span className="pstrut" style="height:3.36em;"></span><span className="mord"><span className="mord">0</span></span></span><span style="top:-4.3811em;"><span className="pstrut" style="height:3.36em;"></span><span className="mord"></span></span><span style="top:-1.8091em;"><span className="pstrut" style="height:3.36em;"></span><span className="mord"></span></span><span style="top:0.2428em;"><span className="pstrut" style="height:3.36em;"></span><span className="mord"><span className="mrel,amsrm">∴</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:4.453em;"><span></span></span></span></span></span><span className="col-align-l"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:4.953em;"><span style="top:-6.953em;"><span className="pstrut" style="height:3.36em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop"><span className="mop,op-symbol,large-op" style="margin-right:0.4445em;position:relative;top:-0.0011em;">∫</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:-0.6106em;"><span style="top:-1.7881em;margin-left:-0.4445em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">x</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9119em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">x</span><span className="mclose">)</span></span></span><span style="top:-4.3811em;"><span className="pstrut" style="height:3.36em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mop"><span className="mop,op-symbol,large-op" style="margin-right:0.4445em;position:relative;top:-0.0011em;">∫</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:-0.6106em;"><span style="top:-1.7881em;margin-left:-0.4445em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">x</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9119em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">x</span><span className="mclose">)</span></span></span><span style="top:-1.8091em;"><span className="pstrut" style="height:3.36em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mop"><span className="mop,op-symbol,large-op" style="margin-right:0.4445em;position:relative;top:-0.0011em;">∫</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:-0.6106em;"><span style="top:-1.7881em;margin-left:-0.4445em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">x</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9119em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">x</span><span className="mclose">)</span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">x</span><span className="mclose">)</span></span></span><span style="top:0.2428em;"><span className="pstrut" style="height:3.36em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.3557em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">x</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.1389em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8502em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">x</span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">]</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:4.453em;"><span></span></span></span></span></span></span></span></span></span></span></span></note><note title="注"><p>本文作者不知道这个引理在文献里是否有标准名字。但它出现得太频繁了，值得给它起个名字，方便引用。</p></note><h2 id="别让过去干扰你">别让过去干扰你</h2><p>看看我们刚才的策略梯度表达式：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\nabla_{\theta} J(\pi_{\theta}) = \underset{\tau \sim \pi_{\theta}}{\mathbb{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t |s_t) R(\tau)\right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.0954em;vertical-align:-1.2671em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8059em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size4">[</span></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size4">]</span></span></span></span></span></span></span><p>沿这个梯度走一步，会把每个动作的对数概率往上推，推的幅度正比于 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">R(\tau)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span></span></span></span>，也就是<em>曾经拿到的所有奖励</em>的总和。但这不太讲道理。</p><p>智能体强化一个动作，理应只看它的<em>后果</em>。采取动作<em>之前</em>拿到的奖励与这个动作的好坏毫无关系：有关系的只有<em>之后</em>来的奖励。</p><p>这个直觉其实藏在数学里：可以证明，策略梯度也能写成</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><munderover><mo>∑</mo><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>=</mo><mi>t</mi></mrow><mi>T</mi></munderover><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>a</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>s</mi><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\nabla_{\theta} J(\pi_{\theta}) = \underset{\tau \sim \pi_{\theta}}{\mathbb{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t |s_t) \sum_{t&#39;=t}^T R(s_{t&#39;}, a_{t&#39;}, s_{t&#39;+1})\right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.1223em;vertical-align:-1.294em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8059em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size4">[</span></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.856em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">=</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.294em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size4">]</span></span></span></span></span></span></span><p>在这个形式里，动作只根据它<em>被采取之后</em>获得的奖励来强化。</p><p>我们把这个形式叫作「reward-to-go 策略梯度」，因为轨迹中某一点之后的奖励总和</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mover accent="true"><mi>R</mi><mo>^</mo></mover><mi>t</mi></msub><mo>≐</mo><munderover><mo>∑</mo><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>=</mo><mi>t</mi></mrow><mi>T</mi></munderover><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>a</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>s</mi><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\hat{R}_t \doteq \sum_{t&#39;=t}^T R(s_{t&#39;}, a_{t&#39;}, s_{t&#39;+1})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0968em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,accent"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.9468em;"><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span></span><span style="top:-3.2523em;"><span className="pstrut" style="height:3em;"></span><span className="accent-body" style="left:-0.1667em;"><span className="mord">^</span></span></span></span></span></span></span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≐</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.1223em;vertical-align:-1.294em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.856em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">=</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.294em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>就叫作从该点起的 <strong>reward-to-go</strong>（只往后算的奖励），而这个策略梯度表达式依赖的正是各状态-动作对的 reward-to-go。</p><note title="你应该知道"><p><strong>但这为什么更好</strong>？策略梯度的一个关键问题是：要把它的样本估计做到低方差，需要多少条采样轨迹。我们出发的那个公式里，强化动作的项正比于过去的奖励，这些项均值为零、方差非零，只会给策略梯度的样本估计添噪音。去掉它们，所需的采样轨迹就少了。</p></note><p>这个论断的（选读）证明见<a href="https://spinningup.openai.com/en/latest/spinningup/extra_pg_proof1.html" rel="nofollow">这里</a>，最终靠的是 EGLP 引理。</p><h2 id="实现-reward-to-go-策略梯度">实现 reward-to-go 策略梯度</h2><p>reward-to-go 策略梯度的 PyTorch 短实现放在 <code>spinup/examples/pytorch/pg_math/2_rtg_pg.py</code>（也可以<a href="https://github.com/openai/spinningup/blob/master/spinup/examples/pytorch/pg_math/2_rtg_pg.py" rel="nofollow">在 GitHub 上查看</a>）。</p><p>相比 <code>1_simple_pg.py</code>，唯一的变化是损失函数里用了不同的权重。代码改动非常小：加一个新函数，再改两行。新函数是：</p><pre className="language-python3 shiki shiki-themes github-light github-dark" code="def reward_to_go(rews):
    n = len(rews)
    rtgs = np.zeros_like(rews)
    for i in reversed(range(n)):
        rtgs[i] = rews[i] + (rtgs[i+1] if i+1 &lt; n else 0)
    return rtgs
" language="python3" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="szBVR">def</span><span class="sScJk"> reward_to_go</span><span class="sVt8B">(rews):
</span></span><span class="line" line="2"><span class="sVt8B">    n </span><span class="szBVR">=</span><span class="sj4cs"> len</span><span class="sVt8B">(rews)
</span></span><span class="line" line="3"><span class="sVt8B">    rtgs </span><span class="szBVR">=</span><span class="sVt8B"> np.zeros_like(rews)
</span></span><span class="line" line="4"><span class="szBVR">    for</span><span class="sVt8B"> i </span><span class="szBVR">in</span><span class="sj4cs"> reversed</span><span class="sVt8B">(</span><span class="sj4cs">range</span><span class="sVt8B">(n)):
</span></span><span class="line" line="5"><span class="sVt8B">        rtgs[i] </span><span class="szBVR">=</span><span class="sVt8B"> rews[i] </span><span class="szBVR">+</span><span class="sVt8B"> (rtgs[i</span><span class="szBVR">+</span><span class="sj4cs">1</span><span class="sVt8B">] </span><span class="szBVR">if</span><span class="sVt8B"> i</span><span class="szBVR">+</span><span class="sj4cs">1</span><span class="szBVR"> &lt;</span><span class="sVt8B"> n </span><span class="szBVR">else</span><span class="sj4cs"> 0</span><span class="sVt8B">)
</span></span><span class="line" line="6"><span class="szBVR">    return</span><span class="sVt8B"> rtgs
</span></span></code></pre><p>然后把原来的第 91、92 行从：</p><pre className="language-python3 shiki shiki-themes github-light github-dark" code="                # 每个 logprob(a|s) 的权重是 R(tau)
                batch_weights += [ep_ret] * ep_len
" language="python3" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sJ8bj">                # 每个 logprob(a|s) 的权重是 R(tau)
</span></span><span class="line" line="2"><span class="sVt8B">                batch_weights </span><span class="szBVR">+=</span><span class="sVt8B"> [ep_ret] </span><span class="szBVR">*</span><span class="sVt8B"> ep_len
</span></span></code></pre><p>改成：</p><pre className="language-python3 shiki shiki-themes github-light github-dark" code="                # 每个 logprob(a_t|s_t) 的权重是从 t 时刻起的 reward-to-go
                batch_weights += list(reward_to_go(ep_rews))
" language="python3" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sJ8bj">                # 每个 logprob(a_t|s_t) 的权重是从 t 时刻起的 reward-to-go
</span></span><span class="line" line="2"><span class="sVt8B">                batch_weights </span><span class="szBVR">+=</span><span class="sj4cs"> list</span><span class="sVt8B">(reward_to_go(ep_rews))
</span></span></code></pre><note title="你应该知道"><p>本节最初写的是 Tensorflow 例子，旧版 Tensorflow 的内容见<a href="https://spinningup.openai.com/en/latest/spinningup/extra_tf_pg_implementation.html#implementing-reward-to-go-policy-gradient" rel="nofollow">这里</a>。</p></note><h2 id="策略梯度中的基线">策略梯度中的基线</h2><p>EGLP 引理的一个直接推论：对任何只依赖状态的函数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">b</span></span></span></span>，有</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><msub><mi>a</mi><mi>t</mi></msub><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mi>b</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\underset{a_t \sim \pi_{\theta}}{\mathbb{E}}\left[\nabla_{\theta} \log \pi_{\theta}(a_t|s_t) b(s_t)\right] = 0</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.5559em;vertical-align:-0.8059em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2963em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.143em;"><span></span></span></span></span></span></span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8059em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord,mathnormal">b</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">]</span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">0</span></span></span></span></span><p>这让我们可以在策略梯度表达式里加上或减去任意多个这样的项，而期望不变：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mrow><mo fence="true">(</mo><munderover><mo>∑</mo><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>=</mo><mi>t</mi></mrow><mi>T</mi></munderover><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>a</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>s</mi><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo>−</mo><mi>b</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo fence="true">)</mo></mrow><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\nabla_{\theta} J(\pi_{\theta}) = \underset{\tau \sim \pi_{\theta}}{\mathbb{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t |s_t) \left(\sum_{t&#39;=t}^T R(s_{t&#39;}, a_{t&#39;}, s_{t&#39;+1}) - b(s_t)\right)\right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.1223em;vertical-align:-1.294em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8059em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size4">[</span></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size4">(</span></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.856em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">=</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.294em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">−</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal">b</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size4">)</span></span></span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size4">]</span></span></span></span></span></span></span><p>这样使用的任何函数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">b</span></span></span></span> 都叫作<strong>基线（baseline）</strong>。</p><p>最常见的基线选择是<a href="/posts/spinning-up-rl-intro#%E4%BB%B7%E5%80%BC%E5%87%BD%E6%95%B0">同策略价值函数</a> <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">V^{\pi}(s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>。回忆一下：它指智能体从状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">s_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 出发、之后一直按策略 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span></span></span></span> 行动所能得到的平均回报。</p><p>经验上，取 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo>=</mo><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">b(s_t) = V^{\pi}(s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">b</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 能切实降低策略梯度样本估计的方差，策略学得更快也更稳。从概念上讲，这个选择也有吸引力：它编码了这样一个直觉，如果智能体得到的就是它预期的，它对此应该「无感」。</p><note title="你应该知道"><p>实践中 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">V^{\pi}(s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 算不出来，只能近似。通常用一个神经网络 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>V</mi><mi>ϕ</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">V_{\phi}(s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0361em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.2222em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">ϕ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>，与策略同时更新（让价值网络始终近似最新策略的价值函数）。</p><p>学习 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>V</mi><mi>ϕ</mi></msub></mrow><annotation encoding="application/x-tex">V_{\phi}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.9694em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.2222em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">ϕ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span></span></span></span> 最简单的方法（VPG、TRPO、PPO 和 A2C 的大多数实现都这么做）是最小化均方误差目标：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>ϕ</mi><mi>k</mi></msub><mo>=</mo><mi>arg</mi><mo>⁡</mo><munder><mrow><mi>min</mi><mo>⁡</mo></mrow><mi>ϕ</mi></munder><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mover accent="true"><mi>R</mi><mo>^</mo></mover><mi>t</mi></msub><mo>∼</mo><msub><mi>π</mi><mi>k</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><msup><mrow><mo fence="true">(</mo><msub><mi>V</mi><mi>ϕ</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo>−</mo><msub><mover accent="true"><mi>R</mi><mo>^</mo></mover><mi>t</mi></msub><mo fence="true">)</mo></mrow><mn>2</mn></msup><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\phi_k = \arg \min_{\phi} \underset{s_t, \hat{R}_t \sim \pi_k}{\mathbb{E}}\left[\left( V_{\phi}(s_t) - \hat{R}_t \right)^2\right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">ϕ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.5148em;vertical-align:-1.0648em;"></span><span className="mop">ar<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6679em;"><span style="top:-2.3479em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">ϕ</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">min</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8882em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.1713em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2963em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.143em;"><span></span></span></span></span></span></span><span className="mpunct,mtight">,</span><span className="mord,mtight"><span className="mord,accent,mtight"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.9468em;"><span style="top:-2.7em;"><span className="pstrut" style="height:2.7em;"></span><span className="mord,mathnormal,mtight" style="margin-right:0.0077em;">R</span></span><span style="top:-2.9523em;"><span className="pstrut" style="height:2.7em;"></span><span className="accent-body" style="left:-0.1667em;"><span className="mord,mtight">^</span></span></span></span></span></span></span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2963em;"><span style="top:-2.357em;margin-left:-0.0077em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.143em;"><span></span></span></span></span></span></span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.0648em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size3">[</span></span><span className="minner"><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size2">(</span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.2222em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">ϕ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">−</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord"><span className="mord,accent"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.9468em;"><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span></span><span style="top:-3.2523em;"><span className="pstrut" style="height:3em;"></span><span className="accent-body" style="left:-0.1667em;"><span className="mord">^</span></span></span></span></span></span></span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size2">)</span></span></span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:1.354em;"><span style="top:-3.6029em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size3">]</span></span></span></span></span></span></span><p>其中 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>k</mi></msub></mrow><annotation encoding="application/x-tex">\pi_k</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 是第 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span></span></span></span> 轮的策略。具体做法是从上一轮的价值参数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ϕ</mi><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow><annotation encoding="application/x-tex">\phi_{k-1}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.9028em;vertical-align:-0.2083em;"></span><span className="mord"><span className="mord,mathnormal">ϕ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span><span className="mbin,mtight">−</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span></span></span></span> 出发，做一步或多步梯度下降。</p></note><h2 id="策略梯度的其他形式">策略梯度的其他形式</h2><p>到目前为止我们看到，策略梯度有如下一般形式：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></munder></mi><mrow><mo fence="true">[</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\nabla_{\theta} J(\pi_{\theta}) = \underset{\tau \sim \pi_{\theta}}{\mathbb{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t |s_t) \Phi_t\right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.0954em;vertical-align:-1.2671em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8059em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size4">[</span></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord">∇</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size4">]</span></span></span></span></span></span></span><p>其中 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\Phi_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 可以取</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub><mo>=</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\Phi_t = R(\tau)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span></span></span></span></span><p>或</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub><mo>=</mo><munderover><mo>∑</mo><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>=</mo><mi>t</mi></mrow><mi>T</mi></munderover><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>a</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>s</mi><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\Phi_t = \sum_{t&#39;=t}^T R(s_{t&#39;}, a_{t&#39;}, s_{t&#39;+1})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.1223em;vertical-align:-1.294em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.856em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">=</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.294em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>或</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub><mo>=</mo><munderover><mo>∑</mo><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>=</mo><mi>t</mi></mrow><mi>T</mi></munderover><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>a</mi><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></msub><mo separator="true">,</mo><msub><mi>s</mi><mrow><msup><mi>t</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo>−</mo><mi>b</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\Phi_t = \sum_{t&#39;=t}^T R(s_{t&#39;}, a_{t&#39;}, s_{t&#39;+1}) - b(s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.1223em;vertical-align:-1.294em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.856em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">=</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.294em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.328em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">−</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">b</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>尽管方差不同，这些选择给出的策略梯度期望值全都相同。事实上，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\Phi_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 还有两个同样合法的选择，也很重要，值得知道。</p><p><strong>1. 同策略动作价值函数</strong>。取</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub><mo>=</mo><msup><mi>Q</mi><msub><mi>π</mi><mi>θ</mi></msub></msup><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\Phi_t = Q^{\pi_{\theta}}(s_t, a_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>也是合法的。（选读）证明见<a href="https://spinningup.openai.com/en/latest/spinningup/extra_pg_proof2.html" rel="nofollow">这个页面</a>。</p><p><strong>2. 优势函数</strong>。回忆一下，动作的<a href="/posts/spinning-up-rl-intro#%E4%BC%98%E5%8A%BF%E5%87%BD%E6%95%B0">优势</a>定义为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>A</mi><mi>π</mi></msup><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo>=</mo><msup><mi>Q</mi><mi>π</mi></msup><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo>−</mo><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">A^{\pi}(s_t,a_t) = Q^{\pi}(s_t,a_t) - V^{\pi}(s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">A</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">−</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>，描述的是（相对当前策略）它平均而言比其他动作好多少。取</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub><mo>=</mo><msup><mi>A</mi><msub><mi>π</mi><mi>θ</mi></msub></msup><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\Phi_t = A^{\pi_{\theta}}(s_t, a_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">A</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>也合法。证明：它等价于先取 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub><mo>=</mo><msup><mi>Q</mi><msub><mi>π</mi><mi>θ</mi></msub></msup><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\Phi_t = Q^{\pi_{\theta}}(s_t, a_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:-0.0359em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>，再加一个价值函数基线，而基线我们随时可以加。</p><note title="你应该知道"><p>用优势函数表述策略梯度极其常见，不同算法有各自估计优势函数的办法。</p></note><note title="你应该知道"><p>想更深入这个主题，应该去读 <a href="https://arxiv.org/abs/1506.02438" rel="nofollow">Generalized Advantage Estimation</a>（GAE）的论文，它的背景章节深入讨论了 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="normal">Φ</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\Phi_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord">Φ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 的各种选择。</p><p>那篇论文接着提出了 GAE，一种在策略优化算法中近似优势函数的方法，如今使用非常广泛。比如 Spinning Up 的 VPG、TRPO、PPO 实现都用它。所以我们强烈建议研究它。</p></note><h2 id="小结">小结</h2><p>本章讲了策略梯度方法的基本理论，并把其中一些早期结果对接到了代码示例上。有兴趣的读者可以从这里继续，去研究后面的结果（价值函数基线、策略梯度的优势形式）如何落实到 Spinning Up 的 <a href="https://spinningup.openai.com/en/latest/algorithms/vpg.html" rel="nofollow">Vanilla Policy Gradient</a> 实现里。</p><style>html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}</style>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[译] 深度强化学习入门（二）：强化学习算法的分类]]></title>
            <description><![CDATA[OpenAI Spinning Up 第二部分的全文翻译：无模型与基于模型的 RL、策略优化与 Q-learning 的取舍，以及规划、专家迭代和经验增强等方法。]]></description>
            <link>https://blog.niina.fun/posts/spinning-up-rl-intro-2</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/spinning-up-rl-intro-2</guid>
            <pubDate>Mon, 28 Sep 2026 01:20:21 GMT</pubDate>
            <content:encoded><![CDATA[<note title="译者说明"><p>本文是 OpenAI <a href="https://spinningup.openai.com/en/latest/" rel="nofollow">Spinning Up in Deep RL</a> 教程第二部分 <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html" rel="nofollow">Part 2: Kinds of RL Algorithms</a> 的全文翻译。原仓库采用 <a href="https://github.com/openai/spinningup/blob/master/LICENSE" rel="nofollow">MIT 协议</a>。分类图保留官方英文原图，正文保留算法缩写与外链；术语首次出现时附英文原词，后文沿用同一译名。</p></note><p>术语和记号打好基础后，接下来可以看更丰富的内容：现代 RL 算法的整体版图，以及算法设计中各种取舍。</p><h2 id="强化学习算法的分类">强化学习算法的分类</h2><p><img alt="现代强化学习算法的一种分类图（官方英文原图）" src="https://spinningup.openai.com/en/latest/_images/rl_algorithms_9_15.svg" /></p><p><em>图 1：现代 RL 算法的一种实用分类，并未穷尽所有方法。算法引用见文末。</em></p><p>先说明一点：现代 RL 算法的模块化特征很难用树形结构表示，因此要画出准确且包罗万象的算法分类图相当困难。为了让分类图能放在一页内，并使这篇导论便于阅读，我们省略了不少进阶主题，例如探索、迁移学习和元学习。本文的目标是：</p><ul><li>突出深度 RL 算法中最基础的设计选择：要学习什么，以及怎样学习；</li><li>展示这些选择各自的取舍；</li><li>将几种重要的现代算法放进这些选择构成的脉络中。</li></ul><h3 id="无模型与基于模型的强化学习">无模型与基于模型的强化学习</h3><p>强化学习算法面临的一个重要分岔是：智能体能否访问环境模型，或者能否学到一个环境模型。这里的环境模型指预测状态转移和奖励的函数。</p><p>使用模型的主要好处，是智能体可以向前规划：设想若干种选择分别会带来什么结果，再明确比较这些选择。随后，智能体还可以把规划所得提炼成一个学到的策略。一个著名例子是 <a href="https://arxiv.org/abs/1712.01815" rel="nofollow">AlphaZero</a>。模型方法若能奏效，相较于不使用模型的方法，样本效率（sample efficiency）可能会大幅提高。</p><p>主要困难在于，智能体通常拿不到环境的真实模型。此时若想使用模型，就只能从经验中学习它；这会带来一系列挑战。最突出的一点是，智能体可能利用模型中的偏差，在学到的模型里表现良好，到了真实环境中却表现欠佳，甚至极差。模型学习本身就很难；即使投入大量时间和算力，也可能得不到相应收益。</p><p>使用环境模型的算法称为<strong>基于模型的方法（model-based methods）</strong>，不使用环境模型的算法称为<strong>无模型方法（model-free methods）</strong>。无模型方法放弃了借助模型提高样本效率的潜在收益，但通常更容易实现和调参。截至这篇导论写作时（2018 年 9 月），无模型方法更受欢迎，也经过了更多开发和测试。</p><h3 id="要学习什么">要学习什么</h3><p>另一个关键分岔是算法要学习什么。常见的学习对象包括：</p><ul><li>策略，可以是随机型策略，也可以是确定型策略；</li><li>动作价值函数（Q 函数）；</li><li>价值函数；</li><li>环境模型；</li><li>或以上对象的组合。</li></ul><h4 id="无模型强化学习要学习什么">无模型强化学习要学习什么</h4><p>无模型 RL 中，表征和训练智能体主要有两种方法。</p><p><strong>策略优化（policy optimization）</strong>。这类方法显式地用策略 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>a</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\pi_{\theta}(a|s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">a</span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 表示智能体。它们通过两种方式优化参数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi></mrow><annotation encoding="application/x-tex">\theta</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span></span></span></span>：直接对性能目标 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">J(\pi_{\theta})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 做梯度上升，或者间接地最大化该目标的局部代理（surrogate）。优化过程几乎总是采用<strong>同策略（on-policy）</strong> 方式：每次更新只使用智能体按照当前最新策略与环境交互时收集的数据。</p><p>策略优化通常还会学习一个函数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>V</mi><mi>ϕ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">V_{\phi}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0361em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.2222em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">ϕ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>，用来近似该策略下的价值函数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">V^{\pi}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>，并据此计算如何更新策略。策略优化方法的例子包括：</p><ul><li><a href="https://arxiv.org/abs/1602.01783" rel="nofollow">A2C / A3C</a>：通过梯度上升直接最大化性能；</li><li><a href="https://arxiv.org/abs/1707.06347" rel="nofollow">PPO</a>：通过最大化一个代理目标函数，间接推动性能目标上升。这个代理目标会保守估计这次策略更新可能引起的性能目标变化。</li></ul><p><strong>Q-learning</strong>。这类方法学习一个函数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>Q</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">Q_{\theta}(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span>，用来近似最优动作价值函数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">Q^*(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span>。它们通常使用基于贝尔曼方程的目标函数。优化过程几乎总是采用<strong>异策略（off-policy）</strong> 方式：每次更新可以使用训练过程中任意时刻收集的数据，而不受智能体采集这些数据时采用何种探索策略的限制。</p><p>最优策略会选择使 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">Q^*(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span> 最大的动作；Q-learning 则用学得的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>Q</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">Q_{\theta}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8778em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 近似 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup></mrow><annotation encoding="application/x-tex">Q^*</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8831em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span></span></span></span>，据此选取动作：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>a</mi><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo>=</mo><mi>arg</mi><mo>⁡</mo><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><mi>a</mi></munder><msub><mi>Q</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a(s) = \arg\max_a Q_{\theta}(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">a</span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mop">ar<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.4306em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">a</span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">max</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span></span><p>Q-learning 方法的例子包括：</p><ul><li><a href="https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf" rel="nofollow">DQN</a>：推动深度 RL 领域发展的经典方法；</li><li><a href="https://arxiv.org/abs/1707.06887" rel="nofollow">C51</a>：一种学习回报分布的变体，其期望就是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup></mrow><annotation encoding="application/x-tex">Q^*</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8831em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span></span></span></span>。</li></ul><h4 id="策略优化与-q-learning-的取舍">策略优化与 Q-learning 的取舍</h4><p>策略优化方法的主要优点是目标明确：它直接优化我们真正关心的对象，因此往往稳定、可靠。相比之下，Q-learning 通过训练 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>Q</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">Q_{\theta}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8778em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 满足自洽方程，间接推动智能体表现变好。这类学习存在多种失效模式，因此通常稳定性较差（相关讨论见文末参考资料 <span>1</span>）。不过，在能够奏效时，Q-learning 的样本效率明显更高，因为它比策略优化更充分地复用数据。</p><h4 id="介于策略优化与-q-learning-之间的方法">介于策略优化与 Q-learning 之间的方法</h4><p>策略优化与 Q-learning 并不互斥，在<a href="https://arxiv.org/abs/1704.06440" rel="nofollow">某些条件下甚至可以等价</a>。处于两者之间的一系列算法，可以在两端方法的优缺点之间作出折中。例如：</p><ul><li><a href="https://arxiv.org/abs/1509.02971" rel="nofollow">DDPG</a> 同时学习一个确定型策略和一个 Q 函数，并利用两者相互改进；</li><li><a href="https://arxiv.org/abs/1801.01290" rel="nofollow">SAC</a> 使用随机策略、熵正则化和其他技巧来稳定学习，在标准基准上的得分高于 DDPG。</li></ul><h3 id="基于模型的强化学习要学习什么">基于模型的强化学习要学习什么</h3><p>与无模型 RL 不同，基于模型的方法很难归纳成少数几类边界清晰的算法；模型的使用方式有许多彼此独立的维度。下面只举几个例子，远未穷尽。每种方法使用的模型都可能是直接给定的，也可能是从经验中学到的。</p><h4 id="背景纯规划">背景：纯规划</h4><p>最基础的做法是不显式表示策略，而是使用 <a href="https://en.wikipedia.org/wiki/Model_predictive_control" rel="nofollow">模型预测控制（model-predictive control，MPC）</a> 这样的纯规划方法来选择动作。MPC 中，智能体每次观测环境时，都会根据模型计算一个最优计划；计划规定从当前时刻起，在一个固定时间窗口内要采取哪些动作。</p><p>规划算法也可以借助学到的价值函数，考虑规划视界之外的未来奖励。随后，智能体只执行计划中的第一个动作，并立即丢弃计划的其余部分。每次准备与环境交互时，智能体都会重新计算计划，以免继续执行旧计划时规划视界已短于预期。</p><ul><li><a href="https://sites.google.com/view/mbmf" rel="nofollow">MBMF</a> 探索了在深度 RL 的标准基准任务中，使用学到的环境模型进行 MPC 的做法。</li></ul><h4 id="专家迭代">专家迭代</h4><p>在纯规划的基础上，一个直接的扩展是显式表示并学习策略 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>a</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\pi_{\theta}(a|s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">a</span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>。智能体在模型中使用规划算法（例如蒙特卡洛树搜索）来规划，并从当前策略中采样候选动作。规划算法给出的动作比策略单独给出的动作更好，因此，规划算法可以充当相对于当前策略的“专家”。</p><p>之后，智能体更新策略，使它更倾向于给出与规划算法相似的动作。</p><ul><li><a href="https://arxiv.org/abs/1705.08439" rel="nofollow">ExIt</a> 使用这种方法训练深度神经网络来玩 Hex。</li><li><a href="https://arxiv.org/abs/1712.01815" rel="nofollow">AlphaZero</a> 也是这种方法的例子。</li></ul><h4 id="用模型生成的经验增强无模型方法">用模型生成的经验增强无模型方法</h4><p>使用无模型 RL 算法训练策略或 Q 函数，同时在更新智能体时采取以下做法之一：</p><ol><li>用模型生成的虚拟经验补充真实经验；</li><li>只使用模型生成的虚拟经验来更新智能体。</li></ol><ul><li><a href="https://arxiv.org/abs/1803.00101" rel="nofollow">MBVE</a> 展示了如何用虚拟经验补充真实经验。</li><li><a href="https://worldmodels.github.io/" rel="nofollow">World Models</a> 展示了如何只用虚拟经验训练智能体，并把这种方式称作“在梦境中训练”（training in the dream）。</li></ul><h4 id="在策略中嵌入规划过程">在策略中嵌入规划过程</h4><p>另一种方法是把规划过程直接嵌入策略，作为策略调用的一个子程序；完整计划作为辅助信息提供给策略。再用任意标准的无模型算法训练策略。这里的关键是，策略可以学会选择在什么时候、以什么方式使用规划结果。</p><p>这样可以减轻模型偏差的影响：如果模型在某些状态下给出的规划不可靠，策略可以学会忽略它。</p><ul><li><a href="https://arxiv.org/abs/1707.06203" rel="nofollow">I2A</a> 是让智能体具备这类“想象”能力的一种方法。</li></ul><h2 id="参考资料">参考资料</h2><p><strong><span>1</span> Q-learning 方法为何会失效</strong></p><ul><li>Tsitsiklis 和 Van Roy 的<a href="https://web.mit.edu/jnt/www/Papers/J063-97-bvr-td.pdf" rel="nofollow">经典论文</a>；</li><li>Szepesvári 的<a href="https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf" rel="nofollow">综述</a>，第 4.3.2 节；</li><li>Sutton 与 Barto 的<a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow">《强化学习：导论》</a>第 11 章，尤其是第 11.3 节关于“致命三元组”（deadly triad）的讨论：函数逼近、自举更新和异策略数据共同导致价值学习算法不稳定。</li></ul><p><strong>分类图中的算法引用</strong></p><ul><li><strong><span>2</span> <a href="https://arxiv.org/abs/1602.01783" rel="nofollow">A2C / A3C</a></strong>（Asynchronous Advantage Actor-Critic）：Mnih 等，2016。</li><li><strong><span>3</span> <a href="https://arxiv.org/abs/1707.06347" rel="nofollow">PPO</a></strong>（Proximal Policy Optimization）：Schulman 等，2017。</li><li><strong><span>4</span> <a href="https://arxiv.org/abs/1502.05477" rel="nofollow">TRPO</a></strong>（Trust Region Policy Optimization）：Schulman 等，2015。</li><li><strong><span>5</span> <a href="https://arxiv.org/abs/1509.02971" rel="nofollow">DDPG</a></strong>（Deep Deterministic Policy Gradient）：Lillicrap 等，2015。</li><li><strong><span>6</span> <a href="https://arxiv.org/abs/1802.09477" rel="nofollow">TD3</a></strong>（Twin Delayed DDPG）：Fujimoto 等，2018。</li><li><strong><span>7</span> <a href="https://arxiv.org/abs/1801.01290" rel="nofollow">SAC</a></strong>（Soft Actor-Critic）：Haarnoja 等，2018。</li><li><strong><span>8</span> <a href="https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf" rel="nofollow">DQN</a></strong>（Deep Q-Networks）：Mnih 等，2013。</li><li><strong><span>9</span> <a href="https://arxiv.org/abs/1707.06887" rel="nofollow">C51</a></strong>（Categorical 51-Atom DQN）：Bellemare 等，2017。</li><li><strong><span>10</span> <a href="https://arxiv.org/abs/1710.10044" rel="nofollow">QR-DQN</a></strong>（Quantile Regression DQN）：Dabney 等，2017。</li><li><strong><span>11</span> <a href="https://arxiv.org/abs/1707.01495" rel="nofollow">HER</a></strong>（Hindsight Experience Replay）：Andrychowicz 等，2017。</li><li><strong><span>12</span> <a href="https://worldmodels.github.io/" rel="nofollow">World Models</a></strong>：Ha 和 Schmidhuber，2018。</li><li><strong><span>13</span> <a href="https://arxiv.org/abs/1707.06203" rel="nofollow">I2A</a></strong>（Imagination-Augmented Agents）：Weber 等，2017。</li><li><strong><span>14</span> <a href="https://sites.google.com/view/mbmf" rel="nofollow">MBMF</a></strong>（Model-Based RL with Model-Free Fine-Tuning）：Nagabandi 等，2017。</li><li><strong><span>15</span> <a href="https://arxiv.org/abs/1803.00101" rel="nofollow">MBVE</a></strong>（Model-Based Value Expansion）：Feinberg 等，2018。</li><li><strong><span>16</span> <a href="https://arxiv.org/abs/1712.01815" rel="nofollow">AlphaZero</a></strong>：Silver 等，2017。</li></ul>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[译] 深度强化学习入门（一）：RL 的关键概念]]></title>
            <description><![CDATA[OpenAI Spinning Up 教程第一部分的全文翻译：智能体与环境、状态与观测、动作空间、策略、轨迹、奖励与回报、价值函数、贝尔曼方程、优势函数，附 MDP 形式化。]]></description>
            <link>https://blog.niina.fun/posts/spinning-up-rl-intro</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/spinning-up-rl-intro</guid>
            <pubDate>Mon, 28 Sep 2026 00:29:40 GMT</pubDate>
            <content:encoded><![CDATA[<note title="译者说明"><p>本文是 OpenAI <a href="https://spinningup.openai.com/en/latest/" rel="nofollow">Spinning Up in Deep RL</a> 教程第一部分 <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro.html" rel="nofollow">Part 1: Key Concepts in RL</a> 的全文翻译，原仓库以 MIT 协议发布。原文的提示框、外链与代码示例全部保留，视频以链接代替嵌入。术语首次出现时附英文原词，之后沿用同一译名。</p></note><p>欢迎来到强化学习导论！这一部分的目标是让你熟悉：</p><ul><li>讨论这一主题所用的语言与记号；</li><li>RL 算法在做什么的高层解释（不过我们大多回避它们<em>如何</em>做到这一问题）；</li><li>以及支撑这些算法的一点点核心数学。</li></ul><p>概括地说，RL 研究的是智能体（agent）以及它们如何通过试错（trial and error）学习。它把这个想法形式化：对智能体的行为给予奖励或惩罚，会让它未来更倾向于重复或放弃该行为。</p><h2 id="rl-能做什么">RL 能做什么</h2><p>近年来，RL 方法收获了大范围的成功。例如，它被用来教计算机在模拟中控制机器人：<a href="https://d4mucfpksywv.cloudfront.net/openai-baselines-ppo/knocked-over-stand-up.mp4" rel="nofollow">模拟环境里，机器人被撞倒后重新站起来（视频）</a>；也在真实世界中这样做：<a href="https://www.youtube.com/watch?v=jwSbzNHGflM" rel="nofollow">真实机器人控制（YouTube）</a>。</p><p>它还因在复杂策略游戏中造出突破性 AI 而闻名，最有名的是 <a href="https://deepmind.com/research/alphago/" rel="nofollow">Go（围棋）</a>和 <a href="https://blog.openai.com/openai-five/" rel="nofollow">Dota</a>；它教会计算机从原始像素<a href="https://deepmind.com/research/dqn/" rel="nofollow">玩 Atari 游戏</a>，也训练过模拟机器人<a href="https://blog.openai.com/deep-reinforcement-learning-from-human-preferences/" rel="nofollow">跟随人类指令</a>。</p><h2 id="关键概念与术语">关键概念与术语</h2><p><img alt="智能体与环境的交互循环" className="dark-ink" src="/images/spinning-up-rl-loop.png" /></p><p>RL 的两个主角是<strong>智能体（agent）和环境（environment）</strong>。环境是智能体身处其中并与之交互的世界。交互的每一步，智能体都会看到世界状态的一个（可能不完整的）观测，然后决定要采取的动作。智能体作用于环境时，环境会随之改变，但它也可能自行变化。</p><p>智能体还会从环境感知到一个<strong>奖励（reward）</strong> 信号：一个数字，告诉它当前的世界状态有多好或多坏。智能体的目标是最大化自己的累积奖励，这个累积量称为<strong>回报（return）</strong>。强化学习方法，就是智能体学习行为以达成这一目标的各种途径。</p><p>要把 RL 做的事说得更具体，还需要引入一批术语。我们要讨论：</p><ul><li>状态与观测；</li><li>动作空间；</li><li>策略；</li><li>轨迹；</li><li>回报的不同形式；</li><li>RL 优化问题；</li><li>以及价值函数。</li></ul><h3 id="状态与观测">状态与观测</h3><p><strong>状态（state）</strong> <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 是对世界的完整描述，没有任何关于世界的信息被它遗漏。<strong>观测（observation）</strong> <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>o</mi></mrow><annotation encoding="application/x-tex">o</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">o</span></span></span></span> 则是对状态的部分描述，可能省略一些信息。</p><p>在深度 RL 中，我们几乎总是用实值向量、矩阵或更高阶张量来表示状态和观测。例如，一个视觉观测可以用其像素值的 RGB（红绿蓝）矩阵表示；机器人的状态可以用它的关节角度和速度表示。</p><p>当智能体能够观测到环境的完整状态时，我们说环境是<strong>完全可观测（fully observed）</strong> 的；当它只能看到部分观测时，我们说环境是<strong>部分可观测（partially observed）</strong> 的。</p><note title="你应该知道"><p>RL 的记号有时会在严格来说更适合写观测符号 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>o</mi></mrow><annotation encoding="application/x-tex">o</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">o</span></span></span></span> 的地方写状态符号 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span>。描述智能体如何决定动作时就常见这种情况：记号上通常写作“动作以状态为条件”，但实践中动作取决于观测，因为智能体拿不到状态。</p><p>本指南遵循标准记号约定，具体指哪个应当能从上下文分辨出来。如果有不清楚的地方，欢迎到<a href="https://github.com/openai/spinningup/issues" rel="nofollow">原仓库</a>提 issue！我们的目标是把人教会，而不是把人绕晕。</p></note><h3 id="动作空间">动作空间</h3><p>不同的环境允许不同种类的动作。一个给定环境中所有合法动作的集合，通常叫作<strong>动作空间（action space）</strong>。有些环境（比如 Atari 和围棋）具有<strong>离散动作空间（discrete action space）</strong>：智能体可用的走法只有有限多个。另一些环境（比如智能体在物理世界中控制机器人）则具有<strong>连续动作空间（continuous action space）</strong>：动作是实值向量。</p><p>这个区分对深度 RL 方法的影响相当深远：有些算法家族只能直接用于其中一种情形，要用到另一种情形上就得动大手术。</p><h3 id="策略">策略</h3><p><strong>策略（policy）</strong> 是智能体用来决定采取什么动作的规则。它可以是确定型（deterministic）的，通常记作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal">μ</span></span></span></span>：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>a</mi><mi>t</mi></msub><mo>=</mo><mi>μ</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a_t = \mu(s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">μ</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>也可以是随机型（stochastic）的，通常记作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span></span></span></span>：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>a</mi><mi>t</mi></msub><mo>∼</mo><mi>π</mi><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a_t \sim \pi(\cdot | s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mopen">(</span><span className="mord">⋅</span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>策略是智能体的大脑，所以拿「policy」一词替换「agent」来说话并不少见，比如『策略正在试图最大化奖励』。</p><p>在深度 RL 中，我们处理的是<strong>参数化策略（parameterized policy）</strong>：这类策略的输出是可计算函数，函数依赖一组参数（比如神经网络的权重和偏置）；通过某种优化算法调整这些参数，就能改变策略的行为。</p><p>这类策略的参数常记作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi></mrow><annotation encoding="application/x-tex">\theta</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span></span></span></span> 或 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ϕ</mi></mrow><annotation encoding="application/x-tex">\phi</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal">ϕ</span></span></span></span>，并写成策略符号的下标来点明这层关系：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable rowspacing="0.25em" columnalign="right left" columnspacing="0em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><msub><mi>a</mi><mi>t</mi></msub></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><msub><mi>μ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><msub><mi>a</mi><mi>t</mi></msub></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow></mstyle></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{aligned}
a_t &amp;= \mu_{\theta}(s_t) \\
a_t &amp;\sim \pi_{\theta}(\cdot | s_t)
\end{aligned}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.7em;vertical-align:-1.1em;"></span><span className="mord"><span className="mtable"><span className="col-align-r"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.6em;"><span style="top:-3.76em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span><span style="top:-2.26em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.1em;"><span></span></span></span></span></span><span className="col-align-l"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.6em;"><span style="top:-3.76em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord,mathnormal">μ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span><span style="top:-2.26em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord">⋅</span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.1em;"><span></span></span></span></span></span></span></span></span></span></span></span><h4 id="确定型策略">确定型策略</h4><p><strong>例子：确定型策略</strong>。下面这段代码用 PyTorch 的 <code>torch.nn</code> 包，为连续动作空间构建一个简单的确定型策略：</p><pre className="language-python3 shiki shiki-themes github-light github-dark" code="pi_net = nn.Sequential(
              nn.Linear(obs_dim, 64),
              nn.Tanh(),
              nn.Linear(64, 64),
              nn.Tanh(),
              nn.Linear(64, act_dim)
            )
" language="python3" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sVt8B">pi_net </span><span class="szBVR">=</span><span class="sVt8B"> nn.Sequential(
</span></span><span class="line" line="2"><span class="sVt8B">              nn.Linear(obs_dim, </span><span class="sj4cs">64</span><span class="sVt8B">),
</span></span><span class="line" line="3"><span class="sVt8B">              nn.Tanh(),
</span></span><span class="line" line="4"><span class="sVt8B">              nn.Linear(</span><span class="sj4cs">64</span><span class="sVt8B">, </span><span class="sj4cs">64</span><span class="sVt8B">),
</span></span><span class="line" line="5"><span class="sVt8B">              nn.Tanh(),
</span></span><span class="line" line="6"><span class="sVt8B">              nn.Linear(</span><span class="sj4cs">64</span><span class="sVt8B">, act_dim)
</span></span><span class="line" line="7"><span class="sVt8B">            )
</span></span></code></pre><p>它构建了一个 MLP（multi-layer perceptron，多层感知机）网络：两个隐藏层各 64 个单元，激活函数是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>tanh</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\tanh</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mop">tanh</span></span></span></span>。如果 <code>obs</code> 是装着一批观测的 Numpy 数组，可以这样用 <code>pi_net</code> 求出这批观测对应的动作：</p><pre className="language-python3 shiki shiki-themes github-light github-dark" code="obs_tensor = torch.as_tensor(obs, dtype=torch.float32)
actions = pi_net(obs_tensor)
" language="python3" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sVt8B">obs_tensor </span><span class="szBVR">=</span><span class="sVt8B"> torch.as_tensor(obs, </span><span class="s4XuR">dtype</span><span class="szBVR">=</span><span class="sVt8B">torch.float32)
</span></span><span class="line" line="2"><span class="sVt8B">actions </span><span class="szBVR">=</span><span class="sVt8B"> pi_net(obs_tensor)
</span></span></code></pre><note title="你应该知道"><p>神经网络的内容如果不熟悉，不必担心：本教程聚焦 RL，不展开神经网络那一侧。可以先跳过这个例子，之后再回来看。不过我们猜想，如果你已经了解，它会有所帮助。</p></note><h4 id="随机型策略">随机型策略</h4><p>深度 RL 中最常见的两种随机型策略是<strong>分类策略（categorical policy）和对角高斯策略（diagonal Gaussian policy）</strong>。</p><p><a href="https://en.wikipedia.org/wiki/Categorical_distribution" rel="nofollow">分类</a>策略可用于离散动作空间，对角<a href="https://en.wikipedia.org/wiki/Multivariate_normal_distribution" rel="nofollow">高斯</a>策略则用于连续动作空间。</p><p>使用和训练随机型策略时，有两项关键计算居于中心地位：</p><ul><li>从策略中采样动作；</li><li>计算特定动作的对数似然（log likelihood）<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>a</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\log \pi_{\theta}(a|s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">a</span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>。</li></ul><p>接下来分别说明这两件事在分类策略和对角高斯策略上怎么做。</p><note title="分类策略"><p>分类策略就像作用在离散动作上的分类器。构建分类策略的神经网络和构建分类器是同一套做法：输入观测，接若干层（视输入类型可以是卷积层或全连接层），最后一个线性层给出每个动作的 logits，再接一个 <a href="https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax" rel="nofollow">softmax</a> 把 logits 转成概率。</p><p><strong>采样</strong>。拿到每个动作的概率后，PyTorch 和 TensorFlow 这类框架都内置了采样工具，例如 <a href="https://pytorch.org/docs/stable/distributions.html#categorical" rel="nofollow">PyTorch 的 Categorical 分布</a>、<a href="https://pytorch.org/docs/stable/torch.html#torch.multinomial" rel="nofollow"><code>torch.multinomial</code></a>、<code>tf.distributions.Categorical</code> 或 <code>tf.multinomial</code> 的文档。</p><p><strong>对数似然</strong>。记最后一层输出的概率为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">P_{\theta}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>。它是一个向量，条目数与动作数相同，因此可以把动作当作向量的索引。动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span></span></span></span> 的对数似然通过索引得到：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>a</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo><mo>=</mo><mi>log</mi><mo>⁡</mo><msub><mrow><mo fence="true">[</mo><msub><mi>P</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow><mi>a</mi></msub></mrow><annotation encoding="application/x-tex">\log \pi_{\theta}(a|s) = \log \left[P_{\theta}(s)\right]_a</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">a</span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.0497em;vertical-align:-0.2997em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">]</span></span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.0017em;"><span style="top:-2.4003em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">a</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2997em;"><span></span></span></span></span></span></span></span></span></span></span></note><note title="对角高斯策略"><p>多元高斯分布（multivariate Gaussian distribution，也叫多元正态分布）由均值向量 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal">μ</span></span></span></span> 和协方差矩阵 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Σ</mi></mrow><annotation encoding="application/x-tex">\Sigma</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">Σ</span></span></span></span> 描述。对角高斯分布是协方差矩阵只有对角线元素的特例，因此可以用一个向量来表示它。</p><p>对角高斯策略总有一个从观测映射到均值动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>μ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\mu_{\theta}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">μ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 的神经网络。协方差矩阵通常有两种表示方式。</p><p><strong>第一种</strong>：只有一个对数标准差向量 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>log</mi><mo>⁡</mo><mi>σ</mi></mrow><annotation encoding="application/x-tex">\log \sigma</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">σ</span></span></span></span>，它<strong>不是</strong>状态的函数：这些 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>log</mi><mo>⁡</mo><mi>σ</mi></mrow><annotation encoding="application/x-tex">\log \sigma</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">σ</span></span></span></span> 是独立的参数。（你应该知道：我们的 VPG（Vanilla Policy Gradient）、TRPO（Trust Region Policy Optimization）和 PPO 的实现用的就是这一种。）</p><p><strong>第二种</strong>：有一个从状态映射到对数标准差 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>log</mi><mo>⁡</mo><msub><mi>σ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\log \sigma_{\theta}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">σ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 的神经网络，它可以选择性地与均值网络共享一些层。</p><p>注意，两种情况下输出的都是对数标准差，并非直接输出标准差。原因是对数标准差可以在 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mo>−</mo><mi mathvariant="normal">∞</mi><mo separator="true">,</mo><mi mathvariant="normal">∞</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(-\infty, \infty)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">(</span><span className="mord">−</span><span className="mord">∞</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord">∞</span><span className="mclose">)</span></span></span></span> 上自由取值，而标准差必须非负；不必强制这类约束，参数训练起来更容易。对对数标准差取指数即可立即得到标准差，所以这种表示不损失任何东西。</p><p><strong>采样</strong>。给定均值动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>μ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\mu_{\theta}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">μ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>、标准差 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\sigma_{\theta}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">σ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>，以及一个来自球形高斯的噪声向量 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>z</mi></mrow><annotation encoding="application/x-tex">z</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.044em;">z</span></span></span></span>（<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>z</mi><mo>∼</mo><mi mathvariant="script">N</mi><mo stretchy="false">(</mo><mn>0</mn><mo separator="true">,</mo><mi>I</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">z \sim \mathcal{N}(0, I)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.044em;">z</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathcal" style="margin-right:0.1474em;">N</span><span className="mopen">(</span><span className="mord">0</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mclose">)</span></span></span></span>），动作样本可以这样计算：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>a</mi><mo>=</mo><msub><mi>μ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo>+</mo><msub><mi>σ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo>⊙</mo><mi>z</mi></mrow><annotation encoding="application/x-tex">a = \mu_{\theta}(s) + \sigma_{\theta}(s) \odot z</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">μ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">σ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">⊙</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.044em;">z</span></span></span></span></span><p>其中 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>⊙</mo></mrow><annotation encoding="application/x-tex">\odot</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6667em;vertical-align:-0.0833em;"></span><span className="mord">⊙</span></span></span></span> 表示两个向量的逐元素乘积。主流框架都有内置的方式生成噪声向量，比如 <a href="https://pytorch.org/docs/stable/torch.html#torch.normal" rel="nofollow"><code>torch.normal</code></a> 或 <code>tf.random_normal</code>；也可以构建分布对象，比如 <a href="https://pytorch.org/docs/stable/distributions.html#normal" rel="nofollow"><code>torch.distributions.Normal</code></a> 或 <code>tf.distributions.Normal</code>，再用它们生成样本。（后一种方式的好处是，这些对象还能替你算对数似然。）</p><p><strong>对数似然</strong>。对均值为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi><mo>=</mo><msub><mi>μ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\mu = \mu_{\theta}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal">μ</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">μ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>、标准差为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mo>=</mo><msub><mi>σ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\sigma = \sigma_{\theta}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">σ</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">σ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 的对角高斯，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span></span></span></span> 维动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span></span></span></span> 的对数似然为</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>a</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo><mo>=</mo><mo>−</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><mrow><mo fence="true">(</mo><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>k</mi></munderover><mrow><mo fence="true">(</mo><mfrac><mrow><mo stretchy="false">(</mo><msub><mi>a</mi><mi>i</mi></msub><mo>−</mo><msub><mi>μ</mi><mi>i</mi></msub><msup><mo stretchy="false">)</mo><mn>2</mn></msup></mrow><msubsup><mi>σ</mi><mi>i</mi><mn>2</mn></msubsup></mfrac><mo>+</mo><mn>2</mn><mi>log</mi><mo>⁡</mo><msub><mi>σ</mi><mi>i</mi></msub><mo fence="true">)</mo></mrow><mo>+</mo><mi>k</mi><mi>log</mi><mo>⁡</mo><mn>2</mn><mi>π</mi><mo fence="true">)</mo></mrow></mrow><annotation encoding="application/x-tex">\log \pi_{\theta}(a|s) = -\frac{1}{2}\left(\sum_{i=1}^k \left(\frac{(a_i - \mu_i)^2}{\sigma_i^2} + 2 \log \sigma_i \right) + k \log 2\pi \right)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">θ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">a</span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.1138em;vertical-align:-1.2777em;"></span><span className="mord">−</span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3214em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">2</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">1</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.686em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size4">(</span></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8361em;"><span style="top:-1.8723em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">i</span><span className="mrel,mtight">=</span><span className="mord,mtight">1</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2777em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size3">(</span></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.4911em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">σ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.7959em;"><span style="top:-2.4231em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.0448em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">−</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord"><span className="mord,mathnormal">μ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose"><span className="mclose">)</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9629em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">2</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">σ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size3">)</span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">lo<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord">2</span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size4">)</span></span></span></span></span></span></span></note><h3 id="轨迹">轨迹</h3><p>轨迹（trajectory）<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>τ</mi></mrow><annotation encoding="application/x-tex">\tau</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span></span></span></span> 是世界中状态与动作构成的序列：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>τ</mi><mo>=</mo><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><msub><mi>a</mi><mn>0</mn></msub><mo separator="true">,</mo><msub><mi>s</mi><mn>1</mn></msub><mo separator="true">,</mo><msub><mi>a</mi><mn>1</mn></msub><mo separator="true">,</mo><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\tau = (s_0, a_0, s_1, a_1, ...)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">1</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">1</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord">...</span><span className="mclose">)</span></span></span></span></span><p>世界的第一个状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mn>0</mn></msub></mrow><annotation encoding="application/x-tex">s_0</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 从<strong>初始状态分布（start-state distribution）</strong> 中随机采样，后者有时记作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ρ</mi><mn>0</mn></msub></mrow><annotation encoding="application/x-tex">\rho_0</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">ρ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>s</mi><mn>0</mn></msub><mo>∼</mo><msub><mi>ρ</mi><mn>0</mn></msub><mo stretchy="false">(</mo><mo>⋅</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">s_0 \sim \rho_0(\cdot)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">ρ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord">⋅</span><span className="mclose">)</span></span></span></span></span><p>状态转移（从时刻 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6151em;"></span><span className="mord,mathnormal">t</span></span></span></span> 的世界状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">s_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 到时刻 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">t+1</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6984em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal">t</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">1</span></span></span></span> 的状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow><annotation encoding="application/x-tex">s_{t+1}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6389em;vertical-align:-0.2083em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span></span></span></span> 之间世界发生了什么）由环境的自然法则支配，且只依赖最近的动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>a</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">a_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>。它可以是确定型的：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">s_{t+1} = f(s_t, a_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6389em;vertical-align:-0.2083em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.1076em;">f</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>也可以是随机型的：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo>∼</mo><mi>P</mi><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">s_{t+1} \sim P(\cdot|s_t, a_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6389em;vertical-align:-0.2083em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord">⋅</span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>动作则由智能体依照其策略产生。</p><note title="你应该知道"><p>轨迹也常被称为<strong>回合（episode）</strong> 或 rollout。</p></note><h3 id="奖励与回报">奖励与回报</h3><p>奖励函数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi></mrow><annotation encoding="application/x-tex">R</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span></span></span></span> 在强化学习中至关重要。它依赖当前的世界状态、刚采取的动作和下一个世界状态：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>r</mi><mi>t</mi></msub><mo>=</mo><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">r_t = R(s_t, a_t, s_{t+1})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>不过它经常被简化成只依赖当前状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>r</mi><mi>t</mi></msub><mo>=</mo><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">r_t = R(s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>，或依赖状态-动作对 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>r</mi><mi>t</mi></msub><mo>=</mo><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">r_t = R(s_t, a_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>。</p><p>智能体的目标是最大化某种「沿轨迹累积的奖励」，但这话实际可以有几种含义。我们对所有这些情形统一记作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">R(\tau)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span></span></span></span>：具体指哪一种，要么能从上下文分辨，要么无关紧要（因为同样的方程适用于所有情形）。</p><p>一种回报是<strong>有限视界不折扣回报（finite-horizon undiscounted return）</strong>：固定步数窗口内获得的奖励之和：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo>=</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi>T</mi></munderover><msub><mi>r</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">R(\tau) = \sum_{t=0}^T r_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.0954em;vertical-align:-1.2671em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span></span><p>另一种是<strong>无限视界折扣回报（infinite-horizon discounted return）</strong>：智能体<em>曾经</em>获得的全部奖励之和，但按奖励到来时刻的远近打折。这种回报形式引入了折扣因子 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi><mo>∈</mo><mo stretchy="false">(</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\gamma \in (0,1)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7335em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0556em;">γ</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">(</span><span className="mord">0</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord">1</span><span className="mclose">)</span></span></span></span>：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo>=</mo><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mi mathvariant="normal">∞</mi></munderover><msup><mi>γ</mi><mi>t</mi></msup><msub><mi>r</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">R(\tau) = \sum_{t=0}^{\infty} \gamma^t r_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.9185em;vertical-align:-1.2671em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.6514em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∑</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">∞</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0556em;">γ</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8436em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span></span><p>可我们为什么会想要折扣因子？难道不就是把<em>所有</em>奖励都拿到手吗？确实想，但折扣因子既直观又便于数学处理。直观层面：现在的现金好过以后的现金。数学层面：无限视界的奖励和<a href="https://en.wikipedia.org/wiki/Convergent_series" rel="nofollow">可能不收敛</a>到有限值，在方程里难以处理；而有了折扣因子，再加上合理条件，无穷和就收敛了。</p><note title="你应该知道"><p>这两种回报形式在 RL 形式化里界限分明，但深度 RL 的实践常把界线弄得相当模糊：比如我们经常把算法设定为优化不折扣的回报，却在估计<strong>价值函数</strong>时使用折扣因子。</p></note><h3 id="rl-问题">RL 问题</h3><p>无论选哪种回报度量（无限视界折扣，还是有限视界不折扣），也无论策略怎么选，RL 的目标都是：挑出一个策略，使智能体按它行动时的<strong>期望回报（expected return）</strong> 最大。</p><p>要谈期望回报，得先谈轨迹上的概率分布。</p><p>假设环境转移和策略都是随机型的。这时，一条 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi></mrow><annotation encoding="application/x-tex">T</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">T</span></span></span></span> 步轨迹的概率是：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>π</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi>ρ</mi><mn>0</mn></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo stretchy="false">)</mo><munderover><mo>∏</mo><mrow><mi>t</mi><mo>=</mo><mn>0</mn></mrow><mrow><mi>T</mi><mo>−</mo><mn>1</mn></mrow></munderover><mi>P</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mi>π</mi><mo stretchy="false">(</mo><msub><mi>a</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">P(\tau|\pi) = \rho_0 (s_0) \prod_{t=0}^{T-1} P(s_{t+1} | s_t, a_t) \pi(a_t | s_t)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.0954em;vertical-align:-1.2671em;"></span><span className="mord"><span className="mord,mathnormal">ρ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mrel,mtight">=</span><span className="mord,mtight">0</span></span></span></span><span style="top:-3.05em;"><span className="pstrut" style="height:3.05em;"></span><span><span className="mop,op-symbol,large-op">∏</span></span></span><span style="top:-4.3em;margin-left:0em;"><span className="pstrut" style="height:3.05em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">T</span><span className="mbin,mtight">−</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span></span><p>期望回报（无论按哪种度量）记作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><mi>π</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">J(\pi)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mclose">)</span></span></span></span>：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><mi>π</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mo>∫</mo><mi>τ</mi></msub><mi>P</mi><mo stretchy="false">(</mo><mi>τ</mi><mi mathvariant="normal">∣</mi><mi>π</mi><mo stretchy="false">)</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><mi>π</mi></mrow></munder></mi><mo stretchy="false">[</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">J(\pi) = \int_{\tau} P(\tau|\pi) R(\tau) = \underset{\tau\sim \pi}{\mathbb{E}}[R(\tau)]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.2719em;vertical-align:-0.9119em;"></span><span className="mop"><span className="mop,op-symbol,large-op" style="margin-right:0.4445em;position:relative;top:-0.0011em;">∫</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:-0.6106em;"><span style="top:-1.7881em;margin-left:-0.4445em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.9119em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mord">∣</span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span></span><span className="mopen">[</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)]</span></span></span></span></span><p>RL 的中心优化问题于是可以写成：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>π</mi><mo>∗</mo></msup><mo>=</mo><mi>arg</mi><mo>⁡</mo><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><mi>π</mi></munder><mi>J</mi><mo stretchy="false">(</mo><mi>π</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\pi^* = \arg \max_{\pi} J(\pi)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7387em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mop">ar<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.4306em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">max</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0962em;">J</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mclose">)</span></span></span></span></span><p>其中的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>π</mi><mo>∗</mo></msup></mrow><annotation encoding="application/x-tex">\pi^*</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6887em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span></span></span></span> 就是<strong>最优策略（optimal policy）</strong>。</p><h3 id="价值函数">价值函数</h3><p>知道一个状态或状态-动作对的<strong>价值（value）</strong> 常常很有用。所谓价值，意思是：如果从该状态或状态-动作对出发，之后一直按某个特定策略行动，所能得到的期望回报。<strong>价值函数（value function）</strong> 以这样或那样的方式出现在几乎每一个 RL 算法里。</p><p>这里有四个值得注意的主要函数。</p><ol><li><strong>同策略价值函数（on-policy value function）</strong> <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">V^{\pi}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>：从状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 出发、之后一直按策略 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span></span></span></span> 行动，能得到的期望回报：<span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><mi>π</mi></mrow></munder></mi><mrow><mo fence="true">[</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo>∣</mo><msub><mi>s</mi><mn>0</mn></msub><mo>=</mo><mi>s</mi><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">V^{\pi}(s) = \underset{\tau \sim \pi}{\mathbb{E}}\left[ R(\tau) \mid s_0 = s \right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∣</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord,mathnormal">s</span><span className="mclose,delimcenter" style="top:0em;">]</span></span></span></span></span></span></li><li><strong>同策略动作价值函数（on-policy action-value function）</strong> <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">Q^{\pi}(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span>：从状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 出发，先采取一个任意动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span></span></span></span>（它可以不来自策略），之后一直按策略 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span></span></span></span> 行动，能得到的期望回报：<span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>Q</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><mi>π</mi></mrow></munder></mi><mrow><mo fence="true">[</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo>∣</mo><msub><mi>s</mi><mn>0</mn></msub><mo>=</mo><mi>s</mi><mo separator="true">,</mo><msub><mi>a</mi><mn>0</mn></msub><mo>=</mo><mi>a</mi><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">Q^{\pi}(s,a) = \underset{\tau \sim \pi}{\mathbb{E}}\left[ R(\tau) \mid s_0 = s, a_0 = a \right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∣</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord,mathnormal">a</span><span className="mclose,delimcenter" style="top:0em;">]</span></span></span></span></span></span></li><li><strong>最优价值函数（optimal value function）</strong> <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>V</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">V^*(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>：从状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 出发、之后一直按环境中<em>最优</em>的策略行动，能得到的期望回报：<span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>V</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo>=</mo><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><mi>π</mi></munder><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><mi>π</mi></mrow></munder></mi><mrow><mo fence="true">[</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo>∣</mo><msub><mi>s</mi><mn>0</mn></msub><mo>=</mo><mi>s</mi><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">V^*(s) = \max_{\pi} \underset{\tau \sim \pi}{\mathbb{E}}\left[ R(\tau) \mid s_0 = s \right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.4306em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">max</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∣</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord,mathnormal">s</span><span className="mclose,delimcenter" style="top:0em;">]</span></span></span></span></span></span></li><li><strong>最优动作价值函数（optimal action-value function）</strong> <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">Q^*(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span>：从状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 出发，先采取一个任意动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span></span></span></span>，之后一直按环境中<em>最优</em>的策略行动，能得到的期望回报：<span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo><mo>=</mo><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><mi>π</mi></munder><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>τ</mi><mo>∼</mo><mi>π</mi></mrow></munder></mi><mrow><mo fence="true">[</mo><mi>R</mi><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo><mo>∣</mo><msub><mi>s</mi><mn>0</mn></msub><mo>=</mo><mi>s</mi><mo separator="true">,</mo><msub><mi>a</mi><mn>0</mn></msub><mo>=</mo><mi>a</mi><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">Q^*(s,a) = \max_{\pi} \underset{\tau \sim \pi}{\mathbb{E}}\left[ R(\tau) \mid s_0 = s, a_0 = a \right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.4306em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">max</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.1132em;">τ</span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.1132em;">τ</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∣</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord,mathnormal">a</span><span className="mclose,delimcenter" style="top:0em;">]</span></span></span></span></span></span></li></ol><note title="你应该知道"><p>谈价值函数时，如果没有专门提到对时间的依赖，指的都是期望<strong>无限视界折扣回报</strong>。有限视界不折扣回报的价值函数必须把时间作为参数传入。你能想到为什么吗？提示：时间到了会发生什么？</p></note><note title="你应该知道"><p>价值函数与动作价值函数之间有两个经常用到的关键联系：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><mi>a</mi><mo>∼</mo><mi>π</mi></mrow></munder></mi><mrow><mo fence="true">[</mo><msup><mi>Q</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">V^{\pi}(s) = \underset{a\sim \pi}{\mathbb{E}}\left[ Q^{\pi}(s,a) \right]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">a</span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">]</span></span></span></span></span></span><p>以及</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>V</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo>=</mo><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><mi>a</mi></munder><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">V^*(s) = \max_a Q^*(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.4306em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">a</span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">max</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span></span><p>这些关系可以直接从刚才给出的定义推出来：你能证明吗？</p></note><h3 id="最优-q-函数与最优动作">最优 Q 函数与最优动作</h3><p>最优动作价值函数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">Q^*(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span> 与最优策略所选的动作之间有一个重要联系。按定义，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">Q^*(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span> 给出的是：从状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 出发、先采取（任意的）动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span></span></span></span>、之后一直按最优策略行动时的期望回报。</p><p>状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 处的最优策略，会选择使「从 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 出发的期望回报」最大的那个动作。因此，如果我们手里有 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup></mrow><annotation encoding="application/x-tex">Q^*</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8831em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span></span></span></span>，就可以直接得到最优动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>a</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a^*(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span>：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>a</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo>=</mo><mi>arg</mi><mo>⁡</mo><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><mi>a</mi></munder><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a^*(s) = \arg \max_a Q^*(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.45em;vertical-align:-0.7em;"></span><span className="mop">ar<span style="margin-right:0.0139em;">g</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.4306em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">a</span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">max</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span></span><p>注意：可能有多个动作同时使 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">Q^*(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6887em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span> 最大，此时它们全都是最优的，最优策略可以随机选择其中任何一个。但总存在一个确定型地选择动作的最优策略。</p><h3 id="贝尔曼方程">贝尔曼方程</h3><p>四个价值函数全都满足一类特殊的自洽方程，称为<strong>贝尔曼方程（Bellman equations）</strong>。贝尔曼方程背后的基本想法是：</p><blockquote><p>你出发点的价值，等于你期望从待在那里得到的奖励，加上你下一落脚点的价值。</p></blockquote><p>同策略价值函数的贝尔曼方程为：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable rowspacing="0.25em" columnalign="right left" columnspacing="0em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mstyle scriptlevel="1"><mtable rowspacing="0.1em" columnalign="center" columnspacing="1em"><mtr><mtd><mstyle scriptlevel="1" displaystyle="false"><mrow><mi>a</mi><mo>∼</mo><mi>π</mi></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="1" displaystyle="false"><mrow><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>∼</mo><mi>P</mi></mrow></mstyle></mtd></mtr></mtable></mstyle></munder></mi><mrow><mo fence="true">[</mo><mi>r</mi><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo><mo>+</mo><mi>γ</mi><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><msup><mi>Q</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>∼</mo><mi>P</mi></mrow></munder></mi><mrow><mo fence="true">[</mo><mi>r</mi><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo><mo>+</mo><mi>γ</mi><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><msup><mi>a</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>∼</mo><mi>π</mi></mrow></munder></mi><mrow><mo fence="true">[</mo><msup><mi>Q</mi><mi>π</mi></msup><mo stretchy="false">(</mo><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo separator="true">,</mo><msup><mi>a</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow><mo fence="true">]</mo></mrow></mrow></mstyle></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{aligned}
V^{\pi}(s) &amp;= \underset{\substack{a \sim \pi \\ s&#39;\sim P}}{\mathbb{E}}\left[r(s,a) + \gamma V^{\pi}(s&#39;)\right] \\
Q^{\pi}(s,a) &amp;= \underset{s&#39;\sim P}{\mathbb{E}}\left[r(s,a) + \gamma \underset{a&#39;\sim \pi}{\mathbb{E}}\left[Q^{\pi}(s&#39;,a&#39;)\right]\right]
\end{aligned}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:4.4332em;vertical-align:-1.9666em;"></span><span className="mord"><span className="mtable"><span className="col-align-r"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:2.4666em;"><span style="top:-4.7766em;"><span className="pstrut" style="height:3.15em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span><span style="top:-1.9277em;"><span className="pstrut" style="height:3.15em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.9666em;"><span></span></span></span></span></span><span className="col-align-l"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:2.4666em;"><span style="top:-4.7766em;"><span className="pstrut" style="height:3.15em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.0926em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mtable"><span className="col-align-c"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0592em;"><span style="top:-3.4565em;"><span className="pstrut" style="height:2.8278em;"></span><span className="mord,mtight"><span className="mord,mathnormal,mtight">a</span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span><span style="top:-2.4486em;"><span className="pstrut" style="height:2.8278em;"></span><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8278em;"><span style="top:-2.931em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">P</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.5592em;"><span></span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.3989em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.0556em;">γ</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8019em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">]</span></span></span></span><span style="top:-1.9277em;"><span className="pstrut" style="height:3.15em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.3557em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">P</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7443em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size2">[</span></span><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.0556em;">γ</span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.356em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">a</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.744em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8019em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8019em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">]</span></span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size2">]</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.9666em;"><span></span></span></span></span></span></span></span></span></span></span></span><p>其中 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>∼</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">s&#39; \sim P</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7519em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7519em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span></span></span></span> 是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>∼</mo><mi>P</mi><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">s&#39; \sim P(\cdot|s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7519em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7519em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord">⋅</span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span> 的简写，表示下一状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></mrow><annotation encoding="application/x-tex">s&#39;</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7519em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7519em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span> 从环境的转移规则中采样；<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo>∼</mo><mi>π</mi></mrow><annotation encoding="application/x-tex">a \sim \pi</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span></span></span></span> 是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo>∼</mo><mi>π</mi><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a \sim \pi(\cdot|s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mopen">(</span><span className="mord">⋅</span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 的简写；<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>a</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>∼</mo><mi>π</mi></mrow><annotation encoding="application/x-tex">a&#39; \sim \pi</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7519em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7519em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span></span></span></span> 是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>a</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>∼</mo><mi>π</mi><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a&#39; \sim \pi(\cdot|s&#39;)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7519em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7519em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∼</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.0019em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mopen">(</span><span className="mord">⋅</span><span className="mord">∣</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7519em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 的简写。</p><p>最优价值函数的贝尔曼方程为：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable rowspacing="0.25em" columnalign="right left" columnspacing="0em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><msup><mi>V</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><mi>a</mi></munder><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>∼</mo><mi>P</mi></mrow></munder></mi><mrow><mo fence="true">[</mo><mi>r</mi><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo><mo>+</mo><mi>γ</mi><msup><mi>V</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><mi><munder><mo><mi mathvariant="double-struck">E</mi></mo><mrow><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>∼</mo><mi>P</mi></mrow></munder></mi><mrow><mo fence="true">[</mo><mi>r</mi><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo><mo>+</mo><mi>γ</mi><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><msup><mi>a</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></munder><msup><mi>Q</mi><mo>∗</mo></msup><mo stretchy="false">(</mo><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo separator="true">,</mo><msup><mi>a</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow></mstyle></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{aligned}
V^*(s) &amp;= \max_a \underset{s&#39;\sim P}{\mathbb{E}}\left[r(s,a) + \gamma V^*(s&#39;)\right] \\
Q^*(s,a) &amp;= \underset{s&#39;\sim P}{\mathbb{E}}\left[r(s,a) + \gamma \max_{a&#39;} Q^*(s&#39;,a&#39;)\right]
\end{aligned}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:3.7787em;vertical-align:-1.6393em;"></span><span className="mord"><span className="mtable"><span className="col-align-r"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:2.1393em;"><span style="top:-4.4493em;"><span className="pstrut" style="height:3.15em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span><span style="top:-2.255em;"><span className="pstrut" style="height:3.15em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.6393em;"><span></span></span></span></span></span><span className="col-align-l"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:2.1393em;"><span style="top:-4.4493em;"><span className="pstrut" style="height:3.15em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.4306em;"><span style="top:-2.4em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">a</span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">max</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.3557em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">P</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7443em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">[</span><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.0556em;">γ</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8019em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;">]</span></span></span></span><span style="top:-2.255em;"><span className="pstrut" style="height:3.15em;"></span><span className="mord"><span className="mord"></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mord"><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6889em;"><span style="top:-2.3557em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mrel,mtight">∼</span><span className="mord,mathnormal,mtight" style="margin-right:0.1389em;">P</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop,mathbb">E</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7443em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size2">[</span></span><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.0556em;">γ</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.4306em;"><span style="top:-2.356em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">a</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6828em;"><span style="top:-2.786em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">max</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.744em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7387em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mbin,mtight">∗</span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8019em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8019em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mclose">)</span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size2">]</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.6393em;"><span></span></span></span></span></span></span></span></span></span></span></span><p>同策略价值函数与最优价值函数的贝尔曼方程之间，关键区别在对动作取不取 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>max</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\max</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mop">max</span></span></span></span>。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>max</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\max</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mop">max</span></span></span></span> 的存在反映了这样一个事实：每当智能体有机会自己挑动作，要想最优地行动，就必须挑那个带来最高价值的动作。</p><note title="你应该知道"><p>「贝尔曼备份（Bellman backup）」这个词在 RL 文献中出现得非常频繁。一个状态或状态-动作对的贝尔曼备份，就是贝尔曼方程的右边：奖励加下一状态的价值。</p></note><h3 id="优势函数">优势函数</h3><p>有时在 RL 中，我们不需要绝对地描述一个动作有多好，只需要知道它平均而言比别的动作好多少。也就是说，我们想知道该动作的相对<strong>优势（advantage）</strong>。我们用<strong>优势函数（advantage function）</strong> 把这个概念精确化。</p><p>对应策略 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span></span></span></span> 的优势函数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>A</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">A^{\pi}(s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">A</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span> 描述的是：在状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 采取特定动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span></span></span></span>，比按 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\pi(\cdot|s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mopen">(</span><span className="mord">⋅</span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 随机选一个动作好多少；这里假设之后一直按 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span></span></span></span> 行动。数学上，优势函数定义为：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msup><mi>A</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo><mo>=</mo><msup><mi>Q</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo><mo>−</mo><msup><mi>V</mi><mi>π</mi></msup><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">A^{\pi}(s,a) = Q^{\pi}(s,a) - V^{\pi}(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">A</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">Q</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">−</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.2222em;">V</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7144em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">π</span></span></span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span></span><note title="你应该知道"><p>这一点后面还会细讲：优势函数对策略梯度方法至关重要。</p></note><h2 id="选读形式化">（选读）形式化</h2><p>到目前为止，我们一直在非正式地讨论智能体所处的环境。但如果去翻文献，你很可能会遇到这个设定下的标准数学形式化：<strong>MDP（Markov Decision Process，马尔可夫决策过程）</strong>。一个 MDP 是五元组 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">⟨</mo><mi>S</mi><mo separator="true">,</mo><mi>A</mi><mo separator="true">,</mo><mi>R</mi><mo separator="true">,</mo><mi>P</mi><mo separator="true">,</mo><msub><mi>ρ</mi><mn>0</mn></msub><mo stretchy="false">⟩</mo></mrow><annotation encoding="application/x-tex">\langle S, A, R, P, \rho_0 \rangle</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">⟨</span><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">A</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">ρ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">⟩</span></span></span></span>，其中：</p><ul><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi></mrow><annotation encoding="application/x-tex">S</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span></span></span></span> 是所有合法状态的集合；</li><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>A</mi></mrow><annotation encoding="application/x-tex">A</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">A</span></span></span></span> 是所有合法动作的集合；</li><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo>:</mo><mi>S</mi><mo>×</mo><mi>A</mi><mo>×</mo><mi>S</mi><mo>→</mo><mi mathvariant="double-struck">R</mi></mrow><annotation encoding="application/x-tex">R : S \times A \times S \to \mathbb{R}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">:</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal">A</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6889em;"></span><span className="mord,mathbb">R</span></span></span></span> 是奖励函数，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>r</mi><mi>t</mi></msub><mo>=</mo><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>a</mi><mi>t</mi></msub><mo separator="true">,</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">r_t = R(s_t, a_t, s_{t+1})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">a</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">+</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>；</li><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mo>:</mo><mi>S</mi><mo>×</mo><mi>A</mi><mo>→</mo><mi mathvariant="script">P</mi><mo stretchy="false">(</mo><mi>S</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">P : S \times A \to \mathcal{P}(S)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">:</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">A</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathcal" style="margin-right:0.0822em;">P</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span><span className="mclose">)</span></span></span></span> 是转移概率函数（transition probability function），<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mi mathvariant="normal">∣</mi><mi>s</mi><mo separator="true">,</mo><mi>a</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">P(s&#39;|s,a)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0019em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7519em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span><span className="mord">∣</span><span className="mord,mathnormal">s</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">a</span><span className="mclose">)</span></span></span></span> 表示从状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 出发采取动作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">a</span></span></span></span> 后转移到状态 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>s</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup></mrow><annotation encoding="application/x-tex">s&#39;</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7519em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7519em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span></span></span></span></span></span></span></span> 的概率；</li><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ρ</mi><mn>0</mn></msub></mrow><annotation encoding="application/x-tex">\rho_0</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">ρ</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 是初始状态分布。</li></ul><p>「马尔可夫决策过程」这个名字来自系统服从<a href="https://en.wikipedia.org/wiki/Markov_property" rel="nofollow">马尔可夫性质（Markov property）</a>这一事实：转移只依赖最近的状态和动作，不依赖更早的历史。</p><style>html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}</style>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM 的 Memory-Compute Trade-off：从 MHA 到 MLA、Sparse 与 Linear Attention]]></title>
            <description><![CDATA[ds4.1f 上线了。它的架构 novel 在哪，便宜和快又是从哪省出来的。]]></description>
            <link>https://blog.niina.fun/posts/llm-memory-compute-tradeoff</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/llm-memory-compute-tradeoff</guid>
            <pubDate>Fri, 11 Sep 2026 03:00:00 GMT</pubDate>
            <content:encoded><![CDATA[<p>GPT6 出来之后，体感上很多人都很焦虑迷茫，真的到了 AGI time，接下来该做什么呢？虽然说 Astra 用下来，在一些 case 上还是会 overfit，但真的已经能胜任大多数人完成的任务了（在数字世界里）。这两天 ds4.1f 正式上线了，如此 novel 的架构让我感到新奇，模型架构与硬件的 co-design 达到一定水平的杰作，让我们来看一下吧。<strong>这篇由 AI 整理。</strong></p><p>从 MHA 出发，一路到 Linear Attention，看每一代分别在砍哪一项成本。</p><h2 id="mha-与-kv-cache">MHA 与 KV Cache</h2><p>MHA（Multi-Head Attention）是这一切的起点，2017 年《Attention Is All You Need》里提出来的。数学上它就是多个独立的单头注意力拼起来。假设输入向量序列是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>1</mn></msub><mo separator="true">,</mo><msub><mi>x</mi><mn>2</mn></msub><mo separator="true">,</mo><mo>⋯</mo><mtext> </mtext><mo separator="true">,</mo><msub><mi>x</mi><mi>l</mi></msub></mrow><annotation encoding="application/x-tex">x_1, x_2, \cdots, x_l</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">1</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner">⋯</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0197em;">l</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>，其中 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mi>i</mi></msub><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><mi>d</mi></msup></mrow><annotation encoding="application/x-tex">x_i \in \mathbb{R}^d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6891em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8491em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8491em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">d</span></span></span></span></span></span></span></span></span></span></span>。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">h</span></span></span></span> 是 head 数，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 是 head 的下标，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mtext> </mtext><mo separator="true">,</mo><mi>h</mi></mrow><annotation encoding="application/x-tex">s = 1, 2, \cdots, h</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord">1</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord">2</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner">⋯</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">h</span></span></span></span>。那么：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>o</mi><mi>t</mi></msub><mo>=</mo><mo stretchy="false">[</mo><msubsup><mi>o</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><msubsup><mi>o</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><mo>⋯</mo><mtext> </mtext><mo separator="true">,</mo><msubsup><mi>o</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mi>h</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">o_t = [o_t^{(1)}, o_t^{(2)}, \cdots, o_t^{(h)}]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">o</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.2948em;vertical-align:-0.25em;"></span><span className="mopen">[</span><span className="mord"><span className="mord,mathnormal">o</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mtight">1</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">o</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mtight">2</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner">⋯</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">o</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">h</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mclose">]</span></span></span></span></span><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msubsup><mi>o</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mrow><mi mathvariant="normal">A</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">n</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">n</mi></mrow><mo stretchy="false">(</mo><msubsup><mi>q</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><msubsup><mi>k</mi><mrow><mo>≤</mo><mi>t</mi></mrow><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><msubsup><mi>v</mi><mrow><mo>≤</mo><mi>t</mi></mrow><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>≜</mo><mfrac><mrow><munder><mo>∑</mo><mrow><mi>i</mi><mo>≤</mo><mi>t</mi></mrow></munder><mi>exp</mi><mo>⁡</mo><mo stretchy="false">(</mo><msubsup><mi>q</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>k</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup><mo stretchy="false">)</mo><msubsup><mi>v</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><munder><mo>∑</mo><mrow><mi>i</mi><mo>≤</mo><mi>t</mi></mrow></munder><mi>exp</mi><mo>⁡</mo><mo stretchy="false">(</mo><msubsup><mi>q</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>k</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">o_t^{(s)} = \mathrm{Attention}(q_t^{(s)}, k^{(s)}_{\le t}, v^{(s)}_{\le t}) \triangleq \frac{\sum_{i \le t} \exp(q_t^{(s)} k_i^{(s)\top}) v_i^{(s)}}{\sum_{i \le t} \exp(q_t^{(s)} k_i^{(s)\top})}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.2906em;vertical-align:-0.2458em;"></span><span className="mord"><span className="mord,mathnormal">o</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.4004em;vertical-align:-0.3556em;"></span><span className="mord"><span className="mord,mathrm">Attention</span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4396em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mrel,mtight">≤</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3556em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4396em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mrel,mtight">≤</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3556em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel,amsrm">≜</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:3.1594em;vertical-align:-1.3297em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.8297em;"><span style="top:-2.11em;"><span className="pstrut" style="height:3.0448em;"></span><span className="mord"><span className="mop"><span className="mop,op-symbol,small-op" style="position:relative;top:0em;">∑</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.162em;"><span style="top:-2.4003em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">i</span><span className="mrel,mtight">≤</span><span className="mord,mathnormal,mtight">t</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3949em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">exp</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span><span style="top:-3.2748em;"><span className="pstrut" style="height:3.0448em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.8297em;"><span className="pstrut" style="height:3.0448em;"></span><span className="mord"><span className="mop"><span className="mop,op-symbol,small-op" style="position:relative;top:0em;">∑</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.162em;"><span style="top:-2.4003em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">i</span><span className="mrel,mtight">≤</span><span className="mord,mathnormal,mtight">t</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3949em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mop">exp</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.3297em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span></span><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msubsup><mi>q</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><msub><mi>x</mi><mi>i</mi></msub><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><msub><mi>d</mi><mi>k</mi></msub></msup><mo separator="true">,</mo><mspace width="1em"></mspace><msubsup><mi>k</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><msub><mi>x</mi><mi>i</mi></msub><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><msub><mi>d</mi><mi>k</mi></msub></msup><mo separator="true">,</mo><mspace width="1em"></mspace><msubsup><mi>v</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><msub><mi>x</mi><mi>i</mi></msub><msubsup><mi>W</mi><mi>v</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><msub><mi>d</mi><mi>v</mi></msub></msup></mrow><annotation encoding="application/x-tex">q_i^{(s)} = x_i W_q^{(s)} \in \mathbb{R}^{d_k}, \quad k_i^{(s)} = x_i W_k^{(s)} \in \mathbb{R}^{d_k}, \quad v_i^{(s)} = x_i W_v^{(s)} \in \mathbb{R}^{d_v}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3217em;vertical-align:-0.2769em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.3211em;vertical-align:-0.3831em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-2.453em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3831em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.3217em;vertical-align:-0.2769em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:1em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.3461em;vertical-align:-0.3013em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.3217em;vertical-align:-0.2769em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:1em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.185em;vertical-align:-0.247em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-2.453em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.247em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8991em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1645em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.143em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><note><p>（省略了缩放因子。常见设置是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>k</mi></msub><mo>=</mo><msub><mi>d</mi><mi>v</mi></msub><mo>=</mo><mi>d</mi><mi mathvariant="normal">/</mi><mi>h</mi></mrow><annotation encoding="application/x-tex">d_k = d_v = d/h</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">d</span><span className="mord">/</span><span className="mord,mathnormal">h</span></span></span></span>。）关键在因果性。自回归生成时，新预测出来的第 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">t+1</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6984em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal">t</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">1</span></span></span></span> 个 token 不会影响已经算好的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>k</mi><mrow><mo>≤</mo><mi>t</mi></mrow><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><annotation encoding="application/x-tex">k^{(s)}_{\le t}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.4004em;vertical-align:-0.3556em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4396em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mrel,mtight">≤</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3556em;"><span></span></span></span></span></span></span></span></span></span> 和 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>v</mi><mrow><mo>≤</mo><mi>t</mi></mrow><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><annotation encoding="application/x-tex">v^{(s)}_{\le t}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.4004em;vertical-align:-0.3556em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4396em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mrel,mtight">≤</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3556em;"><span></span></span></span></span></span></span></span></span></span>，所以这部分可以缓存下来，避免重复计算。这就是 KV Cache。它的大小约等于：</p></note><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>2</mn><mo>×</mo><mi>L</mi><mo>×</mo><mi>h</mi><mo>×</mo><msub><mi>d</mi><mi>k</mi></msub><mo>×</mo><mi>N</mi><mo>×</mo><mtext>bytes</mtext></mrow><annotation encoding="application/x-tex">2 \times L \times h \times d_k \times N \times \text{bytes}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">2</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal">L</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.7778em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal">h</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord,text"><span className="mord">bytes</span></span></span></span></span></span><p>其中 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">L</span></span></span></span> 是层数，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 是序列长度。注意这里面的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span>。模型参数是个常数，选定模型就不再变了；但 KV Cache 随 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 线性增长。Context 足够长的时候，它会变成显存里的主导项，可能超出一张卡甚至一台机（8 张卡）的总显存。举个数。按 GPT-3 的配置算：96 层，96 个 attention head，每个 head 维度 128。单个 token 的 KV Cache 是</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>2</mn><mo>×</mo><mn>96</mn><mo>×</mo><mn>96</mn><mo>×</mo><mn>128</mn><mo>≈</mo><mn>2.36</mn><mtext>M</mtext></mrow><annotation encoding="application/x-tex">2 \times 96 \times 96 \times 128 \approx 2.36\text{M}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">2</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">96</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">96</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">128</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">2.36</span><span className="mord,text"><span className="mord">M</span></span></span></span></span></span><p>个参数。按 2 字节精度存，就是 4.7 MB。一个 token 4.7 MB。128k context 就是 600 GB 量级。这个数字基本说明了为什么长上下文不是&quot;再多加个位置编码&quot;就能解决的。它是个显存和带宽问题。而 GPU 部署有个很不讲道理的原则：能一张卡放下的，就不要跨多张卡；能一台机放下的，就不要跨多台机。因为卡内通信带宽 &gt; 卡间通信带宽 &gt; 机间通信带宽。木桶效应，跨的设备越多，被最慢那段拖累得越狠。</p><p>单卡 H100 内 SRAM 到 HBM 的带宽已经有 3 TB/s，对 short context 来说这依然是瓶颈，更不用说更慢的卡间和机间了。</p><h2 id="prefill-与-decode">Prefill 与 Decode</h2><p>LLM 推理分两个阶段，它们的成本结构不同。<strong>Prefill</strong>：把输入的所有 token 并行算一遍，同时把 KV Cache 建起来。所有 token 一起算，矩阵运算排得很满，算力基本能吃满。它是 <strong>compute bound</strong>。<strong>Decode</strong>：之后逐个生成 token。这里有个容易漏掉的点。</p><p>每生成一个 token，计算量很小，但要读的东西一点都不少：<strong>模型权重</strong>要整体过一遍，KV Cache 也要读一遍。</p><p>不是读一部分权重，也不是只读 attention 那几层。每一步都要把参与计算的权重从 HBM 搬进 SM。算得少，搬得多。所以 decode 是 <strong>memory bound</strong>。同一个模型，两个阶段的瓶颈完全不同。如果一个架构在两种形式下数学等价，那就可以 prefill 用一种算，decode 用另一种算，各取所需。MLA 就是这么干的。</p><p>后面 V4.1 那一节会把这两个瓶颈各自的账算一遍。最后把账单写全：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>C</mi><mtext>total</mtext></msub><mo>=</mo><msub><mi>C</mi><mtext>attention</mtext></msub><mo>+</mo><msub><mi>C</mi><mtext>FFN/MoE</mtext></msub><mo>+</mo><msub><mi>C</mi><mtext>memory movement</mtext></msub></mrow><annotation encoding="application/x-tex">C_{\text{total}} = C_{\text{attention}} + C_{\text{FFN/MoE}} + C_{\text{memory movement}}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0715em;">C</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">total</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0715em;">C</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3175em;"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">attention</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1.0385em;vertical-align:-0.3552em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0715em;">C</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.5198em;margin-left:-0.0715em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">FFN/MoE</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3552em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.9694em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0715em;">C</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">memory movement</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span></span></span></span></span><p>账单上还有另外两项，而且它们经常被无视。第二项 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mtext>FFN/MoE</mtext></msub></mrow><annotation encoding="application/x-tex">C_{\text{FFN/MoE}}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0385em;vertical-align:-0.3552em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0715em;">C</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.5198em;margin-left:-0.0715em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">FFN/MoE</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3552em;"><span></span></span></span></span></span></span></span></span></span> 是前馈层的计算。MoE（Mixture of Experts）就是把这层换成多个专家网络，每个 token 只激活其中少数几个。它跟序列长度 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 成正比，跟模型宽度 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span> 的平方成正比，所以量级是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><msup><mi>d</mi><mn>2</mn></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(Nd^2)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0641em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>。attention 那项是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><msup><mi>N</mi><mn>2</mn></msup><mi>d</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N^2 d)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0641em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span><span className="mord,mathnormal">d</span><span className="mclose">)</span></span></span></span>。注意这两个量级里的指数位置不一样：<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 小的时候，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>N</mi><mn>2</mn></msup><mi>d</mi></mrow><annotation encoding="application/x-tex">N^2 d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8141em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span><span className="mord,mathnormal">d</span></span></span></span> 打不过 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><msup><mi>d</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">Nd^2</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8141em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span></span></span></span>；<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 大起来之后才反过来。</p><p>这也是为什么在短 context 时代大家理所当然地只盯 attention。那时候它确实是瓶颈。但当 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 从 4k 涨到 1M，情况变了；再等到 sparse attention 把 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>N</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">N^2</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8141em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span></span></span></span> 压下去，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><msup><mi>d</mi><mn>2</mn></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(Nd^2)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0641em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 这一项就浮到水面上了。第三项 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mtext>memory movement</mtext></msub></mrow><annotation encoding="application/x-tex">C_{\text{memory movement}}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.9694em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0715em;">C</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">memory movement</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span></span></span></span> 是最容易漏的。GPU 上算一个 FLOP 很快，搬一次数据慢得多，所以&quot;减少 KV Cache&quot;的收益很多时候落在省搬运，而不在省显存。但这件事有个限度：decode 每步要读的权重是固定的，搬不动，KV Cache 压到比它更小之后，继续压的边际收益就下降。这条限度解释了后面很多设计的取舍，也解释了为什么 1M context 之后，战场转向每层的计算和权重：账单上还能砍的地方换位置了。</p><h2 id="mqa-与-gqa">MQA 与 GQA</h2><p>既然 KV Cache 是问题，最朴素的想法就是：少存点。MQA（Multi-Query Attention）直接让所有 attention head 共享同一份 K、V。用公式说，就是把所有 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo separator="true">,</mo><mi>v</mi></mrow><annotation encoding="application/x-tex">k, v</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span></span></span></span> 的上标 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 去掉。它是 2019 年提出来的。那会儿 LLM 还没火，可见减少 KV Cache 早就已经是热门课题了。KV Cache 直接降到原来的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi>h</mi></mrow><annotation encoding="application/x-tex">1/h</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">1/</span><span className="mord,mathnormal">h</span></span></span></span>。单从省显存的角度看，这已经是天花板了。</p><p>代价也很直白：原来一个 token 的每个 head 都有各自的 K、V，现在被压成一份，信息损失不小。</p><p>不过 MQA 的支持者认为这个损失被高估了。理由是不同 head 学到的 K、V 本来就高度相关，共享一份丢不掉太多有效信息；而且这部分损失可以通过继续训练补回来。这个论证后来被证明有一定道理，但&quot;一定道理&quot;和&quot;可以无条件依赖&quot;是两回事。GQA 的出现本身就说明，业界并不敢把压缩率开到 MQA 那么极端。</p><p>有个细节容易被忽略。MQA 共享 K、V 之后，attention 部分的参数量少了将近一半。为了保持总参数量不变，通常会相应增大 FFN 的规模。这能补回来一部分效果损失。但还有人担心压得太狠，影响学习效率和最终效果。于是有了 GQA（Grouped-Query Attention），那是 2023 年的事。GQA 把 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">h</span></span></span></span> 个 head 分成 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span></span></span></span> 组，每组共享同一对 K、V。具体怎么分？</p><p>记第 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">s</span></span></span></span> 个 head 属于第 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\gamma(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0556em;">γ</span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 组：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>γ</mi><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mo fence="true">⌈</mo><mfrac><mrow><mi>s</mi><mi>g</mi></mrow><mi>h</mi></mfrac><mo fence="true">⌉</mo></mrow></mrow><annotation encoding="application/x-tex">\gamma(s) = \left\lceil \frac{sg}{h} \right\rceil</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0556em;">γ</span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.836em;vertical-align:-0.686em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size2">⌈</span></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.1076em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,mathnormal">h</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,mathnormal">s</span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.686em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size2">⌉</span></span></span></span></span></span></span><p>（就是按顺序平均切开：前 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mi mathvariant="normal">/</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">h/g</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">h</span><span className="mord">/</span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span></span></span></span> 个 head 一组，接着 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mi mathvariant="normal">/</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">h/g</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">h</span><span className="mord">/</span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span></span></span></span> 个一组，以此类推。）于是 attention 写成：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msubsup><mi>o</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mrow><mi mathvariant="normal">A</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">n</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">n</mi></mrow><mrow><mo fence="true">(</mo><msubsup><mi>q</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><mtext> </mtext><msubsup><mi>k</mi><mrow><mo>≤</mo><mi>t</mi></mrow><mrow><mo stretchy="false">(</mo><mi>γ</mi><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><mtext> </mtext><msubsup><mi>v</mi><mrow><mo>≤</mo><mi>t</mi></mrow><mrow><mo stretchy="false">(</mo><mi>γ</mi><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></msubsup><mo fence="true">)</mo></mrow></mrow><annotation encoding="application/x-tex">o_t^{(s)} = \mathrm{Attention}\left(q_t^{(s)},\ k^{(\gamma(s))}_{\le t},\ v^{(\gamma(s))}_{\le t}\right)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.2906em;vertical-align:-0.2458em;"></span><span className="mord"><span className="mord,mathnormal">o</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.8em;vertical-align:-0.65em;"></span><span className="mord"><span className="mord,mathrm">Attention</span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;"><span className="delimsizing,size2">(</span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace"> </span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4396em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mrel,mtight">≤</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight" style="margin-right:0.0556em;">γ</span><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">))</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3556em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace"> </span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4396em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mrel,mtight">≤</span><span className="mord,mathnormal,mtight">t</span></span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight" style="margin-right:0.0556em;">γ</span><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">))</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3556em;"><span></span></span></span></span></span></span><span className="mclose,delimcenter" style="top:0em;"><span className="delimsizing,size2">)</span></span></span></span></span></span></span><p>注意 Q 仍然是每个 head 一份，只有 K、V 被分组共用。这个不对称就是 GQA 的全部内容。举个具体的数。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">h = 8</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">h</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">8</span></span></span></span>、<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">g = 2</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">2</span></span></span></span> 时，第 1 到第 4 个 head 共用一对 K、V，第 5 到第 8 个 head 共用另一对。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>=</mo><mi>h</mi></mrow><annotation encoding="application/x-tex">g = h</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">h</span></span></span></span> 时每个 head 各自一对，退化成 MHA；<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">g = 1</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">1</span></span></span></span> 时所有 head 共用一对，退化成 MQA。</p><p>它提供了 MHA 到 MQA 的自然过渡，压缩率不如 MQA，但自由度和效果保证更好。</p><p>llama2/3-70B 用的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">g = 8</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">8</span></span></span></span>。早年有个很整齐的解释：70B 这个体量放不进单卡，得单机 8 卡，而 attention 的 head 本来就是独立运算再拼接的，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">g = 8</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">8</span></span></span></span> 正好让每张卡负责一组 K、V 对应的 head，把卡间通信压到最小。这个解释现在不太站得住了。量化让更大的模型能进单卡，MoE 让 activated 参数远小于总参数，跨机推理也早就不是禁区。</p><p>两个现成的例子：Qwen3.8-Flash-Next 是 125B 的模型、每个 token 只激活 6B，在 B200 上用 TP4 就能跑（<a href="https://www.lmsys.org/blog/2026-08-26-qwen-flash-next" rel="nofollow">SGLang 的 day-0 支持</a>）；DeepSeek-V4-Flash-FP8 同样 TP4 就能跑，H200 上就够。</p><p>所以更准确的说法是：<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span></span></span></span> 跟并行度对齐仍然是个好主意，但&quot;必须单机 8 卡&quot;这种硬约束已经没有了。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">g = 8</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">8</span></span></span></span> 能留下来，一半是因为并行度，一半是习惯。但 MQA 和 GQA 有一个共同的性质，值得单独点出来：</p><p><strong>它们压的是 head 的数量，不是 head 的内容。</strong></p><p>被保留下来的那些 KV，每一个仍然是完整的、未经压缩的向量。共享是<strong>离散</strong>的。要么整份共用，要么各自一份，中间没有别的可能。下一节就是打破这个限制。</p><h2 id="mla">MLA</h2><p>MLA（Multi-head Latent Attention）是 DeepSeek-V2 在 2024 年 5 月提出来的，到现在两年多。技术报告是从低秩投影的角度引入的，以至于有人问：LoRA 提出这么久了，为什么直到 MLA 才有人对 KV Cache 做低秩分解？我认为低秩投影这个角度不贴近本质。因为要说低秩投影的话，只要把 GQA 的所有 K、V 拼成一个长向量，就会发现 GQA 本身也是低秩投影。</p><p>把这个拼起来的向量记作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">c_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>。它由 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span></span></span></span> 组 K、V 依次首尾相接而成：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>c</mi><mi>i</mi></msub><mo>=</mo><mo stretchy="false">[</mo><msubsup><mi>k</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><mo>⋯</mo><mtext> </mtext><mo separator="true">,</mo><msubsup><mi>k</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>g</mi><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><msubsup><mi>v</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><mo>⋯</mo><mtext> </mtext><mo separator="true">,</mo><msubsup><mi>v</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>g</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">]</mo><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><mrow><mi>g</mi><mo stretchy="false">(</mo><msub><mi>d</mi><mi>k</mi></msub><mo>+</mo><msub><mi>d</mi><mi>v</mi></msub><mo stretchy="false">)</mo></mrow></msup></mrow><annotation encoding="application/x-tex">c_i = [k_i^{(1)}, \cdots, k_i^{(g)}, v_i^{(1)}, \cdots, v_i^{(g)}] \in \mathbb{R}^{g(d_k + d_v)}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.3217em;vertical-align:-0.2769em;"></span><span className="mopen">[</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mtight">1</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner">⋯</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">g</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mtight">1</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner">⋯</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">g</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mclose">]</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.938em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">g</span><span className="mopen,mtight">(</span><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span><span className="mbin,mtight">+</span><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1645em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.143em;"><span></span></span></span></span></span></span><span className="mclose,mtight">)</span></span></span></span></span></span></span></span></span></span></span></span></span><p>而它整个是由 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">x_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 一次矩阵乘法得到的。把各组投影矩阵横向拼起来记作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>W</mi><mi>c</mi></msub></mrow><annotation encoding="application/x-tex">W_c</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>c</mi><mi>i</mi></msub><mo>=</mo><msub><mi>x</mi><mi>i</mi></msub><msub><mi>W</mi><mi>c</mi></msub><mo separator="true">,</mo><mspace width="2em"></mspace><msub><mi>W</mi><mi>c</mi></msub><mo>=</mo><mo stretchy="false">[</mo><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><mo>⋯</mo><mtext> </mtext><mo separator="true">,</mo><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>g</mi><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><msubsup><mi>W</mi><mi>v</mi><mrow><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo separator="true">,</mo><mo>⋯</mo><mtext> </mtext><mo separator="true">,</mo><msubsup><mi>W</mi><mi>v</mi><mrow><mo stretchy="false">(</mo><mi>g</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">]</mo><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><mrow><mi>d</mi><mo>×</mo><mi>g</mi><mo stretchy="false">(</mo><msub><mi>d</mi><mi>k</mi></msub><mo>+</mo><msub><mi>d</mi><mi>v</mi></msub><mo stretchy="false">)</mo></mrow></msup></mrow><annotation encoding="application/x-tex">c_i = x_i W_c, \qquad W_c = [W_k^{(1)}, \cdots, W_k^{(g)}, W_v^{(1)}, \cdots, W_v^{(g)}] \in \mathbb{R}^{d \times g(d_k + d_v)}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8778em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:2em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.3461em;vertical-align:-0.3013em;"></span><span className="mopen">[</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mtight">1</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner">⋯</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">g</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-2.453em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mtight">1</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.247em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner">⋯</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-2.453em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">g</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.247em;"><span></span></span></span></span></span></span><span className="mclose">]</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.938em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="mbin,mtight">×</span><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">g</span><span className="mopen,mtight">(</span><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span><span className="mbin,mtight">+</span><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1645em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.143em;"><span></span></span></span></span></span></span><span className="mclose,mtight">)</span></span></span></span></span></span></span></span></span></span></span></span></span><p>因为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>c</mi></msub><mo>=</mo><mi>g</mi><mo stretchy="false">(</mo><msub><mi>d</mi><mi>k</mi></msub><mo>+</mo><msub><mi>d</mi><mi>v</mi></msub><mo stretchy="false">)</mo><mo>&lt;</mo><mi>d</mi></mrow><annotation encoding="application/x-tex">d_c = g(d_k + d_v) &lt; d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">&lt;</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span>，这一步就是在把维度压小。所以 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mi>i</mi></msub><mo>→</mo><msub><mi>c</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">x_i \to c_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 本来就是低秩投影。也就是说，光看&quot;低秩投影&quot;这一步，GQA 和 MLA 没有区别。MLA 的改进落在<strong>低秩投影之后做了什么</strong>。GQA 投影之后干的事很简单：把向量对半分成 K 和 V，每一份再均分成 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span></span></span></span> 份，每份复制 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mi mathvariant="normal">/</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">h/g</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">h</span><span className="mord">/</span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span></span></span></span> 次，凑够 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">h</span></span></span></span> 个 head。分割和复制都是极简的线性变换。</p><p>MLA 的第一步想法就是把它们换成一般的线性变换，让每个 head 拿到的 K、V 各不相同：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msubsup><mi>k</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><msub><mi>c</mi><mi>i</mi></msub><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><msub><mi>d</mi><mi>k</mi></msub></msup><mo separator="true">,</mo><mspace width="1em"></mspace><msubsup><mi>v</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><msub><mi>c</mi><mi>i</mi></msub><msubsup><mi>W</mi><mi>v</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><msub><mi>d</mi><mi>v</mi></msub></msup></mrow><annotation encoding="application/x-tex">k_i^{(s)} = c_i W_k^{(s)} \in \mathbb{R}^{d_k}, \quad v_i^{(s)} = c_i W_v^{(s)} \in \mathbb{R}^{d_v}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3217em;vertical-align:-0.2769em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.3461em;vertical-align:-0.3013em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.3217em;vertical-align:-0.2769em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3448em;"><span style="top:-2.3488em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1512em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:1em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.185em;vertical-align:-0.247em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-2.453em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.247em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8991em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1645em;"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em;"><span className="pstrut" style="height:2.5em;"></span><span className="sizing,reset-size3,size1,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.143em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><p>但这样问题就来了。我们缓存的是投影后的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>k</mi><mi>i</mi></msub><mo separator="true">,</mo><msub><mi>v</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">k_i, v_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>，一旦它们各不相同，KV Cache 就恢复成 MHA 那么大，GQA 的初衷全丢了。MLA 的做法是找一个恒等变换绕过去。在推理阶段：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msubsup><mi>q</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>k</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup><mo>=</mo><mo stretchy="false">(</mo><msub><mi>x</mi><mi>t</mi></msub><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo stretchy="false">(</mo><msub><mi>c</mi><mi>i</mi></msub><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msup><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></msup><mo>=</mo><msub><mi>x</mi><mi>t</mi></msub><mo stretchy="false">(</mo><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup><mo stretchy="false">)</mo><msubsup><mi>c</mi><mi>i</mi><mi mathvariant="normal">⊤</mi></msubsup></mrow><annotation encoding="application/x-tex">q_t^{(s)} k_i^{(s)\top} = (x_t W_q^{(s)})(c_i W_k^{(s)})^\top = x_t (W_q^{(s)} W_k^{(s)\top}) c_i^\top</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3217em;vertical-align:-0.2769em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.4279em;vertical-align:-0.3831em;"></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-2.453em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3831em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span><span className="mclose"><span className="mclose">)</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">⊤</span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.4279em;vertical-align:-0.3831em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-2.453em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3831em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-2.453em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">⊤</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.247em;"><span></span></span></span></span></span></span></span></span></span></span><p>这意味着可以把 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup></mrow><annotation encoding="application/x-tex">W_q^{(s)} W_k^{(s)\top}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3461em;vertical-align:-0.3013em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.5834em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2527em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span></span></span></span> 合并起来当作 Q 的投影矩阵，那么 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">c_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 就取代了 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>k</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">k_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>。同理 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>v</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><msub><mi>c</mi><mi>i</mi></msub><msubsup><mi>W</mi><mi>v</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><annotation encoding="application/x-tex">v_i^{(s)} = c_i W_v^{(s)}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3217em;vertical-align:-0.2769em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.1948em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.5834em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1166em;"><span></span></span></span></span></span></span></span></span></span> 里的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>W</mi><mi>v</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><annotation encoding="application/x-tex">W_v^{(s)}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.1614em;vertical-align:-0.1166em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.5834em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.1166em;"><span></span></span></span></span></span></span></span></span></span> 也能吸收进后面的输出投影，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>v</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">v_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 也可以用 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">c_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 代替。于是推理时 KV Cache 只需要存 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">c_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>。而 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">c_i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 与 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(s)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">(</span><span className="mord,mathnormal">s</span><span className="mclose">)</span></span></span></span> 无关，是所有 head 共享的。</p><p><strong>MLA 在推理阶段恒等变换成了一个 MQA。</strong> 到这里 MLA 完成了什么就很清楚了：它用不同的投影矩阵增强了 GQA 的能力，同时在推理时保持同样大小的 KV Cache。反过来想，如果只需要跟 GQA 相当的能力，那 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>c</mi></msub></mrow><annotation encoding="application/x-tex">d_c</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 就不必取 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo stretchy="false">(</mo><msub><mi>d</mi><mi>k</mi></msub><mo>+</mo><msub><mi>d</mi><mi>v</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">g(d_k + d_v)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>，取更小的值就行。DeepSeek-V2 取了 512。</p><p>这就是 MLA 的核心。低秩压缩为什么有效？因为不同 head 需要的信息本来就有大量重叠。下投影矩阵负责提取这些共同信息，上投影矩阵让各个 head 用各自的方式去使用它。这甚至可能带来一点正则化效果。它强制模型把冗余的共同信息压进同一个低维空间，而不是每个 head 各存一份。</p><p>我一开始以为 MLA 只是&quot;把 KV 存小一点&quot;。后来才反应过来，它换掉的是一整套共享方式。存小只是结果，不是原因。</p><note title="数值精度"><p><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup></mrow><annotation encoding="application/x-tex">W_q^{(s)} W_k^{(s)\top}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3461em;vertical-align:-0.3013em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.5834em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2527em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span></span></span></span> 合并成一个矩阵，理论上只在无限精度下成立。用 BF16 时变换后的精度损失往往挺明显，多层累积后可能放大到可观的程度。</p><p>实际实现一般不按 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mi>t</mi></msub><mo stretchy="false">(</mo><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">x_t (W_q^{(s)} W_k^{(s)\top})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3461em;vertical-align:-0.3013em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.5834em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2527em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 算 Q，而是按 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msub><mi>x</mi><mi>t</mi></msub><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup></mrow><annotation encoding="application/x-tex">(x_t W_q^{(s)}) W_k^{(s)\top}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3461em;vertical-align:-0.3013em;"></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.5834em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2527em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span></span></span></span> 算。虽然是串行的，但在低秩假设下计算量更少，理论精度损失也更小。</p></note><p>不过别急。到这里 MLA 有个绕不开的缺陷：<strong>不兼容 RoPE</strong>。前面那步恒等变换能成立，关键在于 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup></mrow><annotation encoding="application/x-tex">W_q^{(s)} W_k^{(s)\top}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3461em;vertical-align:-0.3013em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.5834em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2527em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span></span></span></span> 是一个跟位置无关的固定矩阵。加上 RoPE 之后就不行了：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msubsup><mi>q</mi><mi>t</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>k</mi><mi>i</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup><mo>=</mo><mo stretchy="false">(</mo><msub><mi>x</mi><mi>t</mi></msub><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>R</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">(</mo><msub><mi>c</mi><mi>i</mi></msub><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>R</mi><mi>i</mi></msub><msup><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></msup><mo>=</mo><msub><mi>x</mi><mi>t</mi></msub><mo stretchy="false">(</mo><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>R</mi><mrow><mi>t</mi><mo>−</mo><mi>i</mi></mrow></msub><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup><mo stretchy="false">)</mo><msubsup><mi>c</mi><mi>i</mi><mi mathvariant="normal">⊤</mi></msubsup></mrow><annotation encoding="application/x-tex">q_t^{(s)} k_i^{(s)\top} = (x_t W_q^{(s)} R_t)(c_i W_k^{(s)} R_i)^\top = x_t (W_q^{(s)} R_{t-i} W_k^{(s)\top}) c_i^\top</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3217em;vertical-align:-0.2769em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4542em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2458em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.4231em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2769em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.4279em;vertical-align:-0.3831em;"></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-2.453em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3831em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose"><span className="mclose">)</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">⊤</span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.4279em;vertical-align:-0.3831em;"></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.938em;"><span style="top:-2.453em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3831em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">−</span><span className="mord,mathnormal,mtight">i</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-2.453em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">i</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">⊤</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.247em;"><span></span></span></span></span></span></span></span></span></span></span><p><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>W</mi><mi>q</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>R</mi><mrow><mi>t</mi><mo>−</mo><mi>i</mi></mrow></msub><msubsup><mi>W</mi><mi>k</mi><mrow><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></mrow></msubsup></mrow><annotation encoding="application/x-tex">W_q^{(s)} R_{t-i} W_k^{(s)\top}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.3461em;vertical-align:-0.3013em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.5834em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">q</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2527em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0077em;">R</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0077em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">−</span><span className="mord,mathnormal,mtight">i</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.0448em;"><span style="top:-2.3987em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span><span style="top:-3.2198em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mopen,mtight">(</span><span className="mord,mathnormal,mtight">s</span><span className="mclose,mtight">)</span><span className="mord,mtight">⊤</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.3013em;"><span></span></span></span></span></span></span></span></span></span> 跟位置差 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi><mo>−</mo><mi>i</mi></mrow><annotation encoding="application/x-tex">t - i</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6984em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal">t</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">−</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6595em;"></span><span className="mord,mathnormal">i</span></span></span></span> 相关，无法合并成一个固定矩阵。放弃 RoPE 换 ALIBI 之类的 attention bias 是一种办法，但实验显示明显不如 RoPE。最后 MLA 采取的是混合方案：每个 head 的 Q、K 新增 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>r</mi></msub></mrow><annotation encoding="application/x-tex">d_r</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">r</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 个维度专门用来加 RoPE，其中 K 新增的这部分在 head 间共享。</p><p>这样没有 RoPE 的维度继续走前面的压缩，带 RoPE 的维度补充位置信息。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>r</mi></msub></mrow><annotation encoding="application/x-tex">d_r</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">r</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 取 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>k</mi></msub><mi mathvariant="normal">/</mi><mn>2</mn><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">d_k/2 = 64</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord">/2</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">64</span></span></span></span>，相比 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>c</mi></msub><mo>=</mo><mn>512</mn></mrow><annotation encoding="application/x-tex">d_c = 512</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">512</span></span></span></span>，增加幅度不大。</p><p>还有一个细节：MLA 最终版把 Q 的输入也改成了低秩投影。这跟减少 KV Cache 无关，主要是为了减少训练期间的参数量和梯度所占显存。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>d</mi><mi>c</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msubsup></mrow><annotation encoding="application/x-tex">d_c&#39;</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.9989em;vertical-align:-0.247em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.7519em;"><span style="top:-2.453em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">′</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.247em;"><span></span></span></span></span></span></span></span></span></span> 取 1536，和 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>c</mi></msub><mo>=</mo><mn>512</mn></mrow><annotation encoding="application/x-tex">d_c = 512</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">512</span></span></span></span> 不同。于是解码阶段的 MLA 是 MQA 形式，Q、K 的 head size 变成 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>c</mi></msub><mo>+</mo><msub><mi>d</mi><mi>r</mi></msub></mrow><annotation encoding="application/x-tex">d_c + d_r</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">r</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>，是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>k</mi></msub></mrow><annotation encoding="application/x-tex">d_k</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 的 4 倍。</p><p><strong>解码的计算量其实是增加的。</strong></p><p>用 DeepSeek-V2 的配置把这笔账算一遍。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mo>=</mo><mn>60</mn></mrow><annotation encoding="application/x-tex">L = 60</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">L</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">60</span></span></span></span> 层，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mo>=</mo><mn>128</mn></mrow><annotation encoding="application/x-tex">h = 128</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">h</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">128</span></span></span></span> 个 head，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>k</mi></msub><mo>=</mo><mn>128</mn></mrow><annotation encoding="application/x-tex">d_k = 128</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">128</span></span></span></span>，MLA 这边 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>c</mi></msub><mo>=</mo><mn>512</mn></mrow><annotation encoding="application/x-tex">d_c = 512</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">512</span></span></span></span>、<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>r</mi></msub><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">d_r = 64</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">r</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">64</span></span></span></span>，BF16 存储，所以每参数 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">b = 2</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">b</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">2</span></span></span></span> 字节。decode 到第 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 个位置时，每 token 每层：</p><table><thead><tr><th></th><th>KV 读取</th><th>attention FLOPs</th></tr></thead><tbody><tr><td>MHA</td><td><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mi>h</mi><msub><mi>d</mi><mi>k</mi></msub><mi>N</mi><mi>b</mi></mrow><annotation encoding="application/x-tex">2 h d_k N b</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord">2</span><span className="mord,mathnormal">h</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mord,mathnormal">b</span></span></span></span></td><td><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>4</mn><mi>h</mi><msub><mi>d</mi><mi>k</mi></msub><mi>N</mi></mrow><annotation encoding="application/x-tex">4 h d_k N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord">4</span><span className="mord,mathnormal">h</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></td></tr><tr><td>MLA decode</td><td><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msub><mi>d</mi><mi>c</mi></msub><mo>+</mo><msub><mi>d</mi><mi>r</mi></msub><mo stretchy="false">)</mo><mi>N</mi><mi>b</mi></mrow><annotation encoding="application/x-tex">(d_c + d_r) N b</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">r</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mord,mathnormal">b</span></span></span></span></td><td><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mi>h</mi><mo stretchy="false">(</mo><msub><mi>d</mi><mi>c</mi></msub><mo>+</mo><msub><mi>d</mi><mi>r</mi></msub><mo stretchy="false">)</mo><mi>N</mi><mo>+</mo><mn>2</mn><mi>h</mi><msub><mi>d</mi><mi>c</mi></msub><mi>N</mi></mrow><annotation encoding="application/x-tex">2h(d_c+d_r)N + 2hd_cN</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">2</span><span className="mord,mathnormal">h</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">r</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord">2</span><span className="mord,mathnormal">h</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></td></tr></tbody></table><p>代进数字：</p><table><thead><tr><th></th><th>MHA</th><th>MLA</th><th>比值</th></tr></thead><tbody><tr><td>KV 读取 / 层 / token</td><td><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>65536</mn><mi>N</mi></mrow><annotation encoding="application/x-tex">65536N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">65536</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span>B</td><td><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1152</mn><mi>N</mi></mrow><annotation encoding="application/x-tex">1152N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">1152</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span>B</td><td><strong>1 : 56.9</strong></td></tr><tr><td>attention FLOPs / 层 / token</td><td><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>65536</mn><mi>N</mi></mrow><annotation encoding="application/x-tex">65536N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">65536</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></td><td><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>278528</mn><mi>N</mi></mrow><annotation encoding="application/x-tex">278528N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">278528</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></td><td><strong>4.25 : 1</strong></td></tr></tbody></table><p>用 roofline 分析一下。H100 SXM 的 BF16 算力约 989 TFLOPS，HBM3 带宽 3.35 TB/s，拐点在：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mfrac><mrow><mn>989</mn><mtext> TFLOPS</mtext></mrow><mrow><mn>3.35</mn><mtext> TB/s</mtext></mrow></mfrac><mo>≈</mo><mn>295</mn><mtext> FLOP/byte</mtext></mrow><annotation encoding="application/x-tex">\frac{989\ \text{TFLOPS}}{3.35\ \text{TB/s}} \approx 295\ \text{FLOP/byte}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.2963em;vertical-align:-0.936em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">3.35</span><span className="mspace"> </span><span className="mord,text"><span className="mord">TB/s</span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">989</span><span className="mspace"> </span><span className="mord,text"><span className="mord">TFLOPS</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.936em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">295</span><span className="mspace"> </span><span className="mord,text"><span className="mord">FLOP/byte</span></span></span></span></span></span><p>算术强度低于 295 就是 memory bound，高于才是 compute bound。再看 MLA decode 这一侧，别忘了每 token 还要把权重读一遍。</p><ul><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span>：当前 context 长度，也就是 decode 到这一步时要看多少个历史 token</li><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mo>=</mo><mn>60</mn></mrow><annotation encoding="application/x-tex">L = 60</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">L</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">60</span></span></span></span>：层数</li><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mtext>kv</mtext></msub><mo>=</mo><mo stretchy="false">(</mo><msub><mi>d</mi><mi>c</mi></msub><mo>+</mo><msub><mi>d</mi><mi>r</mi></msub><mo stretchy="false">)</mo><mi>b</mi><mo>=</mo><mn>576</mn><mo>×</mo><mn>2</mn><mo>=</mo><mn>1152</mn></mrow><annotation encoding="application/x-tex">c_{\text{kv}} = (d_c + d_r)b = 576 \times 2 = 1152</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">kv</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">r</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord,mathnormal">b</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">576</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">2</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">1152</span></span></span></span> 字节：每层、每 token、每多看一个历史位置，要多读多少 KV</li><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mtext>flop</mtext></msub><mo>=</mo><mn>2</mn><mi>h</mi><mo stretchy="false">(</mo><msub><mi>d</mi><mi>c</mi></msub><mo>+</mo><msub><mi>d</mi><mi>r</mi></msub><mo stretchy="false">)</mo><mo>+</mo><mn>2</mn><mi>h</mi><msub><mi>d</mi><mi>c</mi></msub><mo>=</mo><mn>278528</mn></mrow><annotation encoding="application/x-tex">c_{\text{flop}} = 2h(d_c + d_r) + 2hd_c = 278528</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7167em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">flop</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">2</span><span className="mord,mathnormal">h</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0278em;">r</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord">2</span><span className="mord,mathnormal">h</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">c</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">278528</span></span></span></span>：每层、每 token、每多看一个历史位置，要多做多少次浮点运算</li><li><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>W</mi><mo>=</mo><mn>21</mn><mtext>B</mtext><mo>×</mo><mn>2</mn><mtext> 字节</mtext><mo>=</mo><mn>42</mn></mrow><annotation encoding="application/x-tex">W = 21\text{B} \times 2\ \text{字节} = 42</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord">21</span><span className="mord,text"><span className="mord">B</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">2</span><span className="mspace"> </span><span className="mord,text"><span className="mord,cjk_fallback">字节</span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">42</span></span></span></span> GB：每 token 要读的权重字节数（21B 是 activated 参数量），它不随 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 变</li></ul><p>算术强度（记作 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>I</mi></mrow><annotation encoding="application/x-tex">I</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span></span></span></span>）就是一个字节要承担多少次浮点运算：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>I</mi><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mn>2</mn><mi>P</mi><mo>+</mo><msub><mi>c</mi><mtext>flop</mtext></msub><mo>⋅</mo><mi>L</mi><mo>⋅</mo><mi>N</mi></mrow><mrow><mi>W</mi><mo>+</mo><msub><mi>c</mi><mtext>kv</mtext></msub><mo>⋅</mo><mi>L</mi><mo>⋅</mo><mi>N</mi></mrow></mfrac></mrow><annotation encoding="application/x-tex">I(N) = \frac{2P + c_{\text{flop}} \cdot L \cdot N}{W + c_{\text{kv}} \cdot L \cdot N}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.1963em;vertical-align:-0.836em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">kv</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">⋅</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal">L</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">⋅</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">2</span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">flop</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">⋅</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal">L</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">⋅</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.836em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span></span><p>分子是每个 token 要做的浮点次数：权重矩阵乘 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mi>P</mi></mrow><annotation encoding="application/x-tex">2P</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">2</span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span></span></span></span>，加上 attention 那部分。分母是要搬的字节：权重 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>W</mi></mrow><annotation encoding="application/x-tex">W</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span></span></span></span>，加上 KV。代进数字：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>I</mi><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mn>42</mn><mo>×</mo><msup><mn>10</mn><mn>9</mn></msup><mo>+</mo><mn>278528</mn><mo>×</mo><mn>60</mn><mo>×</mo><mi>N</mi></mrow><mrow><mn>42</mn><mo>×</mo><msup><mn>10</mn><mn>9</mn></msup><mo>+</mo><mn>1152</mn><mo>×</mo><mn>60</mn><mo>×</mo><mi>N</mi></mrow></mfrac></mrow><annotation encoding="application/x-tex">I(N) = \frac{42 \times 10^9 + 278528 \times 60 \times N}{42 \times 10^9 + 1152 \times 60 \times N}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.2604em;vertical-align:-0.7693em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.4911em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">42</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">1</span><span className="mord"><span className="mord">0</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.7401em;"><span style="top:-2.989em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">9</span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">1152</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">60</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">42</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">1</span><span className="mord"><span className="mord">0</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">9</span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">278528</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">60</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7693em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span></span><p><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 一大，分子分母里那两项权重就被 KV 和 attention 项压过去了，于是有上界：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><munder><mrow><mi>lim</mi><mo>⁡</mo></mrow><mrow><mi>N</mi><mo>→</mo><mi mathvariant="normal">∞</mi></mrow></munder><mi>I</mi><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><msub><mi>c</mi><mtext>flop</mtext></msub><msub><mi>c</mi><mtext>kv</mtext></msub></mfrac><mo>=</mo><mfrac><mn>278528</mn><mn>1152</mn></mfrac><mo>≈</mo><mn>242</mn><mtext> FLOP/byte</mtext></mrow><annotation encoding="application/x-tex">\lim_{N \to \infty} I(N) = \frac{c_{\text{flop}}}{c_{\text{kv}}} = \frac{278528}{1152} \approx 242\ \text{FLOP/byte}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.4943em;vertical-align:-0.7443em;"></span><span className="mop,op-limits"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.6944em;"><span style="top:-2.3557em;margin-left:0em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.109em;">N</span><span className="mrel,mtight">→</span><span className="mord,mtight">∞</span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span><span className="mop">lim</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7443em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.9436em;vertical-align:-0.836em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.1076em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">kv</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">flop</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.836em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.0074em;vertical-align:-0.686em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3214em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">1152</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">278528</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.686em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">242</span><span className="mspace"> </span><span className="mord,text"><span className="mord">FLOP/byte</span></span></span></span></span></span><p>注意 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">L</span></span></span></span> 被约掉了。<strong>无论多少层，这个上限都一样</strong>，因为 KV 和 attention 计算都随层数线性增长。</p><p><strong>242 小于 295。</strong> 也就是说，不管 context 多长，MLA 的 decode 都落在拐点左边，是 memory bound。那 4.25 倍的计算量加进去就是白加的。算力本来就在空转等内存。</p><p>具体看几个长度：</p><table><thead><tr><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></th><th>算术强度</th><th>内存耗时</th><th>计算耗时</th><th>瓶颈</th></tr></thead><tbody><tr><td>4k</td><td>2.6</td><td>12.6 ms</td><td>0.11 ms</td><td>memory</td></tr><tr><td>32k</td><td>13.3</td><td>13.2 ms</td><td>0.60 ms</td><td>memory</td></tr><tr><td>128k</td><td>43.7</td><td>15.2 ms</td><td>2.3 ms</td><td>memory</td></tr><tr><td>1M</td><td>153</td><td>34.2 ms</td><td>17.8 ms</td><td>memory</td></tr></tbody></table><p>到 1M 的时候计算耗时（17.8 ms）已经追到内存耗时（34.2 ms）的一半了，但还没越过去。</p><p>不过这里只算了 batch size 等于 1。在线服务几乎不会单条跑，batch 是提高算术强度的另一条路子，而且它会改变结论。</p><p>原因很直接：权重每一步只需要读一次，然后同时喂给 batch 里所有序列；KV 读取却是一条序列一份。把 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi></mrow><annotation encoding="application/x-tex">B</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span></span></span></span> 写进去：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>I</mi><mo stretchy="false">(</mo><mi>B</mi><mo separator="true">,</mo><mi>N</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mi>B</mi><mrow><mo fence="true">(</mo><mn>2</mn><mi>P</mi><mo>+</mo><msub><mi>c</mi><mtext>flop</mtext></msub><mi>L</mi><mi>N</mi><mo fence="true">)</mo></mrow></mrow><mrow><mi>W</mi><mo>+</mo><mi>B</mi><mo>⋅</mo><msub><mi>c</mi><mtext>kv</mtext></msub><mi>L</mi><mi>N</mi></mrow></mfrac></mrow><annotation encoding="application/x-tex">I(B, N) = \frac{B\left(2P + c_{\text{flop}} L N\right)}{W + B \cdot c_{\text{kv}} L N}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.263em;vertical-align:-0.836em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.427em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">⋅</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">kv</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal">L</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner"><span className="mopen,delimcenter" style="top:0em;">(</span><span className="mord">2</span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">flop</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mord,mathnormal">L</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose,delimcenter" style="top:0em;">)</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.836em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span></span><p><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi></mrow><annotation encoding="application/x-tex">B</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span></span></span></span> 一大，分母里的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>W</mi></mrow><annotation encoding="application/x-tex">W</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span></span></span></span> 就被摊薄，算术强度趋近于：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>I</mi><mo stretchy="false">(</mo><mi mathvariant="normal">∞</mi><mo separator="true">,</mo><mi>N</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mn>2</mn><mi>P</mi><mo>+</mo><msub><mi>c</mi><mtext>flop</mtext></msub><mi>L</mi><mi>N</mi></mrow><mrow><msub><mi>c</mi><mtext>kv</mtext></msub><mi>L</mi><mi>N</mi></mrow></mfrac></mrow><annotation encoding="application/x-tex">I(\infty, N) = \frac{2P + c_{\text{flop}} L N}{c_{\text{kv}} L N}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mopen">(</span><span className="mord">∞</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.1963em;vertical-align:-0.836em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">kv</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord,mathnormal">L</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">2</span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">flop</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mord,mathnormal">L</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.836em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span></span><p>代进数字：</p><table><thead><tr><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></th><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">B=1</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">1</span></span></span></span></th><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">B=8</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">8</span></span></span></span></th><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">B=64</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">64</span></span></span></span></th><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>=</mo><mn>512</mn></mrow><annotation encoding="application/x-tex">B=512</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">512</span></span></span></span></th><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>→</mo><mi mathvariant="normal">∞</mi></mrow><annotation encoding="application/x-tex">B \to \infty</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord">∞</span></span></span></span></th></tr></thead><tbody><tr><td>4k</td><td>2.6</td><td>20.0</td><td>118</td><td><strong>303</strong></td><td><strong>390</strong></td></tr><tr><td>32k</td><td>13.3</td><td>78.5</td><td>202</td><td>251</td><td>260</td></tr><tr><td>128k</td><td>43.7</td><td>156</td><td>230</td><td>244</td><td>246</td></tr><tr><td>1M</td><td>153</td><td>226</td><td>240</td><td>242</td><td>242</td></tr></tbody></table><p>（加粗的是越过 295 的格子。）</p><p>于是结论要分区间说：</p><ul><li><strong>短 context（4k 这一档）</strong>：batch 开得够大，decode 会翻到 compute bound。这时候那 4.25 倍的计算量是真的要还的。</li><li><strong>32k 及以上</strong>：即使 batch 开到无穷，算术强度也够不到 295。decode 始终 memory bound，那 4.25 倍照样白加。</li></ul><p>原因在渐近值里写得清楚。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 一大，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mtext>flop</mtext></msub><mi>L</mi><mi>N</mi></mrow><annotation encoding="application/x-tex">c_{\text{flop}} L N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.9694em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">flop</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mord,mathnormal">L</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 就压过 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mi>P</mi></mrow><annotation encoding="application/x-tex">2P</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">2</span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span></span></span></span>，权重项被稀释掉，剩下的比值就是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mtext>flop</mtext></msub><mi mathvariant="normal">/</mi><msub><mi>c</mi><mtext>kv</mtext></msub><mo>=</mo><mn>242</mn></mrow><annotation encoding="application/x-tex">c_{\text{flop}}/c_{\text{kv}} = 242</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0361em;vertical-align:-0.2861em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">flop</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span className="mord">/</span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">kv</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">242</span></span></span></span>，跟 batch 无关。</p><p>所以 batch 只在短 context 上有效，而短 context 恰恰是 MLA 收益最小的区间。前面那张加速比表里，4k 只有 1.4×。两件事碰在一起，这个取舍就更清楚了：<strong>MLA 在自己最有价值的长 context 区间永远是 memory bound，那笔计算量从头到尾都不用还。</strong></p><p>长 context 下 batch 也开不大，还有另一个原因：KV Cache 装不下。MLA 在 128k 时每条序列 9.06 GB，32 条就是 290 GB，早就超出单卡。而同样条件下的 MHA 是 515 GB，<strong>一条序列都放不下</strong>。</p><p>所以 MHA 在长 context 下跑不起来，谈不上快慢。</p><p>那&quot;划算&quot;的刻度在哪里？MLA 省的是字节，而字节由权重和 KV 两部分组成。KV 读取追平权重的位置：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>MHA： </mtext><mn>3.93</mn><mtext> MB</mtext><mo>×</mo><mi>N</mi><mo>=</mo><mn>42</mn><mtext> GB</mtext><mo>⇒</mo><mi>N</mi><mo>≈</mo><mn>1.1</mn><mtext> 万</mtext></mrow><annotation encoding="application/x-tex">\text{MHA：}\ 3.93\ \text{MB} \times N = 42\ \text{GB} \Rightarrow N \approx 1.1\ \text{万}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,text"><span className="mord">MHA</span><span className="mord,cjk_fallback">：</span></span><span className="mspace"> </span><span className="mord">3.93</span><span className="mspace"> </span><span className="mord,text"><span className="mord">MB</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">42</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">⇒</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">1.1</span><span className="mspace"> </span><span className="mord,text"><span className="mord,cjk_fallback">万</span></span></span></span></span></span><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>MLA： </mtext><mn>69.1</mn><mtext> KB</mtext><mo>×</mo><mi>N</mi><mo>=</mo><mn>42</mn><mtext> GB</mtext><mo>⇒</mo><mi>N</mi><mo>≈</mo><mn>61</mn><mtext> 万</mtext></mrow><annotation encoding="application/x-tex">\text{MLA：}\ 69.1\ \text{KB} \times N = 42\ \text{GB} \Rightarrow N \approx 61\ \text{万}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,text"><span className="mord">MLA</span><span className="mord,cjk_fallback">：</span></span><span className="mspace"> </span><span className="mord">69.1</span><span className="mspace"> </span><span className="mord,text"><span className="mord">KB</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">42</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">⇒</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">61</span><span className="mspace"> </span><span className="mord,text"><span className="mord,cjk_fallback">万</span></span></span></span></span></span><p>既然两边都是 memory bound，加速比就等于字节数之比：</p><table><thead><tr><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></th><th>MLA 相对 MHA 的每 token 加速比</th></tr></thead><tbody><tr><td>4k</td><td>1.4×</td></tr><tr><td>32k</td><td>3.9×</td></tr><tr><td>128k</td><td>10.9×</td></tr><tr><td>1M</td><td>36.4×</td></tr></tbody></table><p>所以答案是一条具体的曲线：<strong>4k 时几乎白干（1.4×），32k 开始有意义（3.9×），128k 是 11 倍，1M 是 36 倍。</strong></p><p>这条曲线顺带解释了 MLA 的时间线。它 2024 年出来的时候，主流 context 还在 4k 到 32k 之间，那一段的收益并不惊人；让它值回票价的是后面 128k、1M 的军备竞赛。</p><p>另外还有一件事能说明这个设计有多自觉。一般 LLM 架构满足 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mo>×</mo><msub><mi>d</mi><mi>k</mi></msub><mo>=</mo><mi>d</mi></mrow><annotation encoding="application/x-tex">h \times d_k = d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7778em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal">h</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span>，但 DeepSeek-V2 是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>k</mi></msub><mo>=</mo><mn>128</mn><mo separator="true">,</mo><mi>d</mi><mo>=</mo><mn>5120</mn><mo separator="true">,</mo><mi>h</mi><mo>=</mo><mn>128</mn></mrow><annotation encoding="application/x-tex">d_k = 128, d = 5120, h = 128</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord">128</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">d</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord">5120</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">h</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">128</span></span></span></span>，head 数是一般设置的 3 倍。</p><p>因为 MLA 的 KV Cache 大小跟 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">h</span></span></span></span> 无关。增大 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">h</span></span></span></span> 只增加计算量和提升模型能力，不增加 KV Cache，所以不会带来速度瓶颈。MLA 完成的思想升级：<strong>从离散的 head sharing，到连续的 representation compression</strong>。</p><h2 id="csa-与-hca">CSA 与 HCA</h2><h3 id="压-sequence-轴">压 Sequence 轴</h3><p>MLA 压小的是每个 token 存的东西。但 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mn>1</mn></msub><mo separator="true">,</mo><msub><mi>c</mi><mn>2</mn></msub><mo separator="true">,</mo><mo>⋯</mo><mtext> </mtext><mo separator="true">,</mo><msub><mi>c</mi><mi>N</mi></msub></mrow><annotation encoding="application/x-tex">c_1, c_2, \cdots, c_N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">1</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner">⋯</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mord,mathnormal">c</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.109em;">N</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 仍然有 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 个。如果每个 query 还是要看全部历史，那 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>×</mo><mi>N</mi></mrow><annotation encoding="application/x-tex">N \times N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 还在。</p><p><strong>MLA 解决的是 KV 的宽度，没解决历史的长度。</strong></p><p>把 KV Cache 想成一个二维矩阵：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>Sequence</mtext><mo>×</mo><mtext>Feature</mtext></mrow><annotation encoding="application/x-tex">\text{Sequence} \times \text{Feature}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8778em;vertical-align:-0.1944em;"></span><span className="mord,text"><span className="mord">Sequence</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,text"><span className="mord">Feature</span></span></span></span></span></span><p>MQA、GQA、MLA 压的都是 Feature 轴。V4 开始压 Sequence 轴。V4 改成了三类 attention：</p><ul><li><strong>CSA</strong>：先按大约 4:1 沿序列压缩，再用 Lightning Indexer 给每个 query 挑少量 top-k compressed blocks</li><li><strong>HCA</strong>：用更激进的序列压缩（约 128 倍），让 query 对压缩后的全局历史做 attention</li><li><strong>Sliding Window</strong>：保留最近的精细上下文</li></ul><p>这两个压缩率不是估的。Hugging Face 对 V4 的实现文档给出了默认值：CSA 是 4，HCA 是 128。为什么非要留一个 sliding window？因为压缩是有损的。CSA 压到 1/4，HCA 压到 1/128，最近的几个 token 一旦被压进 block，细节就再也找不回来了。而语言模型对紧接着的上下文又格外敏感。代词指谁、缩进几个空格、刚才那个变量叫什么名字、上一个 tool call 返回了什么。</p><p>这些信息全都发生在最后几百个 token 里。</p><p>所以最近的一段必须原样保留。三类 attention 是三段分工，不能互相替换：最近的原样看，中间挑着看，最远的粗着看。这个分工其实很符合直觉。人读一篇长文档也是这样：最后一段逐字读，中间扫小标题，开头的细节早忘了，只记得大意。于是 attention 不再是简单的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>×</mo><mi>N</mi></mrow><annotation encoding="application/x-tex">N \times N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span>，而是三种东西的叠加（记局部窗口大小为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>W</mi></mrow><annotation encoding="application/x-tex">W</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span></span></span></span>，每个 query 选中的 block 数为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi></mrow><annotation encoding="application/x-tex">B</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span></span></span></span>，压缩率为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span></span></span></span>）：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>N</mi><mo>×</mo><mi>W</mi><mspace width="1em"></mspace><mtext>（局部窗口）</mtext></mrow><annotation encoding="application/x-tex">N \times W \quad \text{（局部窗口）}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="mspace" style="margin-right:1em;"></span><span className="mord,text"><span className="mord,cjk_fallback">（局部窗口）</span></span></span></span></span></span><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>N</mi><mo>×</mo><mi>B</mi><mspace width="1em"></mspace><mtext>（稀疏选中的 compressed blocks）</mtext></mrow><annotation encoding="application/x-tex">N \times B \quad \text{（稀疏选中的 compressed blocks）}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:1em;"></span><span className="mord,text"><span className="mord,cjk_fallback">（稀疏选中的</span><span className="mord"> compressed blocks</span><span className="mord,cjk_fallback">）</span></span></span></span></span></span><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>N</mi><mo>×</mo><mfrac><mi>N</mi><mi>r</mi></mfrac><mspace width="1em"></mspace><mtext>（高度压缩的 global branch）</mtext></mrow><annotation encoding="application/x-tex">N \times \frac{N}{r} \quad \text{（高度压缩的 global branch）}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:2.0463em;vertical-align:-0.686em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.686em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:1em;"></span><span className="mord,text"><span className="mord,cjk_fallback">（高度压缩的</span><span className="mord"> global branch</span><span className="mord,cjk_fallback">）</span></span></span></span></span></span><h3 id="复杂度没有变成-on">复杂度没有变成 O(N)</h3><p>如果 HCA 那一路的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span></span></span></span> 是个固定常数，那么它的理论渐进复杂度仍然带有 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>N</mi><mn>2</mn></msup><mi mathvariant="normal">/</mi><mi>r</mi></mrow><annotation encoding="application/x-tex">N^2/r</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0641em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span><span className="mord">/</span><span className="mord,mathnormal" style="margin-right:0.0278em;">r</span></span></span></span>，没有数学意义上变成 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span></span></span></span>。</p><p>但工程量级已经差很多。而 CSA 的 sparse branch 更接近：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><mi>B</mi><mo stretchy="false">)</mo><mo separator="true">,</mo><mspace width="1em"></mspace><mi>B</mi><mo>≪</mo><mi>N</mi></mrow><annotation encoding="application/x-tex">O(NB), \quad B \ll N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mclose">)</span><span className="mpunct">,</span><span className="mspace" style="margin-right:1em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≪</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></span><p>这是 V4 攻击 quadratic attention 的部分。所以&quot;V4 把 attention 变成线性了&quot;这种说法，把两件事混成了一件。严格结论和工程结论要分开讲。</p><h2 id="linear-attention-与-ssm">Linear Attention 与 SSM</h2><p>前面所有方案都建立在同一个前提上：KV 要保存下来。那换个问题。<strong>如果 KV 这么贵，为什么一定要保存它？</strong> RNN 的答案是最极端的：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>h</mi><mi>t</mi></msub><mo>=</mo><mi>tanh</mi><mo>⁡</mo><mo stretchy="false">(</mo><msub><mi>W</mi><mi>x</mi></msub><msub><mi>x</mi><mi>t</mi></msub><mo>+</mo><msub><mi>W</mi><mi>h</mi></msub><msub><mi>h</mi><mrow><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo>+</mo><mi>b</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">h_t = \tanh(W_x x_t + W_h h_{t-1} + b)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">h</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mop">tanh</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">x</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal">x</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.9028em;vertical-align:-0.2083em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">h</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal">h</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">−</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">b</span><span className="mclose">)</span></span></span></span></span><p>过去全部信息压进一个固定大小的状态。不保存 KV，只保存状态。代价是长期记忆很差，而且不能并行。LSTM / GRU 用门控改善了记忆，但没根治，也仍然不能并行。然后 Linear Attention 出现了：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>S</mi><mi>t</mi></msub><mo>=</mo><msub><mi>S</mi><mrow><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo>+</mo><mi>ϕ</mi><mo stretchy="false">(</mo><msub><mi>k</mi><mi>t</mi></msub><mo stretchy="false">)</mo><msubsup><mi>v</mi><mi>t</mi><mi mathvariant="normal">⊤</mi></msubsup></mrow><annotation encoding="application/x-tex">S_t = S_{t-1} + \phi(k_t) v_t^\top</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8917em;vertical-align:-0.2083em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">t</span><span className="mbin,mtight">−</span><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.2083em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1.1491em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">ϕ</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0315em;">k</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0315em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">v</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-2.453em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">⊤</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.247em;"><span></span></span></span></span></span></span></span></span></span></span><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>o</mi><mi>t</mi></msub><mo>=</mo><mi>ϕ</mi><mo stretchy="false">(</mo><msub><mi>q</mi><mi>t</mi></msub><msup><mo stretchy="false">)</mo><mi mathvariant="normal">⊤</mi></msup><msub><mi>S</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">o_t = \phi(q_t)^\top S_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">o</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1.1491em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">ϕ</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0359em;">q</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose"><span className="mclose">)</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">⊤</span></span></span></span></span></span></span></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span></span><p>每步只更新一个 state，复杂度 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span></span></span></span>。看到这个式子会觉得眼熟：它就是 <strong>LSTM 式的 recurrent state</strong> 加 <strong>Attention 式的 query-key retrieval</strong>。</p><p>所以它像一个&quot;现代 LSTM&quot;是有道理的，不是公式恰巧长得像。它重新选择了<strong>压缩历史，而不是保存历史</strong>。代价是信息碰撞。不同的 key 的信息全挤进同一个 state，容量有限，互相干扰。state <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>S</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">S_t</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0576em;">S</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight">t</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 的大小是固定的，跟 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>k</mi></msub><mo>×</mo><msub><mi>d</mi><mi>v</mi></msub></mrow><annotation encoding="application/x-tex">d_k \times d_v</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0315em;">k</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mathnormal,mtight" style="margin-right:0.0359em;">v</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 同阶，不随 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 增长，而它要装的是全部历史的信息。序列越长，需要塞进去的东西越多，碰撞就越严重。具体表现是：想精确取回某一个早先 token 的信息时，取回来的其实是很多 token 叠加后的结果；full attention 能精确地只挑那一个，Linear Attention 做不到。门控能缓解遗忘，缓解不了容量上限造成的记混。</p><p>SSM（State Space Model）是这条线的另一个分支，典型代表是 Mamba。它和 Linear Attention 的数学形式不同，但目标一样：用固定大小的状态代替完整历史，把 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><msup><mi>N</mi><mn>2</mn></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N^2)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0641em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span> 降到 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span></span></span></span>。Mamba 的特点在状态更新的方式上。它让状态转移矩阵随输入变化，相当于给状态加了一层输入相关的门控。这和 LSTM 的门控思路一脉相承，只是换了个数学外衣。</p><p>所以往上看，Mamba、RWKV、RetNet、DeltaNet、Gated DeltaNet、Kimi Linear，看起来像一堆论文名字动物园，其实都在回答同一个问题：<strong>怎么用固定大小的记忆，尽可能少地丢掉历史。</strong> 于是有了 Hybrid：大部分层用廉价的 state memory，少数几层保留对完整历史的随机访问。这不是&quot;RNN 又赢了&quot;：attention 的随机访问能力和 RNN 的固定状态成本各有好处，把两者揉到一起，比宗教战争式地二选一更有产出。</p><h2 id="kimi-与-deepseek">Kimi 与 DeepSeek</h2><p>Kimi 和 DeepSeek 都在压历史，但压完之后的处理方式不一样：<strong>Kimi 把历史压成 state，DeepSeek 压缩之后仍然保持可检索</strong>。</p><h3 id="state-machine-与-indexed-memory">State Machine 与 Indexed Memory</h3><p>Kimi 的 Linear Attention 路线（KDA 那一支）是把历史递推成一个 state，查询时直接对 state 做操作：</p><pre className="language-text" code="历史
 ↓
recurrent state
 ↓
query state
" language="text" meta=""><code __ignoreMap="">历史
 ↓
recurrent state
 ↓
query state
</code></pre><p>DeepSeek 的路线是压缩，但压缩后的东西仍然带索引、可以被检索回来：</p><pre className="language-text" code="历史
 ↓
compressed memory
 ↓
index / retrieval
 ↓
sparse attention
" language="text" meta=""><code __ignoreMap="">历史
 ↓
compressed memory
 ↓
index / retrieval
 ↓
sparse attention
</code></pre><p>我把它们概括成 <strong>State Machine vs. Indexed Memory</strong>。这比说&quot;Linear Attention vs Sparse Attention&quot;有解释力：后者的区别看起来只是稀疏程度不同，前者是两种根本不同的 memory 哲学，一个<strong>把历史变成状态</strong>，一个<strong>把历史变成可检索的索引</strong>。有一个细节能说明 Kimi 这边并不只是&quot;换了个 attention&quot;。</p><p>K3 的技术报告把 refined training and data recipes、general/agentic/coding RL、million-token agentic RL infrastructure、sandbox state，和 KDA、AttnRes、LatentMoE 并列成核心进展。注意这个列表的构成：一半是架构，一半是数据和基础设施。</p><p>如果 KDA 真是能力跃迁的唯一来源，报告没必要把 RL infra 抬到同样的位置。</p><p>反过来看，这也解释了 KDA 这类工作为什么重要。它让 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mtext>M</mtext></mrow><annotation encoding="application/x-tex">1\text{M}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">1</span><span className="mord,text"><span className="mord">M</span></span></span></span></span> context 和 long-horizon rollout 在计算上变得现实，于是以前喂不起的 RL trajectories 变得喂得起。架构在这里是<strong>开路的</strong>，不是<strong>发力的</strong>。DeepSeek 这边也一样，CSA/HCA 让 1M context 变得足够便宜，价值主要落在这里。</p><p>那这两条路线谁能赢？我觉得决胜点不在复杂度写 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span></span></span></span> 还是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><mi>B</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(NB)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mclose">)</span></span></span></span>，而在更朴素的地方：<strong>同样的显存带宽和 FLOPs 下，谁还能准确找回 50 万 token 前那个关键细节</strong>。这是长上下文模型最难在 benchmark 上糊弄过去的地方。</p><h2 id="l-n-d">L × N × D</h2><p>把前面的方案合起来看，它们动的其实是同一个三维对象，也就是 KV 的形状 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mo>×</mo><mi>N</mi><mo>×</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">L \times N \times D</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal">L</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">D</span></span></span></span>：<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">L</span></span></span></span> 是层数，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 是序列长度，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">D</span></span></span></span> 是每个 token 存下来的表示宽度。整条 DeepSeek 路线可以重新写成：GQA/MLA 压 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">D</span></span></span></span>，V4 的 CSA/HCA 压 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span>，V4.1 压 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">L</span></span></span></span> 和计算路径。到 V4.1，问题变成了：</p><p><strong>既然历史已经压缩过一次，为什么每一层还要重复为它付费？</strong></p><p>ds4.1f 采用的是一个新架构族：552B MoE，Causal-Encoder-Decoder，输入侧约 8B activated、输出侧约 16B activated。这个不对称的计算预算是关键。要理解为什么 8B 和 16B 是分开的两个数，得把那两个瓶颈各自的账算一遍。先定符号。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi></mrow><annotation encoding="application/x-tex">P</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span></span></span></span> 是参数量（MoE 里指 activated 部分），<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">b</span></span></span></span> 是每个参数的字节数，<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 是序列长度。<strong>输入侧</strong>面对的是长 prompt。</p><p>1M 个 token 全部要走一遍编码器，计算量约 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mi>P</mi><mi>N</mi></mrow><annotation encoding="application/x-tex">2PN</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">2</span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span>，而要读的权重只有 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mo>×</mo><mi>b</mi></mrow><annotation encoding="application/x-tex">P \times b</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">b</span></span></span></span>。</p><p><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 到百万量级时，算术强度能到几千 FLOPs/byte，早就超过了硬件的算力带宽比。这一侧完全被算力卡住。想在 prefill 上省钱，唯一有效的办法就是把 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi></mrow><annotation encoding="application/x-tex">P</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span></span></span></span> 压下去。<strong>8B 是这么来的。</strong> <strong>输出侧</strong>面对的是逐个生成的 token，情况正好反过来。每生成一个 token，计算量只有 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mi>P</mi></mrow><annotation encoding="application/x-tex">2P</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">2</span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span></span></span></span>，但要读两份东西：模型权重 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mo>×</mo><mi>b</mi></mrow><annotation encoding="application/x-tex">P \times b</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">b</span></span></span></span>，加上 KV Cache。而模型权重是必须整体读一遍的。</p><p>拿 16B activated、BF16 存储来算：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>16</mn><mtext>B</mtext><mo>×</mo><mn>2</mn><mtext> bytes</mtext><mo>=</mo><mn>32</mn><mtext> GB</mtext></mrow><annotation encoding="application/x-tex">16\text{B} \times 2\ \text{bytes} = 32\ \text{GB}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord">16</span><span className="mord,text"><span className="mord">B</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord">2</span><span className="mspace"> </span><span className="mord,text"><span className="mord">bytes</span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">32</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span></span></span></span></span><p>每生成一个 token，光权重就要搬 32 GB。而这一个 token 的计算量是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>×</mo><mn>16</mn><mtext>B</mtext><mo>=</mo><mn>32</mn></mrow><annotation encoding="application/x-tex">2 \times 16\text{B} = 32</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">2</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">16</span><span className="mord,text"><span className="mord">B</span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">32</span></span></span></span> GFLOPs。算术强度大约是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn></mrow><annotation encoding="application/x-tex">1</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">1</span></span></span></span> FLOP/byte。现代 GPU 的算力带宽比是几十到几百 FLOP/byte。差了将近两个数量级，意味着算力几乎全程在等内存，多算一点几乎不花钱。既然算力在空转等数据，就没有理由在生成端省参数量。<strong>16B 是这么来的。</strong></p><note title="简化与推断边界"><p>上面按&quot;每步只生成一个 token&quot;算。speculative decoding 会一次验证多个候选 token，test-time scaling、parallel decoding 也会改变每步的计算和内存特征，但不改变主线：无论一步生成几个 token，权重和 KV Cache 该读的还是要读。speculative decoding 甚至更吃带宽，草稿模型本身要读一遍权重，验证阶段还要再读主模型。</p><p>8B/16B 落在 prefill 和 decode 两侧、方向符合各自瓶颈，这是我的读法，官方没有给出设计理由。</p></note><p>Compute bound 的那一侧压计算，memory bound 的那一侧压不住，就把能力堆上去。传统 decoder-only LLM 对 prompt 里每个 token 和 decode token 用几乎同一套昂贵 backbone，等于强迫两个瓶颈完全不同的阶段共用一套参数。</p><p>如果输入有 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>=</mo><mn>1</mn><mtext>M</mtext></mrow><annotation encoding="application/x-tex">N = 1\text{M}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">1</span><span className="mord,text"><span className="mord">M</span></span></span></span></span>，那么即使 attention 已经 sparse 或 compressed，你依然需要对百万 token 做很多层计算。V4.1 攻击的不只是 attention matrix：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>Q</mi><msup><mi>K</mi><mi mathvariant="normal">⊤</mi></msup></mrow><annotation encoding="application/x-tex">QK^\top</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0935em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal">Q</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0715em;">K</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8991em;"><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">⊤</span></span></span></span></span></span></span></span></span></span></span></span><p>还攻击整个长序列每层 FFN/MoE 的计算。前面账单上的第二项一直被 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>N</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">N^2</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8141em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span></span></span></span> 的光环盖着：attention 那项是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><msup><mi>N</mi><mn>2</mn></msup><mi>d</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N^2 d)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0641em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span><span className="mord,mathnormal">d</span><span className="mclose">)</span></span></span></span>，FFN/MoE 那项是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><msup><mi>d</mi><mn>2</mn></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(Nd^2)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.0641em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mord"><span className="mord,mathnormal">d</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span><span className="mclose">)</span></span></span></span>。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 大到百万量级、<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>N</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">N^2</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8141em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span></span></span></span></span></span></span></span> 又被打下来之后，后者因为每一层都要对每个 token 算一遍，纹丝不动。</p><p>所以 V4.1 做 asymmetric compute，本质上是：attention 都已经省下来了，总不能剩下的网络继续拿同样预算烧百万 token。终于有人检查了完整账单，而不是只盯着最显眼的那一行。</p><p>省下来的成本是多少？ds4.1f 的技术报告（<a href="https://www.alphaxiv.org/abs/2609.deepseek-v4-1-flash" rel="nofollow">DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression</a>）给了具体的数。相比上一代 V4-Flash：</p><ul><li>常驻 <strong>HBM</strong> 的 global KV cache：<strong>890 字节 / token</strong>，约为原来的 <strong>1/4</strong></li><li>落在 <strong>SSD 或 host memory</strong> 的 persistent KV cache：约为原来的 <strong>1/8</strong></li></ul><p>报告把这些收益归因于&quot;entry size、sequence dimension、layer dimension 三个方向上的协同优化&quot;，落到 CSA2 上就是三个模式：投影让 head 之间共享信息（压 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">D</span></span></span></span>）、把多个 token 合成一个 KV entry（压 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span>）、跨层复用 KV（压 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">L</span></span></span></span>）。再加上主 KV cache 本身换成 FP4。</p><p><strong>这不就是前面的 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mo>×</mo><mi>N</mi><mo>×</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">L \times N \times D</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal">L</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">D</span></span></span></span>。</strong> 前面那条&quot;压宽度 → 压长度 → 压深度&quot;的路线，在 ds4.1f 上是第一次三个方向同时动刀。</p><p>压 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal">L</span></span></span></span> 这一个方向前面一直只是断言，说清楚它得看 CSA2 的三种模式：</p><ul><li><strong>Full</strong>：这层自己算主 K、V，投影一个 indexer key，产出一组新的 Top-K 索引</li><li><strong>Reindex</strong>：主 K、V 和 indexer key 都从前一层继承，只重算 indexer query，于是选出一组<strong>不同</strong>的 Top-K</li><li><strong>Reuse</strong>：主 K、V 和 Top-K 索引全部继承，这层几乎不做索引计算</li></ul><p>再配一个分层的 indexer：第一个 Full 层先圈出一个 block 级的候选池，后面的 Reindex 层只在这个池子里打分。于是每个 query 的索引开销被限住，context 是一万还是一百万都一样。</p><p>这就是&quot;每一层不再重复为同一份历史付费&quot;的具体做法：多数层复用索引，少数层重建。</p><p>持久层那 1/8 靠的是另一个部署优化，叫 SWA Bounded Replay：局部窗口的 cache 不再常驻，请求恢复时只重放最近一小段就能近似重建，不必把整条序列重算一遍。</p><p>另外 CED 让 prefill 的复杂度从 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><mo>⋅</mo><mi>L</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N \cdot L)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">⋅</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal">L</span><span className="mclose">)</span></span></span></span> 降到大约 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>N</mi><mo>⋅</mo><mfrac><mi>L</mi><mn>2</mn></mfrac><mo>+</mo><msub><mi>n</mi><mtext>win</mtext></msub><mo>⋅</mo><mfrac><mi>L</mi><mn>2</mn></mfrac><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(N \cdot \frac{L}{2} + n_{\text{win}} \cdot \frac{L}{2})</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">⋅</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1.2173em;vertical-align:-0.345em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.8723em;"><span style="top:-2.655em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">2</span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.394em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">L</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.345em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.5945em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">n</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3175em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">win</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">⋅</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1.2173em;vertical-align:-0.345em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.8723em;"><span style="top:-2.655em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">2</span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.394em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">L</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.345em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mclose">)</span></span></span></span>，长输入的计算量差不多减半。</p><p>那 890 字节 / token 在真实硬件上意味着什么。拿 B200 算一遍（<a href="https://inferencex.semianalysis.com/chips/b200" rel="nofollow">规格</a>）。</p><p>先说部署形态。552B 权重按 FP4 存是 276 GB，单卡 180 GB 装不下，所以要拆卡。<strong>四卡（TP4）就够</strong>：每卡 69 GB 权重，还剩 111 GB 放 KV。</p><p>下面算的都是<strong>每张卡</strong>的数字。TP4 之后每卡只承担四分之一，所以字面值和耗时都除以 4，算术强度不变。</p><p>先把精度定死，因为算力和字节数都随它变：</p><table><thead><tr><th></th><th>精度</th><th>依据</th></tr></thead><tbody><tr><td>模型权重</td><td>FP4</td><td>每参数 0.5 字节</td></tr><tr><td>KV cache</td><td>FP4（MXFP4）</td><td>技术报告的 QAT 就是按 MXFP4 做的，890 字节 / token 是这个精度下的值</td></tr><tr><td>算力基准</td><td>FP4</td><td>B200 dense 9000 TFLOP/s</td></tr></tbody></table><p>三项同一把尺子，拐点和字节数才可比。</p><p>B200 的 BF16 / FP8 / FP4 dense 算力分别是 2250 / 4500 / 9000 TFLOP/s，HBM3e 带宽 8 TB/s。FP4 的拐点是：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mfrac><mrow><mn>9000</mn><mtext> TFLOPS</mtext></mrow><mrow><mn>8</mn><mtext> TB/s</mtext></mrow></mfrac><mo>=</mo><mn>1125</mn><mtext> FLOP/byte</mtext></mrow><annotation encoding="application/x-tex">\frac{9000\ \text{TFLOPS}}{8\ \text{TB/s}} = 1125\ \text{FLOP/byte}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.2963em;vertical-align:-0.936em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">8</span><span className="mspace"> </span><span className="mord,text"><span className="mord">TB/s</span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">9000</span><span className="mspace"> </span><span className="mord,text"><span className="mord">TFLOPS</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.936em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">1125</span><span className="mspace"> </span><span className="mord,text"><span className="mord">FLOP/byte</span></span></span></span></span></span><p>decode 侧每 token 要搬多少？16B activated，FP4 存储：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>16</mn><mtext>B</mtext><mo>×</mo><mn>0.5</mn><mtext> 字节</mtext><mo>=</mo><mn>8</mn><mtext> GB</mtext></mrow><annotation encoding="application/x-tex">16\text{B} \times 0.5\ \text{字节} = 8\ \text{GB}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord">16</span><span className="mord,text"><span className="mord">B</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">0.5</span><span className="mspace"> </span><span className="mord,text"><span className="mord,cjk_fallback">字节</span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">8</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span></span></span></span></span><p>KV 那边，1M context 时是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>890</mn><mo>×</mo><msup><mn>10</mn><mn>6</mn></msup><mo>≈</mo><mn>0.93</mn></mrow><annotation encoding="application/x-tex">890 \times 10^6 \approx 0.93</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">890</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8141em;"></span><span className="mord">1</span><span className="mord"><span className="mord">0</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">6</span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">0.93</span></span></span></span> GB。加起来 8.93 GB。算术强度：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>I</mi><mo>=</mo><mfrac><mrow><mn>2</mn><mo>×</mo><mn>16</mn><mtext>B</mtext></mrow><mrow><mn>8.93</mn><mtext> GB</mtext></mrow></mfrac><mo>≈</mo><mn>3.6</mn><mtext> FLOP/byte</mtext></mrow><annotation encoding="application/x-tex">I = \frac{2 \times 16\text{B}}{8.93\ \text{GB}} \approx 3.6\ \text{FLOP/byte}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.0463em;vertical-align:-0.686em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">8.93</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">2</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">16</span><span className="mord,text"><span className="mord">B</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.686em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">3.6</span><span className="mspace"> </span><span className="mord,text"><span className="mord">FLOP/byte</span></span></span></span></span></span><p>按 8 TB/s 的带宽上限，单流大约 900 token/s。</p><p><strong>3.6 对 1125，差了 300 倍。</strong> B200 的 FP4 算力有 99.7% 用不上。</p><p>换权重精度结论也一样。KV 那 890 字节 / token 已经是 FP4 下的实测值，所以只动权重：</p><table><thead><tr><th>权重精度</th><th>权重字节 / token</th><th>拐点</th><th>算术强度</th><th>差几倍</th></tr></thead><tbody><tr><td>FP4</td><td>8.0 GB</td><td>1125</td><td>3.58</td><td>314×</td></tr><tr><td>FP8</td><td>16.0 GB</td><td>562</td><td>1.89</td><td>298×</td></tr><tr><td>BF16</td><td>32.0 GB</td><td>281</td><td>0.97</td><td>289×</td></tr></tbody></table><p>精度降一半，拐点翻一倍，但算术强度也跟着掉一半，两边抵消。根源是 B200 上这三个精度的「算力 × 每参字节」都是同一个数：<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>9000</mn><mo>×</mo><mn>0.5</mn><mo>=</mo><mn>4500</mn><mo>×</mo><mn>1</mn><mo>=</mo><mn>2250</mn><mo>×</mo><mn>2</mn><mo>=</mo><mn>4500</mn></mrow><annotation encoding="application/x-tex">9000 \times 0.5 = 4500 \times 1 = 2250 \times 2 = 4500</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">9000</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">0.5</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">4500</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">1</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span className="mord">2250</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">2</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">4500</span></span></span></span>。所以<strong>差多少倍基本由架构决定，不由精度决定</strong>。</p><p>流量拆开看：</p><table><thead><tr><th>1M context 下每 token 的流量</th><th>字节</th><th>占比</th></tr></thead><tbody><tr><td>权重</td><td>8.0 GB</td><td>90%</td></tr><tr><td>KV</td><td>0.93 GB</td><td>10%</td></tr></tbody></table><p>模型结构和硬件的 co-design 之后，<strong>KV 已经不是瓶颈</strong>。</p><p>ds4.1f 把 KV 压到 890 字节 / token 之后，就算跑到 1M context，它也只占总流量的 10%。再压一半，总时间只少 6%；压到四分之一，也只少 9%。</p><p>而权重占 90%。把它减半，就是接近一倍的提速。</p><p>字节数直接决定速度。decode 是 memory bound，每个 token 的耗时就是总字节除以带宽：</p><table><thead><tr><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></th><th>权重</th><th>KV</th><th>合计</th><th>每 token</th><th>单流 tok/s</th></tr></thead><tbody><tr><td>4k</td><td>8.00 GB</td><td>0.004 GB</td><td>8.00 GB</td><td>1.00 ms</td><td>1000</td></tr><tr><td>32k</td><td>8.00 GB</td><td>0.029 GB</td><td>8.03 GB</td><td>1.00 ms</td><td>996</td></tr><tr><td>128k</td><td>8.00 GB</td><td>0.117 GB</td><td>8.12 GB</td><td>1.01 ms</td><td>986</td></tr><tr><td>1M</td><td>8.00 GB</td><td>0.933 GB</td><td>8.93 GB</td><td>1.12 ms</td><td>896</td></tr></tbody></table><p><strong>从 4k 到 1M，单流速度只掉 10%。</strong> KV 压缩买到的是跑长 context 不掉速，能跑起来只是附带的结果。</p><p>再看容量。单卡 172 GB 可用显存，按 890 字节 / token：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mfrac><mrow><mn>172</mn><mtext> GB</mtext></mrow><mrow><mn>890</mn><mtext> B</mtext></mrow></mfrac><mo>≈</mo><mn>1.93</mn><mtext> 亿 token</mtext></mrow><annotation encoding="application/x-tex">\frac{172\ \text{GB}}{890\ \text{B}} \approx 1.93\ \text{亿 token}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.0463em;vertical-align:-0.686em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">890</span><span className="mspace"> </span><span className="mord,text"><span className="mord">B</span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">172</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.686em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord">1.93</span><span className="mspace"> </span><span className="mord,text"><span className="mord,cjk_fallback">亿</span><span className="mord"> token</span></span></span></span></span></span><p>一张卡装得下 1.93 亿 token 的 global KV。</p><p>但容量不是硬上限。装不下就往下沉，host DRAM、SSD 一级一级放；请求再来时命中 prefix cache，把 KV 搬回 HBM 就能接着算。技术报告里那两个数本来就是按这个层级给的：常驻 HBM 的 global KV 是 890 字节 / token，而落在 SSD 或 host memory 的 persistent KV 再压到 V4-Flash 的 1/8。</p><p><strong>KV 这一层已经从&quot;装不下就崩&quot;的硬约束，变成了一个分层的存储问题。</strong></p><p>所以这条线走到这里，战场又换了一次位置：从&quot;压 KV&quot;换到&quot;压权重&quot;。前面几代解决的是历史存成什么样，接下来要解决的是 16B activated 这个数本身。</p><p>batch 能不能救？批处理把权重摊薄，但同时把 KV 按 batch 复制：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>I</mi><mo stretchy="false">(</mo><mi>B</mi><mo separator="true">,</mo><mi>N</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mi>B</mi><mo>×</mo><mn>2</mn><mo>×</mo><mn>16</mn><mtext>B</mtext></mrow><mrow><mn>8</mn><mtext> GB</mtext><mo>+</mo><mi>B</mi><mo>×</mo><mn>890</mn><mo>×</mo><mi>N</mi></mrow></mfrac></mrow><annotation encoding="application/x-tex">I(B, N) = \frac{B \times 2 \times 16\text{B}}{8\ \text{GB} + B \times 890 \times N}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.1297em;vertical-align:-0.7693em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">8</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">890</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">2</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">16</span><span className="mord,text"><span className="mord">B</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7693em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span></span><p>代进不同长度（可以直接运行验证）：</p><pre className="language-python shiki shiki-themes github-light github-dark" code="P = 16e9        # activated 参数量
W = 8e9         # FP4 权重，每 token 要读的字节数
kv = 890        # 每 token 的 KV 字节数
inflection = 1125  # B200 FP4 的算力带宽比（FLOP/byte）

def intensity(B, N):
    return B * 2 * P / (W + B * kv * N)

for N in [4e3, 32e3, 128e3, 1e6]:
    cells = &quot;  &quot;.join(f&quot;B={B}:{intensity(B, N):&gt;6.0f}&quot; for B in [1, 8, 64, 512])
    mark = &quot;  &lt;- 越线&quot; if intensity(512, N) &gt; inflection else &quot;&quot;
    print(f&quot;N={N:&gt;8.0f}  {cells}{mark}&quot;)
" language="python" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sVt8B">P </span><span class="szBVR">=</span><span class="sj4cs"> 16e9</span><span class="sJ8bj">        # activated 参数量
</span></span><span class="line" line="2"><span class="sVt8B">W </span><span class="szBVR">=</span><span class="sj4cs"> 8e9</span><span class="sJ8bj">         # FP4 权重，每 token 要读的字节数
</span></span><span class="line" line="3"><span class="sVt8B">kv </span><span class="szBVR">=</span><span class="sj4cs"> 890</span><span class="sJ8bj">        # 每 token 的 KV 字节数
</span></span><span class="line" line="4"><span class="sVt8B">inflection </span><span class="szBVR">=</span><span class="sj4cs"> 1125</span><span class="sJ8bj">  # B200 FP4 的算力带宽比（FLOP/byte）
</span></span><span class="line" line="5"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="6"><span class="szBVR">def</span><span class="sScJk"> intensity</span><span class="sVt8B">(B, N):
</span></span><span class="line" line="7"><span class="szBVR">    return</span><span class="sVt8B"> B </span><span class="szBVR">*</span><span class="sj4cs"> 2</span><span class="szBVR"> *</span><span class="sVt8B"> P </span><span class="szBVR">/</span><span class="sVt8B"> (W </span><span class="szBVR">+</span><span class="sVt8B"> B </span><span class="szBVR">*</span><span class="sVt8B"> kv </span><span class="szBVR">*</span><span class="sVt8B"> N)
</span></span><span class="line" line="8"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="9"><span class="szBVR">for</span><span class="sVt8B"> N </span><span class="szBVR">in</span><span class="sVt8B"> [</span><span class="sj4cs">4e3</span><span class="sVt8B">, </span><span class="sj4cs">32e3</span><span class="sVt8B">, </span><span class="sj4cs">128e3</span><span class="sVt8B">, </span><span class="sj4cs">1e6</span><span class="sVt8B">]:
</span></span><span class="line" line="10"><span class="sVt8B">    cells </span><span class="szBVR">=</span><span class="sZZnC"> &quot;  &quot;</span><span class="sVt8B">.join(</span><span class="szBVR">f</span><span class="sZZnC">&quot;B=</span><span class="sj4cs">{</span><span class="sVt8B">B</span><span class="sj4cs">}</span><span class="sZZnC">:</span><span class="sj4cs">{</span><span class="sVt8B">intensity(B, N)</span><span class="szBVR">:&gt;6.0f</span><span class="sj4cs">}</span><span class="sZZnC">&quot;</span><span class="szBVR"> for</span><span class="sVt8B"> B </span><span class="szBVR">in</span><span class="sVt8B"> [</span><span class="sj4cs">1</span><span class="sVt8B">, </span><span class="sj4cs">8</span><span class="sVt8B">, </span><span class="sj4cs">64</span><span class="sVt8B">, </span><span class="sj4cs">512</span><span class="sVt8B">])
</span></span><span class="line" line="11"><span class="sVt8B">    mark </span><span class="szBVR">=</span><span class="sZZnC"> &quot;  &lt;- 越线&quot;</span><span class="szBVR"> if</span><span class="sVt8B"> intensity(</span><span class="sj4cs">512</span><span class="sVt8B">, N) </span><span class="szBVR">&gt;</span><span class="sVt8B"> inflection </span><span class="szBVR">else</span><span class="sZZnC"> &quot;&quot;
</span></span><span class="line" line="12"><span class="sj4cs">    print</span><span class="sVt8B">(</span><span class="szBVR">f</span><span class="sZZnC">&quot;N=</span><span class="sj4cs">{</span><span class="sVt8B">N</span><span class="szBVR">:&gt;8.0f</span><span class="sj4cs">}</span><span class="sj4cs">  {</span><span class="sVt8B">cells</span><span class="sj4cs">}{</span><span class="sVt8B">mark</span><span class="sj4cs">}</span><span class="sZZnC">&quot;</span><span class="sVt8B">)
</span></span></code></pre><p>输出里只有两格越线，都在 4k 那一行，而且要 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>≥</mo><mn>512</mn></mrow><annotation encoding="application/x-tex">B \ge 512</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8193em;vertical-align:-0.136em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≥</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">512</span></span></span></span>。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>→</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">N \to 0</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">0</span></span></span></span> 时 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>I</mi><mo>→</mo><mn>4</mn><mi>B</mi></mrow><annotation encoding="application/x-tex">I \to 4B</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">4</span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span></span></span></span>，所以 batch 要超过 280 才有可能碰到那条线。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">B=64</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">64</span></span></span></span> 的极限是 256，连门槛都摸不到。</p><p><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 一大就更没戏。<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>→</mo><mi mathvariant="normal">∞</mi></mrow><annotation encoding="application/x-tex">B \to \infty</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord">∞</span></span></span></span> 时 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>I</mi><mo>→</mo><mn>2</mn><mi>P</mi><mi mathvariant="normal">/</mi><mo stretchy="false">(</mo><mn>890</mn><mi>N</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">I \to 2P / (890N)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0785em;">I</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">2</span><span className="mord,mathnormal" style="margin-right:0.1389em;">P</span><span className="mord">/</span><span className="mopen">(</span><span className="mord">890</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span></span></span></span>，1M context 下只有 34，离 1125 差 33 倍。因为 KV 太小，把权重完全摊掉之后剩下的那点 KV 依然喂不饱算力。</p><p>结论比 MLA 那次更极端。MLA 至少留了一个&quot;大 batch 加短 context 会翻到 compute bound&quot;的窗口，而且那个窗口落在 4k，恰好是它收益最小的区间。ds4.1f 的窗口更窄：要 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>≥</mo><mn>512</mn></mrow><annotation encoding="application/x-tex">B \ge 512</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8193em;vertical-align:-0.136em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≥</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">512</span></span></span></span> 且 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 压到 1.4 万以内，才算勉强碰到算力那一侧。</p><p>再算一遍吞吐。每步耗时取带宽和算力里较大的那个：</p><table><thead><tr><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi></mrow><annotation encoding="application/x-tex">B</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.0502em;">B</span></span></span></span></th><th>1M：单序列 tok/s</th><th>1M：总吞吐</th><th>32k：总吞吐</th></tr></thead><tbody><tr><td>1</td><td>896</td><td>896</td><td>996</td></tr><tr><td>8</td><td>517</td><td>4,138</td><td>7,773</td></tr><tr><td>32</td><td>211</td><td>6,761</td><td>28,657</td></tr><tr><td>128</td><td>63</td><td>8,034</td><td>87,276</td></tr><tr><td>512</td><td>16</td><td>8,431</td><td>178,617</td></tr><tr><td><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>→</mo><mi mathvariant="normal">∞</mi></mrow><annotation encoding="application/x-tex">\to \infty</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.3669em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord">∞</span></span></span></span></td><td>趋于 0</td><td>8,572</td><td>274,315</td></tr></tbody></table><p><strong>总吞吐有上限，因为它最终被 KV 的带宽锁住：</strong></p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mtext>吞吐</mtext><mi>max</mi><mo>⁡</mo></msub><mo>→</mo><mfrac><mrow><mn>8</mn><mtext> TB/s</mtext></mrow><mrow><mn>890</mn><mtext> B</mtext><mo>×</mo><mi>N</mi></mrow></mfrac></mrow><annotation encoding="application/x-tex">\text{吞吐}_{\max} \to \frac{8\ \text{TB/s}}{890\ \text{B} \times N}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,text"><span className="mord,cjk_fallback">吞吐</span></span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mop,mtight"><span className="mtight">m</span><span className="mtight">a</span><span className="mtight">x</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">→</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.1963em;vertical-align:-0.7693em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.427em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">890</span><span className="mspace"> </span><span className="mord,text"><span className="mord">B</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">8</span><span className="mspace"> </span><span className="mord,text"><span className="mord">TB/s</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.7693em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span></span><table><thead><tr><th><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span></th><th>4k</th><th>32k</th><th>128k</th><th>1M</th></tr></thead><tbody><tr><td>吞吐上限</td><td>281k</td><td>274k</td><td>68.6k</td><td>8.6k</td></tr></tbody></table><p>4k 和 32k 那两档其实先被算力封顶（281k），再往上才是 KV 带宽封顶。分界在 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>≈</mo><mn>3.2</mn></mrow><annotation encoding="application/x-tex">N \approx 3.2</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">3.2</span></span></span></span> 万，和前面 batch 表里算出来的是同一个数。</p><p><strong>代价在延迟那一列。</strong> 同一个 1M 的请求，batch 从 1 开到 512，单序列速度从 896 掉到 16 token/s。长 context 上吞吐和交互速度是直接对立的：要吃满吞吐，就得牺牲单个用户的等待时间。</p><p>反过来，如果要求单序列不低于 200 token/s，1M context 下 batch 只能开到 30 上下，单卡吞吐就是 6-7k token/s。这个量级和我实际跑 ds4.1f 的手感对得上。</p><p>换成 TP4 部署，每卡的账是这样：</p><table><thead><tr><th></th><th>每卡</th></tr></thead><tbody><tr><td>权重常驻显存（全部 552B 专家）</td><td>69 GB</td></tr><tr><td>留给 KV 的显存</td><td>111 GB</td></tr><tr><td>每步读的权重（16B activated 的四分之一）</td><td><strong>2.00 GB</strong></td></tr><tr><td>每步读的 KV（batch 1，1M context）</td><td><strong>0.23 GB</strong></td></tr><tr><td>合计</td><td>2.23 GB</td></tr><tr><td>每步带宽下限</td><td>0.28 ms</td></tr></tbody></table><p>要分清常驻和每步读，也要交代那个除以 4 是哪来的。</p><p><strong>常驻</strong>：552B 个专家参数必须都在显存里。TP4 把每个权重矩阵切成四份、每卡持一份，所以是</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mfrac><mrow><mn>552</mn><mtext>B</mtext><mo>×</mo><mn>0.5</mn><mtext> 字节</mtext></mrow><mn>4</mn></mfrac><mo>=</mo><mn>69</mn><mtext> GB</mtext></mrow><annotation encoding="application/x-tex">\frac{552\text{B} \times 0.5\ \text{字节}}{4} = 69\ \text{GB}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.0463em;vertical-align:-0.686em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">4</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">552</span><span className="mord,text"><span className="mord">B</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">0.5</span><span className="mspace"> </span><span className="mord,text"><span className="mord,cjk_fallback">字节</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.686em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">69</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span></span></span></span></span><p><strong>每步读</strong>：一个 token 只激活其中 16B。整机每步要读</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>16</mn><mtext>B</mtext><mo>×</mo><mn>0.5</mn><mtext> 字节</mtext><mo>=</mo><mn>8</mn><mtext> GB</mtext></mrow><annotation encoding="application/x-tex">16\text{B} \times 0.5\ \text{字节} = 8\ \text{GB}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7667em;vertical-align:-0.0833em;"></span><span className="mord">16</span><span className="mord,text"><span className="mord">B</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">0.5</span><span className="mspace"> </span><span className="mord,text"><span className="mord,cjk_fallback">字节</span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">8</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span></span></span></span></span><p>而这 16B 激活参数同样散在四张卡上，各卡只读自己那一份，所以每卡读四分之一：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mfrac><mrow><mn>8</mn><mtext> GB</mtext></mrow><mn>4</mn></mfrac><mo>=</mo><mn>2</mn><mtext> GB</mtext></mrow><annotation encoding="application/x-tex">\frac{8\ \text{GB}}{4} = 2\ \text{GB}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.0463em;vertical-align:-0.686em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3603em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">4</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">8</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.686em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">2</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span></span></span></span></span><p>KV 是同一个道理。TP 把 KV head 也分到四张卡上，每卡只存自己那部分，也就只读自己那部分：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mfrac><mrow><mn>890</mn><mtext> B</mtext><mo>×</mo><msup><mn>10</mn><mn>6</mn></msup></mrow><mn>4</mn></mfrac><mo>=</mo><mn>0.23</mn><mtext> GB</mtext></mrow><annotation encoding="application/x-tex">\frac{890\ \text{B} \times 10^6}{4} = 0.23\ \text{GB}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.1771em;vertical-align:-0.686em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.4911em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">4</span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">890</span><span className="mspace"> </span><span className="mord,text"><span className="mord">B</span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">×</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord">1</span><span className="mord"><span className="mord">0</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">6</span></span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.686em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord">0.23</span><span className="mspace"> </span><span className="mord,text"><span className="mord">GB</span></span></span></span></span></span><p><strong>两项加起来，KV 只占 10%。</strong> 和前面 batch 1 的结论一致，TP 不改变这个比例。</p><p>四张卡合起来，节点级 KV 容量是 444 GB，按 890 字节 / token 算就是 4.99 亿 token。单序列的理论上限也从 896 提到约 3,600 token/s。</p><p>这就是 ds4.1f-FP4 在 B200 上的 roofline。</p><p>拐点 1125 FLOP/byte 换一种读法：<strong>搬运 1 个字节的时间，够做 1125 次浮点运算</strong>。于是一个机制只要能用不到 1125 次运算省下 1 个字节的搬运，就净赚。以这个比值衡量，ds4.1f 里的几个设计都在做同一笔交易：</p><ul><li><strong>SWA Bounded Replay</strong>：请求恢复时不重放整条序列，只重算最近 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>n</mi><mtext>win</mtext></msub></mrow><annotation encoding="application/x-tex">n_{\text{win}}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">n</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3175em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">win</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 个 token 来重建局部状态。用一段计算，换掉局部窗口 KV 的持久化。</li><li><strong>FP4 KV</strong>：存储和带宽都降到四分之一，代价是读进来要反量化。用一点计算，换掉四分之三的字节。</li><li><strong>Hierarchical Indexer</strong>：先用一层打分粗筛出候选池，后续层只在这个池子里挑。用一层索引，换掉对全部候选的扫描。</li></ul><p>共同点是把成本从 memory 一侧挪到 compute 一侧。</p><p>前面几代的动作一直是压缩存储。现在存储已经压到接近下限，规则反过来：<strong>用富余的算力，去重建那些不值得搬过来的字节。</strong></p><p>这些省下来的资源最终可以被重新花到：更多 tokens、更好的数据过滤、更长的 RL rollout、更多 environment interaction、更多 rejection sampling、更多 verifier compute、更大的 batch、更复杂的 agent tasks。到这里，可以回到最初的问题了。ds4.1f 省掉的，是账单上几乎每一项。</p><h2 id="capability-与-efficiency">Capability 与 Efficiency</h2><p>业界基本有共识：能力里很大的一部分来自后训练，而不是架构。三个例子。</p><p>GLM-5.3 的发布说明只有一句话，<a href="https://z.ai/blog/glm-5.3" rel="nofollow">Scaling post-training is all we did</a>。基座和 GLM-5.2 一样，没有新增一个预训练 token，Terminal-Bench 3.0 从 4.6 涨到 28.3，六倍。</p><p>Cognition 的 <a href="https://cognition.com/blog/swe-2" rel="nofollow">SWE-2</a> 更能说明分工。它是一个美国实验室从 Kimi K3 的开源权重上后训练出来的，而 Kimi K3 是 Moonshot 放出来的 2.8T 参数模型。Cognition 报的成绩是编码能力离前沿只差一点，成本低 64%；首次真正动手改代码的中位步数从 48 降到 18。</p><p>开源权重到了这个水平，做一款有竞争力的编码产品，贵的环节已经不是预训练，而是后训练和 harness。</p><p>第三个例子是 DeepSeek V4-Flash-0731。它的模型结构和尺寸与 Preview 保持一致，只重新做了后训练，但 Agent benchmark 大幅提升：</p><p>DeepSWE 从 7.3 涨到 54.4，CyberGym 从 38.7 到 76.7，TerminalBench 2.1 从 61.8 到 82.7。模型 backbone 没换。Attention 没换。参数量没换。官方更新日志直接写明结构不变、仅重新后训练。这个例子比任何口号都说明问题：<strong>同一个 backbone 的 post-training 差异，完全可能比两代 architecture 的差异更大。</strong> 所以把&quot;模型进步&quot;拆成四个层次更合适：</p><table><thead><tr><th>层次</th><th>解决什么</th><th>对能力的作用</th></tr></thead><tbody><tr><td>Architecture</td><td>每 FLOP / 每 GB 能做多少事</td><td>提高上限和效率</td></tr><tr><td>Data</td><td>模型到底看到了什么分布</td><td>决定知识、能力覆盖</td></tr><tr><td>Post-training / RL</td><td>学会怎么推理、用工具、完成任务</td><td>直接决定最终行为能力</td></tr><tr><td>Infra</td><td>上面三件事是否真的按设计运行</td><td>防止几个月训练被 bug 偷走</td></tr></tbody></table><p>Infra 这一层最容易被低估。RL 的产出取决于一条很长的系统，公式只占其中一环：<code>prompt → rollout → tool environment → reward → sampling → advantage → gradient → model update</code>。任何地方一个小 bug，最后都会变成同一句话：&quot;奇怪，我们这个新的 RL 算法怎么效果一般？&quot; 而那个 bug 可能是 reward 算错、tool state 泄漏、rollout truncation 错、mask 错、sampling distribution 不一致、tokenizer offset 错、advantage normalization 错、stale policy 太严重，或者 sandbox reset 有问题。实际上不是算法一般，是代码在偷偷犯罪。所以更准的因果链是架构先降低成本，省下的成本换来更有效的训练，训练再沉淀成能力；&quot;架构直接产生智能&quot;是另一种叙事。架构对能力的作用经常是<strong>间接的</strong>。如果问&quot;这一代模型为什么明显更好用了&quot;，重要性排序大致是 Data &gt; Post-training/RL &gt; Eval/Verifier &gt; Training infra &gt; Architecture，答案多半落在前面几项，而不在&quot;我们把 attention 又改了一个 acronym&quot;。旁边还有一组更好记的说法，把每一层对应到一个&quot;前沿&quot;：</p><ul><li>Architecture 决定 <strong>efficiency frontier</strong>：每单位算力能处理多少有效训练信号</li><li>Data 决定 <strong>learning frontier</strong>：模型有机会学到什么</li><li>RL / post-training 决定 <strong>behavior frontier</strong>：模型会不会把能力真正用出来</li><li>Infra 决定<strong>你是否真的获得前三者</strong>：实验结果到底是不是你以为的实验</li></ul><p>Infra 这一条听起来最不起眼，出问题时损失最大。一个漂亮的新 loss 写在论文里只需要半页；一个可靠的 million-token agent rollout system，可能需要几十个人跟各种分布式 bug、CUDA bug、环境 bug、数据 bug 打几个月架。论文不会把凌晨三点的 NCCL timeout 写进公式里，但模型能力很可能就死在那里。KDA 和 CSA/HCA 的作用前面已经说过：让 1M context 和 long-horizon RL rollout 在计算上变得现实。放进 infra 的语境再看一层：这套系统跑不起来，再好的 loss 也只是半页论文。</p><p><strong>架构是 compute multiplier，乘出来的是训练预算，不是智能本身。</strong></p><h2 id="算力预算">算力预算</h2><p>如果架构主要在省钱，那&quot;该不该冒架构的险&quot;就变成了一个纯粹的预算问题。假设已验证的 Transformer scaling recipe 能拿到 95 分。新架构理论上能到 100，但有 20% 概率训练炸掉，只剩 70。如果算力富裕，为什么要冒险？把已验证的架构再 scale 10 倍，可能直接拿到 98，而且更可靠。这其实是一种&quot;富人的保守&quot;。而算力紧张的实验室没有 10× compute 可以挥霍，只能想办法让 1× compute 做出 3× compute 的效果，架构创新的 ROI 因此变得非常高。从优化目标看更直白。算力富裕的一方更接近：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>max</mi><mo>⁡</mo><mtext>Capability</mtext><mo separator="true">,</mo><mspace width="1em"></mspace><mtext>s.t. Compute 很大</mtext></mrow><annotation encoding="application/x-tex">\max \text{Capability}, \quad \text{s.t. Compute 很大}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mop">max</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,text"><span className="mord">Capability</span></span><span className="mpunct">,</span><span className="mspace" style="margin-right:1em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,text"><span className="mord">s.t. Compute </span><span className="mord,cjk_fallback">很大</span></span></span></span></span></span><p>所以可以接受较低的 architecture efficiency，只要 scaling 稳。而算力受限的一方更像：</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>max</mi><mo>⁡</mo><mfrac><mtext>Capability</mtext><mtext>Compute</mtext></mfrac></mrow><annotation encoding="application/x-tex">\max \frac{\text{Capability}}{\text{Compute}}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:2.2519em;vertical-align:-0.8804em;"></span><span className="mop">max</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:1.3714em;"><span style="top:-2.314em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,text"><span className="mord">Compute</span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord,text"><span className="mord">Capability</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.8804em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span></span><p>于是被逼出 MLA、MoE、Sparse、Linear、各种 compression。这和芯片行业很像。资源充裕时加晶体管，资源受限时改 architecture。所以很多所谓架构路线之争，底层其实是不同算力预算下的经济学。把这件事再拆细一点，会发现竞争其实发生在四种资本上：</p><ul><li><strong>Compute Capital</strong>：GPU、HBM、network、datacenter。</li><li><strong>Data Capital</strong>：真实用户、agent trajectories、enterprise workflow、feedback。</li><li><strong>Algorithmic Capital</strong>：architecture、training recipe、RL、verifier。</li><li><strong>Infrastructure Capital</strong>：稳定地把几千几万张 GPU、RL environment、rollout 全跑起来。</li></ul><p>算力富裕的一方在前两项和第四项上有巨大积累；算力受限的一方被迫在第三项特别凶。这也是为什么&quot;美国靠暴力 scaling、中国靠聪明架构&quot;这种说法太简化了。更接近事实的写法是：<strong>一边在用资源优势降低研究风险，另一边在用资源约束提高创新收益。</strong></p><p>而且&quot;数据&quot;这一项也没法简单说成谁多谁少。真正稀缺的早就不是互联网文本了。Wikipedia、Common Crawl、GitHub 这些 public corpus，大家拿到的量级差不多。现在值钱的是高质量 reasoning traces、真实 agent trajectories、用户交互反馈、专业任务数据、tool-use traces、可验证环境。这些东西不是爬虫能给你的，是产品规模给的。</p><p>所以更要命的不是&quot;谁有更多互联网 token&quot;，而是&quot;谁有更多高价值的 interaction token&quot;。但这里有个反转。</p><p>一旦某个架构优化被证明有效，情况就变了。DeepSeek 做出 MLA，或者 Kimi 做出成熟的 Linear Attention，算力多的一方并不会因为自己算力多就拒绝使用。它们完全可以拿&quot;更多算力 × 更高效架构&quot;叠加使用。于是原本弱者为了生存开发出来的 efficiency innovation，最后反而可能让强者 scale 得更狠。问题最后又回到：省下来的钱，值不值得花。</p><h2 id="小结">小结</h2><p>开头说 ds4.1f 是模型架构与硬件 co-design 的杰作。看完整条路线，我觉得这个说法可以讲得更具体一点：它省掉的不是某一项。是账单上的好几项，而且是分几代攒下来的。把整条路线画成一张图是这样：</p><pre className="language-text" code="MHA              每个 head 一份 K、V
 │
 ▼
MQA / GQA        压 head 数：离散地共享
 │
 ▼
MLA              压 D 轴：连续压缩成 latent
 │
 ▼
CSA / HCA        压 N 轴：压成 block，再挑着看
 │
 ▼
Linear / SSM     压掉保存本身：只留一个 state
 │
 ▼
V4.1             压 L 轴和计算阶段：不再每层每步都付同样的钱
" language="text" meta=""><code __ignoreMap="">MHA              每个 head 一份 K、V
 │
 ▼
MQA / GQA        压 head 数：离散地共享
 │
 ▼
MLA              压 D 轴：连续压缩成 latent
 │
 ▼
CSA / HCA        压 N 轴：压成 block，再挑着看
 │
 ▼
Linear / SSM     压掉保存本身：只留一个 state
 │
 ▼
V4.1             压 L 轴和计算阶段：不再每层每步都付同样的钱
</code></pre><p>同样的路线可以读成一连串追问：每个 head 都需要独立 KV 吗——MQA/GQA 说不用；每个 token 都需要完整 KV 吗——MLA 说不用；每个 query 都需要看所有 token 吗——CSA/HCA 说不用；每个 token 都需要被永久保存吗——Linear/SSM 说不用；每个 layer 都需要独立 memory、prefill 和 decode 必须用同样计算吗——V4.1 说不用。每一代都在放松一个假设，<strong>保存信息、压缩信息、找回信息</strong>三个动作在其中循环。</p><p>前面一直把架构、数据、算力当成三样不同的东西比轻重，这个切法可能一开始就偏了。它们互相兑换：</p><p><strong>数据是架构。</strong> 用什么数据、怎么组织、怎么采样，直接决定模型长成什么样。同一套 attention，换一个数据分布就是另一个模型。数据管线里每个决定，效果上和改架构是同一个量级。</p><p><strong>架构是算力。</strong> 这条前面已经推过一遍。MLA 把 KV 压到 1/57，ds4.1f 把 decode 的算术强度压到离拐点 300 倍，省下来的都是算力。架构不改模型会做什么，只改做同一件事要花多少钱。它是算力的另一种记账方式。</p><p><strong>算力是数据。</strong> 省下来的算力不会消失，会被重新花到更多 tokens、更长的 rollout、更多 verifier compute。算力的下游就是数据。</p><p>三个词绕成一个环。能力和架构谁更重要，这个问题问错了对象：三者是同一笔资源的三种形态，换名字不改变总量，变的只是兑换率。成本压到足够低之后，架构这条线上剩下的竞争，只是决定那笔资源能兑换多少次。</p><style>html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}</style>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[有趣的音乐节奏]]></title>
            <link>https://blog.niina.fun/posts/interesting-music-rhythm</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/interesting-music-rhythm</guid>
            <pubDate>Thu, 16 Jul 2026 03:47:24 GMT</pubDate>
            <content:encoded><![CDATA[<p>套拍子可以说是音乐中化腐朽为神奇的一种玩法，可以让简单的乐句也变得十分的丰富和有趣，我个人十分喜欢套拍的听感和玩法。但是对怎么玩套拍本身还是存在诸多疑惑。</p><p>虽然自己一直有听过什么听四打三、听三打四，四套三、三套三、二对三等等之类的套拍说法，但其实对于它们本身指的是什么？以及具体的组成还是有些模糊。</p><p>在音乐中，节奏(rhythm)一般指声音在时间上是如何排列组合的，其实算一个抽象的概念。具象一些的实践，就是节拍(meter)。可以说节拍(meter)是节奏(rhythm)在音乐中的形状。节拍指的是重复的节奏和拍子是如何有规律地被划分的，乐理上有小节、拍号、速度等等书面描述记法将之具像化。</p><p>套拍子简单来说就是复合节奏(polyrhythm)、复合节拍(polymeter)。</p><p>借用一张图直观的感受一下。</p><p><img alt="复合节拍和复合节奏" src="https://i0.wp.com/ethanhein.com/wp/wp-content/uploads/2024/04/3-to-4-polymeter-vs-3-to-4-polyrhythm.png" /></p><ul><li>复合节拍(Based on Beat)：每一拍是对齐的（或者说 grid 对齐），但是强拍（小节）不对齐。</li><li>复合节奏(Based on Bar)：每一拍不对齐（或者说 grid 不对齐），但强拍（小节）是对齐的。</li></ul><p>两者显然有很大的区别。Polymeter 的情况下，tempo（速度）是不变的，每一个 beat 也是对齐的。我想所谓听三打四、听四打三就是属于复合节拍。</p><p>而复合节奏的话其实很常见，比如说三连音就是短时的复合节奏。而常说的四对三</p><h2 id="复合节拍听四打三实例">复合节拍：听四打三实例</h2><p>复合节拍（polymeter）是另一回事：两把&quot;乐器&quot;拍子对拍子、速度一致，但各自数自己的小节，小节线是错开的。也就是说可能一个四拍子的东西，你当作了三拍子听（反之亦然），就是复合节拍的效果。下面给出一个听四打三的例子。</p><poly-score :staff="true"><pre className="language-poly-score shiki shiki-themes github-light github-dark" code="4/4 @144 b4 c4 d4 e4 | d4 c4 b4 c4 | d4 e4 d4 c4 | b4 c4 d4 e4 |
3/4 @144 bars=6 | |
4/4 @144
" highlights="0" language="poly-score" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span>4/4 @144 b4 c4 d4 e4 | d4 c4 b4 c4 | d4 e4 d4 c4 | b4 c4 d4 e4 |
</span></span><span class="line" line="2"><span>3/4 @144 bars=6 | |
</span></span><span class="line" line="3"><span>4/4 @144
</span></span></code></pre></poly-score><h2 id="复合节奏">复合节奏</h2><p>复合节奏（PolyRhythm）又叫复合速度(PolyTempo)，也就是说在保持强拍一致的情况下，同时存在不同 Tempo 的声部。</p><poly-score :staff="true"><pre className="language-poly-score shiki shiki-themes github-light github-dark" code="4/4 @160 bars=4 inst=tone c4 e4 d4 f4 | c4 e8 c4 b&#39;4. | c4 e4 d4 f4 | c4 e8 c4 b&#39;4 g8
3/4 @120
4/4 @160
" highlights="0" language="poly-score" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span>4/4 @160 bars=4 inst=tone c4 e4 d4 f4 | c4 e8 c4 b&#39;4. | c4 e4 d4 f4 | c4 e8 c4 b&#39;4 g8
</span></span><span class="line" line="2"><span>3/4 @120
</span></span><span class="line" line="3"><span>4/4 @160
</span></span></code></pre></poly-score><p>速度不同也行，时间轴按真实时间比例拉开，错拍的过程一目了然：</p><poly-score><pre className="language-poly-score shiki shiki-themes github-light github-dark" code="4/4 @80
3/4 @60
" language="poly-score" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span>4/4 @80
</span></span><span class="line" line="2"><span>3/4 @60
</span></span></code></pre></poly-score><style>html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}</style>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM 可解释性（一）：SAE与单语义]]></title>
            <description><![CDATA[读 Anthropic《Towards Monosemanticity》。从难以解释的神经元出发, 讲清superposition为什么让单个神经元混成一团, SAE怎么把多语义的激活分解成单语义特征。]]></description>
            <link>https://blog.niina.fun/posts/towards-monosemanticity</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/towards-monosemanticity</guid>
            <pubDate>Sun, 12 Jul 2026 13:00:00 GMT</pubDate>
            <content:encoded><![CDATA[<p>这就是 <strong>polysemanticity(多义性)</strong>。Anthropic 这篇的工作是:既然神经元本身不好直接解释,那我们就用稀疏自编码器(SAE)把激活&quot;解混&quot;成一组<strong>单语义特征(monosemantic features)</strong>,并给出证据说这些特征才是模型真正在用的计算单元。</p><h2 id="一可解释性的前提假设">一、可解释性的前提假设</h2><p>机械可解释性(mechanistic interpretability)最早的想法很朴素:打开模型,逐个神经元看它对什么输入有反应,像生物学家找&quot;祖母细胞&quot;那样。但这条路撞墙了,因为绝大多数神经元是多义的,一个神经元可能同时对 DNA 序列、阿拉伯文、人名首字母大写都有反应。</p><p>为什么? Anthropic 在更早的 <a href="https://transformer-circuits.pub/2022/toy_model/index.html#strategic-ways-out" rel="nofollow">Toy Models of Superposition</a> 里给了一个很干净的假设:</p><blockquote><p><strong>模型要表达的概念(features)比它拥有的维度(neurons)多得多。</strong></p></blockquote><p>于是模型把一堆概念&quot;挤&quot;进一个低维空间里——这就是 <strong>superposition(叠加)</strong>。直觉上,如果每个概念是高维空间里一个<strong>几乎正交</strong>的方向,那么 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span> 维空间可以塞下远多于 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span> 个几乎正交的方向(只要允许它们之间有一点干扰),代价是每个方向不再对齐到任何一根坐标轴(也就是不再等于某个神经元)。这件事不是拍脑袋--苏剑林在<a href="https://kexue.fm/archives/7076" rel="nofollow">《n 维空间下两个随机向量的夹角分布》</a>里算过:两个随机单位向量的夹角 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi></mrow><annotation encoding="application/x-tex">\theta</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span></span></span></span> 的密度正比于 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mrow><mi>sin</mi><mo>⁡</mo></mrow><mrow><mi>d</mi><mo>−</mo><mn>2</mn></mrow></msup><mi>θ</mi></mrow><annotation encoding="application/x-tex">\sin^{d-2}\theta</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.9069em;"></span><span className="mop"><span className="mop">sin</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.9069em;"><span style="top:-3.1208em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="mbin,mtight">−</span><span className="mord,mtight">2</span></span></span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">θ</span></span></span></span>,关于 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi><mi mathvariant="normal">/</mi><mn>2</mn></mrow><annotation encoding="application/x-tex">\pi/2</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0359em;">π</span><span className="mord">/2</span></span></span></span> 对称,均值就是 90°。但光有均值不够--二维、三维的均值也是 90°。真正让结论成立的是<strong>方差</strong>:文章证明方差随维度 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span> 趋于 0,近似为 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><mi>d</mi><mo>−</mo><mn>2</mn></mrow></mfrac></mrow><annotation encoding="application/x-tex">\frac{1}{d-2}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1.2484em;vertical-align:-0.4033em;"></span><span className="mord"><span className="mopen,nulldelimiter"></span><span className="mfrac"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.8451em;"><span style="top:-2.655em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="mbin,mtight">−</span><span className="mord,mtight">2</span></span></span></span><span style="top:-3.23em;"><span className="pstrut" style="height:3em;"></span><span className="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.394em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">1</span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.4033em;"><span></span></span></span></span></span><span className="mclose,nulldelimiter"></span></span></span></span></span>(弧度²),<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi><mo>=</mo><mn>1000</mn></mrow><annotation encoding="application/x-tex">d=1000</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">1000</span></span></span></span> 时标准差只有约 1.8°,夹角几乎被钉死在 90°。</p><p>另外有 JL 引理：塞下 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span></span></span></span> 个向量只需要 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>l</mi><mi>o</mi><mi>g</mi><mi>N</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(logN)</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord,mathnormal" style="margin-right:0.0278em;">O</span><span className="mopen">(</span><span className="mord,mathnormal" style="margin-right:0.0197em;">l</span><span className="mord,mathnormal">o</span><span className="mord,mathnormal" style="margin-right:0.0359em;">g</span><span className="mord,mathnormal" style="margin-right:0.109em;">N</span><span className="mclose">)</span></span></span></span> 维空间。</p><blockquote><p><strong>高维空间里随便抓两个方向就几乎正交,所以能塞下远多于 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span> 个几乎正交的方向。</strong></p></blockquote><p>随手模拟一下:在不同维度 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span> 里各取一万个随机向量,看随机一对的夹角落在 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">[</mo><mn>80</mn><mi mathvariant="normal">°</mi><mo separator="true">,</mo><mn>100</mn><mi mathvariant="normal">°</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">[80°,100°]</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mopen">[</span><span className="mord">80°</span><span className="mpunct">,</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord">100°</span><span className="mclose">]</span></span></span></span> 的比例:</p><pre className="language-python shiki shiki-themes github-light github-dark" code="import numpy as np

np.random.seed(0)
for d in [2, 3, 10, 100, 1000]:
    n = 10000
    x = np.random.randn(n, d)
    x /= np.linalg.norm(x, axis=1, keepdims=True)
    i, j = np.random.randint(n, size=100000), np.random.randint(n, size=100000)
    ang = np.degrees(np.arccos(np.clip((x[i] * x[j]).sum(1), -1, 1)))
    print(f&quot;d={d:5d}: 夹角落在 [80°, 100°] 的比例 = {(np.abs(ang - 90) &lt;= 10).mean():.4f}&quot;)
" language="python" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="szBVR">import</span><span class="sVt8B"> numpy </span><span class="szBVR">as</span><span class="sVt8B"> np
</span></span><span class="line" line="2"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="3"><span class="sVt8B">np.random.seed(</span><span class="sj4cs">0</span><span class="sVt8B">)
</span></span><span class="line" line="4"><span class="szBVR">for</span><span class="sVt8B"> d </span><span class="szBVR">in</span><span class="sVt8B"> [</span><span class="sj4cs">2</span><span class="sVt8B">, </span><span class="sj4cs">3</span><span class="sVt8B">, </span><span class="sj4cs">10</span><span class="sVt8B">, </span><span class="sj4cs">100</span><span class="sVt8B">, </span><span class="sj4cs">1000</span><span class="sVt8B">]:
</span></span><span class="line" line="5"><span class="sVt8B">    n </span><span class="szBVR">=</span><span class="sj4cs"> 10000
</span></span><span class="line" line="6"><span class="sVt8B">    x </span><span class="szBVR">=</span><span class="sVt8B"> np.random.randn(n, d)
</span></span><span class="line" line="7"><span class="sVt8B">    x </span><span class="szBVR">/=</span><span class="sVt8B"> np.linalg.norm(x, </span><span class="s4XuR">axis</span><span class="szBVR">=</span><span class="sj4cs">1</span><span class="sVt8B">, </span><span class="s4XuR">keepdims</span><span class="szBVR">=</span><span class="sj4cs">True</span><span class="sVt8B">)
</span></span><span class="line" line="8"><span class="sVt8B">    i, j </span><span class="szBVR">=</span><span class="sVt8B"> np.random.randint(n, </span><span class="s4XuR">size</span><span class="szBVR">=</span><span class="sj4cs">100000</span><span class="sVt8B">), np.random.randint(n, </span><span class="s4XuR">size</span><span class="szBVR">=</span><span class="sj4cs">100000</span><span class="sVt8B">)
</span></span><span class="line" line="9"><span class="sVt8B">    ang </span><span class="szBVR">=</span><span class="sVt8B"> np.degrees(np.arccos(np.clip((x[i] </span><span class="szBVR">*</span><span class="sVt8B"> x[j]).sum(</span><span class="sj4cs">1</span><span class="sVt8B">), </span><span class="szBVR">-</span><span class="sj4cs">1</span><span class="sVt8B">, </span><span class="sj4cs">1</span><span class="sVt8B">)))
</span></span><span class="line" line="10"><span class="sj4cs">    print</span><span class="sVt8B">(</span><span class="szBVR">f</span><span class="sZZnC">&quot;d=</span><span class="sj4cs">{</span><span class="sVt8B">d</span><span class="szBVR">:5d</span><span class="sj4cs">}</span><span class="sZZnC">: 夹角落在 [80°, 100°] 的比例 = </span><span class="sj4cs">{</span><span class="sVt8B">(np.abs(ang </span><span class="szBVR">-</span><span class="sj4cs"> 90</span><span class="sVt8B">) </span><span class="szBVR">&lt;=</span><span class="sj4cs"> 10</span><span class="sVt8B">).mean()</span><span class="szBVR">:.4f</span><span class="sj4cs">}</span><span class="sZZnC">&quot;</span><span class="sVt8B">)
</span></span></code></pre><p>输出大致是 d=2: 0.11、d=3: 0.17、d=10: 0.39、d=100: 0.92、d=1000: 1.00--维度一上来,几乎每一对都贴着 90°。</p><p>方向挤在一起就不再严格正交,激活一个特征会顺带蹭到别的特征一点--这是叠加付出的<strong>干扰</strong>。几乎正交的条件不够,真正让 superposition 成立的是<strong>稀疏</strong>:同时激活的特征很少,每对方向本来就接近 90°。</p><p>所以 polysemanticity 是模型在有限容量下<strong>主动选择</strong>的压缩策略:</p><ul><li>神经元 = 坐标轴方向 → 因为叠加,一个轴上叠了好几个概念 → 多义。</li><li>特征 = 叠加后那个几乎正交的方向 → 才是&quot;一个概念&quot;的载体 → 单义。</li></ul><p>另外，LLM 其实是对超大规模数据的压缩，superposition 可以说是压缩过程中自然的行为；对应的，我们追求可解释性就需要进行解压缩。</p><p>结论很清楚:<strong>要看懂模型,得先把叠加解开,把坐标轴换成特征方向。</strong></p><h2 id="二稀疏自编码器把混合的激活解混">二、稀疏自编码器:把混合的激活&quot;解混&quot;</h2><p>上面说模型把概念稀疏叠加进激活,写成式子就是 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mo>≈</mo><mi>W</mi><mi>f</mi></mrow><annotation encoding="application/x-tex">x \approx Wf</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4831em;"></span><span className="mord,mathnormal">x</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="mord,mathnormal" style="margin-right:0.1076em;">f</span></span></span></span>:字典 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>W</mi><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><mrow><mi>d</mi><mo>×</mo><mi>m</mi></mrow></msup></mrow><annotation encoding="application/x-tex">W \in \mathbb{R}^{d\times m}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.7224em;vertical-align:-0.0391em;"></span><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8491em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8491em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span><span className="mbin,mtight">×</span><span className="mord,mathnormal,mtight">m</span></span></span></span></span></span></span></span></span></span></span></span> 过完备(<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi><mo>≫</mo><mi>d</mi></mrow><annotation encoding="application/x-tex">m \gg d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5782em;vertical-align:-0.0391em;"></span><span className="mord,mathnormal">m</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≫</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span>),编码 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord,mathnormal" style="margin-right:0.1076em;">f</span></span></span></span> 稀疏(<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∥</mi><mi>f</mi><msub><mi mathvariant="normal">∥</mi><mn>0</mn></msub><mo>≪</mo><mi>m</mi></mrow><annotation encoding="application/x-tex">\|f\|_0 \ll m</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord">∥</span><span className="mord,mathnormal" style="margin-right:0.1076em;">f</span><span className="mord"><span className="mord">∥</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">0</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≪</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">m</span></span></span></span>)。这正是经典<strong>字典学习</strong>(dictionary learning)的形式,而解混它就是把这个方程反过来解--顺手的工具是 <strong>sparse autoencoder(SAE)</strong>。对模型某一层(这篇是小型 SoLU transformer 的 MLP 层)的激活向量 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mo>∈</mo><msup><mi mathvariant="double-struck">R</mi><mi>d</mi></msup></mrow><annotation encoding="application/x-tex">x \in \mathbb{R}^{d}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.5782em;vertical-align:-0.0391em;"></span><span className="mord,mathnormal">x</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">∈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8491em;"></span><span className="mord"><span className="mord,mathbb">R</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8491em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mathnormal,mtight">d</span></span></span></span></span></span></span></span></span></span></span></span>, 训练一个 autoencoder:</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>encode: </mtext><mi>f</mi><mo>=</mo><mrow><mi mathvariant="normal">R</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">L</mi><mi mathvariant="normal">U</mi></mrow><mo stretchy="false">(</mo><msub><mi>W</mi><mtext>enc</mtext></msub><mtext> </mtext><mi>x</mi><mo>+</mo><msub><mi>b</mi><mtext>enc</mtext></msub><mo stretchy="false">)</mo><mo separator="true">,</mo><mspace width="2em"></mspace><mtext>decode: </mtext><mover accent="true"><mi>x</mi><mo>^</mo></mover><mo>=</mo><msub><mi>W</mi><mtext>dec</mtext></msub><mtext> </mtext><mi>f</mi><mo>+</mo><msub><mi>b</mi><mtext>dec</mtext></msub></mrow><annotation encoding="application/x-tex">\text{encode: } f = \mathrm{ReLU}(W_{\text{enc}}\, x + b_{\text{enc}}), \qquad \text{decode: } \hat{x} = W_{\text{dec}}\, f + b_{\text{dec}}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord,text"><span className="mord">encode: </span></span><span className="mord,mathnormal" style="margin-right:0.1076em;">f</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathrm">ReLU</span></span><span className="mopen">(</span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">enc</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal">x</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:1em;vertical-align:-0.25em;"></span><span className="mord"><span className="mord,mathnormal">b</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">enc</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mclose">)</span><span className="mpunct">,</span><span className="mspace" style="margin-right:2em;"></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,text"><span className="mord">decode: </span></span><span className="mord,accent"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6944em;"><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="mord,mathnormal">x</span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="accent-body" style="left:-0.2222em;"><span className="mord">^</span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span className="mord"><span className="mord,mathnormal" style="margin-right:0.1389em;">W</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">dec</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span className="mspace" style="margin-right:0.1667em;"></span><span className="mord,mathnormal" style="margin-right:0.1076em;">f</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span className="mord"><span className="mord,mathnormal">b</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,mtight">dec</span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span></span><p>目标函数是<strong>重建误差 + 稀疏惩罚</strong>:</p><span className="katex-display"><span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi mathvariant="script">L</mi><mo>=</mo><munder><munder><mrow><mi mathvariant="normal">∥</mi><mi>x</mi><mo>−</mo><mover accent="true"><mi>x</mi><mo>^</mo></mover><msubsup><mi mathvariant="normal">∥</mi><mn>2</mn><mn>2</mn></msubsup></mrow><mo stretchy="true">⏟</mo></munder><mtext>重建</mtext></munder><mo>+</mo><mi>λ</mi><munder><munder><mrow><mi mathvariant="normal">∥</mi><mi>f</mi><msub><mi mathvariant="normal">∥</mi><mn>1</mn></msub></mrow><mo stretchy="true">⏟</mo></munder><mtext>稀疏</mtext></munder></mrow><annotation encoding="application/x-tex">\mathcal{L} = \underbrace{\|x - \hat{x}\|_2^2}_{\text{重建}} + \lambda \underbrace{\|f\|_1}_{\text{稀疏}}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6833em;"></span><span className="mord,mathcal">L</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">=</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:2.4404em;vertical-align:-1.5763em;"></span><span className="minner,munder"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.8641em;"><span style="top:-1.4237em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,cjk_fallback,mtight">重建</span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="minner,munder"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.8641em;"><span className="svg-align" style="top:-2.102em;"><span className="pstrut" style="height:3em;"></span><span className="stretchy" style="height:0.548em;min-width:1.6em;"><span className="brace-left" style="height:0.548em;"><svg xmlns="http://www.w3.org/2000/svg" width="400em" height="0.548em" viewBox="0 0 400000 548" preserveAspectRatio="xMinYMin slice"><path d="M0 6l6-6h17c12.688 0 19.313.3 20 1 4 4 7.313 8.3 10 13
 35.313 51.3 80.813 93.8 136.5 127.5 55.688 33.7 117.188 55.8 184.5 66.5.688
 0 2 .3 4 1 18.688 2.7 76 4.3 172 5h399450v120H429l-6-1c-124.688-8-235-61.7
-331-161C60.687 138.7 32.312 99.3 7 54L0 41V6z"></path></svg></span><span className="brace-center" style="height:0.548em;"><svg xmlns="http://www.w3.org/2000/svg" width="400em" height="0.548em" viewBox="0 0 400000 548" preserveAspectRatio="xMidYMin slice"><path d="M199572 214
c100.7 8.3 195.3 44 280 108 55.3 42 101.7 93 139 153l9 14c2.7-4 5.7-8.7 9-14
 53.3-86.7 123.7-153 211-199 66.7-36 137.3-56.3 212-62h199568v120H200432c-178.3
 11.7-311.7 78.3-403 201-6 8-9.7 12-11 12-.7.7-6.7 1-18 1s-17.3-.3-18-1c-1.3 0
-5-4-11-12-44.7-59.3-101.3-106.3-170-141s-145.3-54.3-229-60H0V214z"></path></svg></span><span className="brace-right" style="height:0.548em;"><svg xmlns="http://www.w3.org/2000/svg" width="400em" height="0.548em" viewBox="0 0 400000 548" preserveAspectRatio="xMaxYMin slice"><path d="M399994 0l6 6v35l-6 11c-56 104-135.3 181.3-238 232-57.3
 28.7-117 45-179 50H-300V214h399897c43.3-7 81-15 113-26 100.7-33 179.7-91 237
-174 2.7-5 6-9 10-13 .7-1 7.3-1 20-1h17z"></path></svg></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">∥</span><span className="mord,mathnormal">x</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">−</span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mord,accent"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.6944em;"><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="mord,mathnormal">x</span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="accent-body" style="left:-0.2222em;"><span className="mord">^</span></span></span></span></span></span></span><span className="mord"><span className="mord">∥</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.8641em;"><span style="top:-2.453em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span><span style="top:-3.113em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">2</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.247em;"><span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.898em;"><span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.5763em;"><span></span></span></span></span></span><span className="mspace" style="margin-right:0.2222em;"></span><span className="mbin">+</span><span className="mspace" style="margin-right:0.2222em;"></span></span><span className="base"><span className="strut" style="height:2.3263em;vertical-align:-1.5763em;"></span><span className="mord,mathnormal">λ</span><span className="mspace" style="margin-right:0.1667em;"></span><span className="minner,munder"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.75em;"><span style="top:-1.4237em;"><span className="pstrut" style="height:3em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,text,mtight"><span className="mord,cjk_fallback,mtight">稀疏</span></span></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="minner,munder"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.75em;"><span className="svg-align" style="top:-2.102em;"><span className="pstrut" style="height:3em;"></span><span className="stretchy" style="height:0.548em;min-width:1.6em;"><span className="brace-left" style="height:0.548em;"><svg xmlns="http://www.w3.org/2000/svg" width="400em" height="0.548em" viewBox="0 0 400000 548" preserveAspectRatio="xMinYMin slice"><path d="M0 6l6-6h17c12.688 0 19.313.3 20 1 4 4 7.313 8.3 10 13
 35.313 51.3 80.813 93.8 136.5 127.5 55.688 33.7 117.188 55.8 184.5 66.5.688
 0 2 .3 4 1 18.688 2.7 76 4.3 172 5h399450v120H429l-6-1c-124.688-8-235-61.7
-331-161C60.687 138.7 32.312 99.3 7 54L0 41V6z"></path></svg></span><span className="brace-center" style="height:0.548em;"><svg xmlns="http://www.w3.org/2000/svg" width="400em" height="0.548em" viewBox="0 0 400000 548" preserveAspectRatio="xMidYMin slice"><path d="M199572 214
c100.7 8.3 195.3 44 280 108 55.3 42 101.7 93 139 153l9 14c2.7-4 5.7-8.7 9-14
 53.3-86.7 123.7-153 211-199 66.7-36 137.3-56.3 212-62h199568v120H200432c-178.3
 11.7-311.7 78.3-403 201-6 8-9.7 12-11 12-.7.7-6.7 1-18 1s-17.3-.3-18-1c-1.3 0
-5-4-11-12-44.7-59.3-101.3-106.3-170-141s-145.3-54.3-229-60H0V214z"></path></svg></span><span className="brace-right" style="height:0.548em;"><svg xmlns="http://www.w3.org/2000/svg" width="400em" height="0.548em" viewBox="0 0 400000 548" preserveAspectRatio="xMaxYMin slice"><path d="M399994 0l6 6v35l-6 11c-56 104-135.3 181.3-238 232-57.3
 28.7-117 45-179 50H-300V214h399897c43.3-7 81-15 113-26 100.7-33 179.7-91 237
-174 2.7-5 6-9 10-13 .7-1 7.3-1 20-1h17z"></path></svg></span></span></span><span style="top:-3em;"><span className="pstrut" style="height:3em;"></span><span className="mord"><span className="mord">∥</span><span className="mord,mathnormal" style="margin-right:0.1076em;">f</span><span className="mord"><span className="mord">∥</span><span className="msupsub"><span className="vlist-t,vlist-t2"><span className="vlist-r"><span className="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight">1</span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:0.898em;"><span></span></span></span></span></span></span></span><span className="vlist-s">​</span></span><span className="vlist-r"><span className="vlist" style="height:1.5763em;"><span></span></span></span></span></span></span></span></span></span><p>几个关键点:</p><ol><li><strong>字典比原维度大很多。</strong> MLP 宽度 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi><mo>≈</mo><mn>2048</mn></mrow><annotation encoding="application/x-tex">d \approx 2048</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span><span className="mspace" style="margin-right:0.2778em;"></span><span className="mrel">≈</span><span className="mspace" style="margin-right:0.2778em;"></span></span><span className="base"><span className="strut" style="height:0.6444em;"></span><span className="mord">2048</span></span></span></span>,但字典可以开到几十万甚至上百万个特征(扩张系数 expansion factor = 字典大小 / <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span>,这篇扫到量级几百倍)。这正对应&quot;概念数 &gt; 神经元数&quot;的叠加假设——故意给一个过完备(overcomplete)的字典。</li><li><strong>ReLU 强制非负 + L1 强制稀疏。</strong> 重建要求字典能还原 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.4306em;"></span><span className="mord,mathnormal">x</span></span></span></span>,L1 要求每个样本只激活很少几个特征。两个约束一起,逼着字典学出<strong>稀疏组合</strong>——每个激活向量被表示成&quot;少数几个特征方向&quot;的叠加。(<span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>λ</mi></mrow><annotation encoding="application/x-tex">\lambda</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">λ</span></span></span></span> 量级在 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>10</mn><mrow><mo>−</mo><mn>3</mn></mrow></msup></mrow><annotation encoding="application/x-tex">10^{-3}</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.8141em;"></span><span className="mord">1</span><span className="mord"><span className="mord">0</span><span className="msupsub"><span className="vlist-t"><span className="vlist-r"><span className="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span className="pstrut" style="height:2.7em;"></span><span className="sizing,reset-size6,size3,mtight"><span className="mord,mtight"><span className="mord,mtight">−</span><span className="mord,mtight">3</span></span></span></span></span></span></span></span></span></span></span></span>,具体值看模型规模。)</li><li><strong>L0 远小于 <span className="katex"><span className="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span className="katex-html" ariaHidden="true"><span className="base"><span className="strut" style="height:0.6944em;"></span><span className="mord,mathnormal">d</span></span></span></span>,但方差解释率(FVE)很高。</strong> 这是最漂亮的结果:每个 token 的 MLP 激活,只需要<strong>几十到几百个特征</strong>稀疏激活就能重建绝大部分方差。如果神经元真是计算单元,你不可能用这么少的方向重建得这么好——这反过来支持&quot;特征才是真实单元&quot;。</li></ol><h2 id="三sae-与语义的映射">三、SAE 与语义的映射</h2><p>光重建好不够,得证明每个特征<strong>确实对应一个可解释的概念</strong>。他们做了大量 case study,挑几个最能说明问题的:</p><table><thead><tr><th>特征</th><th>激活条件</th><th>为什么重要</th></tr></thead><tbody><tr><td><strong>DNA 特征</strong></td><td>输入是 <code>ATGC...</code> 这样的 DNA 序列</td><td>跨 token、跨上下文都稳定,甚至能拿来给文本&quot;是否含 DNA&quot;打标</td></tr><tr><td><strong>阿拉伯文特征</strong></td><td>文本里出现阿拉伯字符</td><td>与主题无关,纯粹认&quot;书写系统&quot;这个抽象</td></tr><tr><td><strong>专有名词 / 缩写特征</strong></td><td>首字母大写的人名、全大写缩写</td><td>语法-词性层面的抽象</td></tr><tr><td><strong>无用 token 特征</strong></td><td>模型词表里那些训练时几乎从不出现的&quot;垃圾&quot; token</td><td>关键对照:这种 token 上特征的激活模式异常,说明特征不是无脑触发</td></tr></tbody></table><p>最有说服力的是 <strong>无用 token(unuseful tokens)</strong> 实验:Anthropic 在词表里故意塞了一批训练数据里几乎不出现的 token。如果特征只是字典学习的数学产物,它对这些 token 应该也&quot;随便&quot;激活;但实际观察到的激活模式是有结构的、能和&quot;这个 token 从没被真正学过&quot;对上。这是对&quot;特征是真实单元、不是字典 artifact&quot;的一个反证辩护。</p><p>合起来,论文给出的画像:每一个 SAE 特征,在一个<strong>很窄但很一致</strong>的语义条件下激活——这就是 <strong>monosemantic(单语义)</strong>。和原始神经元的 polysemantic 形成鲜明对比。</p><h2 id="四sae-真的有用">四、SAE 真的有用</h2><p>这一节是我觉得最该细读的部分,也是最容易被科普文章跳过的。</p><p>最自然的反对意见是:<strong>你开一个过完备字典 + L1,当然能把任何信号拆成&quot;稀疏的零件&quot;,但这些零件是模型在用的东西,还是你强加的基?</strong>(basis dependence / 字典依赖)</p><p>他们的回应分两层:</p><ol><li><strong>特征分裂(feature splitting)是良性的。</strong> 字典开得越大,一个粗特征会&quot;分裂&quot;成几个更细的特征(比如&quot;阿拉伯文&quot; → &quot;阿拉伯文-宗教文本&quot;&quot;阿拉伯文-代码注释&quot;……)。这看起来像过拟合,但分裂后每个子特征依然单义、依然可解释——只是粒度更细。<strong>所以字典大小控制的是&quot;放大镜的倍率&quot;,不是凭空造概念。</strong></li><li><strong>行为层面的因果验证。</strong> 光说&quot;可解释&quot;是旁观证据,真正站住脚的是:能不能拿特征去<strong>干预</strong>模型行为。把某个特征在推理时强行激活或抑制(clamp / ablation),输出会按预期改变。<em>(注:大规模 steering 实验——包括那个有名的&quot;毒性特征&quot;案例——主要在 2024 年的续作 Scaling Monosemanticity 里,本篇规模较小,但因果验证的思路已立住。)</em></li></ol><p>我的看法:这两点合起来基本回应了 basis dependence,但没完全消解。<strong>特征是&quot;字典相关的&quot;这件事是事实</strong>,只是&quot;字典相关&quot;不等于&quot;任意&quot;——好的字典会让特征和模型的真实计算对齐。这有点像量子力学里的表象问题:换基会变,但物理不变;SAE 找的是那个让特征尽量&quot;自然&quot;的基。</p><h2 id="五延伸思考">五、延伸思考</h2><p>总的来说，SAEs 为 Transformer 的可解释性提供了可能性。我们朴素直觉(神经元=概念)被一个更准确结构(在神经元中被叠加某一方向=概念)取代。可解释性的基本单元从 neuron 让位给了 feature direction。</p><p>不过，SAE 只是&quot;放大镜&quot;。它给的是一个<strong>可调粒度</strong>的视图(字典大小=倍率)。模型的特征不是一个有确定数目的东西,而是依观察尺度展开的——这跟物理里&quot;尺度决定你看到什么&quot;是一回事。别把某一次 SAE 的特征数当成模型&quot;真正的概念数&quot;。SAE 只是把多语义的拆分成更细的特征，便于我们解释，以及在不修改参数的情况下进行操作。</p><hr /><blockquote><p><strong>模型把比神经元多得多的概念稀疏地压进激活空间,所以神经元是多义的;稀疏自编码器通过&quot;过完备字典 + L1 稀疏&quot;把这个叠加解混,得到的特征在很大概率上才是模型真正在用的、单义的计算单元——而&quot;是不是真在用&quot;,最终只能靠因果干预来回答。</strong></p></blockquote><style>html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}</style>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[简单聊聊对 AI Agent 行业的理解]]></title>
            <description><![CDATA[写这篇文章的初衷，是我发觉逐渐忘了前两个月的 AI/LLMs 是什么样的能力和情况了。意识中似乎某些在突然加速，但理性和直觉告诉我这不对，作为行内人，我知道这是一个渐进的、累积的过程。我想自己回忆一下，Agent的发展。]]></description>
            <link>https://blog.niina.fun/posts/my-thoughts-on-the-ai-industry</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/my-thoughts-on-the-ai-industry</guid>
            <pubDate>Sat, 20 Jun 2026 04:14:25 GMT</pubDate>
            <content:encoded><![CDATA[<p>我印象里 AI Agent 的概念在 2024 年的时候就逐步走向成熟了。凭我模糊的回忆说一下（不保证正确）。</p><p>一开始 22 年 ChatGPT 就以 ChatBot 的形式爆火。那个时候还称不上现在的 Agent，充其量就是浓缩了大量世界知识的 ChatBot。23 年 24 年开始有了&quot;Agent&quot;应用，一开始的那个忘了叫什么名字了（Logo 长得像个骆驼的），是 AI 写小说的。我是看见一个学长研究生毕业做的，这个我印象很深。从这里开始，我们不仅仅只是用一个集人类精华的存在去 Chat，而是通过一种组织的形式，发挥它的知识和能力，开始有意识根据模型本身的能力设计一个系统，让模型在这个系统中完成任务，这可以说是早期的 harness 了。</p><p>而 tool-calling 之类的能力，也是在这种系统性地使用模型能力的时候的自发实践。就像很早的 GPT-Academic，就是自己实现 function，配合 prompt 完成对 paper 的拆解。这份能力太过广泛和基础，下游的使用促使上游模型训练端让模型 tool-calling 的能力进一步加强，成为 ReasoningLLM 的标准能力。后面则是对这份 tool-calling 能力的扩展，从原本的工具调用会出错，到单次工具调用成功，再到多次工具调用——模型在工具使用上已经成了专家。</p><p>而能够使用工具，就是 Agentic 能力的一个关键点。在此之前的所谓 Agent，不过是 prompt APPs 罢了。让我来梳理之前 LLM 发展的趋势。</p><h2 id="_2024augmented-llmtool-userag">2024：Augmented LLM、tool use、RAG</h2><p>在2022年的时候，ChatGPT5天达到了1M的DAU。人们意识到这是一个伟大产品的诞生。在学术上，人们达成了共识，未来LLM发展，除了模型本身的能力，很大程度上是围绕模型增强模型的能力。2023-2024年出现了部分如今的agent harness。有人开发一个软件系统，让人能够和AI持续交互，同时</p><p>我在 24 年暑假关注到了 AstrBot 这个项目，那个时候 LLMs 的能力刚好够到能进行工具调用的程度，但鲜少有 QQBot 的项目把 tool-call LLM 作为一等公民接入 QQ，AstrBot 做了。比较早期的就跟进了 MCP，沙箱执行，工具调用。我当时顺手给 AstrBot 提了个 typo fix，修了下启动时的 logo，哈哈。但我后续在使用 AstrBot 的时候感到了厌倦——我没有感受到它带给了我什么。让 Bot 更像人？和人交互？我根本用不上。</p><p>2024 年有些 PaaS Agent APP 平台，比如 Dify、Coze 等等。RAG 的概念也是作为增强模型能力的一个方法为人所熟知，一个是可以解决 AI 幻觉问题，一个是解决 AI 上下文不够长的问题。</p><note><p>2026 年 10 月 29 日，Coze 发布《企业版下线公告》：企业标准版、企业旗舰版11月30日起停止新售，旧版套餐2026年12月31日下线，新版最晚2027年12月31日全面下线。</p></note><p>而这个时候，token 的消耗量并没有那么大, dsr1 context length 128k，单次对话就调用一个工具。但现在不一样，2026年Q2以后，市面上的 AI 都能在一个 turn 中进行多个乃至上百个 step，而每一次 step 就会通过无状态的 LLM API 发送一次请求，这样的 Token 使用量是 O(step^2) 级别增长的。2026年一天的 token 使用量，比过去几年的使用量加起来还要多。</p><note><p>当需要多 step 的 agent 场景的时候，缓存命中是极其关键的。Serving LLM 端的缓存命中，可以在同样的负载下 serving 更多的 token, 同时不会影响 serving 整体的性能。Serving LLM 的时候缓存命中是极其关键的，意味着更低的 TTFT。对于客户端（Harness）来说，缓存命中意味着更低的价格。如今在多个 turn 的情况下，Anthropic 格式的 API 通常能够达到 95% 以上的 cache hit, OpenAI Response 格式的 API 通常能够达到 99%。一般 coding agent 场景很容易达到很大的缓存命中率。</p></note><p>另外，IDE 也在因为 AI 的能力增长而发生改变。21年以前，是通过 VS Code 插件的形式，提供 Tab 补全/预测 等功能。而2023/2024年左右，Cursor/Windsurf这种面向 AI 的 IDE 就已经出现。并且在2026年 coding agent 爆发的时候，也受到影响，迭代了至少三个大版本的 IDE，甚至连 VS Code也受到了影响，现在发生了很大的变化。具体可以看 捕蛇者说 的某一期。</p><h2 id="_2025agentic-ai">2025：Agentic AI</h2><p>Open AI o1发布不久后，DeepSeek R1发布，把 LLM 推到了 Reasoner 的程度，LLM 极度火热，再一次进入了大众的视野。</p><p>在 2025 年 Q2，Agentic AI 的概念已经被提出来了（与此同时是大量的 agentic 框架，这是 25Q2 有人列举的<a href="https://docs.google.com/spreadsheets/d/1B37VxTBuGLeTSPVWtz7UMsCdtXrqV5hCjWkbHN8tfAo/edit?ref=blog.langchain.dev&amp;pli=1&amp;gid=0#gid=0" rel="nofollow">不同 agent 框架之间的对比</a>），并且就是我们现在常谈论的 agent，而非之前的改改 system prompt 的 &#39;agent&#39;。那个时候还有很多 agent 框架，不管是非常早就出来的 LangChain、还是算比较新的 autogen、LangGraph 这些都是很方便去构建 workflow 式的 agent 的平台，开发 agent 的门槛变得很低。但这个时候业界反应还没有那么大（2025Q2）——相应的就是机会非常多，因为绝大多数人都没有做过 agent。而今一年过去了，体感上各种培训班已经满天飞了。</p><p>另外一个，token 消耗量在模型长上下文普及和长程任务能力越来越强的现在激增。我一天开发一个项目最高能烧掉 1B 的 token，这换在 ChatBot 时代是无法想象的。而海量的 token 就代表着海量的数据，而且还是之前现实世界没有的，agentic 的 long horizon 任务执行的轨迹数据，这是新一轮的数据飞轮。随着 Transformer 架构的不断升级，现在 dsv4 解决了 KV Cache 算力、显存消耗太大的问题（之前只是解决 Attn 上的算力和显存问题，而长上下文中 KV Cache 带来的问题没有解决），能够在 128k 的上下文做预训练，这是以前不敢想的。人类产生数据的速度也在激增。这有点像烧着能源，让大模型有了左脚踩右脚的能力，当然中间少不了人类做 pilot 和 steer，以保证模型和人类 align。</p><p>2025 年 vibe coding 的概念就已经深入人心。25 五一的时候，有同学问我课程项目（前端后端数据库）能不能全用 vibe coding，我说试试，应该能。他最后全 vibe coding 成功了。那个时候就有所谓&#39;前端已死&#39;的论调，并在 Gemini 3（2025Q4）的时候极为强烈。</p><p>之后出现的 AI 产品五花八门。AI 陪伴类的一直都有（酒馆、MiniMax 的星野 etc.），这种情感类的反馈周期长，算法的 reward 太多太广太复杂，很难有好的确切的标准做 SFT，并且需要长周期的反馈，很难直接 eval 效果。还有 AI Researcher，印象里这个 OpenAI 先做的，然后 Kimi 似乎是跟进的不错的那个。Kimi Research，记得当时他们的模型做到了一次对话能够调用上百个工具，出了个产品 OK Computer，来做单次请求的长程解决，这个时候（2025Q3）似乎各家就能做到单次 30 分钟以上的持续执行了。2025Q4 的 skills 概念的提出让 Agentic AI 的易用性进一步降低，（skills 这个概念真神了）skills，可以说是 LLM Infra 中的 FlashAttention。将修改模型上下文的权利下放（而这是 agentic 系统中至关重要的），通过 skills 可以直接给模型传输经验、方法、流程、领域知识、思路，等等等等。直接让通用模型的能力轻松落地。而且通过三步渐进暴露的方式做了漂亮的 context engineering，是 Agent 在 C 端变得高可用的一个至关重要的概念。</p><p>写到这里，我发现模型变动的速度太快了。两个月甚至一个月模型厂商就能把新一代的 post-training 的模型端上来。模型能力的提升从上游往下游不断传播，影响巨大。</p><p>一个类似玩笑的说法是，这个月你用 AI Coding 感到棘手的项目，你大可放置一段时间，等新一代模型发布的时候，过去的难题可能就迎刃而解了。这反应在行业里面很残酷——就像做垂直领域的一些 AI Agent 产品，现在正在被通用的 Codex 吃掉市场份额一样。</p><p>其实也告诉了我们时间很重要。这个月是这样，下个月情况可能就会反转，像 Fable 5 这样的事情，之前已经发生过，未来一定还会发生。但是这个具体的变动谁也不清楚，不身处在水中，怎么能摸到第一线的变化？</p><h2 id="破圈">破圈</h2><p>2026Q1 Opus 4.6 发布之后，我第一次感觉到 Agentic AI 真正意义上的破圈了，好像 AI 真的突破了某种能力的上限，而那种上限格外诱人，各个行业和领域的人都开始为之疯狂。当然这种 FOMO 并非 2026 才发生的，更早就有了。</p><p>另一个破圈的是 OpenClaw（ClawBot），这玩意我记得在 Dev 中有过一次传播，但似乎大家没有这么在意。隔了一两个月在 Q1 和 Q2 之间的时候，OpenClaw 突然破圈了，引起了一波 Claw 潮。我其实到现在也没有想明白是为什么？难道是真的发现了 AI Agent 的能力已经足够在生产生活中起到作用了吗？但是资本似乎从此刻就开始狂欢。智谱、MiniMax 接连港股上市，而今估值快万亿。</p><h2 id="agent-在做什么">Agent 在做什么</h2><p>上面啰嗦了这么多，其实真正想聊的是这个：抛开产品和概念，Agent 到底是什么，以及它现在到底能做什么。</p><p>只说 agent 类型的话，可以分作 workflow 和 agent 两种，以及两者之间的东西。但我觉得这个二分法本身没那么重要——更重要的是想清楚一个问题：<strong>你的业务里，哪些环节需要 agentic，需要到什么程度？</strong></p><important><p>随着模型能力越来越强，plan、todo、workflow 这种规划能力已经被模型内化，模型会在自己的 CoT 中动态的规划，确认自己进行到了哪一步，而不一定需要显式的 workflow 框架，比如 Dify、Coze 等等 Agent 编排框架进行编排。更多是通过某种方式提供所需要的上下文。比如算子相关的 Kernel Agent 中，除了算子优化能力的补足（提供优化方式/技巧/优化方法论），还需要记录做了哪些优化，哪些尝试，当前在什么阶段。</p></important><p>现在的 agent harness 提供了基本的限制。更多的限制，需要根据自己的业务开展。</p><p><strong>业务流程清晰、规律、稳定、门槛不高</strong>。这些场景用 workflow 就够了。workflow 的优势不是能力，是可控。每一步的输入输出是确定的，出问题你能定位到具体环节。而一个完全自主的 agent，它做对了你不知道为什么对，做错了你也不知道在哪一步开始偏的。对于大多数企业场景，可控性比智能程度更值钱。当然如果胡乱让一个 Agent 做十次都没有出错，在做好了基本的错误处理后，其实可以放弃 workflow。</p><p><strong>垂直领域</strong>不同。通用模型 + skills 在垂直领域的效果往往不好，而且可预计的未来通用模型在这些领域的数据都比较稀缺。这时候要么训练/微调，要么结合领域知识的 RAG。如果做不到微调模型，就只能用 workflow 去逼近效果。这不是最好的方案，但是是可行的方案。但是随着基础模型的能力越来越强，能够做的工作不是提高模型能力，而是在具体的场景中提供上下文，对接数据，工具。</p><p><strong>共识：上下文是核心</strong>。不管是 workflow 还是 agent，让模型接收到正确的、充足的、合适的上下文，决定了整个系统的上限。上下文给少了模型瞎猜，给多了模型被噪声淹没，给错了方向整个就偏了。这件事说起来简单，实际上极其依赖对业务的理解——你得知道什么信息对决策是关键的，什么信息是噪音。这不是技术问题，是领域知识问题。</p><p>另一个同样关键但更难的是<strong>评估</strong>。怎么知道你的 agent 系统在变好还是变坏？模型一更新，prompt 一改，工具一换，行为就可能变了。如果没有一套评估手段，就是在碰运气。但评估 agent 比评估模型难得多，因为 agent 的输出不是单点的，是多步的、有状态的、和环境交互的。单次工具调用对不对？连续调用的顺序对不对？最终结果对不对？三个维度的评估，目前行业里也没有很好的统一方案。</p><note><p>说起 eval, 基模厂发布模型的时候都会附上 benchmark 来说明自己模型的能力。2026 年，很多 bench（比如tb）都通过容器（sandbox）的方式给 AI 提供完整的环境去做评估。那些 bench 具体的 tasks 都有公布，看看测过的 tasks，初步确认模型能力。</p></note><p>这也是为什么我说 OpenClaw 的破圈让我困惑。如果 agent 系统的基础设施——上下文工程、评估体系——都还处在这种&quot;大家各做各的&quot;阶段，那它真的准备好进入生产生活了吗？还是说我们又一次高估了短期，而低估了长期？</p><p>后记：写这篇文章的初衷，是我发觉逐渐忘了前两个月的 AI/LLMs 是什么样的能力和情况了。行业的节奏快到你来不及消化今天的东西，明天又变了。我想把现在的理解写下来，是为了过几个月回头再看的时候，能看到自己当时站在哪里，看到了什么，又看漏了什么。人的记忆真的忘的太快了。</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RoPE]]></title>
            <link>https://blog.niina.fun/posts/rotational-positional-encoding</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/rotational-positional-encoding</guid>
            <pubDate>Wed, 17 Jun 2026 09:31:40 GMT</pubDate>
            <content:encoded><![CDATA[<p>Attn 本身不表示位置，不包含时序信息。在进入 Attn Block 之前，需要将文本的位置信息编码进 Embedding vector 里。</p><p>早期如 BERT 使用的是绝对位置编码，用一个 Embedding 去表示位置，无法外推，并且还需要额外存储位置信息。相对位置编码，用两个 token 之间的相对距离去编码解决了外推的问题，但实现很麻烦。</p><p>我们知道 Attn 本身就是抽象出 Query、Key、Value，计算 QK 相关性找 V 这个过程，这里用了矩阵乘法。对于矩阵我们有很多 trick。同时旋转 Q、K 两个矩阵，他们矩阵点乘的结果就能实现相对位置编码，而旋转过程中频率的衰减自然展现出了绝对位置的信息。</p><p>RoPE 利用了矩阵的数学特性，漂亮地做好了位置编码这件事。只需要做旋转就可以。空间复杂度 O(1)，算法复杂度都降了很多。把大的 QK 矩阵分块改成一对一对的到二维空间做旋转，这些维度对从高到低不同频率衰减包含在其中。高频负责近距离，中频平衡精度和范围，低频确定远距离位置。外推能力强，没有引入额外的复杂度，非常美丽。</p><p>如果位置信息能从 off 到其他模块，RoPE 这块还能减负。比如使用 Linear Attn，如 Gated DeltaNet，自带很强的位置理解。Attn 就可以把长程的位置理解 off 掉，用 Partial RoPE 保留高频部分，只理解近邻的位置信息即可。</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[大语言模型入门基础]]></title>
            <link>https://blog.niina.fun/posts/introduction-to-large-language-model</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/introduction-to-large-language-model</guid>
            <pubDate>Tue, 16 Jun 2026 13:36:11 GMT</pubDate>
            <content:encoded><![CDATA[<p>本文介绍大语言模型(LLM)的基础内容。</p><h2 id="insight">Insight</h2><p>对于推理时的 LLMs 来说，它只做一件事：预测下一个 token。这听起来是一件非常简单的一件事。为什么 LLMs 却在文本模态上表现的如此全能呢？</p><p>力大砖飞？道法自然？或许都对。LLMs 在海量数据上做了 pretrain。截止到 2026 年 4 月份，国内开源的几家模型公司在预训练阶段训练的 token 数量大约都在 30T-35TB 之间。模型每个参数预训练消耗的 token 都远大于 20 了。并且我相信这个数据使用量会在 DeepSeek V4 分享他们新的稀疏方法后，继续增长。另外，qwen 小模型的实践告诉了我们：预训练时，随着数据量的增多，即使是小模型，其能力也在稳步提升。</p><h2 id="数据飞轮">数据飞轮</h2><p>在模型的长上下文突破(RoPE、YaRN), 基座模型 Agentic 能力提升， LLM Infra(vLLM、SGLang etc.) Day0 级跟进支持等等突破之后，一批全新的、过去没有的长上下文 Agent 任务的数据在飞速产生。这个时候谁拥有更多数据（并且能聪明地运用），那么他们下一代的模型就会更强，这是个正反馈的过程，也被称为数据飞轮。</p><p>数据飞轮正在拉大不同模型厂商之间的差距。</p><h2 id="看到新的-token-产生的时候发生了什么">看到新的 Token 产生的时候，发生了什么？</h2><p>使用 LLM 时，输入的是人类的文本，输出的是人类的文本，这个过程中模型到底做了什么呢？</p><p>在看到文本的前一步，token 是经过采样后得到的 next token ID，根据 ID 一做 detokenizer 就得到看见的文本了。</p><p>得到 token id 的前一步，是经过 Softmax 计算出的一个概率矩阵，根据不同的解码策略（贪婪解码：取概率最大的； 投机解码：需要 MTP 模块； 或者按概率采样等等），选择 next token id。再上一步就是 logits。我们平常说的 Temperature（对 logits 矩阵做除法）、Top-K（只在概率最大的 K 个 Token 中选）、Top-P（只在累积概率 P 以内的 Token 中选）都是在这步做的。</p><p>logits 输出前是 LM Head（Linear 层），它将 hidden Linear 的特征数扩大到词表的大小。在此之前是文本经过 Embedding 向量化和 PE（Positional Encode）后进入若干个 Transformer Block 后又过了一遍 RMSNorm。</p><p>Transformer Block 很简单。就是 Attn 机制加上前馈网络（FFN）罢了。</p><pre className="language-python shiki shiki-themes github-light github-dark" code="# Input: x
# Transformer Block
def forward(self, x):
    x = x + Attn(RMSNorm(x))  # 简单的残差
    x = x + FFN(RMSNorm(x))
    return x
" language="python" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="sJ8bj"># Input: x
</span></span><span class="line" line="2"><span class="sJ8bj"># Transformer Block
</span></span><span class="line" line="3"><span class="szBVR">def</span><span class="sScJk"> forward</span><span class="sVt8B">(self, x):
</span></span><span class="line" line="4"><span class="sVt8B">    x </span><span class="szBVR">=</span><span class="sVt8B"> x </span><span class="szBVR">+</span><span class="sVt8B"> Attn(RMSNorm(x))  </span><span class="sJ8bj"># 简单的残差
</span></span><span class="line" line="5"><span class="sVt8B">    x </span><span class="szBVR">=</span><span class="sVt8B"> x </span><span class="szBVR">+</span><span class="sVt8B"> FFN(RMSNorm(x))
</span></span><span class="line" line="6"><span class="szBVR">    return</span><span class="sVt8B"> x
</span></span></code></pre><p>Attn 机制的话，现在有 DSA、MLA 等等。不过未来的趋势还是掌管稀疏的神——DeepSeek 的新 Attention 的时代，这成本降的太夸张了。</p><p>干了什么？混合注意力（CSA：压缩稀疏注意力， HCA：高度压缩注意力）。不仅仅是上一代注意力的稀疏性，还直接对 Attn 进行压缩。表明了一个潜在的观点，对于长上下文（百万级），即使利用其稀疏性能够将 Attn 的算法复杂度降到 O(L)级。但是面对长上下文存储的 KV Cache 还是无能为力。V4 直接对 Attn 进行压缩，减少 KV Cache 数量。（Tips: KV Cache 可以近似看成 Context 的物理实体）</p><pre className="language-python shiki shiki-themes github-light github-dark" code="def forward(self, x):
    # 计算多头注意力 -&gt; xq, xk, xv
    xq, xk = RMSNorm(xq), RMSNorm(xk) # qk-norm
    
    xq, xk = apply_RoPE(xq, xk, cos, sin) # 加入相对位置编码
    #... 存 KV Cache。  
    # KV Cache 一般存显存。但是现在有存在SSD上的，比如DeepSeek，使得缓存命中成本惊人的低，是未来降低推理成本的一个方向

    output = F.scaled_dot_product_attention(xq, xk, xk, is_casual=True) # FlashAttention
    output = output.transpose(1, 2).reshape(bsz, seq_lean, -1)

    return output, past_kv
" language="python" meta="" style=""><code __ignoreMap=""><span class="line" line="1"><span class="szBVR">def</span><span class="sScJk"> forward</span><span class="sVt8B">(self, x):
</span></span><span class="line" line="2"><span class="sJ8bj">    # 计算多头注意力 -&gt; xq, xk, xv
</span></span><span class="line" line="3"><span class="sVt8B">    xq, xk </span><span class="szBVR">=</span><span class="sVt8B"> RMSNorm(xq), RMSNorm(xk) </span><span class="sJ8bj"># qk-norm
</span></span><span class="line" line="4"><span class="sVt8B">    
</span></span><span class="line" line="5"><span class="sVt8B">    xq, xk </span><span class="szBVR">=</span><span class="sVt8B"> apply_RoPE(xq, xk, cos, sin) </span><span class="sJ8bj"># 加入相对位置编码
</span></span><span class="line" line="6"><span class="sJ8bj">    #... 存 KV Cache。  
</span></span><span class="line" line="7"><span class="sJ8bj">    # KV Cache 一般存显存。但是现在有存在SSD上的，比如DeepSeek，使得缓存命中成本惊人的低，是未来降低推理成本的一个方向
</span></span><span class="line" line="8"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="9"><span class="sVt8B">    output </span><span class="szBVR">=</span><span class="sVt8B"> F.scaled_dot_product_attention(xq, xk, xk, </span><span class="s4XuR">is_casual</span><span class="szBVR">=</span><span class="sj4cs">True</span><span class="sVt8B">) </span><span class="sJ8bj"># FlashAttention
</span></span><span class="line" line="10"><span class="sVt8B">    output </span><span class="szBVR">=</span><span class="sVt8B"> output.transpose(</span><span class="sj4cs">1</span><span class="sVt8B">, </span><span class="sj4cs">2</span><span class="sVt8B">).reshape(bsz, seq_lean, </span><span class="szBVR">-</span><span class="sj4cs">1</span><span class="sVt8B">)
</span></span><span class="line" line="11"><span emptyLinePlaceholder="true">
</span></span><span class="line" line="12"><span class="szBVR">    return</span><span class="sVt8B"> output, past_kv
</span></span></code></pre><h2 id="optimizer">Optimizer</h2><p>除了 Embeding 模型和 LM Head Linear 层以及其余 Linear 层、Bias 用 AdamW 优化器，其余矩阵用 Muon(dim&gt;=2)。Muon 能够保证矩阵正交性，可以加速收敛，Loss 下降更稳定。但是 Muon 较新，一些后端实现的性能不好（比如 PyTorch 2.12 的 mps 后端， 替换 AdamW 之后训练速度变慢了）。</p><p>不过使用 Muon 是趋势，相比 AdamW 能够减少一半 optim 的参数使用，训练也更稳定。</p><h2 id="mtp-多-token-预测">MTP 多 Token 预测</h2><p>一次推理输出多个 Token。</p><p>如何实现？加上一个 MTP 模块（可以当只有一块注意力块的 LLM）：输入是模型预测出的 t+1，经过类似的步骤但是 Transformer Block 只有一块（减少计算量以及显存开销），得到 t+2。嵌套调用后可以一次推理得到多个 token。和主模型的 logits 一对比后，结果一样的就保留，不一样的就重新预测。因为序列相近，语义大概率类似，因此成功率很高，一般可以实现 2-3 倍的加速比。</p><style>html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}</style>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[如何开始一个开源项目？]]></title>
            <description><![CDATA[浅谈我对开源项目的认知。]]></description>
            <link>https://blog.niina.fun/posts/opensourcesoftwarestarttutorial</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/opensourcesoftwarestarttutorial</guid>
            <pubDate>Thu, 17 Apr 2025 09:00:00 GMT</pubDate>
            <content:encoded><![CDATA[<p>接触开源零零散散也快有三年了，虽然听上去时间很长，但是我实际参与的开源项目寥寥无几，只是描述我的见闻而已。</p><p>最基础的就是些开源相关的常识，比如开源协议，开源使用的工具等等，如果有意学习我相信大多数人很快就能掌握。我也就大致列了一些在下面。当然，最快速的方法是寻找一些新兴的正在开发的开源项目，观察其规范、行为乃至参与其中，这样的学习效率是最高的。</p><h2 id="开源协议">开源协议</h2><h3 id="常见的开源协议">常见的开源协议</h3><ul><li><strong>MIT License</strong>：这是一个非常宽松的协议，允许任何人几乎无限制地使用、复制、修改和分发代码。</li><li><strong>Apache License 2.0</strong>：与 MIT 类似，但它增加了专利授权条款，并要求保留原始版权声明。</li><li><strong>GPL（GNU General Public License）</strong>：强制衍生作品必须以相同的许可证发布，确保开源精神得以延续。</li><li><strong>LGPL（GNU Lesser General Public License）</strong>：适用于库，它允许专有软件链接到该库而不需要开放其源码。</li><li><strong>BSD Licenses</strong>：提供了两种版本，新 BSD 协议包含了禁止使用贡献者名字进行推广的条款。</li><li><strong>AGPLv3（GNU Affero General Public License version 3）</strong>:扩展了 GPLv3（GNU General Public License version 3）的条款，特别针对通过网络提供服务的应用程序。</li></ul><h2 id="开源的平台">开源的平台</h2><h3 id="常用的开源平台">常用的开源平台</h3><ul><li><strong>GitHub</strong>：最流行的代码托管平台之一，支持 Git 版本控制。</li><li><strong>GitLab</strong>：提供更多的 CI/CD 集成选项，并且可以自托管。</li></ul><h2 id="开源使用的工具">开源使用的工具</h2><h3 id="版本控制">版本控制</h3><p>使用 Git 进行版本控制, 跟踪代码更改历史, 多人合作.</p><h3 id="git-hook">Git Hook</h3><p>Hook 脚本可以在特定时间发生时自动运行, 顾名思义 Hook(钩子)就是在某事发生必然发生的事, 用途比如提交前检查代码风格或者提交后部署代码.</p><h3 id="git-ignore">Git Ignore</h3><p>使用 <code>.gitignore</code> 文件列出不应被 Git 追踪的文件, 例如编译产物或 IDE 配置文件.</p><h3 id="issue-模板">Issue 模板</h3><p>创建 issue 模版可以帮助开源社区贡献者清晰描述问题, 包括复现步骤和期望行为. 规范的 issue 不仅可以帮助提问者快速的解决问题, 节省维护者了解问题的精力, 还节省了后续查询相关 issues 的用户的时间. 遇到过不少不规范 issue 的项目, 去查 issue 查个半天会发现一些提问者描述的既不清晰, 而且在没有给出解决方案(不管可行不可行)的情况下就把 issue 关闭了.</p><h3 id="pull-request-模板">Pull Request 模板</h3><p>PR 模板有助于确保所有必要的信息都被包含在内，如变更描述、测试结果等。可以方便审查代码.</p><h3 id="cicd">CI/CD</h3><p>持续集成和持续部署工具，如 GitHub Actions、GitLab CI、Jenkins 等，可以自动化测试和部署流程。比如自动编译 Release, 自动部署, 自动打包 Docker 镜像等等.</p><h2 id="宣传">宣传</h2><blockquote><p>东亚文化是一种集体主义，社会鼓吹的是踏实、谦虚、低调，不要突出自己。所以很多人喜欢在集体中默默无闻的奉献，这是最经典的一种驯化。长期演变下来，你会发现，身边很多人是不擅长或者说不屑于展示自我的，因为我们的文化里常常将&quot;Selling yourself&quot;等同于一种负面的道德品格。
当你做完一件事情其实还不够，你得&quot;Show your work&quot;，把它&quot;贩卖&quot;出去。</p></blockquote><p>我亲眼目睹了一个开源项目在宣传前后 star 数的井喷，在那个项目的开发者还没有下场宣传的时候，它不到一千的 star，但是在宣传之后，那个项目的 star 在短短两天之内就到了五千 star。&quot;酒香也怕巷子深&quot;，因此好的宣传也是做好一个开源项目必要的。</p><h2 id="开源软件如何进行宣传">开源软件如何进行宣传？</h2><ul><li><strong>社交媒体</strong>：国内利用 Bilibili、知乎、小红书, 国外利用 Twitter、LinkedIn 等社交网络分享项目进展。</li><li><strong>技术博客</strong>：撰写关于项目的博客文章，解释其用途和技术细节。</li><li><strong>SEO 优化</strong>：确保项目页面针对搜索引擎进行了优化，以便更容易被发现。</li><li><strong>多交朋友</strong>：与其他相关的开源项目或公司建立合作关系扩大影响力。</li></ul><p>当然开源项目是经过众人不断用实践检验的。清晰的文档，用户友好的指引，会有助于新人的到来。规范的代码实践和沟通机制，会使得贡献者和维护者更有效的发展项目（对于正在不断开发的项目）。</p><p>按照国内的环境，对于个人来说，开源是一项要求极高，且和做慈善没有区别的事。这种情况估计还会持续下去。</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[生活的崩坏与自我毁灭]]></title>
            <description><![CDATA[是什么在牵引着我？我的行为，我的思想？我的感受？我所处的地方？它们在何处？]]></description>
            <link>https://blog.niina.fun/posts/self-destruction</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/self-destruction</guid>
            <pubDate>Thu, 19 Dec 2024 03:00:00 GMT</pubDate>
            <content:encoded><![CDATA[<p>今天是单周，我按计划翘掉了不签到的早八，但是睡过了签到的早十。一觉起来翻看手机，发现室友在签到的时候问我，“你上课又没来吗”，我回答是。</p><p>如果只是归咎原因，那么可以归于我的毫不在意和单双周的难记忆。但我，在感到有必要要去做的事情的时候，是会早早醒来的。我应当是感到了那样的生活是在消耗我，起码是消耗现在的我。</p><p><strong>我对我将临的事情，并无期待，反而是反抗</strong>。而反抗，这种消极的反抗，造成的是：我生活的崩塌从 regular 到 irregular，从主流的道路到谁也不确定的道路。我生发了一种对生活、乃至“世界”的背叛感。我很难不去追问，到底是怎么了，为什么我会这样做呢。</p><p>当我询问我自己这样的问题时，我内心其实已经有了答案和想法。当一个人生发出一种行为的时候，尚若他追问自己的言行，是一定能够归咎出原因的，即使脑海中假想推演出来的原因和对过去的模拟非常有可能是错误且偏离现实的。</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[重启博客v2]]></title>
            <description><![CDATA[忙里偷闲学习前端，还是决定结合想法手搓一个自己的博客。]]></description>
            <link>https://blog.niina.fun/posts/blogrelifever2</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/blogrelifever2</guid>
            <pubDate>Sun, 24 Nov 2024 15:00:00 GMT</pubDate>
            <content:encoded><![CDATA[<p>很遗憾的一件事情——我发现以前的我并没有写出什么值得发表出来的博客，更遗憾的事情，现在的我也没有写出来这样的文字。</p><p>大抵是从初中开始，我就对文字保有偌大的趣味。我认为那是一件值得品味，需要经验积累才能表征出的东西。我想写自己的小说，传达自己的思想。但自我判断后，认为仍需更多的现实经验积累（但其实更重要的是相应的锻炼，比如说写作训练），便将想法设置为遥远的理想，始终铭记，但将其搁置在行动清单外。</p><p>大学刚入学后，学着网上胡乱的言论（还有身边的例子），我也开始写博客。那时候的我，对于博客的写作，其实只是大多停留在复制粘贴的搬运。毕竟经验与感想是需要累计和思考的，没有一定时间的浸润是很难以自己的想法表达出来其他人的想法的。而自我日常的思绪又过于混乱，不花时间做整理，到最后连自己都看不懂。这样记录实在是无趣。</p><p>对于自我的寻找是更关键的。尽管我知道了自我的位置，但是我还是没有以——有足够归属感的、卸下自己防备的方式去接受自我的位置。这很大程度上动摇了我未来决策的基础。</p><p>这实在遗憾。</p><p>好了，谈了不少呓语。下面提一下这次博客的搭建。当然是放弃了 <code>Hexo</code>，感到厌倦，还是不满意。很多时候就是一个想法出现，决定做不做。过后就会很快的把它丢出“缓存”。</p><p>理想的博客要求是简单、简洁、书写方便。如此，需要 <code>Markdown</code> 文件支持，也需要一些基本的功能，比如说 <code>RSS</code> 订阅、评论、分类、标签、日夜模式切换等等。应该会在后续上线。</p><p>技术栈：Nuxt3(+@nuxt/content) + TailwindCSS</p><p>非常简单的技术栈。摸鱼的时候玩的很爽。www</p><p>就是这样，博客还存在一些问题，但边用边解决吧。</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[生存手册——欢迎来到大学]]></title>
            <description><![CDATA[大学同时生产三样东西：可验证技能、认证信号和驯化资本。你付了四年的钱，真正拿到手的是哪一样，取决于你毕业时要进的那个市场在为什么付费。]]></description>
            <link>https://blog.niina.fun/posts/survival-welcome-to-university</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/survival-welcome-to-university</guid>
            <pubDate>Sat, 12 Oct 2024 11:00:00 GMT</pubDate>
            <content:encoded><![CDATA[<p>2008 年，几个上海交大本科生写了一本 62 页的生存手册，开篇第一句是：&quot;国内绝大部分大学的本科教学，不是濒临崩溃，而是早已崩溃。&quot;</p><p>它真正耐用的判断在更靠后的地方：&quot;在你毕业之时，那一串苍白的分数其实就已经作废了。&quot;这句判断在 2008 年就已经写出来了，比大多数关于大学的经济学讨论更早问到了同一件事：大学在教什么，大学在筛什么。它离一个正在读书的人最近的时刻，是排保研名次、投简历、被用人单位挑的时候。那本手册写给 2008 年刚入学的人，这句话今天照样适用。</p><p>这两个问句指向的东西不同。教什么，看的是四年之后你身上留下了什么；筛什么，看的是雇主凭什么在一群人里把你挑出来。这两件事在理想情况下是同一件事，现实里常常不是。</p><h2 id="不拿学位但获得教育你选哪个">不拿学位但获得教育，你选哪个</h2><p>先看一个说不通的选择：一边是不拿学位但获得教育，一边是得到学位却无法接受教育。多数人会选择后者。</p><p>如果上学的主要目的是学习，这个选择说不通。学位只是对所学内容的一个粗略估量，但学生为了拿到它，会策略性地选&quot;轻松得 A&quot;的课，雇主则几乎不关心这些课具体是什么。一边是学生按给分松紧挑课，一边是雇主不看课程内容，两边对着同一个东西出手，而那个东西不是课上教的内容。</p><p>再看一组数据。全球范围内，个人多受一年教育，未来收入预计上涨 8% 到 12%；而国家层面，公民平均多受一年教育，国民收入仅增长 1% 到 3%。</p><p>如果教育主要在提升生产力，个体多读一年，宏观上就该一起涨。事实没有。它能说明的只有一件事：个人收益里有相当一部分来自他在人群里的位置，而不是他生产出来的东西。</p><note><p>这两个数来自不同口径：8% 到 12% 是按个体工资算出来的私人教育回报率，1% 到 3% 是把国民平均受教育年限放进跨国增长回归得到的系数。搭在一起不能算出教育对国家的回报率。宏观这一侧的估计会被教育数据的测量误差压低，Krueger 和 Lindahl 在 2001 年综述教育与增长的文献时，结论反而是宏观系数至少不弱于微观回报率。把这两组数字并置起来的论证方式，更接近 Bryan Caplan 在《The Case Against Education》里的用法。</p></note><p>Hungerford 和 Solon 在 1987 年发现了一个更直接的证据：完成四年学位的人与读了三年零十一个月但未获学位的人之间，存在显著的薪资差距，这个差距无法用多出来的那一个月教育来解释。这件事不排除能力差别和雇主的偏好，它只是把&quot;证书&quot;这件东西单独拎出来，交给下一节去解释。</p><tip><p>这篇论文发表在 1987 年的《Review of Economics and Statistics》，标题里用的词是 sheepskin effects，字面意思是羊皮纸，也就是文凭本身。用这一现象解释学历价值的文献通常归到信号假说那一侧，它最初要解释的正是&quot;多读的那一年拿到证书&quot;和&quot;多读的那一年没拿到证书&quot;之间的收入差。</p></tip><h2 id="学历的信号价值随持有人数量下降">学历的信号价值随持有人数量下降</h2><p>多出来的那一个月解释不了这份差距，差额落在那张纸上。这么看，雇主买的是别的东西。</p><p>经济学家罗宾·汉森和凯文·西姆勒在《脑中的大象》第 13 章给出了一个解释框架，起点是一套标准的人力资本叙事：学生上学是为了提升自我，雇主愿意为提升过自我的员工付高薪。</p><p>这套说法解释不了上一节的三个现象。1973 年，Spence 提出的信号传递模型提供了一条替代解释。</p><note title="信号传递模型的两个前提"><p>模型能成立靠两件事：雇主相信学历和更高的工作能力相关，而能力低的人拿到学历的成本更高。第二个前提是信号可信的关键，缺了它，学历就只是一张谁都拿得到的纸。原模型只有一个未观测变量，就是能力；智力、责任心、能否延迟满足这些是后来的经验研究往里填的候选特质，模型本身对教育有没有提升生产力并不表态。</p></note><p>在这个模型里，雇主无法直接观测求职者的智力、责任心、延迟满足的能力和遵守规则的习惯，而这些特质对工作表现很重要。学历恰好提供了一个可用的代理指标：能够完成多年学业、通过一系列考试、遵守学校纪律的人，大概率具备这些特质。课上读到的东西没有被雇主看到，被看到的是这个人能不能被训练；而这个判断能成立多久，取决于学位持有人有多少。</p><p>于是学生读四年书，真正在做的事情就是向未来的雇主发送一组可比较的信号。雇主用学位完成记录、学校层级和成绩排名来筛选，不是因为它们与工作内容直接相关，而是因为它们能低成本地预测一个人是否可培训。</p><p>信号之外，还有一个更少被讨论的功能：驯化。信号能解释雇主为什么看学历，解释不了教学质量不升而认证照旧这回事，因为学校在这里还有第二项产出。书中转述了一项观察，欠发达地区的未受教育工人在简单重复劳动中效率低下，不按时可靠地工作，偏好间接暗示而非直接命令，难以接受与其预设文化身份冲突的任务；学校在发达社会中系统性地矫正了这些行为。</p><p>老师奖励遵守纪律，训练学生接受测量、评分和排名，十多年的训练把&quot;按时出勤、完成截止任务、服从命令链&quot;变成默认习惯。</p><p>这套习惯进入雇佣关系后，雇主获得的是一个已经被预训练过的劳动力。信号解释雇主为什么挑学历，驯化解释学校为什么能持续供给被雇主接受的劳动力，两者合起来，教育扩张就可以在教学质量不升的情况下照旧进行。</p><h2 id="几家公司开始不用这张证书">几家公司开始不用这张证书</h2><p>前两节讲的是一套还在运转的机制。2025 年开始，招聘端有一部分公司选择绕开它。</p><p>2025 年，深圳 17 岁高中生陈广宇因持续在技术社区进行高质量技术分享与写作，以实习生身份加入 Kimi 团队。他在&quot;注意力残差&quot;研究中被标注为&quot;共同第一作者&quot;，这项研究正是支撑 Kimi K3 的两项关键架构创新之一。Kimi K3 以 1679 分登顶 Arena.ai 的 Frontend Code Arena 前端编程榜单，马斯克在这条榜单下留言&quot;Impressive&quot;。</p><p>这不是孤例。Kimi 背后的月之暗面随后推出&quot;穿越计划&quot;，针对 2027 届毕业生，首批开放 16 个名额，入选者通过 3 到 6 个月实习考察后，即使尚未毕业也直接获得公司期权，不限专业、不限学历、不限经验。Palantir 在 2025 年开启高中生招聘计划，直接进入核心部门做主力产品研发，500 人报名只录取 22 人。</p><caution title="三组被传得比原始报道更满的说法"><p>「穿越计划」的一手报道来自《晚点》，官方口径是拟推出、拟授予，3 到 6 个月考察期只在二手摘要里出现。Palantir 那条可核验的是第二期每月 5,400 美元津贴，转正年薪 17 万美元只见于中文二手报道。Google、Meta、OpenAI「两年累计录用 300 多名未完成大学学业的年轻人」找不到原始出处，也没有说明&quot;录用&quot;是实习还是全职，本节没有采用这一条。前两条按原稿保留，第三条已在正文中删去。</p></caution><p>这两家不再把学历当作判断人的入口，改用能直接看到的产出和测试结果。它们挑人的依据变了，招聘的门也就跟着开了。</p><p>陈广宇被 Kimi 录用，不是因为他的学历，是因为他在技术社区持续输出的技术内容。Palantir 招的高中生需要有编程和统计基础，还要通过 4 个月的专项培训和实战项目考核才能转正。两家看的是候选人实际做过什么、能做什么，而不是他读了多少年书。</p><p>李飞飞公开表示，招聘软件工程师时更看重候选人的 AI 技能而非大学学位。DeepSeek 的 Harness 团队负责人崔添翼在微博上回应招聘问题时表示，岗位描述中未说明则不存在硬性学历门槛，刚完成高考的优秀高中生也可以申请实习。</p><p>按技能招人已经不是个别公司的偏好，而是招聘口径在改。腾讯副总裁蒋杰观察到，在校年轻人做的方案与工业界的代差越来越小，对大模型的理解已经和工业界非常接近。</p><p>当高校培养体系从领先产业半步变成滞后产业一步，企业等待毕业生&quot;成熟&quot;的成本变得不可接受。缺口不大的时候，课程更新慢一两拍并不致命；AI 行业把这个缺口拉大到以年计，企业就开始自己从高中生和在校生里挑人。</p><h2 id="信号失灵是有条件的">信号失灵是有条件的</h2><p>学历作为信号之所以能起作用，前提是它能把人区分开。录取率很低、学位持有人稀少的时候，一个本科学位就能告诉雇主：这个人大概率具备基本智力、责任心和服从能力。</p><p>高等教育扩张把学位持有人的数量抬上去，同一个本科学位能分出的差距就越来越小。录取率从百分之几涨到接近普及，硕士又接着扩招，雇主手里那张筛子的孔就越换越细。当几乎所有人都有本科学位，它就不再能区分任何人，而 AI 行业是第一批撞上这件事的行业：那里的技能水平差异可以当场测出来，学历这道代用的筛子显得多余。</p><p>世界经济论坛《2025 年未来就业报告》给出一个宏观口径：81% 的雇主在招聘中评估技能时预计优先考量工作经验，43% 预计会依赖大学学位的完成情况。Indeed 的数据显示，2024 年 1 月五成二的招聘岗位已无学历要求，要求本科的岗位五年间从两成下降至一成八。这两组数字支持的是雇主对学位的依赖在下降，并且都只是雇主自报的招聘口径，不等于学位已经不起作用。</p><important><p>这两个数字回答的是&quot;评估技能时会优先采用哪些方式&quot;，是雇主对 2025 到 2030 年的自报偏好，不是招聘的实际结果。同一份报告里还有一条相反的观察：15 个增长最快的岗位里，14 个仍然主要要求大学学位。Indeed 那一组的单位是招聘广告文本，看的是广告里有没有提到正式学历要求。</p></important><h2 id="那么你该怎么办">那么你该怎么办</h2><p>这对学生的行为指导不是&quot;不要上大学&quot;，而是理解大学在做什么，然后决定自己需要什么。</p><p>大学同时生产三样东西：可验证技能、认证信号、驯化资本。可验证技能就是前面说的&quot;教什么&quot;，对应那个不拿学位但获得教育的选择；认证信号是雇主用来筛人的那张纸；驯化资本是这套训练留给你的工作习惯。比例因专业、学校和目标市场而异。</p><tip title="怎么估自己那一样占比更高"><p>看你要进的那个市场，用它验证能力的成本来判断。验证成本低的地方，也就是代码能跑、证能考、手术台上有带教的岗位，雇主会直接测，技能那一项的权重就高。验证成本高、或者一次招错代价大的入口岗，雇主会退回学历和实习经历这类代用指标，信号那一项更值钱。驯化资本在所有雇佣关系里都算分，只有在需要你自己定方向、而不是执行既定流程的岗位上，它才从资产变成负债。</p></tip><p>如果目标市场有硬技能验证机制，例如软件开发的代码测试、会计的执业考试、医学的执照和住院训练，技能价值占比更高，纯粹依赖学历证书的策略回报递减。</p><p>如果目标市场是依赖学历筛选的传统白领入口岗位，学历信号仍有价值，但竞争者在增加，信号的区分度在下降。</p><p>驯化资本在几乎所有雇佣关系中都有价值，但过度驯化在需要独立判断和创造性输出的岗位上可能成为负债。</p><p>三条路合起来是一句：学位照拿，但不要把全部筹码押在它上面。读书期间持续做出别人能看到的产出，把可验证的东西攒在手上，这件事从今天就能开始做。</p><p>陈广宇的经历提供了一个具体的参照。他在中学时兴趣广泛，借用家长身份注册 Shopify 店铺，组织青少年开源社区，做短视频账号四天涨了 29 万粉丝。这些经历本身不走同一条路，共同点是每一件都在真实场景里留下过别人能看到的产出，他后来被看见靠的是这个。</p><p>他在一篇自传性文章里写道：&quot;我先前的所有经历并没有浪费，它们训练了我一个能力：怎么快速进入一个陌生领域，并找到前进的路径。&quot;他靠的不是提前学完大学课程，而是持续在真实场景里做事。</p><p>2008 年那个&quot;分数作废&quot;的判断，说的其实就是这件事：分数作为信号载体，在发出之后对个体就不再具有私人价值；学位作为筛选装置，在雇主端的流通效率也在下降。两者可以同时成立。</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[240205 复活]]></title>
            <description><![CDATA[好吧，断更了许久，我又终于回来了，我又终于回来了。]]></description>
            <link>https://blog.niina.fun/posts/revival</link>
            <guid isPermaLink="true">https://blog.niina.fun/posts/revival</guid>
            <pubDate>Mon, 05 Feb 2024 11:00:00 GMT</pubDate>
            <content:encoded><![CDATA[<p>好吧，断更了许久，我又终于回来了，我又终于回来了。真跟理由。已经不想再提起，别的话，也不多说，那就直接来新的一张吧。</p><p>上次是 23 年 2 月 3 号离家的。而就在前天三号的时候，我从上海出发返回家乡。整整一年的时间。</p><p>回到故乡之后，发现确实有很多东西改变了。</p><p>但是也有很多东西没有改变的。比如人就老旧房屋人们心中不变的观念。重男轻女啊，那一些。依旧可亲的方言呀，那些口音。那些人还有很多街边的设施经常去了店家。他们都没有变，或许我见的东西少了。当然变得也有很多一些店铺换新了。有些倒闭了，小区里面的。住户有一些在出租的他们的房屋房屋年近过年。好些人都回家了。里面的车停的越来越少，显得空荡荡的。</p><p>天气仍旧很好。</p><p>家对面开了十多年的超市，一年过后居然把店铺里面的一半给砍了出来，用墙割了出来，隔出来那片空间开一家新的烧烤店。</p><p>学校旁边的烂尾楼依旧还是没有人处理。其中建设的小吃街，半死不活的名作者。家附近开一家土特产店，还也说不上来什么。总之确实是有发现，但是这样的发展是好是坏，我说不上。</p><p>看了很多家店铺招聘的广告贴出来的很少，就算有贴出来的也不会把工资直接写出来，而是都都是面试之后商议。我不知道是因为给出的工资低廉到写出来都觉得让人羞耻，还是因为别的原因他们选择面议。</p><p>临近春节应该是在繁忙中透露着一些期待放假的浮躁才对。</p><p>我想这段时间应该可以去做些别的事情。但我却没有找到门路。</p><p>不过确实是有新气象的，在路上居然可以看见有小孩子骑着山地车。确实是让我耳目一新的事。人们的兴趣爱好多了。花钱的地方多。提高人素质的地方也多。</p><p>我在家里待不下。在想着图书馆，图书馆的借书卡究竟被去年的我放在哪里？如果真真切切的要做一些和学习工作相关的事情，还是得找一个适合的地方去做。</p><p>我意识到我绝对不能在一个地方久待。</p>]]></content:encoded>
        </item>
    </channel>
</rss>