secret-deus
decca30bef
Replace demo notes with 146 vault notes and diagram assets; keep credentials only in local .env.
122 lines
7.5 KiB
XML
122 lines
7.5 KiB
XML
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 680 600" width="100%" role="img">
|
||
<title>LLM 训练三阶段流水线</title>
|
||
<desc>预训练→SFT→RLHF：从万亿 token 的原始数据到对齐人类偏好的完整流程</desc>
|
||
<style>
|
||
.th { font-family: -apple-system, sans-serif; font-size: 13px; font-weight: 500; fill: #26215C; }
|
||
.ts { font-family: -apple-system, sans-serif; font-size: 11px; fill: #5F5E5A; }
|
||
.tm { font-family: -apple-system, monospace; font-size: 10px; fill: #26215C; }
|
||
.tc { font-family: -apple-system, sans-serif; font-size: 9px; fill: #888780; }
|
||
.tw { font-family: -apple-system, sans-serif; font-size: 10px; fill: #ffffff; }
|
||
.tb { font-family: -apple-system, sans-serif; font-size: 11px; font-weight: 500; fill: #26215C; }
|
||
</style>
|
||
|
||
<!-- ===== Phase 1: Pre-training ===== -->
|
||
<rect x="20" y="20" width="640" height="160" rx="10" fill="#E6F1FB" stroke="#85B7EB" stroke-width="0.5"/>
|
||
<rect x="20" y="20" width="640" height="30" rx="10" fill="#185FA5"/>
|
||
<rect x="20" y="38" width="640" height="12" fill="#185FA5"/>
|
||
<text class="tw" x="340" y="40" text-anchor="middle">阶段 1：预训练 (Pre-training) — 耗时：数周到数月</text>
|
||
|
||
<text class="ts" x="50" y="68">数据：</text>
|
||
<rect x="100" y="56" width="120" height="22" rx="4" fill="#ffffff" stroke="#85B7EB" stroke-width="0.3"/>
|
||
<text class="tc" x="160" y="71" text-anchor="middle">网页 (Common Crawl)</text>
|
||
<rect x="230" y="56" width="80" height="22" rx="4" fill="#ffffff" stroke="#85B7EB" stroke-width="0.3"/>
|
||
<text class="tc" x="270" y="71" text-anchor="middle">代码 (GitHub)</text>
|
||
<rect x="320" y="56" width="80" height="22" rx="4" fill="#ffffff" stroke="#85B7EB" stroke-width="0.3"/>
|
||
<text class="tc" x="360" y="71" text-anchor="middle">书籍</text>
|
||
<rect x="410" y="56" width="80" height="22" rx="4" fill="#ffffff" stroke="#85B7EB" stroke-width="0.3"/>
|
||
<text class="tc" x="450" y="71" text-anchor="middle">论文</text>
|
||
<text class="tc" x="510" y="71">≈ 数万亿 Token</text>
|
||
|
||
<text class="ts" x="50" y="94">目标：</text>
|
||
<text class="tc" x="100" y="94">Next-Token Prediction — 给定前面的 token，预测下一个</text>
|
||
<text class="tc" x="100" y="108">Loss = Cross-Entropy(预测, 真值) ｜ 优化器：AdamW / Muon</text>
|
||
|
||
<text class="ts" x="50" y="128">产出：</text>
|
||
<rect x="100" y="116" width="160" height="28" rx="6" fill="#378ADD" stroke="#0C447C" stroke-width="0.5"/>
|
||
<text class="tw" x="180" y="134" text-anchor="middle">Base Model（基座模型）</text>
|
||
<text class="tc" x="270" y="134">会续写但不会对话，不懂指令</text>
|
||
|
||
<text class="ts" x="50" y="158">典型：</text>
|
||
<text class="tc" x="100" y="158">LLaMA-7B: 1T tokens, ~26 天 (1024×A100) ｜ DeepSeek-V3: 14.8T tokens, ~2 月 (2048×H800)</text>
|
||
|
||
<!-- Arrow 1→2 -->
|
||
<line x1="340" y1="180" x2="340" y2="206" stroke="#888780" stroke-width="1.5" marker-end="url(#arr)"/>
|
||
|
||
<!-- ===== Phase 2: SFT ===== -->
|
||
<rect x="20" y="208" width="640" height="120" rx="10" fill="#FAEEDA" stroke="#EF9F27" stroke-width="0.5"/>
|
||
<rect x="20" y="208" width="640" height="30" rx="10" fill="#854F0B"/>
|
||
<rect x="20" y="226" width="640" height="12" fill="#854F0B"/>
|
||
<text class="tw" x="340" y="228" text-anchor="middle">阶段 2：有监督微调 (SFT) — 耗时：几小时到几天</text>
|
||
|
||
<text class="ts" x="50" y="256">数据：</text>
|
||
<rect x="100" y="244" width="200" height="22" rx="4" fill="#ffffff" stroke="#EF9F27" stroke-width="0.3"/>
|
||
<text class="tc" x="200" y="259" text-anchor="middle">(指令 Instruction, 回答 Response) 对</text>
|
||
<text class="tc" x="310" y="259">≈ 1 万 - 100 万 对</text>
|
||
|
||
<text class="ts" x="50" y="280">目标：</text>
|
||
<text class="tc" x="100" y="280">仍然是 Next-Token Prediction，但 Loss 只在 Response 部分计算 ｜ 可全量微调或 LoRA</text>
|
||
|
||
<text class="ts" x="50" y="300">产出：</text>
|
||
<rect x="100" y="288" width="160" height="28" rx="6" fill="#BA7517" stroke="#633806" stroke-width="0.5"/>
|
||
<text class="tw" x="180" y="306" text-anchor="middle">Instruction Model</text>
|
||
<text class="tc" x="270" y="306">会回答问题、遵循指令，但对齐不够好</text>
|
||
|
||
<defs>
|
||
<marker id="arr" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||
<path d="M2 1L8 5L2 9" fill="none" stroke="#888780" stroke-width="1.2" stroke-linecap="round"/>
|
||
</marker>
|
||
</defs>
|
||
|
||
<!-- Arrow 2→3 -->
|
||
<line x1="340" y1="328" x2="340" y2="354" stroke="#888780" stroke-width="1.5" marker-end="url(#arr)"/>
|
||
|
||
<!-- ===== Phase 3: RLHF ===== -->
|
||
<rect x="20" y="356" width="640" height="225" rx="10" fill="#EEEDFE" stroke="#AFA9EC" stroke-width="0.5"/>
|
||
<rect x="20" y="356" width="640" height="30" rx="10" fill="#534AB7"/>
|
||
<rect x="20" y="374" width="640" height="12" fill="#534AB7"/>
|
||
<text class="tw" x="340" y="376" text-anchor="middle">阶段 3：人类偏好对齐 (RLHF/DPO/GRPO) — 耗时：几天到几周</text>
|
||
|
||
<!-- Sub-step 3a -->
|
||
<rect x="40" y="396" width="180" height="80" rx="6" fill="#EEEDFE" stroke="#7F77DD" stroke-width="0.3"/>
|
||
<text class="tb" x="130" y="414" text-anchor="middle">3a. 训练 Reward Model</text>
|
||
<text class="tc" x="55" y="432">数据：(提问, 回答A, 回答B, A>B)</text>
|
||
<text class="tc" x="55" y="448">训练目标：预测人类偏好</text>
|
||
<text class="tc" x="55" y="464">Bradley-Terry 模型</text>
|
||
|
||
<line x1="220" y1="436" x2="248" y2="436" stroke="#888780" stroke-width="0.8" marker-end="url(#arr-small)"/>
|
||
|
||
<!-- Sub-step 3b -->
|
||
<rect x="250" y="396" width="180" height="80" rx="6" fill="#EEEDFE" stroke="#7F77DD" stroke-width="0.3"/>
|
||
<text class="tb" x="340" y="414" text-anchor="middle">3b. PPO / GRPO 训练</text>
|
||
<text class="tc" x="265" y="432">Policy 生成回答</text>
|
||
<text class="tc" x="265" y="448">Reward Model 打分</text>
|
||
<text class="tc" x="265" y="464">GRPO 用组内相对优势</text>
|
||
|
||
<line x1="430" y1="436" x2="458" y2="436" stroke="#888780" stroke-width="0.8" marker-end="url(#arr-small)"/>
|
||
|
||
<!-- Sub-step 3c -->
|
||
<rect x="460" y="396" width="180" height="80" rx="6" fill="#EEEDFE" stroke="#7F77DD" stroke-width="0.3"/>
|
||
<text class="tb" x="550" y="414" text-anchor="middle">3c. 迭代 + DPO</text>
|
||
<text class="tc" x="475" y="432">重复 3a-3b 多轮</text>
|
||
<text class="tc" x="475" y="448">或用 DPO 直接优化</text>
|
||
<text class="tc" x="475" y="464">(省去 Reward Model)</text>
|
||
|
||
<text class="ts" x="50" y="498">产出：</text>
|
||
<rect x="100" y="486" width="180" height="28" rx="6" fill="#7F77DD" stroke="#3C3489" stroke-width="0.5"/>
|
||
<text class="tw" x="190" y="504" text-anchor="middle">Aligned Model（对齐模型）</text>
|
||
<text class="tc" x="290" y="504">更安全、更有用、更符合人类偏好</text>
|
||
|
||
<text class="ts" x="50" y="532">选择：</text>
|
||
<text class="tc" x="100" y="532">PPO（经典，需 4 个模型加载）｜ GRPO（DeepSeek 提出，去 Critic，省一半显存）</text>
|
||
<text class="tc" x="100" y="548">DPO（更简单，不需要 Reward Model，直接用偏好对训练）｜ OPD（DeepSeek-V4，多教师蒸馏替代 RL）</text>
|
||
|
||
<text class="ts" x="50" y="566">关键挑战：</text>
|
||
<text class="tc" x="100" y="566">RL 训练不稳定（loss spike）｜ Reward Hacking（刷分不干正事）｜ 分布偏移（生成越来越偏）</text>
|
||
|
||
<defs>
|
||
<marker id="arr-small" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="5" markerHeight="5" orient="auto">
|
||
<path d="M2 1L8 5L2 9" fill="none" stroke="#888780" stroke-width="1" stroke-linecap="round"/>
|
||
</marker>
|
||
</defs>
|
||
|
||
</svg> |
