grpo-rlvr-training

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

GRPO & RLVR Training

GRPO & RLVR 训练

This skill assumes

finetuning-method-selection

already routed here because the target behavior has a verifiable pass/fail signal — not demonstrations (

lora-qlora-recipes

) or preference pairs (

preference-optimization

). What follows is when RL is the right tool, the reference recipe, the mandatory reward-inspection gate, and how to pick a GRPO variant when the base recipe misbehaves.

Input: a routing decision (RLVR via GRPO) plus a verifier (code executor, test suite, schema checker, or grader) for the target task. Output format: a validated GRPO config — the kwarg values in

references/grpo-memory.md

and the reward functions in

references/reward-functions.md

, not free-form advice — that

llm-finetuning-training-engineer

consumes directly.

本技能假设

finetuning-method-selection

已将任务路由至此，因为目标行为具备可验证的通过/失败信号——而非演示样本（

lora-qlora-recipes

）或偏好配对（

preference-optimization

）。下文将介绍何时选择RL作为合适工具、参考方案、强制奖励检查环节，以及当基础方案表现异常时如何选择GRPO变体。

输入：路由决策（基于GRPO的RLVR）+ 目标任务验证器（代码执行器、测试套件、schema检查器或评分器） 输出格式：经过验证的GRPO配置——即

references/grpo-memory.md

中的参数值与

references/reward-functions.md

中的奖励函数，而非自由形式建议——可直接供

llm-finetuning-training-engineer

使用。

When RL Applies

RL的适用场景

GRPO+RLVR only pays off when task success is algorithmically checkable — a unit test passes, a parser accepts the output, a tool call matches an expected schema, a math answer matches a ground truth. If grading the output requires human judgment or a subjective rubric, that's an eval-harness and judge-calibration problem first — see

eval-harness-first

— not a reason to skip straight to RL.

Before opening a GRPO run, confirm the model can sometimes succeed on the target task already. RL sharpens an existing capability by reweighting toward the samples that already work; it does not install a capability from zero.

The model never succeeds, even at low temperature across many samples: the gap is format or task understanding, not policy refinement. Route back to SFT first (
```
lora-qlora-recipes
```
) and only return to this skill once the base success rate is nonzero.
The model succeeds sometimes, inconsistently: this is the GRPO sweet spot — proceed to The Recipe below.

The standing rule for the whole plugin: DPO for taste, GRPO for reasoning. If the signal is a preference between two acceptable outputs, that's

preference-optimization

, not this skill.

GRPO+RLVR仅在任务成功可通过算法验证时能带来收益——比如单元测试通过、解析器接受输出、工具调用匹配预期schema、数学答案与真值一致。若输出评分需要人工判断或主观标准，那首先是评估工具链与评判校准问题——请参考

eval-harness-first

——而非直接使用RL的理由。

在启动GRPO训练前，需确认模型有时能成功完成目标任务。RL通过向已成功的样本加权来强化现有能力；它无法从零构建一项能力。

模型始终无法成功，即使在低温度下多次采样也是如此：问题出在格式或任务理解上，而非策略优化。请先路由至SFT（
```
lora-qlora-recipes
```
），仅当基础成功率非零时再返回本技能。
模型有时能成功，但表现不稳定：这正是GRPO的最佳适用场景——请继续阅读下方的《参考方案》。

插件的通用规则：DPO用于优化偏好，GRPO用于优化推理。若信号是两个可接受输出之间的偏好，那属于

preference-optimization

的范畴，而非本技能。

The Recipe

参考方案

The reference recipe is TRL's

GRPOTrainer

with vLLM-backed generation:

python

from trl import GRPOConfig, GRPOTrainer

grpo_args = GRPOConfig(
    output_dir="./outputs-grpo",
    use_vllm=True,
    vllm_mode="colocate",       # single GPU; "server" for multi-GPU
    num_generations=8,          # floor — fewer starves the group-relative baseline
    learning_rate=5e-7,         # settled range for GRPO
    beta=0.01,                  # KL coefficient vs the reference policy
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    bf16=True,
    logging_steps=10,
    seed=3407,
)

trainer = GRPOTrainer(
    model=SFT_CHECKPOINT,
    args=grpo_args,
    reward_funcs=[format_reward, correctness_reward],   # references/reward-functions.md
    train_dataset=prompts,       # prompt-only — GRPO generates its own completions
    processing_class=tokenizer,
)

trainer.train()

vllm_mode="colocate"
runs generation and training on the same GPU — the default for a single-GPU box.
vllm_mode="server"
points at a separate vLLM server process and is the multi-GPU path — generation and training don't compete for the same device.
num_generations
≥ 8 is a floor, not a suggestion: GRPO's advantage estimate is relative to the group mean, and fewer than 8 samples per prompt produces a noisy baseline.
Reward is composite — a format reward (did the output parse / match the required structure) plus a correctness reward (did the answer verify). A well-formed-but-wrong answer and a malformed one should not score identically; correctness alone loses that signal.
learning_rate=5e-7
and beta=0.01
are the settled starting point; deviate only after the base run is stable and reward-inspected (below).

Memory sizing for this recipe by target size class:

references/grpo-memory.md

参考方案是基于TRL的

GRPOTrainer

结合vLLM驱动的生成：

python

from trl import GRPOConfig, GRPOTrainer

grpo_args = GRPOConfig(
    output_dir="./outputs-grpo",
    use_vllm=True,
    vllm_mode="colocate",       # 单GPU；多GPU场景使用"server"
    num_generations=8,          # 下限——样本过少会导致组相对基线噪声过大
    learning_rate=5e-7,         # GRPO的稳定取值范围
    beta=0.01,                  # 与参考策略的KL系数
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    bf16=True,
    logging_steps=10,
    seed=3407,
)

trainer = GRPOTrainer(
    model=SFT_CHECKPOINT,
    args=grpo_args,
    reward_funcs=[format_reward, correctness_reward],   # 参考references/reward-functions.md
    train_dataset=prompts,       # 仅提示——GRPO会自行生成补全内容
    processing_class=tokenizer,
)

trainer.train()

vllm_mode="colocate"
：在同一GPU上运行生成与训练——单GPU设备的默认配置。
vllm_mode="server"
：指向独立的vLLM服务器进程，适用于多GPU场景——生成与训练不会竞争同一设备资源。
num_generations
≥ 8：这是下限而非建议值：GRPO的优势估计基于组均值，每个提示的样本少于8个会导致基线噪声过大。
奖励为复合式——包含格式奖励（输出是否可解析/匹配要求的结构）与正确性奖励（答案是否通过验证）。格式正确但答案错误的输出，与格式错误的输出不应获得相同分数；仅使用正确性奖励会丢失这一信号。
**
```
learning_rate=5e-7
```
与
```
beta=0.01
```
**是稳定的起始参数；仅当基础训练稳定且完成奖励检查（见下文）后才考虑调整。

不同规模模型的内存配置参考：

references/grpo-memory.md

。

The Inspection Rule

检查规则

Run the reward function against 50–100 sampled outputs and manually read the results before starting the actual training run. This is a gate, not a one-time sanity check.

If the reward function's judgment disagrees with a human reading of that sample, fix the reward function first. Training against an uninspected reward, or tuning hyperparameters to compensate for one silently scoring the wrong thing, is how a run reward-hacks: the model optimizes cleanly toward the wrong target, and that doesn't surface as a training-loop bug.

This inspection is a Phase 1 gate input for

/finetune

— the same 50–100-sample read that catches a broken reward function here is what that command checks for before it lets a GRPO brief proceed.

Complete reward function implementations to inspect against — exact-match, schema-validation, unit-test-execution, a length-penalty wrapper, and a rubric-as-reward judge pattern:

references/reward-functions.md

在启动实际训练前，使用奖励函数对50-100个采样输出进行验证并人工查看结果。这是强制环节，而非一次性的sanity check。

若奖励函数的判断与人工对样本的解读不一致，请先修复奖励函数。基于未检查的奖励进行训练，或调整超参数来弥补奖励函数的隐性错误，会导致训练出现奖励欺骗：模型会精准地向错误目标优化，且不会表现为训练循环的bug。

该检查是

/finetune

命令的第一阶段准入条件——此处用于检测奖励函数问题的50-100样本人工审核，也是该命令在允许GRPO任务推进前的必查项。

可供检查的完整奖励函数实现——包括精确匹配、schema验证、单元测试执行、长度惩罚包装器，以及基于评分标准的奖励评判模式：

references/reward-functions.md

。

Variant Selection

变体选择

The base recipe above is the default. Reach for a variant only when a specific failure mode shows up, not preemptively:

Failure mode	Variant	Why
Entropy collapse / degenerate long chain-of-thought	DAPO	Decouples clip bounds and relaxes the KL penalty that over-regularizes exploration on long reasoning traces
Reward or output length trends up regardless of quality	Dr.GRPO	Removes GRPO's length-normalization bias so reward tracks correctness, not completion length
Training a mixture-of-experts model	GSPO	Moves the importance-sampling ratio to the sequence level instead of per-token — per-token ratios are unstable on MoE routing, so GSPO is required here, not optional

Start with plain GRPO. Watch for the specific symptom — collapsing entropy on long CoT, a length-reward correlation, or MoE instability — and only then swap in the matching variant above. Don't pre-select a variant before the base recipe has actually shown the failure mode.

上述基础方案是默认选项。仅当出现特定故障模式时才选择变体，而非预先选择：

故障模式	变体	原因
熵坍缩/退化的长思维链	DAPO	解耦裁剪边界并放松KL惩罚，避免对长推理轨迹的探索过度正则化
奖励或输出长度随训练上升，与质量无关	Dr.GRPO	移除GRPO的长度归一化偏差，使奖励与正确性挂钩而非补全长度
训练混合专家（MoE）模型	GSPO	将重要性采样比率移至序列层面而非逐token层面——逐token比率在MoE路由中不稳定，因此GSPO是必需选项而非可选

先从标准GRPO开始。观察是否出现特定症状——长CoT的熵坍缩、长度与奖励的相关性、MoE不稳定——再替换为对应的变体。不要在基础方案出现故障模式前预先选择变体。

VLM RL Is Reference-Only

VLM RL仅作参考

Vision-language RL is not executed by this plugin in v1 — it's documented here for context, not as a runnable path. Tooling is fragmented across ms-swift and EasyR1-derived forks with no one-line TRL command yet, and naive text-only GRPO applied to a VLM tends to reward-hack by optimizing the text-reasoning trace while ignoring the image — the model learns to sound right without looking at the input. A VLM RL run is a research spike outside this skill's supported recipe, not a variant of The Recipe above.

视觉语言模型（VLM）的RL在v1版本中不被本插件支持——此处仅作背景说明，并非可执行路径。相关工具在ms-swift与EasyR1衍生分支中较为分散，尚无一行式TRL命令可用；将纯文本GRPO直接应用于VLM容易导致奖励欺骗：模型会优化文本推理轨迹而忽略图像输入——即模型学会“听起来正确”却不关注输入图像。VLM的RL训练属于本技能支持方案之外的研究探索，而非上述参考方案的变体。

References

参考资料

```
references/reward-functions.md
```
— complete Python reward functions (exact-match correctness, schema validation, unit-test execution, a length-penalty wrapper, and a rubric-as-reward judge pattern) to inspect under The Inspection Rule before any training run.
```
references/grpo-memory.md
```
— memory sizing by target size class, vLLM sleep-mode and optimizer-state tactics, Unsloth's long-context RL chunking, and the DGX Spark bandwidth caveat for decode-heavy rollouts.

Related skills:

finetuning-method-selection

routes here once a verifiable pass/fail signal exists;

preference-optimization

is the sibling skill for preference pairs rather than verifiable rewards;

eval-harness-first

covers judge calibration for any reward that isn't purely code-checkable. On DGX Spark, defer to the

dgx-spark-ops

plugin's skills, when installed, for the memory/thermal remediation ladder this skill's memory table doesn't cover.

```
references/reward-functions.md
```
——完整的Python奖励函数（精确匹配正确性、schema验证、单元测试执行、长度惩罚包装器、基于评分标准的奖励评判模式），需在任何训练前按照《检查规则》进行验证。
```
references/grpo-memory.md
```
——按模型规模分类的内存配置、vLLM休眠模式与优化器状态策略、Unsloth的长上下文RL分块方法，以及DGX Spark在解码密集型rollout中的带宽注意事项。