Merge branch 'main' into release/3.11

Author: Jintao-Huang

docs/source/Instruction/Command-line-parameters.md

@@ -617,6 +617,7 @@ reward模型参数将在PPO、GRPO中使用。
 - log_entropy: 记录训练中的熵值变化动态，默认为False，具体参考[文档](./GRPO/GetStarted/GRPO.md#logged-metrics)
 - rollout_importance_sampling_mode: 训推不一致校正模式，可选项为 `token_truncate`、`token_mask`、`sequence_truncate`、`sequence_mask`。默认为None，不启用校正。具体参考[文档](./GRPO/AdvancedResearch/training_inference_mismatch.md)
 - rollout_importance_sampling_threshold: 重要性采样权重的阈值，用于截断或屏蔽极端权重。默认为2.0。
+- log_rollout_offpolicy_metrics: 当 `rollout_importance_sampling_mode` 未设置时，是否记录训推不一致诊断指标（KL、PPL、χ²等）。当设置了 `rollout_importance_sampling_mode` 时，指标会自动记录。默认为False。
 
 ##### 奖励函数参数
 内置的奖励函数参考[文档](./GRPO/DeveloperGuide/reward_function.md)

docs/source/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.md

@@ -187,6 +187,16 @@ ESS 值越大（接近1），表示重要性采样权重分布越均匀，样本
 
 ## 使用方式
 
+### 仅记录诊断指标
+
+如果只想监控训推不一致的程度，而不启用重要性采样校正，可以设置：
+
+```
+--log_rollout_offpolicy_metrics true
+```
+
+这将记录上述所有诊断指标（KL、PPL、χ² 等），但不会对损失函数进行任何修正。
+
 ### 启用重要性采样校正
 
 在GRPO训练中，设置以下参数启用校正机制：
@@ -196,6 +206,7 @@ ESS 值越大（接近1），表示重要性采样权重分布越均匀，样本
 --rollout_importance_sampling_threshold （默认为2）
 ```
 
+当设置了 `rollout_importance_sampling_mode` 时，诊断指标会自动记录，无需额外设置 `log_rollout_offpolicy_metrics`。
 
 参考资料
 

docs/source/Instruction/GRPO/DeveloperGuide/reward_function.md

@@ -2,6 +2,8 @@
 ## 自定义奖励函数
 奖励函数接受模型生成的文本 completions 其他数据集中的列以及训练器状态作为参数(kwargs)进行打分, 其中[训练器状态](https://huggingface.co/docs/transformers/main/main_classes/callback#transformers.TrainerState)包含训练的步数等信息。
 
+> Megatron GRPO 使用 self._step 获取当前训练步数
+
 注意：模型输入相关的列（比如query，response）会被处理为 messages 键，原数据集中的 assistant response 会被舍弃，请使用额外的列进行保留。
 相关处理的列名参考[文档](../../../Customization/Custom-dataset.md#query-response格式)
 

docs/source/Instruction/GRPO/GetStarted/GRPO.md

@@ -258,14 +258,14 @@ swift rlhf \
 如果设置了`top_entropy_quantile`参数<1.0, 则会记录entropy threshold的值
 - entropy/threshold: 分位点处的 entropy 值，小于该值的 token 将不会被计算 loss
 
-训推一致性指标，前缀为rollout_correction (ms-swift>=3.11)
+训推一致性指标，前缀为rollout_correction (ms-swift>=3.11)，需设置`log_rollout_offpolicy_metrics=true`或`rollout_importance_sampling_mode`：
 - `kl` / `k3_kl`：训练策略与 rollout 策略之间的 KL 散度（直接估计器 / K3 估计器）
 - `training_ppl` / `rollout_ppl`：训练策略和 rollout 策略的困惑度
 - `log_ppl_diff`：log PPL 差异，反映分布偏移程度
 - `ppl_ratio`：PPL 比率
 - `chi2_token` / `chi2_seq`：Token/Sequence 级别的 χ² 散度
 
-IS 校正指标(需设置rollout_importance_sampling_mode)
+IS 校正指标（需设置`rollout_importance_sampling_mode`）：
 - `is_weight_mean`：平均重要性采样权重
 - `ess`：有效样本大小（Effective Sample Size）
 - `clipped_frac`：被截断或屏蔽的样本比例

docs/source/Megatron-SWIFT/Command-line-parameters.md

@@ -386,6 +386,9 @@ Megatron训练参数继承自Megatron参数和基本参数（**与ms-swift共用
 - delta: [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291)中双侧 GRPO 上界裁剪值。若设置，建议大于 1 + epsilon。默认为None。
 - importance_sampling_level: 控制重要性采样比计算，可选项为 `token` 和 `sequence`，`token` 模式下保留原始的每个 token 的对数概率比，`sequence` 模式下则会对序列中所有有效 token 的对数概率比进行平均。[GSPO论文](https://arxiv.org/abs/2507.18071)中使用sequence级别计算来稳定训练，默认为`token`。
 - scale_rewards：指定奖励的缩放策略。可选值包括 `group`（按组内标准差缩放）、`batch`（按整个批次的标准差缩放）、`none`（不进行缩放）。在 ms-swift < 3.10 版本中，该参数为布尔类型，`true` 对应 `group`，`false` 对应 `none`。默认值与 `advantage_estimator` 绑定：`grpo` 对应 `group`，`rloo` 对应 `none`，`reinforce_plus_plus` 对应 `batch`。
+- rollout_importance_sampling_mode: 训推不一致校正模式，可选项为 `token_truncate`、`token_mask`、`sequence_truncate`、`sequence_mask`。默认为None，不启用校正。具体参考[文档](../Instruction/GRPO/AdvancedResearch/training_inference_mismatch.md)。
+- rollout_importance_sampling_threshold: 重要性采样权重的阈值，用于截断或屏蔽极端权重。默认为2.0。
+- log_rollout_offpolicy_metrics: 当 `rollout_importance_sampling_mode` 未设置时，是否记录训推不一致诊断指标（KL、PPL、χ²等）。当设置了 `rollout_importance_sampling_mode` 时，指标会自动记录。默认为False。
 
 内置奖励函数参数参考[文档](../Instruction/Command-line-parameters.md#奖励函数参数)
 

docs/source_en/Instruction/Command-line-parameters.md

@@ -632,6 +632,7 @@ The hyperparameters for the reward function can be found in the [Built-in Reward
 - log_entropy: Logs the entropy values during training. The default is False. For more information, refer to the [documentation](./GRPO/GetStarted/GRPO.md#logged-metrics).
 - rollout_importance_sampling_mode: Training-inference mismatch correction mode. Options are `token_truncate`, `token_mask`, `sequence_truncate`, `sequence_mask`. Default is None (disabled). For details, refer to the [documentation](./GRPO/AdvancedResearch/training_inference_mismatch.md).
 - rollout_importance_sampling_threshold: Threshold for importance sampling weights, used for truncating or masking extreme weights. Default is 2.0.
+- log_rollout_offpolicy_metrics: Whether to log training-inference mismatch diagnostic metrics (KL, PPL, χ², etc.) when `rollout_importance_sampling_mode` is not set. When `rollout_importance_sampling_mode` is set, metrics are always logged. Default is False.
 
 
 ##### Reward function parameters

docs/source_en/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.md

@@ -187,6 +187,16 @@ A larger ESS value (closer to 1) indicates more uniform importance sampling weig
 
 ## Usage
 
+### Logging Diagnostic Metrics Only
+
+If you only want to monitor the degree of training-inference mismatch without enabling importance sampling correction, you can set:
+
+```
+--log_rollout_offpolicy_metrics true
+```
+
+This will log all diagnostic metrics (KL, PPL, χ², etc.) without modifying the loss function.
+
 ### Enabling Importance Sampling Correction
 
 Enable the correction mechanism with the following parameters:
@@ -196,6 +206,7 @@ Enable the correction mechanism with the following parameters:
 --rollout_importance_sampling_threshold (default 2)
 ```
 
+When `rollout_importance_sampling_mode` is set, diagnostic metrics are automatically logged without needing to set `log_rollout_offpolicy_metrics`.
 
 ## References
 

docs/source_en/Instruction/GRPO/DeveloperGuide/reward_function.md

@@ -1,6 +1,8 @@
 # Reward Function
 ## Custom Reward Function
-The reward function takes as arguments (via kwargs) the model-generated completions, other columns from the dataset, and the training state, and calculates a reward score. The [trainer state]() includes information such as the current training step.
+The reward function takes as arguments (via kwargs) the model-generated completions, other columns from the dataset, and the training state, and calculates a reward score. The [trainer state](https://huggingface.co/docs/transformers/main/main_classes/callback#transformers.TrainerState) includes information such as the current training step.
+
+> If you are using the Megatron backend, use self._step to get the current training step.
 
 Note: The columns related to model input (such as query and response) are converted to the messages key. The original assistant response in the dataset will be discarded, so please use extra columns if you wish to retain it.
 The relevant column names for processing can be found in the [document](../../../Customization/Custom-dataset.md#Query-Response)

docs/source_en/Instruction/GRPO/GetStarted/GRPO.md

@@ -256,14 +256,14 @@ If the `log_entropy` parameter is set, additional entropy-related metrics will b
 If `top_entropy_quantile` is set to a value smaller than 1.0, the entropy threshold value will also be recorded:
 - entropy/threshold: Tokens with entropy below this value will be excluded from the loss calculation.
 
-Training-inference consistency metrics, prefixed with rollout_correction (ms-swift>=3.11):
+Training-inference consistency metrics, prefixed with rollout_correction (ms-swift>=3.11), requires setting `log_rollout_offpolicy_metrics=true` or `rollout_importance_sampling_mode`:
 - `kl` / `k3_kl`: KL divergence between training policy and rollout policy (direct estimator / K3 estimator)
 - `training_ppl` / `rollout_ppl`: Perplexity of training policy and rollout policy
 - `log_ppl_diff`: Log PPL difference, reflects the degree of distribution shift
 - `ppl_ratio`: PPL ratio
 - `chi2_token` / `chi2_seq`: Token/Sequence-level χ² divergence
 
-IS correction metrics (requires setting rollout_importance_sampling_mode):
+IS correction metrics (requires setting `rollout_importance_sampling_mode`):
 - `is_weight_mean`: Average importance sampling weight
 - `ess`: Effective Sample Size
 - `clipped_frac`: Fraction of samples that were truncated or masked

docs/source_en/Megatron-SWIFT/Command-line-parameters.md

@@ -410,6 +410,9 @@ In addition to inheriting the training parameters, the following parameters are
 - delta: Bilateral GRPO upper bound clipping value from the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291). If set, it is recommended to be greater than 1 + epsilon. Default is None.
 - importance_sampling_level: Controls importance sampling ratio calculation. Options are `token` and `sequence`. In `token` mode, the original log probability ratio for each token is preserved. In `sequence` mode, the log probability ratios of all valid tokens in the sequence are averaged. The [GSPO paper](https://arxiv.org/abs/2507.18071) uses sequence-level calculation to stabilize training. Default is `token`.
 - scale_rewards: Specifies the reward scaling strategy. Options include `group` (scale by within-group standard deviation), `batch` (scale by batch-wide standard deviation), and `none` (no scaling). In ms-swift < 3.10, this parameter is boolean, where `true` corresponds to `group` and `false` corresponds to `none`. The default value is bound to `advantage_estimator`: `grpo` corresponds to `group`, `rloo` corresponds to `none`, and `reinforce_plus_plus` corresponds to `batch`.
+- rollout_importance_sampling_mode: Training-inference mismatch correction mode. Options are `token_truncate`, `token_mask`, `sequence_truncate`, `sequence_mask`. Default is None (disabled). For details, refer to the [documentation](../Instruction/GRPO/AdvancedResearch/training_inference_mismatch.md).
+- rollout_importance_sampling_threshold: Threshold for importance sampling weights, used for truncating or masking extreme weights. Default is 2.0.
+- log_rollout_offpolicy_metrics: Whether to log training-inference mismatch diagnostic metrics (KL, PPL, χ², etc.) when `rollout_importance_sampling_mode` is not set. When `rollout_importance_sampling_mode` is set, metrics are always logged. Default is False.
 
 Built-in reward function parameters refer to the [documentation](../Instruction/Command-line-parameters.md#reward-function-parameters).