Skip to content

v1.2.0

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 17 Apr 01:13
· 752 commits to main since this release
v1.2.0
aca4515
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

New SSDTrainer — Simple Self-Distillation

Screenshot 2026-04-16 at 9 08 04 PM

A new experimental SSDTrainer implements the method described in Embarrassingly Simple Self-Distillation Improves Code Generation. SSD samples completions from the model itself at a training-time temperature/truncation setting, then fine-tunes on those raw, unverified samples with standard cross-entropy loss. No reward model, verifier, teacher model, or RL: just prompts and the model.

from datasets import Dataset
from trl.experimental.ssd import SSDConfig, SSDTrainer

dataset = Dataset.from_dict({
    "prompt": [
        [{"role": "user", "content": "Write a function to add two numbers."}],
        [{"role": "user", "content": "Write a function to check if a number is prime."}],
    ],
})

trainer = SSDTrainer(
    model="Qwen/Qwen3-4B-Instruct",
    args=SSDConfig(
        output_dir="ssd-model",
        temperature=0.6,      # T_train from the paper
        top_k=20,
        top_p=0.95,
        learning_rate=5e-6,
    ),
    train_dataset=dataset,
)
trainer.train()

by @kashif in #5505

Drop, don't truncate, overlong tool results in GRPOTrainer

When tool calls produce more tokens than max_completion_length allows, GRPOTrainer now rolls back the tool messages/images added in the current iteration instead of trying to truncate them. This removes ~80 lines of fragile, image-boundary-aware bookkeeping in favor of a ~15-line snapshot-and-rollback. Since overlong samples almost always get rewarded as failures anyway, the learning signal is effectively unchanged — but the code is dramatically simpler and no longer needs per-VLM-family vision-token lookup tables.

by @qgallouedec in #5521

Expanded tool-calling model support: LLaMA 3.1 / 3.2 & DeepSeek-V3

Continuing the effort from v1.1:

  • LLaMA 3.1 and 3.2 tool-calling response schemas, with dedicated templates for identity matching. Note that these templates only support a single tool call and no content alongside the tool call — limitations inherited from the models' native templates. By @qgallouedec in #5518
  • DeepSeek-V3 training chat template with {% generation %} markers, enabling assistant-only loss masking for DeepSeek-V3 models. By @RudrenduPaul in #5527

As a result of a tightened detection (see fixes below), the list of templates reported as tool-calling capable is now correct — notably, the basic Llama 3 template is no longer falsely classified as tool-calling capable.

KTO/DPO alignment push

A major cleanup sweep keeps KTOTrainer and DPOTrainer in lockstep, same initialization patterns, same config surface, same precompute behavior:

  • Add precompute_ref_batch_size to KTO (#5530)
  • Align ref_model initialization (#5534)
  • Align model initialization (#5533)
  • Support None args (#5531)
  • Remove generate_during_eval (#5551)
  • Remove model and ref adapter names (#5552)
  • Don't load ref_model when precompute_ref_log_probs is set in DPO/KTO (#5542)

All by @albertvillanova.

Other

Fixes

  • Fix supports_tool_calling falsely accepting templates that drop assistant tool_calls by @qgallouedec in #5517
  • Fix add_response_schema for VLM processors — the schema was being set on the outer processor instead of the inner tokenizer, so it had no effect. This also collapses a handful of __init__/decode-gate workarounds. By @qgallouedec in #5520
  • Remove xfail condition for Gemma 4 response_schema regex bug by @qgallouedec in #5510
  • Remove unused dependencies for judges from dev requirements by @qgallouedec in #5515

Deprecations

  • Deprecate use_transformers_paged in GRPOConfig and RLOOConfig (and remove entirely from experimental OnlineDPOConfig, GOLDConfig, SelfDistillationConfig). Will be removed from the remaining configs in v2.0.0. In a small A/B benchmark (Qwen3-0.6B GRPO), the paged path is ~20% slower and uses ~6x more peak VRAM than the default; it's also superseded by transformers continuous batching. By @qgallouedec in #5544

Documentation and Examples

CI

What's Changed

Full Changelog: v1.1.0...v1.2.0

Contributors

kashif, rtrompier, and 4 other contributors
Assets 2
Loading