trl internal testing

posted an update about 20 hours ago

Post

2076

Frontier models use distillation as a step of their post-training pipelines.

In 2026 it has three jobs: compress a big model into a small one, merge RL experts into a single model, and let a model teach itself.

I wrote up which frontier models use each one and how: https://huggingface.co/blog/sergiopaniego/distillation-2026

It pairs with Class 2 of the Training an Agent series Ben and I are doing, where we teach these techniques hands-on with TRL!

1 reply

posted an update 3 days ago

Post

3342

🎉 KTO is now part of the stable TRL API

As of Promote KTO to stable API, KTOTrainer and KTOConfig have graduated from trl.experimental to the stable trl API. https://github.com/huggingface/trl/pull/6175

This one closes out a long road. Over the past 6+ months, the "Align KTO with DPO" effort landed ~90 PRs methodically bringing KTO up to the standard we hold for stable trainers, one carefully-scoped change at a time:
- Feature parity with DPO: full VLM support (incl. multi-image), sync_ref_model, PEFT + Liger, ZeRO-3 + PEFT dtype fix, pad_to_multiple_of, activation offloading, IterableDataset and dict eval_dataset, remove_unused_columns, and reference-logprob precomputation at init.
- Consistency with DPO: aligned method order and signatures, tokenization, _prepare_dataset, PEFT handling, ref-model preparation for distributed training, and config layout — plus a new DataCollatorForKTO and output format. Metrics moved into _compute_loss and simplified to direct averages via the shared _metrics attribute.
- Removing legacy baggage: dropped encoder-decoder support, BOS/EOS handling, null_ref_context, generate_during_eval, model_init, preprocess_logits_for_metrics, model/ref adapter names, and several dead config knobs.
- Coverage: a full test suite mirroring DPO, text collator tests, VLM tests, and slow tests.
- The promotion itself: the experimental → stable move (#6175) and shim cleanup (#6287), handled so downstream users get a clean deprecation path.

Honestly, this has been one of the more complex tasks I've taken on since joining the team, not because any single change was hard, but because it demanded sustained consistency across a ~2,000-line trainer, with every branch, comment, and edge case kept in lockstep with DPO.

Huge thanks to everyone who reviewed along the way (especially @qgallouedec ), the incremental review cadence is exactly what kept this maintainable.

KTO now sits on equal footing with our other flagship trainers. 🚀

2 replies

in trl-internal-testing/tiny-DeepseekV3ForCausalLM-0528 6 days ago

Upload DeepseekV3ForCausalLM

#4 opened 6 days ago by

in trl-internal-testing/tiny-DeepseekV3ForCausalLM 6 days ago

Upload DeepseekV3ForCausalLM

#3 opened 6 days ago by

in trl-internal-testing/tiny-DeepseekV3ForCausalLM-0528 6 days ago

Upload DeepseekV3ForCausalLM

#2 opened 2 months ago by

Upload DeepseekV3ForCausalLM

#3 opened 2 months ago by

Upload DeepseekV3ForCausalLM

#1 opened 2 months ago by

in trl-internal-testing/tiny-DeepseekV3ForCausalLM 6 days ago

Upload tiny DeepseekV3ForCausalLM

#2 opened 3 months ago by

in trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 9 days ago

Upload .gitattributes with huggingface_hub

#2 opened about 2 months ago by

Upload README.md with huggingface_hub

#3 opened about 2 months ago by

Upload added_tokens.json with huggingface_hub

#4 opened about 2 months ago by

Upload chat_template.jinja with huggingface_hub

#5 opened about 2 months ago by

Listed on OpenModelMap

#7 opened 20 days ago by

duola15

Great Turkish model — mobile deployment perspective

#8 opened 12 days ago by

3morixd

Upload config.json with huggingface_hub

#6 opened about 2 months ago by

posted an update 13 days ago

Post

290

TRL v1.7.0 is out‼️

+ continuous batching makes GRPO and RLOO 1.25x faster at -16 GB
+ proper MoE post-training across GRPO/RLOO/AsyncGRPO
+ new GMPO trainer
+ AsyncGRPO weight sync + padding-free
+ more

https://github.com/huggingface/trl/releases/tag/v1.7.0

wrote a small article about the continuous batching for GRPO feature

https://huggingface.co/blog/sergiopaniego/cb-trl-grpo

Viewer • Updated 15 days ago • 95 • 21.4k • 1

updated a dataset 15 days ago

trl-internal-testing/zen-multi-image

posted an update 20 days ago

Post

297

Continuous batching just landed in TRL for GRPO!

At 64 generations it runs faster and uses less VRAM than plain generate, no vLLM needed

How it works and when to reach for it, below

https://huggingface.co/blog/sergiopaniego/cb-trl-grpo