About this episode

This episode discusses the concept of On-Policy Delta Distillation in reinforcement learning and its advantages over conventional methods.

🤗 Upvotes: 28 | cs.LG, cs.CL Authors: Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han Title: On-Policy Delta Distillation Arxiv: http://arxiv.org/abs/2607.15161v1 Abstract: On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$^2$). Experiments across mathematics, science, and code-reasoning…

More episodes of Daily Paper Cast

• RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources · July 21, 2026 · 19 min
• RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM · July 21, 2026 · 20 min
• Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories · July 21, 2026 · 21 min
• Loop the Loopies! · July 20, 2026 · 20 min
• xHC: Expanded Hyper-Connections · July 20, 2026 · 21 min
• Cura 1T: Specialized Model for Agentic Healthcare · July 20, 2026 · 21 min

Explore listener stats, chart rankings, contacts and more on the Daily Paper Cast podcast page.