Large Language Models Explore by Latent Distilling

Large Language Models Explore by Latent Distilling

From Daily Paper Cast by Jingwen Liang, Gengyu Wang

May 1, 2026 · 23 min · Episode 1823

About this episode

This episode discusses a new decoding approach for large language models that enhances semantic diversity during generation.

🤗 Upvotes: 56 | cs.CL, cs.AI, cs.LG Authors: Yuanhao Zeng, Ao Lu, Lufei Li, Zheng Zhang, Yexin Li, Kan Ren Title: Large Language Models Explore by Latent Distilling Arxiv: http://arxiv.org/abs/2604.24927v1 Abstract: Generating diverse responses is crucial for test-time scaling of large language models (LLMs), yet standard stochastic sampling mostly yields surface-level lexical variation, limiting semantic exploration. In this paper, we propose Exploratory Sampling (ESamp), a decoding approach that explicitly encourages semantic diversity during generation. ESamp is motivated by the well-known observation that neural networks tend to make lower-error predictions on inputs similar to those encountered before, and incur higher prediction error on novel ones. Building on this property, we train a lightweight Distiller at test time to predict deep-layer hidden representations of the LLM from its shallow-layer representations to model the LLM's depth-wise representation transitions. During decoding, the Distiller continuously adapts to the mappings induced by the current generation context. ESamp uses the prediction error as a novelty signal to reweight candidate token extensions…

People in this episode

Hosts: Jingwen Liang, Gengyu Wang

Topics covered

  • large language models
  • semantic diversity
  • exploratory sampling
  • neural networks
  • decoding approach

Keywords

  • large language models
  • semantic exploration
  • Exploratory Sampling
  • neural networks
  • decoding

Mentioned in this episode

Organizations: Arxiv

More episodes of Daily Paper Cast

Explore listener stats, chart rankings, contacts and more on the Daily Paper Cast podcast page.