“Learning zero, and what SLT gets wrong about it” by Dmitry Vaintrob

About this episode

Dmitry Vaintrob discusses Singular Learning Theory and its implications for understanding data degeneracy in machine learning.

This is a first in a pair of posts I'm hoping to write about Singular Learning Theory (SLT) and singularities as a model of data degeneracy. If I get to it, the second post is going to be more general-audience; this one is more technical. Introduction To me, SLT is an important source of toy models which point at an interesting class of new statistical phenomena in learning. It is also a valuable correction to an older and (at this point) largely-defunct story of learning being fully controlled by Hessian eigenvalues and "nonsingular basins". Practitioners of SLT have been instrumental for developing and refining the practice of Bayesian sampling (used by physicists in papers like this one) to empirical models. And the theory's founder Sumio Watanabe is a once-in-a-generation genius who saw and mathematically justified crucial statistical and information-theoretic concepts in learning before long before they appeared in "mainstream" ML theory. However there is a frequently repeated statement in SLT papers – one that doesn't affect empirical results – which I think is wrong in a load-bearing way. This is the statement that models that appear in machine learning are singular in the…

People in this episode

Guest: Dmitry Vaintrob

Topics covered

• Singular Learning Theory
• data degeneracy
• Bayesian sampling
• machine learning
• statistical phenomena

Keywords

• Singular Learning Theory
• data degeneracy
• Bayesian sampling
• Hessian eigenvalues
• machine learning

More episodes of LessWrong (30+ Karma)

• “The Uncertainty That Matters Isn’t Fundamental” by jimmy · June 13, 2026 · 22 min
• [Linkpost] “US government directive to suspend access to Fable 5 and Mythos 5” by Capybasilisk · June 13, 2026 · 0 min
• “Claude Fable 5 and Mythos 5: The System Card” by Zvi · June 12, 2026 · 60 min
• “Citations Needed: Magic Encyclopedias to Save the World” by Oliver Sourbut · June 12, 2026 · 10 min
• “Simulating Simulators” by kromem · June 12, 2026 · 27 min
• “Implications of Continual Learning for LLM Agents: Introduction” by RohanS, Rauno Arike, Owen Terry, Achu Menon, Zhijing Jin, Francis Rhys Ward, Seth Herd · June 12, 2026 · 14 min