Episode 35: Beyond Online Experimentation: Generative Software That Optimizes Itself

About this episode

Martin Tingley discusses the limitations of human involvement in experimentation and the potential of generative AI to optimize software development.

Martin Tingley, Head of Windows Experimentation at Microsoft and former Head of the Experimentation Platform Analysis Team at Netflix, talks about why humans are the bottleneck in experimentation, and how a five-level maturity framework points the way toward self-optimizing software. Our conversation traces the path from basic hypothesis testing to a frontier where Generative AI creates, evaluates, and refines product variants in a closed loop. We explore the architectural shift required to move from testing single variants to optimizing entire parameter spaces, and how startups are already using AI to generate production-ready landing pages for Fortune 500 companies in hours rather than weeks. Tingley also shares a strategic lens on "experimentation programs," explaining how plotting the distribution of treatment effects across different product areas can serve as a powerful tool for capital allocation and high-level strategy. LINKS Martin on LinkedIn Want Your Company to Get Better at Experimentation? by Iavor Bojinov, David Holtz, Ramesh Johari, Sven Schmit and Martin Tingley (Harvard Business Review) Avoid the Pitfalls of A/B Testing by Iavor Bojinov, Guillaume Saint-Jacques…

People in this episode

Host: Delphina

Guest: Martin Tingley

Topics covered

• online experimentation
• generative software
• self-optimizing software
• AI in product development
• capital allocation
• A/B testing

Keywords

• experimentation
• generative AI
• hypothesis testing
• parameter optimization
• landing pages
• Fortune 500
• treatment effects

Mentioned in this episode

Organizations: Microsoft, Netflix, Harvard Business Review

More episodes of High Signal: Data Science | Career | AI

• Episode 40: The Economic Reality of AI: Friction, Talent, and the Future of the Firm · May 26, 2026 · 59 min
• Episode 39: The 100-Year Lead: What Baseball Teaches Us About the Future of AI · May 12, 2026 · 56 min
• Episode 38: Why AI Won’t Fix Your Data Culture, It Will Only Amplify It (And What To Do About It) · April 16, 2026 · 46 min
• Episode 37: Engineered Intelligence and The Data Science Problem in AI · April 2, 2026 · 46 min
• Episode 36: AI and the Judgment Problem in Data Science · March 19, 2026 · 1h 4m
• Episode 34: Duolingo and the Future of Personalized Education with AI · February 10, 2026 · 46 min