Episode 72: Why Agents Solve the Wrong Problem (and What Data Scientists Do Instead)
From Vanishing Gradients by Hugo Bowne-Anderson
March 20, 2026 · 1h 34m
About this episode
Hugo Bowne-Anderson and Bryan Bischof discuss the shortcomings of AI agents in data science evaluations and propose more realistic problem-solving approaches.
I often see what I would consider to be b******t evals , especially in data, like write this dumb SQL . Almost every one of these dumb SQL questions that I’ve seen for benchmarks are just so either obviously easy or overwhelmingly adversarial. They just, they don’t feel valuable as a data scientist , it’s something that you probably would never ask a real data scientist to do. So I went out my way to create real ones. Let me read one to you. Bryan Bischof , Head of AI at Theory Ventures , joins Hugo to talk about what happened when 150 people spent six hours using AI agents to answer real data science questions across SQL tables , log files , and 750,000 PDFs . They Discuss: * Failure Funnels , pinpoint where agent reasoning breaks down using causal-chain binary evaluations instead of vague 1-5 scales; * Median Score: 23 out of 65 , what happened when world-class engineers turned agents loose on real data work, and why general-purpose coding agents with human prodding beat fancy frameworks; * Zero-Cost Submissions Kill Trust , without a penalty for wrong answers, agents hill-climb to correct submissions through brute force instead of building confidence; * Data Science is…
People in this episode
Host: Hugo Bowne-Anderson
Guest: Bryan Bischof
Topics covered
- data science
- AI agents
- SQL
- evaluation methods
- problem framing
- causal reasoning
Keywords
- data science
- AI agents
- SQL
- evaluation
- problem framing
- trust
- causal reasoning
Mentioned in this episode
Organizations: Theory Ventures
More episodes of Vanishing Gradients
- The Future of Agentic Data Science · May 25, 2026 · 1h 5m
- Agent-Harness.ipynb* · May 20, 2026 · 1h 20m
- Agentic Engineering and the Lost Art of Verification · May 12, 2026 · 1h 32m
- Next Level AI Evals for 2026 · April 23, 2026 · 54 min
- Privacy Theater Is Not Privacy Engineering: What It Actually Takes to Ship Safe AI · April 15, 2026 · 1h 7m
- LLM Architecture in 2026: What You Need to Know with Sebastian Raschka · April 13, 2026 · 1h 18m
Explore listener stats, chart rankings, contacts and more on the Vanishing Gradients podcast page.