LLM as a Judge: Why Your AI Might Be Marking Its Own Homework

LLM as a Judge: Why Your AI Might Be Marking Its Own Homework

From Coding Chats by John Crickett

April 30, 2026 · 1h 7m

About this episode

John talks to Laura Dietz about the implications of using AI for evaluation in software development.

Coding Chats episode 76 - John talks to Laura Dietz - a computer science professor whose work focuses on whether AI evaluation metrics actually tell the truth. She's known for her critical take on "LLM as a judge" — not because she thinks it's useless, but because she wants numbers that mean something rather than numbers that just make a system look good. The conversation tackles some uncomfortable realities for software engineers: using an LLM to write code and another to review it is a circular trap, prompt engineering shouldn't be a computer scientist's day job, and every time you reject your code AI's output, you're quietly generating the training data that shapes its successor. Chapters 00:00 Introduction to Laura Dietz and Her Journey 03:12 Exploring LLMs as Judges 06:16 Challenges in Evaluating Search Systems 08:49 The Evolution of User Queries and Expectations 11:46 The Role of LLMs in Information Retrieval 14:44 Defining Quality in Search Results 17:27 The Complexity of User Intent 19:54 Human-AI Collaboration in Code Review 22:53 The Future of LLMs in Software Development 25:23 Balancing Human and AI Roles 28:20 Innovative Approaches to AI Evaluation 34:10 The…

People in this episode

Host: John Crickett

Guest: Laura Dietz

Topics covered

  • AI evaluation
  • LLM
  • software development
  • code review
  • human-AI collaboration
  • prompt engineering

Keywords

  • AI evaluation metrics
  • LLM as a judge
  • software engineering
  • training data
  • code review
  • human-AI collaboration

Mentioned in this episode

Organizations: computer science

More episodes of Coding Chats

Explore listener stats, chart rankings, contacts and more on the Coding Chats podcast page.