Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik

About this episode

This episode discusses the newly released Apache Airflow common AI provider and its technical background.

In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next. Kaxil Naik , Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kumar Gopidesu , Lead Data Engineer at Experian and Apache Airflow PMC member, join us to walk through the provider's first release and the technical decisions behind it. Key Takeaways: 00:00 Introduction. 04:05 The common AI provider was born from a real production problem. 07:10 Airflow already had the primitives needed for durable agent execution, making it the natural foundation for AI orchestration.  09:15 The LLM schema compare operator uses Apache DataFusion to fetch source schemas. 11:07 Apache DataFusion was chosen for its speed. 13:09 Hook tool sets expose Airflow's provider hooks to agents with an allowed methods list that blocks destructive operations. 15:20 Passing durable=True to an LLM operator caches tool calls and LLM outputs mid-task.  18:13 The provider offers three abstraction levels.  21:20 The provider currently requires Airflow 3 — the team is open to adding Airflow 2.11 support if demand is high…

People in this episode

Guests: Kaxil Naik, Pavan Kumar Gopidesu

Topics covered

• Apache Airflow
• AI orchestration
• data engineering
• technical decisions
• common AI provider
• production problem

Keywords

• AI provider
• Airflow 3
• durable agent execution
• LLM schema compare operator
• MCP server configs

Mentioned in this episode

Organizations: Astronomer, Experian

Products: Apache Airflow, Apache DataFusion

More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• Managing a Customer Analytics Platform with Airflow at Skimlinks · June 11, 2026 · 23 min
• Building a custom Tableau provider for Airflow at JLR · June 4, 2026 · 21 min
• Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira · May 28, 2026 · 33 min
• Enhancing DAGs for Data Processing with William Orgertrice III at Cargill · May 21, 2026 · 26 min
• Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer · May 14, 2026 · 28 min
• Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development · May 7, 2026 · 25 min