Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik
About this episode
This episode discusses the newly released Apache Airflow common AI provider and its technical background.
In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next. Kaxil Naik , Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kumar Gopidesu , Lead Data Engineer at Experian and Apache Airflow PMC member, join us to walk through the provider's first release and the technical decisions behind it. Key Takeaways: 00:00 Introduction. 04:05 The common AI provider was born from a real production problem. 07:10 Airflow already had the primitives needed for durable agent execution, making it the natural foundation for AI orchestration. 09:15 The LLM schema compare operator uses Apache DataFusion to fetch source schemas. 11:07 Apache DataFusion was chosen for its speed. 13:09 Hook tool sets expose Airflow's provider hooks to agents with an allowed methods list that blocks destructive operations. 15:20 Passing durable=True to an LLM operator caches tool calls and LLM outputs mid-task. 18:13 The provider offers three abstraction levels. 21:20 The provider currently requires Airflow 3 — the team is open to adding Airflow 2.11 support if demand is high…
More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
- Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas
- Introducing Airflow 3.2
- Reflections on a Decade of Data Engineering at Seattle Data Guy
- Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam
- Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias
- Automating Threat Intelligence Using Airflow with Karan Alang
Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.