Scaling Airflow at Wix for Analytics and AI with Ethan Shalev
About this episode
Ethan Shalev discusses how Wix operates Airflow at scale and the challenges of migrating to Airflow 3 while leveraging AI for development.
Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge. In this episode, we’re joined by Ethan Shalev , Data Engineer at Wix , to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development. Key Takeaways: 00:00 Introduction. 02:13 Wix structures data engineering across multiple product-focused organizations. 03:40 Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning. 04:31 Migration creates an opportunity to remove long-standing legacy Airflow code. 05:32 Internal playbooks and Cursor rules standardize and speed up DAG migrations. 07:39 Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling. 09:16 Deferrable operators reduce scheduler congestion in large Airflow environments. 12:54 AI-generated code still requires review and strong testing practices. 14:52 Moving to managed Airflow reduces operational burden on internal platform teams. 15:57 Improving multi-tenancy and UI personalization remains a key Airflow need. Resources Mentioned: Ethan…
More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
- Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik
- Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas
- Introducing Airflow 3.2
- Reflections on a Decade of Data Engineering at Seattle Data Guy
- Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam
- Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias
Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.