Scaling Airflow at Wix for Analytics and AI with Ethan Shalev
From The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI by Astronomer
February 26, 2026 · 18 min · Season 1 · Episode 70
About this episode
Ethan Shalev discusses how Wix operates Airflow at scale and the challenges of migrating to Airflow 3 while leveraging AI for development.
Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge. In this episode, we’re joined by Ethan Shalev , Data Engineer at Wix , to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development. Key Takeaways: 00:00 Introduction. 02:13 Wix structures data engineering across multiple product-focused organizations. 03:40 Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning. 04:31 Migration creates an opportunity to remove long-standing legacy Airflow code. 05:32 Internal playbooks and Cursor rules standardize and speed up DAG migrations. 07:39 Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling. 09:16 Deferrable operators reduce scheduler congestion in large Airflow environments. 12:54 AI-generated code still requires review and strong testing practices. 14:52 Moving to managed Airflow reduces operational burden on internal platform teams. 15:57 Improving multi-tenancy and UI personalization remains a key Airflow need. Resources Mentioned: Ethan…
People in this episode
Guest: Ethan Shalev
Topics covered
- data orchestration
- Airflow migration
- AI in data engineering
- scalability
- data pipelines
- multi-tenancy
Keywords
- Airflow 3
- DAGs
- data engineering
- AI-generated code
- managed Airflow
- scheduler congestion
- multi-tenancy
Mentioned in this episode
Organizations: Wix, Apache Airflow, Astronomer, Trino, Apache Iceberg, Cursor
More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
- Managing a Customer Analytics Platform with Airflow at Skimlinks · June 11, 2026 · 23 min
- Building a custom Tableau provider for Airflow at JLR · June 4, 2026 · 21 min
- Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira · May 28, 2026 · 33 min
- Enhancing DAGs for Data Processing with William Orgertrice III at Cargill · May 21, 2026 · 26 min
- Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer · May 14, 2026 · 28 min
- Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development · May 7, 2026 · 25 min
Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.