Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira
From The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI by Astronomer
May 28, 2026 · 33 min · Season 1 · Episode 81
About this episode
Mateus Ferreira discusses the orchestration of over 2,000 Airflow pipelines at Luiza Labs, focusing on data reliability and AI integration.
Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Data Engineer at Luiza Labs (the technology arm of Magazine Luiza, one of Brazil's largest retailers), joins Marc to talk about the patterns his team uses to run 2,000+ Airflow pipelines across more than four petabytes of data. Key Takeaways: 00:00 Introduction 01:11 Mateus introduces himself and Luiza Labs, the technology arm of Magazine Luiza (Magalu), one of Brazil's largest retailers (founded 1957). 1,000+ physical stores, multi-region operations, and a data team that has to handle the variability that comes with all of it. 04:33 Lu Brain, Magalu's AI initiative built around their character Lu, and how AI fits into the data work. 06:47 The data reliability engineering channel where AI summarizes Airflow errors with confidence scores and posts a suggested fix in chat. 08:30 How Airflow became the heart of orchestration. Coming from Control-M in banking, then GCP, then consolidating on Cloud Composer to centralize roughly 2,000…
People in this episode
Host: Marc
Guest: Mateus Ferreira
Topics covered
- Airflow orchestration
- data engineering
- AI integration
- pipeline management
- data reliability
Keywords
- Airflow
- pipelines
- data engineering
- AI
- data reliability
- Luiza Labs
- Magazine Luiza
- Cloud Composer
- metadata
Mentioned in this episode
Organizations: Luiza Labs, Magazine Luiza, Control-M, GCP, Cloud Composer, Lu Brain
Places: Brazil
More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
- Managing a Customer Analytics Platform with Airflow at Skimlinks · June 11, 2026 · 23 min
- Building a custom Tableau provider for Airflow at JLR · June 4, 2026 · 21 min
- Enhancing DAGs for Data Processing with William Orgertrice III at Cargill · May 21, 2026 · 26 min
- Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer · May 14, 2026 · 28 min
- Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development · May 7, 2026 · 25 min
- Building Airflow CTL with Buğra Öztürk at Mollie · April 30, 2026 · 20 min
Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.