Reflections on a Decade of Data Engineering at Seattle Data Guy
From The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI by Astronomer
April 3, 2026 · 26 min · Season 1 · Episode 75
About this episode
The episode reflects on a decade of data engineering, discussing the evolution of the field and the role of Apache Airflow.
Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent. In this episode, Benjamin Rogojan , Owner and Data Consultant at Seattle Data Guy , joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is still dominant and how AI is reshaping the workflows and responsibilities of modern data engineers. Key Takeaways: 00:00 Introduction. 03:00 Airflow becomes valuable when workflows involve many pipelines, teams and dependencies. 05:00 Data engineers are still focused on making data accessible and aligning work with business needs. 05:30 Batch pipelines remain the most common approach even as real-time use cases grow. 07:45 Many “real-time” requests are actually event-driven batch workflows. 09:00 Airflow replaced many custom-built pipeline systems with built-in dependency management. 11:00 Modern orchestration tools often build on Airflow concepts or differentiate from them. 14:00 AI can assist with writing SQL and pipelines but still requires experienced engineers. 15:30 Organizations are…
People in this episode
Guest: Benjamin Rogojan
Topics covered
- data engineering
- Apache Airflow
- AI in data workflows
- batch processing
- data accessibility
Keywords
- data engineering
- Apache Airflow
- batch processing
- AI
- data workflows
- data accessibility
- cloud platforms
Mentioned in this episode
Organizations: Seattle Data Guy
Products: Apache Airflow
More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
- Managing a Customer Analytics Platform with Airflow at Skimlinks · June 11, 2026 · 23 min
- Building a custom Tableau provider for Airflow at JLR · June 4, 2026 · 21 min
- Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira · May 28, 2026 · 33 min
- Enhancing DAGs for Data Processing with William Orgertrice III at Cargill · May 21, 2026 · 26 min
- Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer · May 14, 2026 · 28 min
- Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development · May 7, 2026 · 25 min
Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.