Reflections on a Decade of Data Engineering at Seattle Data Guy
About this episode
The episode reflects on a decade of data engineering, discussing the evolution of the field and the role of Apache Airflow.
Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent. In this episode, Benjamin Rogojan , Owner and Data Consultant at Seattle Data Guy , joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is still dominant and how AI is reshaping the workflows and responsibilities of modern data engineers. Key Takeaways: 00:00 Introduction. 03:00 Airflow becomes valuable when workflows involve many pipelines, teams and dependencies. 05:00 Data engineers are still focused on making data accessible and aligning work with business needs. 05:30 Batch pipelines remain the most common approach even as real-time use cases grow. 07:45 Many “real-time” requests are actually event-driven batch workflows. 09:00 Airflow replaced many custom-built pipeline systems with built-in dependency management. 11:00 Modern orchestration tools often build on Airflow concepts or differentiate from them. 14:00 AI can assist with writing SQL and pipelines but still requires experienced engineers. 15:30 Organizations are…
More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
- Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik
- Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas
- Introducing Airflow 3.2
- Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam
- Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias
- Automating Threat Intelligence Using Airflow with Karan Alang
Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.