The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off.Key Takeaways:00:00 Introduction.01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry.03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run.05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action.06:58 Why XComs don't solve this problem: they get reinitialized on every retry.09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403.11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase.15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs.21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator.23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run.25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting.Resources Mentioned:AstronomerAirflow 3.3 Release NotesThe Task State StoreUpdates to the asset partitions featureRetry policiesMulti-language support3.3 WebinarUpgrading from Airflow 2 to 3 ebookThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow

A strong data-driven mindset underpins how fintech teams scale analytics, infrastructure and decision-making across the business.In this episode, Jaime Oliveira, Lead Data Engineer at Uphold, joins us to discuss how Uphold structures its data organization and orchestration strategy. Jaime shares how the team uses Airflow and dbt to support analytics, reporting and data activation while evolving their approach as the stack grows.Key Takeaways:00:00 Introduction.01:23 A data-driven mindset supports product development and business decisions.02:55 Diverse ingestion pipelines enable scalable analytics.04:18 A single orchestration platform simplifies analytics workflows.05:17 Early experience with orchestration tools shapes engineering practices.08:16 Analytics orchestration works best when aligned with transformation workflows.09:25 Infrastructure choices involve tradeoffs in testing, visibility and overhead.16:39 More collaborative workflow tools could improve accessibility and autonomy.Resources Mentioned:Jaime Oliveirahttps://www.linkedin.com/in/jaime-oliveira-b075855a/Uphold | LinkedInhttps://www.linkedin.com/company/upholdinc/Uphold | Websitehttps://uphold.comApache Airflowhttps://airflow.apache.orgdbthttps://www.getdbt.comSnowflakehttps://www.snowflake.comKuberneteshttps://kubernetes.ioAstronomer Cosmoshttps://astronomer.github.io/astronomer-cosmosCosmos e-bookhttps://www.astronomer.io/ebooks/orchestrating-dbt-with-airflow-using-cosmos/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow

Conviva operates at a massive scale, delivering outcome-based intelligence for digital businesses through real-time and batch data processing. As new use cases emerged, the team needed a way to extend a streaming-first architecture without rebuilding core systems.In this episode, Han Zhang joins us to explain how Conviva uses Apache Airflow as the orchestration backbone for its batch workloads, how the control plane is designed and what trade-offs shaped their platform decisions.Key Takeaways:00:00 Introduction.01:17 Large-scale data platforms require low-latency processing capabilities.02:08 Batch workloads can complement streaming pipelines for additional use cases.03:45 An orchestration framework can act as the core coordination layer.06:12 Batch processing enables workloads that streaming alone cannot support.08:50 Ecosystem maturity and observability are key orchestration considerations.10:15 Built-in run history and logs make failures easier to diagnose.14:20 Platform users can monitor workflows without managing orchestration logic.17:08 Identity, secrets and scheduling present ongoing optimization challenges.19:59 Configuration history and change visibility improve operational reliability.Resources Mentioned:Han Zhanghttps://www.linkedin.com/in/zhanghan177Conviva | Websitehttp://www.conviva.comApache Airflowhttps://airflow.apache.org/Celeryhttps://docs.celeryq.dev/Temporalhttps://temporal.io/Kuberneteshttps://kubernetes.io/LDAPhttps://ldap.com/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow

The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards.Max Calehuff, Lead Data Engineer at Vivian Health, joins us to discuss how his team uses Airflow for ML ops, regulatory compliance and large-scale data orchestration. He also shares insights into upgrading to Airflow 3 and the importance of balancing flexibility with security in a healthcare environment.Key Takeaways:00:00 Introduction.04:21 The role of Airflow in managing ETL pipelines and ML retraining.06:23 Using AWS SageMaker for ML training and deployment.07:47 Why Airflow’s versatility makes it ideal for MLOps.10:50 The importance of documentation and best practices for engineering teams.13:44 Automating anonymization of user data for compliance.15:30 The benefits of remote execution in Airflow 3 for regulated industries.18:16 Quality-of-life improvements and desired features in future Airflow versions.Resources Mentioned:Max Calehuffhttps://www.linkedin.com/in/maxwell-calehuff/Vivian Health | LinkedInhttps://www.linkedin.com/company/vivianhealth/Vivian Health | Websitehttps://www.vivian.comApache Airflowhttps://airflow.apache.org/Astronomerhttps://www.astronomer.io/AWS SageMakerhttps://www.google.com/aclk?sa=L&ai=DChsSEwj3-fbz1tiQAxWXlKYDHXUBBVoYACICCAEQABoCdGI&ae=2&aspm=1&co=1&ase=2&gclid=Cj0KCQiA5abIBhCaARIsAM3-zFWbfj2olUvX4dqoiYNaE3q2fMf_ZifRjmbKNQCVX7D6ZMClaUXUkFkaAuwmEALw_wcB&cid=CAASQuRoMccxWhBvMq-1Uez3XOZti1ul7mTDotKvSMoDHv0q2xCsyS2FzMptO5dJf3tmfkLRu22TtD8ChTmdjvs6YetTjQ&cce=2&category=acrcp_v1_35&sig=AOD64_2xE2xolEEVbpDb56qXQluxTzs-Aw&q&nis=4&adurl&ved=2ahUKEwj7le3z1tiQAxWXcvUHHfZePbAQ0Qx6BAgUEAEdbtLabshttps://www.getdbt.com/Cosmoshttps://github.com/astronomer/astronomer-cosmosSplithttps://www.split.io/Snowflakehttps://www.snowflake.com/en/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow