What's New in Apache Airflow® 3.3
About this episode
This episode discusses the new features in Apache Airflow® 3.3 that enhance production pipeline management.
Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer]( astronomer.io ), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off. Key Takeaways: 00:00 Introduction. 01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry. 03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run. 05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action. 06:58 Why XComs don't solve this problem: they get reinitialized on every retry. 09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403. 11:42 Subclassing the retry rule for…
People in this episode
Host: Kenten Danas
Guest: Marc Lamberti
Topics covered
- Apache Airflow
- data engineering
- production pipelines
- task state management
- multi-language orchestration
Keywords
- Airflow 3.3
- task state store
- retry policies
- asset partitioning
- multi-language support
Mentioned in this episode
Organizations: Astronomer
Products: Apache Airflow® 3.3
More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
- Orchestrating Retail Data Pipelines at Saks Global
- Running Airflow 3 in a regulated environment at OTPP
Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.