About this episode

Kowsy Narayan discusses the challenges and strategies of running Apache Airflow in a regulated environment at the Ontario Teachers' Pension Plan.

Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan]( otpp.com ), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution. Key Takeaways: 00:00 Introduction. 01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement. 02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience. 02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle. 04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC. 05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode. 08:00 Upgrading from Airflow 2.9 to Airflow 3, using…

People in this episode

Host: Kenten Danas

Guest: Kowsy Narayan

Topics covered

• cloud migration
• data engineering
• Apache Airflow
• compliance
• remote execution
• data platform

Keywords

• Airflow 3
• cloud architecture
• data orchestration
• compliance requirements
• remote execution
• data platform team
• performance boost

Mentioned in this episode

Organizations: Ontario Teachers' Pension Plan

Products: Apache Airflow, Snowflake, dbt, dbt Cosmos, Astro CLI

Places: Kubernetes

More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• Orchestrating Retail Data Pipelines at Saks Global · July 16, 2026 · 25 min
• What's New in Apache Airflow® 3.3 · July 9, 2026 · 27 min

Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.