Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam
About this episode
Ashir Alam discusses managing data quality and governance at Credit Karma using Airflow.
Data quality is not optional when you manage credit data at scale. In this episode, Ashir Alam , Senior Data Engineer at Credit Karma , joins us to share how his team acts as the gatekeeper for credit data ingestion, how they standardize data quality with Airflow and DAG Factory and how they scale safely across thousands of DAGs. We explore how governance, PII protection and orchestration come together inside a modern data platform. Key Takeaways: 00:00 Introduction. 01:00 Overview of Credit Karma’s products and financial data ecosystem. 02:00 The team acts as gatekeepers for ingesting data from TransUnion and Equifax. 03:00 Why PII handling and controlled downstream access led to adopting Airflow. 04:00 BigQuery as the warehouse and Airflow as the primary orchestrator. 05:00 Why data quality and governance are critical in financial systems. 07:00 Why Airflow was selected: ease of use and unified ETL plus data quality. 09:00 Introduction to DAG Factory and YAML-based DAG generation. 10:00 GitHub executor creates PR-driven DAG workflows with CI checks. 12:00 BigQuery operators, structured checks and custom Slack and PagerDuty alerts. 13:00 Failed checks stop ETL pipelines and…
More episodes of The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
- Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik
- Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas
- Introducing Airflow 3.2
- Reflections on a Decade of Data Engineering at Seattle Data Guy
- Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias
- Automating Threat Intelligence Using Airflow with Karan Alang
Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.