About this episode

Julian Larralde discusses Skimlinks' customer analytics platform and the use of Airflow in their data infrastructure.

Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to event-driven Assets, and a YAML-based DAG factory the team built to onboard new publishers without rewriting Python. Key Takeaways 00:00 Introduction. 00:45 What Skimlinks does and how it operates as an affiliate marketing network aggregator for publishers. 02:12 Julian's team and the data platform they own: a reporting portal that serves ~2,000 weekly publisher users. 03:07 The stack: real-time ingestion into BigQuery, Airflow as the orchestrator, raw / silver / gold layers, and Apache Druid as the serving database for sub-second BI queries. 04:50 Reusing the same data marts for ~100 internal customers across marketing, finance, operations, and account management. 06:25 Airflow as the single orchestrator: BigQuery operators for SQL business logic, plus raw file exports for the largest publishers. 08:08 Moving from external task sensors to datasets (now Assets) and what the migration actually…

People in this episode

Guest: Julian Larralde

Topics covered

• customer analytics
• data engineering
• Airflow
• reporting platform
• event-driven architecture
• data migration

Keywords

• Skimlinks
• Airflow
• BigQuery
• data infrastructure
• event-driven Assets
• YAML-based DAG factory
• reporting platform
• data migration

Mentioned in this episode

Organizations: Skimlinks, Apache Airflow, BigQuery, Apache Druid, Rundeck, Airflow

Explore listener stats, chart rankings, contacts and more on the The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI podcast page.