From pandas to Arrow: Wes McKinney on the Future of Data Infrastructure

About this episode

Wes McKinney discusses the evolution of data infrastructure from pandas to Apache Arrow and the future of data file formats.

Summary In this episode of Tech on the Rocks , Kostas and Nitay sit down with Wes McKinney the creator of pandas and co-creator of Apache Arrow and Ibis , and long-time leader in the Python data ecosystem. Wes walks us through his journey from building pandas in 2008 to rethinking how we represent and move columnar data with Arrow, and why Arrow is fundamentally different from file formats like Parquet and ORC . We get into the future of data file formats , DataFusion and the new generation of query engines, the rise of open data lakes (Iceberg, Delta, Hudi), and why “big metadata” is becoming just as important as big data. Wes also shares candid thoughts on open source sustainability , how companies and infrastructure projects really survive, and how AI coding agents like Claude Code are changing the day-to-day work of software engineers, especially for complex systems work. If you care about the foundations of modern data infrastructure, or you’ve ever called import pandas as pd, this is an episode you won’t want to miss. Chapters 00:00 Intro — Wes McKinney & his journey in the Python data ecosystem 02:15 How pandas evolved & why UX first mattered for data science 06:14…

People in this episode

Hosts: Kostas, Nitay

Guest: Wes McKinney

Topics covered

• data infrastructure
• open source sustainability
• columnar data
• data file formats
• AI coding agents
• Python data ecosystem

Keywords

• pandas
• Apache Arrow
• data infrastructure
• open source
• AI coding agents
• columnar data
• DataFusion
• metadata

Mentioned in this episode

Organizations: Apache Arrow, pandas, Ibis, DataFusion, Iceberg, Delta, Hudi, Posit

More episodes of Tech on the Rocks

• Physical AI and the Future of Robotics with Sergey Arkhangelskiy of Positronic · June 5, 2026 · 52 min
• Building the Open Lakehouse for the AI Era with Shubham Baldava from DataZip / OLake · May 21, 2026 · 58 min
• From Session Replays to Autonomous Improvement: Shipping the First AI Product Engineer with Milana · April 24, 2026 · 1h 0m
• From Exabyte Storage to Reactive Backends: Jamie Turner on Building Convex After Dropbox · April 9, 2026 · 59 min
• From Art to Science: Wild Moose and the Future of AI-Powered Debugging · March 17, 2026 · 53 min
• From Notebooks to Production: Xorq’s lockfile Approach for Reproducible, Portable ML Pipelines · January 29, 2026 · 57 min