The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering
About this episode
Wes McKinney discusses the evolution of data engineering, the impact of AI on software development, and the importance of good engineering practices.
The creator of Pandas and co-creator of Apache Arrow , Wes McKinney , joins the Data Engineering Central Podcast for an in-depth conversation about how modern data engineering came to exist, where AI is taking software development, and why good engineering still matters more than ever. We start with Wes’ journey from building GoldenEye fan websites as a teenager to creating Pandas while working at a quantitative hedge fund, and eventually launching Apache Arrow, one of the foundational technologies behind today’s modern data ecosystem. Along the way, we discuss Cloudera, Parquet, DuckDB, DataFusion, Spark, and how the industry evolved from Hadoop to today’s lakehouse architectures. Thanks for reading Data Engineering Central! This post is public so feel free to share it. The second half of the conversation dives deep into AI. Wes explains why large language models make experienced engineers more productive but won’t magically replace software engineering, why architecture and good taste are becoming more valuable than writing individual lines of code, and why projects like DuckDB and * Apache Arrow remains incredibly difficult to recreate with AI alone. We also discuss…
More episodes of Data Engineering Central Podcast
- What Happens When a Software Engineer Builds a Company Alone?
- From COBOL to Copilot: 30 Years of Data, BI, and AI with David Langer
- Semantic Layers, Agents, and the Future of Analytics
- The Future of the Lakehouse: Delta Lake, Rust, and Data Platforms at Scale
Explore listener stats, chart rankings, contacts and more on the Data Engineering Central Podcast podcast page.