Data Engineering Central Podcast

Most data teams think they’re building value. In reality, they’ve become ticket queues.In this episode, Chris Gambill explains his storied career in tech and data through the years, dealing with data at Fortune 500 company scale, and breaking out on his own.We cover career growth, what separates senior engineers from true strategic operators, and the biggest mistakes people make early on. We discuss the classic problems that have plagued data teams for decades and why it’s all still a struggle.Today’s podcast is sponsored by Estuary.Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below.Build millisecond-latency, scalable, future-proof data pipelines in minutes.Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best-in-class CDC (Change Data Capture).Estuary unifies today’s batch and streaming paradigms so that your systems, current and future, are synchronized around the same datasets, updating in milliseconds.We also dig into Databricks vs Snowflake, what matters and what doesn’t, and how to think about modern data architecture without falling for marketing hype.* On the AI side, we talk about why most LLMs, in the context of developer lifecycles, have changed how we do data, and also about what human skills cannot be replaced.If you care about leveling up beyond just building pipelines, this one is for you.Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode of Data Engineering Central, I sit down with Kevin, who spent 13 years working at Apple before walking away at the end of 2025.* Not to jump to another job.* Not to start a company.* But to take a step back from everything.Kevin shares his full journey—from growing up in the suburbs of Atlanta to building a career at Apple, and ultimately reaching the point where he could walk away financially and mentally.You can follow along with Kevin below.We dive deep into what it’s really like working in tech: the high salaries, the lifestyle creep, the pressure, and the surprising reality that even people making great money often have no clear financial plan.This conversation also explores the rise of FIRE (Financial Independence, Retire Early), how Kevin discovered it through Mr. Money Mustache, and why his perspective on it has changed over time.Thanks for reading Data Engineering Central! This post is public so feel free to share it.What starts as a path to freedom can easily turn into a scarcity mindset—and that’s something most people don’t talk about.We also get into:* Why high income does not equal financial freedom* The hidden trap of lifestyle inflation in tech* The simple investing strategy that actually works (and why most people ignore it)* Why many engineers are “close” to freedom—but never pull the trigger* The psychology of money, status, and why people stay stuck* How a failed project and burnout became a turning point* And how Kevin went from overworked and unhealthy… to climbing mountains and preparing to backpack 1,000 milesThis is not your typical “get rich quick” or “retire at 30” conversation. It’s a grounded, honest look at money, work, and what it actually takes to build a life you don’t need to escape from.If you work in tech, think about FIRE, or just feel like you’re stuck on the treadmill, this one will hit home.Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode, I sit down with Matt Martin, Staff Engineer, data architect, ETL practitioner, and author of a new book on DuckDB coming soon, to talk about the past, present, and future of data engineering.Matt has spent decades building and architecting data platforms across technologies such as SQL Server, Oracle, DB2, Hadoop, Redshift, and BigQuery, and now focuses on modern tools such as DuckDB and single-node analytics.We discuss how the data industry has evolved, what actually makes data platforms succeed, and where tools like DuckDB, Polars, Databricks, and Snowflake fit into the future of analytics.We also dive into the impact of AI on coding and data engineering, and whether distributed compute clusters will remain dominant — or if more workloads will move toward high-performance single-node systems.Topics Covered* Matt’s early career and journey into data engineering* The evolution of data warehousing and ETL frameworks* Traditional enterprise data systems vs modern cloud platforms* DuckDB and the rise of single-node analytics* Polars vs DuckDB: where each tool shines* Databricks vs Snowflake* AI-assisted coding and its impact on engineers* The current data engineering job market* Lessons learned from decades of building data systems* Writing a book on DuckDB This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode of the Data Engineering Central Podcast, I sit down with Yuki (Yuki Kakegawa) to talk about his journey into tech, the tools and platforms he’s worked with, and where he thinks data engineering and AI are headed next.We cover:• How Yuki got into tech• Early career lessons and pivots• Tools and technologies he’s worked with over the years• How data engineering has evolved• The impact of AI on software development• What engineers should focus on right now• Advice for those building their careers in dataYuki shares practical insights on navigating the industry, staying adaptable, and thinking long-term about your technical growth.If you’re a data engineer, aspiring engineer, or just interested in where AI and modern software are going, this one’s for you.Yuki writes on …LinkedIn - https://www.linkedin.com/in/yukikakegawa/https://yukikakegawa.me/#blogThanks for reading Data Engineering Central! This post is public so feel free to share it.🔔 Subscribe for more interviews with leaders in data engineering, AI, and modern data platforms. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode of the Data Engineering Central Podcast, I sit down with Maxine Meurer, DevOps engineer, author, and educator behind I Love DevOps, for a wide-ranging conversation about careers, infrastructure, automation, and what it actually means to build systems that last.This isn’t a buzzword-heavy DevOps chat. It’s a grounded, honest discussion between two engineers about how people really get into tech, how careers evolve over time, and why modern infrastructure is as much about systems thinking and human judgment as it is about tools.We talk through Maxine’s journey from early technical curiosity to hands-on DevOps work, dealing with “ClickOps” to automation-first infrastructure, and how writing and teaching reshaped the way she thinks about engineering.What we cover in this episode:* 🛠️ From ClickOps to DevOps — what that transition actually looks like in the real world* 🧠 Why DevOps is fundamentally about systems and people, not just pipelines and YAML* 📚 How Maxine went from self-teaching to authoring practical guides like LLMs for Humans and The DevOps Career Switch Blueprint* 🤯 Common mistakes engineers make when learning DevOps, cloud, and distributed systems* 🔍 Testing failures, production realities, and where modern infrastructure still breaks down* 🤖 What AI and LLMs actually change for engineers, and what’s mostly hype* 🧭 Career advice for engineers without a traditional background* 🔮 Where DevOps and platform engineering are heading over the next 3–5 yearsThroughout the conversation, Maxine brings a refreshing, human-centered perspective to topics that are often over-abstracted or oversold. We dig into the tradeoffs behind tooling choices, the reality of production systems, and the importance of learning how to think, not just what to deploy.If you’re navigating a DevOps or infrastructure career, wrestling with modern stacks, or trying to make sense of AI’s role in engineering, this episode offers clarity, context, and hard-won insight.Learn more about Maxine’s work:* Writing & guides: * LinkedIn: https://www.linkedin.com/in/maxinemeurer/* Gumroad resources: https://mameurer.gumroad.comThanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode of the Data Engineering Central Podcast, I sit down with R. Tyler Croy for a wide-ranging conversation on the present—and future—of modern data platforms.Tyler is a long-time open-source contributor to projects such as delta-rs. You can watch him on YouTube, read his blog, or work directly with him through his consultancy, Buoyant Data.Tyler has spent years deep in the open-source data ecosystem, contributing to projects such as Delta Lake and thinking critically about how real-world data systems are built and maintained. This isn’t a hype-driven conversation—it’s a grounded discussion about what’s working, what’s breaking, and what’s coming next.We dig into:* What the Lakehouse architecture gets right—and where it still falls short* Why multimodal data (text, images, audio, video, embeddings) changes everything* How open table formats like Delta Lake fit into the next generation of data platforms* The growing gap between data tooling hype and day-to-day data engineering reality* What skills and architectural thinking will matter most for data engineers over the next decadeIf you’re building or operating modern data platforms—and trying to separate real signal from noise—this episode is for you.Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode of the Data Engineering Central Podcast, I sit down with Andy Leonard — someone who’s been building systems long before “data engineering” was even a job title.Andy’s career didn’t start in software at all. It started with physical circuits, literally wiring systems as an electrician, before moving into programming, databases, and eventually decades of hands-on data engineering work.This conversation isn’t about trends or hype cycles. It’s about how the fundamentals of data work have evolved, what hasn’t changed, and what you only learn after years of building, breaking, fixing, and rebuilding real systems.We talk about how the industry got here, how tools have changed, where they haven’t helped as much as advertised, and what newer data engineers can learn from a long, practical career spent close to the metal.If you’re interested in perspective, experience, and lessons earned the hard way — this one’s for you.Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode, I sit down with Scott Haines — O’Reilly author, Databricks MVP, and veteran of Yahoo, Nike, and Twilio — for a wide-ranging conversation on the real state of modern data engineering. We dig into open-source ecosystems, Lakehouse architectures, the evolution of Spark, streaming, what’s broken and what’s working in today’s data tooling, and the lessons Scott has learned scaling platforms at some of the biggest companies in the world.If you care about data engineering, architecture, OSS, or the future of the modern data stack, you’ll love this one.Thanks for reading Data Engineering Central! This post is public so feel free to share it.Make sure to follow Scott here on Substack, and over on GitHub. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

This is a free preview of a paid episode. To hear more, visit dataengineeringcentral.substack.comHello! A new episode of the Data Engineering Central Podcast is dropping today, we will be covering a few hot topics!* Apache Iceberg Catalogs* new Boring Catalog* new full Iceberg support from Databricks/Unity Catalog* Databricks SQL Scripting* DuckDB coming to a Lake House near you* Lakebase from DatabricksGoing to be a great show, come along for the ride!Thanks …

This is a free preview of a paid episode. To hear more, visit dataengineeringcentral.substack.comIt’s time for another episode of the Data Engineering Central Podcast. In this episode, we cover …* Rust-based tool called UV to replace pip and poetry etc* Apache X-Table and the Future of the Lake House* How is AI going to affect you?Thanks for being a consumer of Data Engineering Central; your support means a lot. Please share this podcast with your friend…

In todays episode of Data Engineering Central Podcast we talk about a few hot topics, AWS S3 Tables, Databricks raising money, are Data Contracts Dead, and the Lake House Storage Format battle!It's a good one, buckle up! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

It’s time for another episode of Data Engineering Central Podcast, our third one! Topics in this episode …* Should you use DuckDB or Polars?* Small Engineering Changes (PR Reviews)* Daft vs Spark on Databricks with Unity Catalog (Delta Lake)* Primary and Foreign keys in the Lake HouseEnjoy! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe