Data Engineering Central Podcast

In this episode of the Data Engineering Central Podcast, I sit down with David Jaitillake to explore the future of data engineering, analytics, and AI. David has spent nearly two decades working across data teams, from analyst roles in the early SQL Server days to leading teams, founding startups, serving as VP of AI at Cube, and now co-founding Quarry.We discuss why semantic layers have suddenly become one of the most important concepts in modern data platforms, how tools like Claude Code are transforming engineering workflows, and why the core problems in data haven’t really changed despite massive advances in technology.David shares his perspective on where agentic workflows are headed, what AI means for junior engineers entering the field, and why experienced practitioners may be more valuable than ever before. We also dive into the evolution of data platforms, lessons learned from startups, the promise of tools like DuckDB and MotherDuck, and how organizations should think about adopting AI responsibly.Thanks for reading Data Engineering Central! This post is public so feel free to share it.Whether you’re a data engineer, analytics engineer, engineering leader, or someone trying to understand where the industry is headed, this conversation offers a practical and honest look at what’s coming next.What We Cover* David’s journey from analyst to startup founder* The rise of semantic layers and why they matter* Why data modeling is still critical in the AI era* How AI coding agents are changing engineering work* What Claude Code is enabling today* The future of agentic data pipelines* Why DuckDB and MotherDuck are gaining traction* The challenges facing junior engineers* Career advice for data professionals at every stage* Whether David is optimistic about the future of AI and dataConnect with David:* LinkedIn: https://www.linkedin.com/in/david-jayatillake/* Substack: Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode, I sit down with industry veteran Robin Moffatt — Sr. Principal Advisor in Streaming Data Technologies (Kafka, etc.) and a longtime voice in the data engineering community, to unpack the journey from old-school data architectures to today’s real-time streaming ecosystems. From early mainframe data processing and COBOL through the rise of Apache Kafka, streaming ETL, and event-driven systems, Robin shares lived experience from across decades of building, scaling, and evolving data platforms.We dive into:* 🧠 How the role of software engineering has shifted with the rise of distributed, real-time systems* 📊 Why event streaming and platforms like Kafka aren’t just messaging systems, but the backbone of modern data architectures* 🚀 How the community’s tooling and mental models have had to evolve — from static databases and nightly jobs to continuous, always-on streaming applications* 🤖 A candid look at how AI and real-time data are intersecting, shaping both tooling and expectations for the next decade* 🔮 Robin’s perspective on where the industry is headed — beyond buzzwords toward real engineering maturityAlong the way, we get historical context, real-world lessons from conference stages and community forums, and a perspective on building resilient, scalable systems that power today’s data-rich applications.If you’ve ever wondered how we got from batch jobs to continuous event streams, or what it really takes to build modern pipelines that support AI workflows, this conversation with Robin is a must-listen.For more from Robin:* 📍 His personal blog & talks: https://rmoff.net/* 🔗 LinkedIn profile: https://www.linkedin.com/in/robinmoffattThanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode of the Data Engineering Central Podcast, I sit down with Hoyt Emerson, founder of The Full Data Stack and Early Signal, for a wide-ranging conversation on data, analytics, and creating content in the tech world.We talk candidly about:* What actually matters in modern data and analytics* Why so much “data content” misses the mark* The difference between noise and real signal* What works (and doesn’t) when building a technical audience* Writing, consistency, and credibility in the data space* Why opinions + experience beat trends and buzzwordsIf you’re a data engineer, analyst, or technologist who’s curious about both building better data systems and communicating ideas that resonate, this episode goes deep on the lessons learned from doing both.This is less about hacks—and more about craft, judgment, and long-term thinking.Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

In this episode of the Data Engineering Central Podcast, I interview a Data OG, someone who’s been around the data space forever, and we talked about all things data, past, present, and future.I’m joined by Thomas Horton a longtime friend and one of the most well-rounded data professionals I know. Over the course of his career, Tom has worn just about every hat in data: developer, DBA, analyst, and everything in between. He’s lived through the era of on-prem databases, the rise of analytics, and the constant reinvention that defines modern data engineering today.We talk about what’s changed, what hasn’t, and why many of the “new” problems in data feel oddly familiar. We also dig into lessons learned the hard way, lessons that are just as relevant for early-career data engineers as they are for seasoned practitioners navigating today’s ever-expanding stacks.On a personal note, a huge portion of what I know about relational databases and analytics can be traced back to Tom. This conversation is part reflection, part history lesson, and part reality check on where the data industry is headed next.* If you’re interested in the past, present, and future of data—and what really matters beneath all the tooling, this is an episode you won’t want to miss.Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

Hello! A new episode of the Data Engineering Central Podcast is dropping today. We will be covering a few hot topics!* Cluster Fatigue* The Death of Open SourceGoing to be a great show, come along for the ride!Thanks for reading Data Engineering Central! This post is public so feel free to share it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe