Developer Voices

How hard is it to write a good database engine? Hard enough that sometimes it takes several versions to get it just right. Paul Dix joins us this week to talk about his journey building InfluxDB, and he's refreshingly frank about what went right, and what went wrong. Sometimes the real database is the knowledge you pick up along the way.... Paul walks us through InfluxDB's evolution from error logging system to time-series datasbase, and from Go to Rust, with unflinching honesty about the major lessons they learnt along the way. We cover technical details like Time-Structure Merge Trees, to business issues like what happens when your database works but your pricing model is broken. If you're interested in how databases work, this is full of interesting details, and if you're interested in how projects evolve from good idea to functioning business, it's a treat. -- Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join InfluxData: https://www.influxdata.com/ InfluxDB: https://www.influxdata.com/products/influxdb/ DataFusion: https://datafusion.apache.org/ DataFusion Episode: https://www.youtube.com/watch?v=8QNNCr8WfDM Apache Arrow: https://arrow.apache.org/ Apache Parquet: https://parquet.apache.org/ BoltDB: https://github.com/boltdb/bolt LevelDB: https://github.com/google/leveldb RocksDB: https://rocksdb.org/ Gorilla: A Fast, Scalable, In-Memory Time Series Database (Facebook paper): https://www.vldb.org/pvldb/vol8/p1816-teller.pdf Paul on LinkedIn: https://www.linkedin.com/in/pauldix/ Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Ever wondered why data integration is still such a nightmare in 2025? Marty Pitt has built something that might finally solve it. TaxiQL isn't just another query language - it's a semantic layer that lets you query across any system without caring about field names, API differences, or where the data actually lives. Instead of writing endless mapping code between your microservices, databases, and APIs, you describe what your data *means* and let TaxiQL figure out how to get it. In this conversation, Marty walks through the "All Powerful Spreadsheet" moment that sparked TaxiQL, how semantic types work in practice, and why this approach might finally decouple producers from consumers in large organizations. We dive deep into query execution, data lineage, streaming integration, and the technical challenges of building a system that can connect anything to anything. If you've ever spent months mapping fields between systems or maintaining brittle integration code, this one's for you. – Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join – TaxiLang Homepage: https://taxilang.org/ TaxiLang Playground: https://playground.taxilang.org/examples/message-queue-and-database Taxi Lang GitHub repository: https://github.com/taxilang/taxilang OpenAPI Specification (formerly Swagger): https://swagger.io/specification/ YOW! Conference - Australian software conference series: https://yowconference.com/ Spring Framework Kotlin support: https://spring.io/guides/tutorials/spring-boot-kotlin/ Ubiquitous Language (DDD Concept): https://martinfowler.com/bliki/UbiquitousLanguage.html Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ – 0:00 Intro

How do you retrofit a clustered data-processing system to use cheap commodity storage? That's the big question in this episode as we look at one of the many attempts to build a version of Kafka that uses object storage services like S3 as its main disk, sacrificing a little latency for cheap, infinitely-scalable disks. There are several companies trying to walk down that road, and it's clearly big business - one of them recently got bought out for a rumoured $250m. But one of them is actively trying to get those changes back into the community, as are pushing to make Apache Kafka speak object storage natively. Joining me to explain why and how are Josep Prat and Filip Yonov of Aiven. We break down what it takes to make Kafka's storage layer optional on a per-topic basis, how they're making sure it's not a breaking change, and how they plan to get such a foundational feature merged. – Announcement Post: https://aiven.io/blog/guide-diskless-apache-kafka-kip-1150 Aiven's (Temporary) Fork, Project Inkless: https://github.com/aiven/inkless/blob/main/docs/inkless/README.md Kafka Improvement Process (KIP) Articles: * KIP-1150: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics * KIP-1163: Diskless Core: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1163%3A+Diskless+Core * KIP-1164: Topic Based Batch Coordinator: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1164%3A+Topic+Based+Batch+Coordinator * KIP-1165: Object Compaction for Diskless: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1165%3A+Object+Compaction+for+Diskless Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Filip on LinkedIn: https://www.linkedin.com/in/filipyonov Josep on LinkedIn: https://www.linkedin.com/in/jlprat/ Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

I'm joined this week by one of the authors of Apache Kafka In Action, to take a look at the state of Kafka, event systems & stream-processing technology. It's an approach (and a whole market) that's had at least a decade to mature, so how has it done? What does Kafka offer to developers and businesses, and which parts do they actually care about? What have streaming data systems promised and what have they actually delivered? What's still left to build? – Apache Kafka in Action: https://www.manning.com/books/apache-kafka-in-action Pat Helland, Data on the Inside vs Data on the Outside: https://queue.acm.org/detail.cfm?id=3415014 Out of the Tar Pit: https://curtclifton.net/papers/MoseleyMarks06a.pdf Martin Kleppmann, Turning the Database Inside-Out: https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html Data Mesh by Zhamak Dehghani: https://www.amazon.co.uk/Data-Mesh-Delivering-Data-Driven-Value/dp/1492092398 Quix Streams: https://github.com/quixio/quix-streams XTDB: https://xtdb.com/ Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Anatoly's Website: https://zelenin.de/ Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ Kris on Twitter: https://twitter.com/krisajenkins

Jupyter's become an incredibly popular programming and data science tool, but how does it actually work? How have they built an interactive language execution engine? And if we understand the architecture, what else could it be used for? Joining me to look inside the Jupyter toolbox are Afshin Darian and Sylvain Corlay, two of Jupyters long-standing contributors and project-steerers. They've going to take us on a journey that starts with today's userbase, goes through the execution protocol and ends with a look at what Jupyter will be in the future - an ambitious framework for interactive, collaborative applications and more. – Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Jupyter Homepage: https://jupyter.org/ Jupyter Xeus: https://github.com/jupyter-xeus/xeus Jupyter AI: https://github.com/jupyterlab/jupyter-ai Jupyter CAD: https://github.com/jupytercad/JupyterCAD Jupyter GIS: https://github.com/geojupyter/jupytergis/ Jupyter GIS Announcement: https://blog.jupyter.org/real-time-collaboration-and-collaborative-editing-for-gis-workflows-with-jupyter-and-qgis-d25dbe2832a6 QGIS: https://qgis.org/ ZeroMQ: https://zeromq.org/ Sylvain on LinkedIn: https://www.linkedin.com/in/sylvaincorlay Darian on LinkedIn: https://www.linkedin.com/in/afshindarian Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Graphite is a new image editor with an interesting architecture - it's a classic UI-driven app, an image-manipulation language, and a library of programmable graphics primitives that any Rust coder could use, extend or add to. The result is something that you can use like Photoshop or Inkscape, or make use of in batch pipelines, a bit like ImageMagick. Joining me to discuss it are Keavon Chambers & Dennis Kobert, who are hammering away on building a project that's potentially as demanding as Photoshop, but with a more ambitious architecture. How can they hope to compete? Perhaps in the short term by doing what regular image And is the future of image editing modular? – Graphite Homepage: https://graphite.rs/ Graphite Web Version: https://editor.graphite.rs/ Graphite on Github: https://github.com/GraphiteEditor/Graphite Signed Distance Fields: https://jasmcole.com/2019/10/03/signed-distance-fields/ Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Trustfall is a library based on a simple question - what happens if we can query absolutely anything? If you could join REST APIs and databases with filesystems and dockerfiles? It's possible in theory because those are all just datasources. Predrag Gruevski is trying to make it easy by building a universal query engine, with pluggable datasources, all in Rust. This week we dive into Trustfall to figure out how it works. How do you model nearly anything as a datasource? How do you make it easy to extend? And what does it take to optimize a query that's going to be spread out over multiple systems and potentially multiple servers? Questions, questions, questions - all about the act of asking our systems questions. 😃 – Trustfall Source: https://github.com/obi1kenobi/trustfall Cargo Semver Checks: https://crates.io/crates/cargo-semver-checks How to Query Almost Everything: https://predr.ag/querying/ SemVer In Rust (talk): https://predr.ag/blog/semver-in-rust-tooling-breakage-and-edge-cases/#cargo-semver-checks-lints-are-database-queries-in-disguise Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ Kris on Twitter: https://twitter.com/krisajenkins

If you want to build really large software systems well, you have to stop thinking of them as just software systems. Beyond a certain size, everything your software touches becomes part of the wider system. You're part of the system, your users are part of the system, and every other employee & department & priority eventually forms part of that system. And that can make it incredibly difficult to make changes, or even to understand which changes will actually matter. That might be overwhelming, but there's hope. Understanding how systems work and how to improve them is something that can be learnt, and improved at. So in the pursuit of better systems understanding, I'm joined this week by Diana Montalion, coder, architect, and author of Learning Systems Thinking. She takes us through how she sees systems, and how we can get better at discovering and understanding our own systems, as we both chew through some difficult systems we've worked on in the hope of figuring out how to do it better next time… – Learning Systems Thinking (book): https://www.amazon.co.uk/Learning-Systems-Thinking-Essential-Professionals-ebook/dp/B0D9KWZRT2 Diana's Website: https://montalion.com/ Scientific Management (Tailorism): https://en.wikipedia.org/wiki/Scientific_management Jay Wright Forrester: https://en.wikipedia.org/wiki/Jay_Wright_Forrester Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Diana on LinkedIn: https://www.linkedin.com/in/dianamontalion/ Diana on YouTube: https://www.youtube.com/@dianamontalion Kris on BlueSky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/