#562: DuckLake: The Lakehouse That's Just SQL and Parquet
About this episode
The episode discusses DuckLake, a data lake solution that simplifies querying by using SQL and Parquet without unnecessary metadata files.
How many files does your query read before it reads any data? On some data lakes, you go through JSON and metadata files first, just to learn which Parquet files matter. DuckLake asks one SQL question instead. The metadata lives in a real database. The data stays in plain Parquet. That's the entire format. Pedro Holanda joined DuckDB in 2018, when it was still a research prototype at CWI. He's the lead DuckLake developer. Guillermo Sanchez Dionis works on DuckLake and the new Quack protocol. With Quack as the catalog, DuckLake handles 200 transactions a second under heavy contention. No other open table format comes close.
More episodes of Talk Python To Me
- #561: TonIO, a Multi-threaded Async Runtime for Python
- #560: Building a Research OS: From Django to 30,000 Samples
- #559: 12 Things You Should (and Shouldn't) Do in AWS
- #558: Hyper-Personal Software with Python
- #557: Security of everything at PyCon 2026
- #556: Updates on Django's Async Story
Explore listener stats, chart rankings, contacts and more on the Talk Python To Me podcast page.