26: A Trillion Events

26: A Trillion Events

From NerdOut@Spotify by Spotify R&D

February 8, 2024 · 39 min · Season 1 · Episode 26

About this episode

The episode discusses Spotify's journey in building a reliable event delivery system capable of handling a trillion events globally.

How did we learn to do event delivery at scale at Spotify? It’s been a journey. When you do something like tap the play button in the Spotify app, that’s an event. And getting that event data is fundamental to the Spotify experience. Without it, we wouldn’t be able to make music recommendations, pay artists fairly, or track down pesky, hard-to-find bugs. At the most basic level, this seems like a straightforward process: record an event, send that event data to a server somewhere, do something useful with it. Easy, right? But now, multiply that process by 50 million events per second. So, how do we make sure all that important data is delivered reliably, from our client apps to the cloud? Host and principal engineer Dave Zolotusky talks with 9-year Spotify veteran Riccardo Petrocco about our journey building a event delivery system that can reliably handle a trillion events around the world, moving from Kafka to the cloud, building systems that are simple enough so that nobody tries to find a way around them and encourages “doing the right thing”, the definition of “quality data”, the value of moving up the stack and focusing less on the data pipes and more on what’s in them, and…

People in this episode

Host: Dave Zolotusky

Guest: Riccardo Petrocco

Topics covered

  • event delivery
  • data management
  • Spotify technology
  • cloud systems
  • data reliability

Keywords

  • event delivery
  • Spotify
  • data systems
  • Kafka
  • cloud
  • data reliability
  • music recommendations

Mentioned in this episode

Organizations: Spotify, Kafka, Backstage

More episodes of NerdOut@Spotify

Explore listener stats, chart rankings, contacts and more on the NerdOut@Spotify podcast page.