About this episode
This episode discusses the complexities of Google's crawling infrastructure and the misconceptions surrounding Googlebot.
Developers often talk about Googlebot as if it were a single program you could just run as "googlebot.exe", but that is not how Google's crawling actually works. In this episode of Search Off the Record, Martin and Gary from the Search Relations team unpack how Google's crawling infrastructure is really built and operated. They cover why "Googlebot" is a misnomer and how it relates to a central crawling software-as-a-service used by many Google products, how crawl behavior is controlled centrally to avoid overwhelming sites (throttling, handling 503s, and "don't break the internet" safeguards) and more! If you build for the web, work on SEO, or just want a more accurate mental model of how Google crawls pages, this behind‑the‑scenes discussion is for you. Resources: Crawlers → https://developer.google.com/crawling Episode transcript → https://goo.gle/sotr107-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO #GoogleSearch Speakers: Martin Splitt…
People in this episode
Host: Martin Splitt
Guest: Gary Illyes
Topics covered
- Google crawling
- SEO
- web development
- Googlebot
- crawling infrastructure
- search relations
Keywords
- Googlebot
- crawling
- SEO
- web development
- search relations
- infrastructure
- throttling
Mentioned in this episode
Organizations: Google, Google Search
Products: Googlebot
More episodes of Search Off the Record
- How AI Is Changing Google Search and SEO - transcript · May 1, 2026
- How AI Is Changing Google Search and SEO · May 1, 2026 · 33 min
- Analysing Robots.txt at scale with HTTP Archive and BigQuery - transcript · April 23, 2026
- Analysing Robots.txt at scale with HTTP Archive and BigQuery · April 23, 2026 · 28 min
- Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained - transcript · March 30, 2026
- Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained · March 30, 2026 · 32 min
Explore listener stats, chart rankings, contacts and more on the Search Off the Record podcast page.