Analysing Robots.txt at scale with HTTP Archive and BigQuery
From Search Off the Record by Google
April 23, 2026 · 28 min
About this episode
The episode explores the analysis of robots.txt files using HTTP Archive and BigQuery, focusing on common directives and user-agents for modern crawling and AI bots.
In this episode of Search Off the Record, Martin and Gary turn a simple robots.txt question into a data‑driven deep dive using HTTP Archive, WebPageTest, custom JavaScript metrics, and BigQuery. They explore how millions of real robots.txt files are actually written in 2025–2026, which directives and user‑agents are most common, and what that means for modern crawling and AI bots. Perfect for beginner to mid‑level developers and SEOs, you'll learn how large‑scale web measurement works (HTTP Archive, Chrome UX Report, Web Almanac), and how to turn raw crawl data into actionable SEO insights. Subscribe for more candid conversations about crawling, indexing, and the data behind how Google Search and the web really work. Resources: Web Almanac → https://almanac.httparchive.org/en/2025/ Robotstxt custom metric for the HTTP Archive → https://github.com/HTTPArchive/custom-metrics/pull/191 robots.txt parser change → https://github.com/google/robotstxt/commit/4af32e54b715442bb04cd0470e99192f0ffb9792#commitcomment-178586774 Episode transcript → https://goo.gle/sotr108-transcript Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel →…
People in this episode
Host: Martin Splitt
Guest: Gary Illyes
Topics covered
- robots.txt analysis
- web crawling
- SEO insights
- data-driven analysis
- BigQuery
- HTTP Archive
Keywords
- robots.txt
- SEO
- crawling
- BigQuery
- HTTP Archive
- data analysis
- web measurement
Mentioned in this episode
Organizations: Google, HTTP Archive, WebPageTest, Chrome UX Report, Web Almanac
More episodes of Search Off the Record
- How AI Is Changing Google Search and SEO - transcript · May 1, 2026
- How AI Is Changing Google Search and SEO · May 1, 2026 · 33 min
- Analysing Robots.txt at scale with HTTP Archive and BigQuery - transcript · April 23, 2026
- Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained - transcript · March 30, 2026
- Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained · March 30, 2026 · 32 min
- Google crawlers behind the scenes - transcript · March 12, 2026
Explore listener stats, chart rankings, contacts and more on the Search Off the Record podcast page.