Analysing Robots.txt at scale with HTTP Archive and BigQuery - transcript
From Search Off the Record by Google
April 23, 2026
About this episode
This episode discusses the analysis of robots.txt files at scale using HTTP Archive and BigQuery.
People in this episode
Host: Google
Topics covered
- robots.txt
- HTTP Archive
- BigQuery
- web crawling
- data analysis
Keywords
- robots.txt
- HTTP Archive
- BigQuery
- web crawling
- data analysis
- SEO
- website management
Mentioned in this episode
Organizations: HTTP Archive, BigQuery
More episodes of Search Off the Record
- How AI Is Changing Google Search and SEO - transcript · May 1, 2026
- How AI Is Changing Google Search and SEO · May 1, 2026 · 33 min
- Analysing Robots.txt at scale with HTTP Archive and BigQuery · April 23, 2026 · 28 min
- Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained - transcript · March 30, 2026
- Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained · March 30, 2026 · 32 min
- Google crawlers behind the scenes - transcript · March 12, 2026
Explore listener stats, chart rankings, contacts and more on the Search Off the Record podcast page.