Crawl Budget
Last updated: March 2025
Definition
How many pages Google will bother crawling on your site in a given period. Large sites with thousands of pages need to manage this carefully. If Googlebot wastes crawl budget on low-value pages (filters, pagination, duplicates), your important pages get crawled less frequently.
Why It Matters
For sites under 10,000 pages, crawl budget rarely matters. For larger sites, it's critical. New content on pages that Google crawls infrequently takes weeks to get indexed. Poor crawl budget management means your best content sits in the queue behind junk pages.
How to Improve
- Block low-value pages from crawling via robots.txt. Faceted navigation, internal search results, and parameter URLs are common culprits.
- Fix broken links and redirect chains. Googlebot follows them and wastes crawl budget on dead ends.
- Submit an XML sitemap with only your important pages. Help Google prioritize what to crawl.
- Monitor crawl stats in Google Search Console. Look for spikes in crawled-but-not-indexed pages.