What Is Crawl Budget?
Crawl budget is how many pages a search engine will crawl on your site in a given timeframe. Mostly a large-site problem, with a real example.
Crawl budget is the number of pages a search engine will crawl on a site within a given timeframe. It’s determined by a mix of how important and trustworthy the site is (crawl demand) and how much crawling its server can comfortably handle without slowing down (crawl rate limit).
Why crawl budget matters
Wasting crawl budget on low-value pages, like faceted filter URLs or thin duplicate pages, means important, genuinely valuable pages may get crawled less often, delaying how quickly new or updated content gets indexed and reflected in rankings. This mostly matters for very large sites; smaller sites with a few hundred pages rarely run into crawl budget limits at all, and spending time optimizing crawl budget on a small site is usually wasted effort better spent elsewhere. The businesses that genuinely need to manage this are large e-commerce catalogs, news publishers, and sites with heavy faceted navigation generating thousands of near-duplicate URLs.
What determines crawl budget
- Crawl demand. How much Google wants to crawl your site, driven by the site’s perceived popularity, freshness needs, and overall authority.
- Crawl rate limit. How much crawling your server can handle without degrading performance for real visitors, Google backs off automatically if it detects server strain.
- Site size and URL count. Larger sites with more total URLs need a proportionally larger crawl budget to get fully covered in a reasonable timeframe.
- Low-value URL volume. Faceted navigation, session parameters, and thin duplicate pages consume crawl budget without adding real coverage of unique content.
How to manage crawl budget on large sites
- Block low-value parameter URLs in robots.txt or via URL parameter handling, so crawlers aren’t repeatedly hitting near-duplicate variants.
- Fix broken links and redirect chains. Every crawled dead-end or unnecessary redirect hop wastes budget that could go toward real content.
- Consolidate thin or duplicate pages rather than leaving hundreds of low-value URLs live and crawlable.
- Keep your XML sitemap clean and current, listing only genuinely important, indexable URLs.
- Improve server response times. A faster server allows Google to crawl more pages within the same crawl rate limit.
Does your site actually have a crawl budget problem?
| Site size | Crawl budget typically a concern? |
|---|---|
| Under a few thousand pages | Rarely, not usually worth dedicated effort |
| Tens of thousands of pages | Possibly, worth checking crawl stats periodically |
| Hundreds of thousands+ pages | Often, especially with heavy faceted navigation |
Real example
For example, a site with 2 million pages but a crawl budget effectively covering only 50,000 pages per month will have large sections that go unvisited or get crawled only rarely, delaying how quickly new or updated content gets indexed and starts appearing in search results at all.
Crawl budget and AI search
AI crawlers operate as a separate category from traditional search crawlers, each with its own crawl allocation and behavior, so a site can have healthy crawl budget usage with Googlebot while still being under-crawled by an AI system’s bot, or vice versa. Large sites now need to think about crawl efficiency across multiple distinct crawler types rather than a single unified budget.
FAQ
Does my small business site need to worry about crawl budget?
Almost certainly not. Crawl budget concerns are largely a large-site problem, sites under a few thousand pages are very unlikely to run into meaningful crawl budget limitations.
How do I check my site’s crawl budget usage?
Google Search Console’s Crawl Stats report, under Settings, shows crawl requests over time, broken down by response type and file type, the primary first-party way to monitor this.
Can I increase my crawl budget directly?
Not directly by requesting it, but improving server speed, fixing errors, and reducing low-value URLs can effectively increase how much of your genuinely valuable content gets crawled within the existing budget.
Does slow server response time hurt crawl budget?
Yes. Google reduces its crawl rate automatically when it detects server strain, so a consistently slow server can directly shrink how much of your site gets crawled in a given period.
Related terms
If you’re not running a site with hundreds of thousands of pages, crawl budget probably isn’t your bottleneck. Don’t spend time optimizing a problem you don’t actually have.