What Is a Crawl Trap?
A crawl trap is a structural fault on your site that lets a crawler generate an effectively unlimited number of URLs from a finite amount of real content.
A crawl trap is a structural fault on your site that lets a crawler generate an effectively unlimited number of URLs from a finite amount of real content. It is not a penalty, and it is not crawler misbehaviour: every one of those URLs returns 200, and your own code is producing them on request.
Why crawl traps drain a site
A crawler has a finite amount of attention for your site. A trap spends it on URLs that will never earn a click. The visible symptom is rarely the trap itself. It surfaces as new pages taking weeks to appear, older pages quietly dropping out of the index, and Search Console reporting a discovered URL count far larger than anything you ever published.
The second cost is diagnostic. Once tens of thousands of junk URLs are in the crawl, every report you read is polluted by them. Coverage counts stop meaning much. Average position averages over pages you never intended to exist. Before you start pruning, check which URLs are actually earning clicks, because the trap and the pages you care about usually sit in completely different directories.
How a crawl trap forms
Traps are built by combination. Any feature that can append one value to a URL, then append a second value on top of it, multiplies the address space far faster than it adds content. Six patterns account for most of what you will find.
- Unbounded calendars. An events widget with a next-month link has no stopping condition. A crawler follows it forward one month at a time, indefinitely, and every empty month returns 200.
- Sort and filter controls rendered as links. Ten filter values do not create ten URLs. They create every combination of those values, in every order they can be applied.
- Parameters written into internal links. Session identifiers and tracking tags mint a fresh URL for identical content on every visit, and each one is crawlable.
- Relative link errors. A path missing its leading slash resolves against the current directory, so /shoes/ becomes /shoes/shoes/, then /shoes/shoes/shoes/, without limit.
- Crawlable internal search. If your search results live at a GET URL, every query anyone has ever linked to is a page, and those pages link to more searches.
- Pagination without an end. A page parameter that returns an empty template with a 200 status instead of a 404 gives the crawler an infinite corridor.
None of these fail an audit. Each URL is reachable by a crawler, returns a valid status, and renders a real template. The fault is that the set of them has no boundary. Google will keep discovering them for as long as something links to them, and discovery is cheap.
| What you see in the URL | What is generating it |
|---|---|
| /events/2033/07/ or ?date=2033-07 | A calendar with no last-event boundary |
| ?sort= and ?order= stacked together | Sort controls output as ordinary anchor tags |
| /shoes/shoes/shoes/ | A relative link missing its leading slash |
| ?s= or /search/?q= | Internal site search exposed to crawlers |
| ?sessionid= or ?utm_ on internal links | Tracking parameters baked into navigation |
The blunt version
Two features cause most of the traps in the wild: an events calendar whose next-month link never runs out, and sort parameters rendered as ordinary links. Both are infinite by construction. Neither is fixed by robots.txt, which is what almost everyone reaches for first.
Blocking a pattern in robots.txt tells Google not to fetch those URLs. It does not tell Google they should not exist. A disallowed URL that is linked from anywhere can still end up indexed as an address with no content attached. Worse, because the page is never fetched, any noindex tag or canonical you put on it is never read. You have hidden the problem from your own reports while leaving it in Google’s.
The fix is upstream, in the template. Bound the calendar so it stops after the last real event and returns 404 beyond that. Render filter controls so selecting one does not produce a crawlable href. Where a filtered view must stay reachable, keep one canonical version and let the rest go. That is developer work, which is exactly why the one-line robots.txt edit keeps winning: it ships in an afternoon and looks like action. The incentive is scheduling, not engineering. If the groundwork is not in place yet, the fundamentals of crawling and indexation come before any of this.
Example
Say a 300-page venue site adds an events calendar. The widget renders a next-month arrow as a plain link, and the archive has no end date. Search Console starts reporting hundreds of thousands of discovered URLs against 300 published pages, nearly all of them empty months years into the future. Traffic does not collapse, but new venue pages take a month to get indexed, and the coverage report is unreadable. Nothing on the site is broken in a way any tool flags. The template simply answers every date it is asked for. Capping the archive at the last scheduled event, and returning 404 past it, removes the entire set in one deployment.
FAQ
How do I know if I have a crawl trap?
Compare the number of URLs Google has discovered against the number of pages you actually publish. If discovery is several multiples of your real page count, something is generating addresses for you. Then sort crawled URLs by directory and look for dates in the future or stacked parameters.
Does robots.txt fix a crawl trap?
It stops the fetching, not the generating. Disallowed URLs can still be indexed as bare addresses if anything links to them, and a blocked page is never fetched, so a noindex tag on it is never seen. Use robots.txt to contain an active bleed while the template fix is built.
Are small sites safe from crawl traps?
No. Site size offers no protection at all, because the trap does not scale with your content, it scales with your parameters. A 200-page site with one unbounded calendar can generate more crawlable URLs than a 50,000-product catalogue with clean faceted navigation. It is a property of the template.
Related terms
- JavaScript Rendering — why some trap URLs only appear once the page has been executed, not fetched.
- Index Bloat — what a trap leaves behind after Google has finished crawling it.
- Crawlability — whether a crawler can reach a page, which every trapped URL passes easily.
If the proposed fix for your crawl trap is a robots.txt rule, you are being sold containment as a cure. Ask which template is producing the URLs, and when it is scheduled to change.