What Is Robots.txt?
Robots.txt tells crawlers which parts of a site they can access. It's not a security measure, and it doesn't guarantee indexing, here's what it does.
Robots.txt is a plain-text file placed at a site’s root (example.com/robots.txt) that tells search engine crawlers which parts of the site they’re allowed or disallowed to access. It’s a set of instructions for well-behaved bots, not a security measure or an access control mechanism.
Why robots.txt matters
A misconfigured robots.txt file can accidentally block search engines from crawling an entire site, one of the most common and most damaging technical SEO mistakes, often introduced accidentally during a site migration or redesign when a staging-environment “Disallow: /” rule gets carried over to production. It’s worth checking regularly, especially right after any development work touching the site’s root configuration. The other common misunderstanding worth correcting: disallowing a URL in robots.txt does not guarantee it won’t be indexed. If other sites link to a disallowed page, Google can still index the URL itself, just without crawling its content, showing up in results with no title or description.
How robots.txt works
- A crawler checks for robots.txt first, before crawling any other page on the domain.
- Rules are matched against user-agent, allowing different instructions for different bots (Googlebot, Bingbot, specific AI crawlers).
- Disallow and Allow directives specify which paths a given user-agent should or shouldn’t crawl.
- Well-behaved crawlers respect these rules, though robots.txt relies on voluntary compliance, it isn’t an enforced access restriction the way password protection is.
Common robots.txt mistakes
- A leftover “Disallow: /” from staging. Blocks the entire site from being crawled, often discovered only after traffic mysteriously drops.
- Assuming it prevents indexing. Use a noindex meta tag for that; robots.txt only controls crawling, not indexing.
- Blocking CSS or JavaScript files needed for rendering. Can prevent Google from properly rendering and understanding the page’s layout and content.
- Treating it as a security measure. Never rely on robots.txt to hide sensitive content, the file itself is publicly readable and simply requests crawlers avoid the listed paths.
Robots.txt vs. noindex tag vs. password protection
| Method | Prevents crawling? | Prevents indexing? | Prevents human access? |
|---|---|---|---|
| Robots.txt | Yes, for compliant bots | Not guaranteed | No |
| Noindex tag | No, page still crawled | Yes | No |
| Password protection | Yes, effectively | Yes, effectively | Yes |
Real example
For example, a rule like “Disallow: /admin/” inside robots.txt asks well-behaved crawlers not to access anything under the /admin/ folder, though it doesn’t prevent that URL from being indexed if other sites happen to link to it, for genuine confidentiality, password protection or a noindex tag is the correct tool instead.
Robots.txt and AI search
Robots.txt has become a genuinely active decision point for AI visibility: site owners can now explicitly allow or block specific AI crawlers (like GPTBot, ClaudeBot, or Google-Extended) separately from standard search crawlers. Blocking an AI crawler is a legitimate content-control choice, but it directly means that content becomes ineligible for citation in that system’s generated answers, a tradeoff worth making deliberately rather than by default or oversight.
FAQ
Can robots.txt hide a page from Google entirely?
Not reliably. It blocks crawling, but the URL can still get indexed without content if external links point to it. Use a noindex tag for genuine removal from search results.
How do I check my site’s robots.txt for problems?
Google Search Console includes a robots.txt report showing the current file and any parsing errors; you can also simply visit yourdomain.com/robots.txt directly in a browser.
Can I block specific AI crawlers without blocking Google?
Yes. Robots.txt supports separate rules per user-agent, so you can allow Googlebot while disallowing a specific AI crawler by name, or vice versa.
Does every website need a robots.txt file?
It’s not strictly required, if absent, crawlers generally assume full access, but having an explicit file, even a permissive one, is standard practice and avoids ambiguity.
Related terms
Check your robots.txt file right after every site migration, not just when traffic mysteriously tanks. A single leftover disallow rule can quietly block your entire site.