What Is GPTBot?
GPTBot is the OpenAI crawler that collects web content which may be used to train its generative AI foundation models. It is one of four crawlers OpenAI operates.
GPTBot is the OpenAI crawler that collects web content which may be used to train its generative AI foundation models. It is one of four crawlers OpenAI operates, and it is not the one that fetches a page when ChatGPT looks something up for a user. Blocking it is a decision about training data, not about whether you appear in answers.
Why the GPTBot decision matters
Most sites make this call once, in a hurry, usually after a board-level conversation about AI and content theft. The rule gets written as a blanket block, and nobody revisits it.
The cost of getting it wrong runs in both directions. Block the wrong agent and you remove yourself from the surface that sends actual clicks, while leaving the one you were worried about untouched. Allow everything without deciding, and you have made a licensing choice about your own archive by accident. OpenAI documents each setting as independent of the others, which means this is four decisions, not one. Treating it as a single switch is where most of the damage happens. If you would rather keep track of what actually changed than react to the next headline about crawlers, that is the useful habit here.
What each OpenAI crawler does
OpenAI publishes the list, the purposes and the IP addresses for each agent. The differences are the whole point.
| User agent | What OpenAI says it does | Used for training? |
|---|---|---|
| GPTBot | Crawls content that may be used in training generative AI foundation models | Yes |
| OAI-SearchBot | Surfaces websites in search results in ChatGPT’s search features | No |
| ChatGPT-User | Acts on user requests to visit a page in ChatGPT and custom GPTs | No |
| OAI-AdsBot | Validates the safety of pages submitted as ads on ChatGPT | No |
- Decide per agent, not per company. OpenAI’s own example is allowing OAI-SearchBot so you appear in search results while disallowing GPTBot to signal that your content should not train models.
- Write separate robots.txt groups. One user-agent block per crawler name. A single wildcard rule applies your strictest intention to all four, including the two that only ever help you.
- Understand the ChatGPT-User exception. OpenAI states that because those actions are initiated by a user, robots.txt rules may not apply. That agent is closer to a person clicking a link than to a crawler working through a queue.
- Verify before you conclude anything. User-agent strings are trivially forged. OpenAI publishes the addresses each crawler uses at openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json and openai.com/adsbot.json. Check the requesting IP against the relevant file before you treat a log line as real.
- Allow for lag. OpenAI notes that for search results it can take roughly 24 hours from a robots.txt update for its systems to adjust. Changing the file and checking an hour later proves nothing.
- Separate access from outcome. Letting a crawler in is a precondition, never a cause. What decides whether your material is quoted back to someone is how these systems choose sources, which is a different discipline entirely.
The blunt version
Blocking GPTBot does not take you out of ChatGPT answers. It was never going to. GPTBot collects training data; ChatGPT’s search feature uses OAI-SearchBot, and a page fetched because a user asked for it comes through ChatGPT-User. Three jobs, three agents, three independent settings. Anyone selling a GPTBot block as AI visibility control has either not read the documentation or is counting on you not having read it.
The expensive mistake runs the other way. A site decides to keep AI out, writes one wildcard disallow, and quietly removes itself from ChatGPT search, which is the surface that carries links and sends people back. The training crawler you objected to was one line. The blanket rule took the referral traffic with it.
Use a GPTBot block when the content is the product: a paid archive, licensed data, original research you sell. Accept that you are trading away background familiarity in future models, and decide that the trade is worth it. Allow GPTBot when discovery matters more to you than exclusivity, which for most commercial sites it does. Either way, leave OAI-SearchBot alone unless you genuinely do not want the clicks. That distinction is the sort of thing that gets lost in a pitch deck, which is roughly what AI did to SEO agencies.
Example
Say a trade publisher with a paywalled research archive decides it will not fund model training for free. Legal asks for AI crawlers to be blocked. The implementation is one wildcard disallow covering everything with ‘GPT’ or ‘OAI’ in the name. Six weeks later, referrals from ChatGPT have gone to nothing, and nobody connects the two events because the rule was described internally as a training block. The archive was never the exposure. The free explainer pages were, and those were the pages ChatGPT’s search feature had been linking to. The fix is two lines: disallow GPTBot on the paid directories, allow OAI-SearchBot everywhere.
FAQ
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects content that may be used for model training. ChatGPT’s search features use OAI-SearchBot, and pages fetched because a user asked for them come through ChatGPT-User. OpenAI documents each of these settings as independent, so a GPTBot rule changes training eligibility and nothing about whether you appear in answers.
How do I confirm a GPTBot hit in my logs is genuine?
Check the requesting IP address against the list OpenAI publishes at openai.com/gptbot.json. The user-agent string on its own proves nothing, because anything can send any string it likes. Each OpenAI crawler has its own published address file, so verify against the file for the agent the request claims to be.
How long does a robots.txt change take to apply?
OpenAI states that for search results it can take around 24 hours from a robots.txt update for its systems to adjust. Test after that window, not before. And test the live file over HTTPS rather than a staging copy, because the wrong environment is the most common reason a rule appears to be ignored.
Related terms
- Social Proof — the on-page evidence a model repeats when it summarises what you do.
- Statement of Work — where an agency’s AI visibility promises should be written down as deliverables.
- LLM Citation — the outcome crawler access makes possible but never produces on its own.
If someone tells you blocking GPTBot controls how you appear in ChatGPT, they are describing the wrong crawler. Ask which of the four agents they mean, and watch what happens next.