Search and training crawlers
Understand Googlebot, OAI-SearchBot and GPTBot as different controls.
“Allow AI bots” is too broad to be a useful access policy. Providers use different agents for different purposes, and one company can offer separate controls for search discovery and model training. Write down the provider, the purpose you intend to allow or restrict and the documented agent before changing any rule.
Google Search has its own controls
Google identifies Googlebot robots.txt directives as the control for crawling for Search, including its AI features. It describes snippet and indexing controls separately, and points to Google-Extended for training and grounding in some other Google systems. Read the current Google AI features guidance before assuming a training-related setting removes content from AI Overviews or AI Mode.
Crawling, indexing and displaying information are different questions. Google’s robots.txt introduction explains that a blocked URL can still appear in search results. A robots.txt file is not a security boundary. Content that must remain private needs appropriate access protection rather than a polite request to crawlers.
OpenAI separates search from training
OpenAI describes OAI-SearchBot as the crawler used to surface websites in ChatGPT search. GPTBot crawls material that may be used to train its generative foundation models. OpenAI says these settings are independent: a site can allow search access while disallowing GPTBot. These are the purposes described in its official crawler overview.
The same documentation distinguishes ChatGPT-User, which can visit pages in response to user actions and is not an automatic search-discovery crawler. It notes that robots.txt rules may not apply to those user-initiated requests. Do not treat all requests containing the company’s name as the same process or assume one directive covers them all.
Make a deliberate, narrow decision
Hypothetical policy: a publisher wants public pages available to ChatGPT search while indicating that content should not be used for OpenAI model training. Its documented choices could allow OAI-SearchBot and disallow GPTBot. This illustrates separate controls; it is not a recommendation for every publisher or a guarantee of search inclusion, data removal or retrospective untraining.
Before applying that policy, inspect the actual robots.txt groups, rules for relevant paths and any conflicting infrastructure restrictions. Read the provider’s current matching and access guidance rather than copying a snippet without context. Keep private paths protected independently, and consider whether the rule unintentionally affects resources needed to read an otherwise public page.
Verify requests without overclaiming
A declared user-agent name alone does not prove a request’s identity. OpenAI publishes IP ranges for its agents in the crawler documentation. Where identity matters, verify with the provider’s current method and preserve the relevant evidence. A request blocked at a hosting layer may never reach the application’s logs, so an empty log is incomplete evidence.
After a change, record the exact rule, affected paths, provider and date. An access policy communicates your choice; observing that a crawler fetched a page confirms only that request. Neither establishes indexing, citation, training history or a business outcome. Recheck the documentation when your policy changes, because bot purposes and provider behavior can evolve.