How Should You Manage ChatGPT and AI Crawler Access?
Separate search/retrieval access from training and product-control signals. Define robots.txt policy deliberately and verify actual reachability.
Training Bots vs. Real-Time Search Crawlers
The most common mistake webmasters make is treating all AI bots identically. AI companies maintain two distinct categories of crawlers:
- Training Crawlers (e.g. GPTBot, ClaudeBot): Scrape public text in bulk to train future foundation models.
- Search & Retrieval Bots (e.g. OAI-SearchBot, Claude-SearchBot, PerplexityBot): Execute real-time queries to cite and summarize web content when users ask questions.
Configuring Robots.txt Safely
If you want citations in ChatGPT Search and Perplexity without allowing bulk training on your copyrighted materials, configure your robots.txt as follows:
# Allow real-time search retrieval
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Restrict or block offline training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /The Fallback Wildcard (*) Trap
According to RFC 9309, crawlers evaluate the most specific User-agent group matching their identity. If a crawler has no specific group, it falls back to User-agent: *. If your wildcard block disallows /, all AI search bots without specific allow directives will be blocked completely.
To resolve detected blockers with root cause analysis, acceptance tests and rollbacks for your engineering team, $99 Full Site Fix Mandate is available.