# robots.txt for AI crawlers

> A robots.txt can treat AI search and AI training differently, because each company uses separate user agents for each. Copy the file below to the root of your site, move each agent into the group that matches your policy, and keep your normal search engines under the catch-all group.

Updated November 8, 2026. Canonical: https://sniptide.com/templates/robots-txt-ai-crawlers
Open it in the Sniptide editor: https://app.sniptide.com/new?template=robots-txt-ai-crawlers
Files as JSON: https://sniptide.com/templates/robots-txt-ai-crawlers.json

## Files

### robots.txt

```
# robots.txt for acme.com
# Search engines and AI search can read the site.
# Crawlers that collect training data can't.

# AI search and assistants fetching a page for a user
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /app/
Disallow: /api/

# Crawlers that collect model training data
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Disallow: /app/
Disallow: /api/

Sitemap: https://acme.com/sitemap.xml
```

## Search, assistants and training are separate choices [#search-assistants-and-training-are-separate-choices]

Most AI companies now run more than one crawler. One collects pages to train models, another builds an index for AI search answers, and a third fetches a page when a person asks an assistant about it. Blocking every AI agent also removes you from ChatGPT search and Perplexity answers, which for many products are now a source of visitors. The file above makes the common choice: be found and cited, but don't feed training sets.

## What each user agent is [#what-each-user-agent-is]

| User agent | Company | What it does |
| --- | --- | --- |
| `OAI-SearchBot` | OpenAI | Indexes pages for ChatGPT search results. |
| `ChatGPT-User` | OpenAI | Fetches a page when a ChatGPT user or GPT asks for it. |
| `GPTBot` | OpenAI | Collects pages to train models. |
| `Claude-SearchBot` | Anthropic | Indexes pages for search inside Claude. |
| `Claude-User` | Anthropic | Fetches a page when a Claude user asks for it. |
| `ClaudeBot` | Anthropic | Collects pages to train models. |
| `PerplexityBot` | Perplexity | Indexes pages for Perplexity answers. |
| `Perplexity-User` | Perplexity | Fetches a page when a Perplexity user asks for it. |
| `Google-Extended` | Google | Not a crawler. A token that controls whether Googlebot's pages train Gemini, without affecting Google Search. |
| `Applebot-Extended` | Apple | A token that controls whether Applebot's pages train Apple's models, without affecting Siri and Spotlight search. |
| `CCBot` | Common Crawl | Builds the open Common Crawl dataset, which many models train on. |
| `Bytespider` | ByteDance | Collects pages, widely reported as training data. |

Companies add and rename agents, so check each company's crawler documentation when you review this file.

## What robots.txt can't do [#what-robotstxt-cant-do]

robots.txt is a request, not access control. The companies above say their crawlers follow it, but nothing enforces that, and agents that fetch a page on a user's behalf may treat it differently from crawlers. Pages you need to keep private belong behind a login, not in a `Disallow` line. Blocking a crawler also doesn't remove what it collected before.

Changes can take a day or more to apply, since crawlers cache the file.

## How the groups work [#how-the-groups-work]

A crawler follows the most specific group that names it and ignores the rest, which is why the training group can say `Disallow: /` while the catch-all allows everything. Several `User-agent` lines in a row share the rules below them. Each group repeats the `/app/` and `/api/` rules because a crawler in a named group never reads the `*` group.

## Variant: block all AI agents [#variant-block-all-ai-agents]

To opt out of AI search as well as training, move every agent into one group:

```text title="robots.txt"
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://acme.com/sitemap.xml
```

To help the agents you allow read the site well, publish an [llms.txt](/templates/llms-txt) next to this file.