{"name":"robots-txt-ai-crawlers","title":"robots.txt for AI crawlers","description":"A robots.txt that lets AI search and assistants read your site while blocking crawlers that collect training data, with the user agent of each.","category":"discovery","updated":"2026-11-08","url":"https://sniptide.com/templates/robots-txt-ai-crawlers","files":[{"name":"robots.txt","language":"text","content":"# robots.txt for acme.com\n# Search engines and AI search can read the site.\n# Crawlers that collect training data can't.\n\n# AI search and assistants fetching a page for a user\nUser-agent: OAI-SearchBot\nUser-agent: ChatGPT-User\nUser-agent: Claude-SearchBot\nUser-agent: Claude-User\nUser-agent: PerplexityBot\nUser-agent: Perplexity-User\nAllow: /\nDisallow: /app/\nDisallow: /api/\n\n# Crawlers that collect model training data\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: CCBot\nUser-agent: Google-Extended\nUser-agent: Applebot-Extended\nUser-agent: Bytespider\nDisallow: /\n\n# Everyone else, including Googlebot and Bingbot\nUser-agent: *\nAllow: /\nDisallow: /app/\nDisallow: /api/\n\nSitemap: https://acme.com/sitemap.xml\n"}]}