robots.txt for AI crawlers: a copy-paste template (2026)

“Should we block AI bots?” is usually asked as one question. It’s really two: do you want AI assistants to find and cite your pages, and do you want your content used to train future models? robots.txt lets you answer them separately, as long as you know which bot does what.

Here’s every major AI crawler, three templates for the most common choices, and how to check they’re actually being respected.

Which AI crawler does what

Three details from that table matter most:

• Search and training are separate. OpenAI says you can allow OAI-SearchBot while blocking GPTBot; disallowing GPTBot “indicates a site’s content should not be used in training”.
• Google-Extended doesn’t touch Google Search. Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. Blocking it won’t remove you from AI Overviews; those run on Googlebot.
• User-triggered fetchers behave differently. ChatGPT-User and Perplexity-User fetch a page because a person asked, and both companies say robots.txt may not apply to them.

Template 1: Allow AI search and training (maximum visibility)

Best for most B2B and consumer brands that want to be found and recommended everywhere. If your current robots.txt has no rules for these bots, they’re already allowed, but stating it explicitly makes intent clear to the next person who edits the file. Replace the example private paths with your own, and remember to repeat them in any named group you add later:

User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: /admin/
Disallow: /cart/
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://yoursite.com/sitemap.xml

Template 2: Allow AI search, block training (the common middle ground)

For publishers, research firms and anyone whose content is the product. You stay eligible for citations in ChatGPT, Claude, Perplexity and Google, while opting out of training:

# AI search: allowed (needed to be cited)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Allow: /

# Model training: blocked
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://yoursite.com/sitemap.xml

CCBot is Common Crawl’s crawler, whose public dataset is widely used for training. Blocking training is a reasonable choice, but know the trade-off: what a model learns in training shapes which brands it “remembers”, which affects recommendations even when it doesn’t search. We explain that in why ChatGPT recommends your competitor.

Template 3: Block all AI crawlers

Only if you genuinely don’t want to appear in AI answers, for example on a members-only or licensed-content site. This removes you from ChatGPT, Claude and Perplexity search:

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Note what this doesn’t do: Google’s AI Overviews and AI Mode use Googlebot, so blocking Google-Extended won’t take you out of them. Blocking Googlebot would remove you from Google Search entirely.

How crawlers read your robots.txt (the rule that trips people up)

A crawler looks for the group that names it. If it finds one, it follows only that group and ignores the User-agent: * rules entirely. If it doesn’t find one, it falls back to *. That’s the standard, written up as RFC 9309.

This matters the moment you add an AI section. Say your file blocks /admin/ for everyone, and you add a group that allows OAI-SearchBot:

User-agent: *
Disallow: /admin/
Disallow: /checkout/

User-agent: OAI-SearchBot
Allow: /

OAI-SearchBot now reads only its own group, so /admin/ and /checkout/ are open to it. Repeat the private paths inside every named group:

User-agent: *
Disallow: /admin/
Disallow: /checkout/

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /checkout/
Allow: /

You can stack several User-agent lines above one set of rules, as above, which keeps the file short and means you only maintain one list of private paths per group. Keep real secrets behind a login, not just a Disallow line: robots.txt is public and only well-behaved bots follow it.

Partial access: allow some sections, block others

You don’t have to choose all or nothing. A common setup lets AI search read everything public that helps buyers (product pages, pricing, docs, blog) and keeps out pages that would only produce bad answers, like internal search results, old campaign pages and gated content:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /search
Disallow: /lp/
Disallow: /resources/gated/
Allow: /

Be careful blocking whole folders like /blog/ or /docs/ to “protect” content. Those are often the pages AI answers cite most, because they answer specific questions.

Other AI crawlers you’ll see in your logs

The table above covers the assistants most buyers use. Your logs will show others. Look up each one in its owner’s documentation before you decide, because names and purposes change:

A simple rule: allow the search crawlers for any assistant your buyers use, and make training and dataset decisions once, as a company, rather than bot by bot as they appear.

Checking that a bot is really who it says

Anyone can put “GPTBot” in a user agent string. If you see heavy traffic claiming to be an AI crawler, check it before you allow or block it at the firewall:

1. Compare the IP against the published list. OpenAI and Perplexity publish IP ranges for their crawlers on the documentation pages linked above.
2. Use your CDN’s verified-bot list. Cloudflare and other CDNs mark known crawlers as verified, which is easier than maintaining IP lists yourself.
3. Block fakes, not the real bot. A scraper pretending to be an AI crawler is a firewall problem. Don’t respond by blocking the real crawler in robots.txt.

Anthropic notes that blocking by IP address may not work reliably for its crawlers, because the addresses can change, and recommends robots.txt instead.

Which template should you use?

If you sell something and want buyers to hear about it in AI answers, Template 1 or 2. Most companies blocking AI bots today did it by accident, through a CDN default or a copied robots.txt, not as a decision.

Should you allow training crawlers?

This is the question teams argue about longest, so here’s a way to settle it in one meeting. Allowing training crawlers means future versions of a model may learn about your brand, product and category from your pages. Blocking them means they learn about you only from what other sites say.

For most companies that sell software or services, the content on their site is marketing they want repeated. Allowing training costs them little. For publishers, the calculation is different, and Template 2 exists for them. Whatever you choose, write the decision down so the next person who edits robots.txt doesn’t undo it by accident.

Where to edit robots.txt on common platforms

Each subdomain needs its own file. docs.yoursite.com and help.yoursite.com don’t inherit the rules from yoursite.com, so check them separately.

How to check robots.txt is working

1. Open your live file at yoursite.com/robots.txt and confirm the rules are there. Caching or a CMS setting sometimes serves an old version.
2. Check the syntax. Each User-agent line needs its own rules below it, and paths are case-sensitive. One missing line can block your whole site.
3. Test a real request as the crawler, to catch CDN blocks that ignore robots.txt:

curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://yoursite.com/
# 200 = reachable   403 or 503 = blocked by your CDN or firewall

1. Check your CDN’s bot settings. Cloudflare began blocking AI crawlers by default for new domains in July 2025. If you’re on Cloudflare, check the AI crawler setting in your dashboard.
2. Give it a day. OpenAI says its systems take about 24 hours to pick up robots.txt changes.
3. Watch your logs over the next week for visits from the bots you allowed.

robots.txt vs llms.txt

robots.txt controls access and AI crawlers obey it. llms.txt only suggests what to read and, according to every study so far, AI search crawlers rarely request it. If you only fix one file, fix robots.txt. The data is in does llms.txt work?

Common mistakes

• Blocking GPTBot and thinking you’ve left ChatGPT search. Search uses OAI-SearchBot.
• Blocking Google-Extended and expecting to leave AI Overviews. AI Overviews use Googlebot.
• Copying a “block all AI” list from a forum. Many include search crawlers, which removes you from AI answers.
• A Disallow: / under User-agent: * left over from a staging site. It blocks everything, including AI crawlers.
• Fixing robots.txt but not the CDN. Always test with a real request.
• Forgetting subdomains. Your docs and help centre each need their own robots.txt, and they’re often the pages AI answers cite.
• Allowing a bot but leaving private paths open to it. Named groups ignore User-agent: *, so repeat your Disallow lines inside each group.

Once crawlers can reach you, make sure they can read what they find: our technical AI search checklist covers rendering, indexing and page structure next.