GPTBot vs OAI-SearchBot: which OpenAI crawler should you block?

A lot of sites blocked “ChatGPT” in 2023 by adding GPTBot to robots.txt. At the time that was the only OpenAI crawler most people had heard of. Now there are four, and blocking the wrong one either leaves you out of ChatGPT search or doesn’t do what you intended.

This guide explains each one in plain terms, then gives you the exact robots.txt lines for each decision you might make.

The four OpenAI bots

OpenAI states the key point directly: you can allow OAI-SearchBot “in order to appear in search results while disallowing GPTBot”, so your content isn’t used for training.

GPTBot vs OAI-SearchBot side by side

Which one should you block?

Blocking OAI-SearchBot but allowing GPTBot is the one combination that rarely makes sense: your content could still shape future models, but ChatGPT search can’t cite or link to you.

robots.txt for each choice

Allow search and training

User-agent: OAI-SearchBot
User-agent: GPTBot
Disallow: /admin/
Allow: /

Allow search, block training

User-agent: OAI-SearchBot
Disallow: /admin/
Allow: /

User-agent: GPTBot
Disallow: /

Block both

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

Allow search, block training on premium content only

User-agent: OAI-SearchBot
Disallow: /members/
Allow: /

User-agent: GPTBot
Disallow: /members/
Disallow: /research/
Allow: /

Remember that a crawler with its own group in robots.txt ignores the User-agent: * rules. Repeat any private paths, like /admin/, inside each named group. Our robots.txt template for all AI crawlers covers Anthropic, Perplexity and Google too.

Audit your current robots.txt in five minutes

1. Open yoursite.com/robots.txt in a browser.
2. Search for “GPTBot”, “OAI-SearchBot” and “ChatGPT”. Note every group that names them and what it allows or disallows.
3. Check the User-agent: * group. If it disallows / and there’s no OAI-SearchBot group, you’re blocking ChatGPT search.
4. Look for wildcard “AI bot” lists copied from forums or plugins. They often include OAI-SearchBot alongside training crawlers.
5. Write down the current result in one line: “Search: allowed / blocked. Training: allowed / blocked.” If it doesn’t match what you want, change it using the templates above.

Repeat for each subdomain. It’s common to find the main site allowing OAI-SearchBot while the help centre, hosted on another platform, blocks it.

Four example setups

In every case, search access is allowed for the pages buyers should find. The training decision is the only one that varies, and it follows whether the content itself is what the business sells.

What about ChatGPT-User?

ChatGPT-User fetches a page when a user’s request in ChatGPT or a Custom GPT needs it, for example when someone pastes your URL and asks for a summary. It isn’t used to build a search index or to train models in the way the other two are. Because these fetches are triggered by a person, OpenAI says robots.txt rules may not apply to them in the same way.

For most sites, ChatGPT-User visits are a good sign: someone was interested enough in your page to send ChatGPT to it. If you see them in your logs on pricing or product pages, those pages are being read by buyers through ChatGPT.

How to check what’s visiting your site

Search your server or CDN logs for each user agent and count requests by page:

grep -E "OAI-SearchBot|GPTBot|ChatGPT-User" access.log \
  | grep -oE "(OAI-SearchBot|GPTBot|ChatGPT-User)" | sort | uniq -c

Then test that your robots.txt and CDN behave as intended:

curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://yoursite.com/pricing
# 200 = reachable   403/503 = blocked at your CDN or firewall

A 200 means the request got through your CDN. Whether the bot is then allowed to use the page is decided by robots.txt, so check both.

How to verify it’s really OpenAI

Anyone can fake a user agent. If you see heavy traffic claiming to be GPTBot, compare the IP addresses with the lists OpenAI publishes for each bot (openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json). Traffic from other IPs isn’t OpenAI, and you can block it at your firewall without touching robots.txt.

1. Download the JSON list for the bot you want to verify.
2. Check whether the request IP falls inside one of the listed ranges.
3. Allow verified ranges in your CDN or firewall; treat everything else claiming the name as a scraper.
4. Recheck the lists every few months, because ranges change.

What blocking GPTBot does and doesn’t do

Blocking GPTBot is often treated as a way to “opt out of ChatGPT”. It’s narrower than that. Here’s what changes and what stays the same:

So if your concern is ChatGPT describing your brand wrongly, blocking GPTBot won’t fix it, and may make it worse, because the model then learns about you only from third parties. The fix for wrong descriptions is accurate, crawlable pages and consistent information on the sites that write about you. We cover that in why ChatGPT recommends your competitor.

How long changes take

OpenAI says it can take “~24 hours from a site’s robots.txt update” for its systems to adjust. If you’ve just unblocked OAI-SearchBot, give it a day before testing ChatGPT search, then a few weeks for your pages to appear in answers as they’re crawled.

Unblocking GPTBot doesn’t change anything you can see right away. It only affects future training, so there’s nothing to test except that the bot can reach your pages.

The same split at other AI companies

OpenAI isn’t alone. Most AI companies now separate search from training, so you can make the same decision for each:

If you decide to allow search and block training for OpenAI, apply the same rule to the others in the same edit, so your robots.txt says one consistent thing. Our full AI crawler template has every line.

Record the decision

robots.txt files get edited by developers, plugins and agencies who don’t know why a line is there. Leave a comment at the top of the file so the decision survives:

# AI crawler policy (decided [month year] by [team]):
# - Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot): ALLOW
#   so we can be cited in AI answers.
# - Training crawlers (GPTBot, ClaudeBot, Google-Extended): [ALLOW/BLOCK]
#   because [reason].
# Ask [owner] before changing these rules.

Should you allow GPTBot?

This is the real decision, and it’s a business one, not a technical one. Allowing GPTBot means future OpenAI models may learn about your products, pricing and positioning from your own pages. Blocking it means they learn about you only from what other sites say.

• Allow it if your pages are marketing, product information and documentation you’d happily see repeated.
• Block it if your content is the product: research, news, courses, premium articles.
• Block some paths if you have both, like a public blog and a paid research library.

Write the decision down with the reason, so the next person who edits robots.txt doesn’t undo it. If legal, marketing and product disagree, settle it in one short meeting using the table above, then put the outcome at the top of the file as shown earlier. A clear written policy beats a perfect one that only lives in someone’s head.

When to revisit the decision

Treat your OpenAI crawler settings as a policy you review, not a one-time fix. Revisit them when any of these happen:

• OpenAI adds or renames a bot. Check its crawler documentation every few months; OAI-AdsBot is a recent addition.
• Your business model changes, for example you launch paid research or a members’ area.
• You sign a content licensing deal, which may set its own crawling terms.
• You redesign or migrate the site. New platforms often ship their own robots.txt and quietly replace yours.
• ChatGPT referral traffic drops suddenly. Check robots.txt and your CDN before anything else.

Common mistakes

• Only having a GPTBot rule. That says nothing about search. Add an explicit OAI-SearchBot group so your intent is clear.
• Forgetting the CDN. A bot-blocking setting can stop OAI-SearchBot even when robots.txt allows it.
• Leaving private paths open. Named groups ignore User-agent: *; repeat your Disallow lines.
• Forgetting subdomains. docs. and help. subdomains need their own robots.txt.
• Never checking logs. Logs tell you whether the bots you allowed are actually visiting, and which pages they read most.

Once OAI-SearchBot can reach you, the next question is whether it can read what it finds. Our guide to JavaScript and AI crawlers covers that.