Cloudflare AI bot settings: block training, not search

Cloudflare sits in front of a large share of websites, so its bot settings decide whether many AI crawlers can reach you at all. Those settings changed in 2026. If you set them up a while ago, or your site was added with defaults, it’s worth checking them now. It takes about fifteen minutes, and a wrong setting here can quietly remove you from AI answers without any error you’d notice.

The goal for most businesses is simple: let AI search find and cite you, and make a deliberate choice about training. Here’s how to do that in Cloudflare, step by step.

The three policy categories

The Training “Disallow” option is the important one. Cloudflare says it writes “a ‘no training’ preference” to your robots.txt, so that cooperating mixed-use crawlers, those used for both search and training, “can still access your content for search indexing”. Mixed-use crawlers that don’t meet Cloudflare’s transparency criteria are still blocked.

The right setup for most businesses

If you sell something and want buyers to hear about you in ChatGPT, Perplexity or Claude, Search and Agent should be allowed. Blocking them is the setting most likely to cost you citations.

What the defaults are

According to Cloudflare, new customers without ad monetisation get no blocks by default. Sites that select “I monetize from pages with ads on this domain” during onboarding get Training set to Disallow by default. If your site was set up earlier, check what’s actually configured, because older settings may still apply.

The older “Block AI bots” toggle under Security Settings, which offered Block on all pages, Block on pages with ads, or Allow, was marked as deprecating on September 15, 2026, with the new category policies taking its place.

Step by step: check and set your policies

1. Log in to Cloudflare and open the domain.
2. Go to Security Settings, filter by Bot traffic, and select Configure AI bot policies, as described in Cloudflare’s managed robots.txt documentation.
3. Set Search and Agent to Allow, unless you have a specific reason not to.
4. Set Training to Allow or Disallow, based on the table above.
5. Decide on Bot Preference Sync (next section).
6. Save, then test with the commands below.

Bot Preference Sync: Cloudflare writes your robots.txt

Bot Preference Sync “generates managed robots.txt directives” from your Search, Agent and Training policies when at least one is set to Block, Block on pages with ads, or, for Training, Disallow. If you already have a robots.txt, Cloudflare prepends its managed rules before yours, combining both in a single response.

When Training is set to Disallow, the managed rules include disallow lines for training crawlers such as Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, along with Cloudflare’s Content Signals Policy explaining the search, ai-input and ai-train categories.

• Turn it on if you want one place, Cloudflare, to control your AI preferences and keep the crawler list current.
• Leave it off if you maintain robots.txt carefully yourself and don’t want rules added in front of yours.
• Either way, read the final file at yoursite.com/robots.txt after saving.

AI Crawl Control: see who’s visiting

AI Crawl Control shows which AI services access your site, lets you allow or block them individually, and tracks whether crawlers follow your robots.txt. It’s available on all Cloudflare plans. Open AI Crawl Control in the dashboard and use the Overview, Crawlers and Metrics tabs. To block a single crawler, go to the Crawlers tab and, in the Action column, select Block.

Use it monthly. You’re looking for two things: search crawlers you want, like OAI-SearchBot, PerplexityBot and Claude-SearchBot, visiting and allowed; and anything blocked that you didn’t intend to block. Free plans identify crawlers by user agent; Enterprise plans with Bot Management get more advanced detection.

Test that AI search can reach you

for ua in OAI-SearchBot PerplexityBot Claude-SearchBot Googlebot bingbot; do
  echo "$ua: $(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://yoursite.com/pricing)"
done
# 200 = reachable   403/503 = blocked or challenged

A fake user agent from your own computer won’t pass Cloudflare’s bot verification the way the real crawler does, so treat a 200 as encouraging rather than proof. AI Crawl Control’s view of real crawler requests is the better confirmation.

Don’t block Google by accident

Google’s AI Overviews and AI Mode use Googlebot, the same crawler as normal Search. Google-Extended is separate: Google says it controls Gemini’s use of content and “does not impact a site’s inclusion in Google Search”. So:

• Disallowing training can include Google-Extended without affecting Google Search or AI Overviews.
• Never block Googlebot unless you want to leave Google Search entirely.
• After changing any bot setting, inspect a key URL in Search Console to confirm Googlebot can still fetch it.

Separate OpenAI’s search and training bots

OpenAI documents that you can allow OAI-SearchBot “in order to appear in search results while disallowing GPTBot”. Cloudflare’s categories follow the same split: OAI-SearchBot is search, GPTBot is training. If ChatGPT citations matter to you, confirm OAI-SearchBot isn’t blocked anywhere in Cloudflare, including in custom firewall rules. Our GPTBot vs OAI-SearchBot guide has the details.

Three common scenarios

A SaaS company on the free plan

The marketing site sits behind Cloudflare with settings nobody has looked at since launch. ChatGPT never cites the docs, which live on a subdomain. The check finds the main site fine, but an old custom rule on the docs subdomain challenges all “known bots”. Removing that rule and setting Search and Agent to Allow on both hostnames fixes access. Training stays on Allow, because everything on the site is marketing and documentation the company wants AI to learn from.

An ad-supported publisher

The publisher wants traffic from AI search but doesn’t want articles used for training. They set Search to Allow, Agent to Block on pages with ads, and Training to Disallow, and turn on Bot Preference Sync so their robots.txt carries the no-training preference. Cooperating mixed-use crawlers can still index the site for search.

A store whose ChatGPT traffic dropped

Referrals from chatgpt.com fall to nearly zero over a week. AI Crawl Control shows OAI-SearchBot requests being blocked since a new rate-limiting rule went live during a sale. Exempting verified crawlers from the rule restores access, and referrals return over the following weeks as pages are recrawled.

A settings audit checklist

• AI bot policies: Search and Agent set as intended; Training set deliberately.
• Bot Preference Sync: on or off by decision, and the final robots.txt read through.
• Custom WAF rules: none blocking verified search crawlers.
• Rate limiting: verified crawlers exempt, or limits high enough not to catch them.
• Bot fight modes: checked for side effects on AI search crawlers.
• Every hostname: main site, docs, help centre and blog checked separately.
• AI Crawl Control: wanted crawlers visiting and allowed.
• Search Console URL Inspection: Googlebot can fetch key pages.

Check your custom rules too

Policy settings aren’t the only way to block bots. Custom WAF rules, rate limiting and “bot fight” modes can block AI crawlers even when your AI policies allow them. Review any rule that matches on user agent, known bots or country, and make sure verified search crawlers aren’t caught by it.

If you’ve been blocking AI search

If the audit shows AI search crawlers were blocked, don’t panic. Fix the setting, then help the crawlers catch up:

1. Change the policy or rule, and confirm in AI Crawl Control that requests are now allowed.
2. Check your robots.txt allows the same crawlers; OpenAI says its systems take about 24 hours to pick up robots.txt changes.
3. Submit your key URLs to Bing through IndexNow, since Microsoft is among ChatGPT search’s providers.
4. Re-run your buyer prompts in ChatGPT and Perplexity after two to four weeks and compare with before.

Citations come back as pages are recrawled, so there’s no instant switch. How quickly depends on how often each crawler visits your site and how long the block was in place. Track it with the same prompts each time, and keep a note of the date you unblocked, so you can tell recovery apart from normal month-to-month movement.

If you’re not on Cloudflare

Other CDNs and hosts have similar controls under different names: bot management, firewall rules, attack challenge modes or “AI crawler” toggles. The principles are the same. Find every setting that can block or challenge bots, make sure verified search crawlers are allowed, decide on training separately, and test with real crawler traffic in your logs rather than trusting the settings page alone.

A monthly routine

Spend ten minutes a month in AI Crawl Control. Check that OAI-SearchBot, PerplexityBot, Claude-SearchBot and Bingbot are visiting your key pages, that nothing you want is blocked, and that no new crawler is hitting the site unusually hard. After any change to firewall or rate-limiting rules, repeat the curl test and look at AI Crawl Control the next day. Settings that block AI search usually get added during an incident, a sale or a security review, and nobody remembers to check what else they caught.

Common mistakes

• Never checking after onboarding. Defaults and features have changed, and old settings carry over.
• Confusing Training with Search. Disallowing training doesn’t have to cost you citations.
• Leaving old custom rules in place that block all “known bots”.
• Not reading the final robots.txt after turning on Bot Preference Sync.
• Testing only the homepage. Test pricing, comparison and docs pages too, since those are the pages buyers’ questions lead to.

Once crawlers can reach you, make sure they can read what they find. Our technical AI search checklist covers rendering, indexing and structure next.