Technical SEO checklist for AI search: 12 checks that matter

Technical AI search advice tends to come as a long list where everything sounds equally urgent: llms.txt, schema, “AI-friendly” formatting, special files. Most of it doesn’t matter. A few things matter a lot, and if they’re broken, nothing else you do will get you cited.

This checklist ranks the work by impact. Do the must-dos this week, the should-dos this month, and ignore the rest until there’s evidence they help. It’s written for marketers and developers who want a clear list they can hand to each other.

The checklist at a glance

Must do: let AI crawlers in

1. Allow AI search crawlers in robots.txt. This is the hard gate. OpenAI says sites that block its search crawler “will not be shown in ChatGPT search answers”. Each AI company runs separate bots for search and for training, so you can allow one and block the other:

# Allow AI search (needed to be cited)
# Repeat any private Disallow paths here too
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Optional: opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Our robots.txt template for AI crawlers covers every major bot and the trade-offs of blocking training.

2. Check your CDN and firewall. Cloudflare started blocking AI crawlers by default for new domains in July 2025, and other providers have similar bot settings. A CDN block overrides your robots.txt. Open your bot or firewall settings and allow verified AI search crawlers, then test with the command in check 3.

Must do: make your content readable

3. Serve content in HTML, not just JavaScript. AI crawlers download JavaScript files but don’t run them. Vercel’s analysis found that neither OpenAI’s nor Anthropic’s crawler executed any JavaScript. Test your key pages:

curl -s -A "OAI-SearchBot" https://yoursite.com/pricing | grep -i "a sentence from your page"

No match means the text isn’t in the HTML a crawler receives. A 403 or challenge page means you’re being blocked (check 2). The full test and fixes by framework are in can ChatGPT read JavaScript?

Must do: get indexed everywhere AI looks

4. Google. Google’s AI Overviews and AI Mode only cite pages that are “indexed and eligible to be shown in Google Search with a snippet”. In Search Console, check that your key pages are indexed and not blocked by a stray noindex or nosnippet.

5. Bing. OpenAI lists Microsoft among ChatGPT search’s search partners, and Microsoft Copilot runs on Bing. Verify your site in Bing Webmaster Tools (you can import it from Search Console), submit your sitemap and turn on IndexNow so new pages are found quickly. You also get Bing’s AI Performance report, which counts how often Copilot cites your pages.

6. Fix broken links and redirects. AI crawlers hit missing pages often: in Vercel’s data, about 34% of their fetches returned 404 errors. Crawl your site, redirect old URLs to their replacements, and keep your sitemap current.

Should do: structure pages so they can be quoted

7. One clear page per buyer question. AI answers pull passages from specific pages. A page that answers “[you] vs [rival]” directly is far easier to cite than a homepage that mentions everything. Comparison and alternatives content took 33% of citations in Something Inc’s study of 4,100 B2B queries, more than any other format.

8. Answer first, and use real HTML. Put the direct answer in the first two sentences under each heading. Use real <table> elements for comparisons and pricing, not images or script-built grids. Use descriptive headings that match the question (“How much does X cost?”), so a crawler can find the passage that answers it.

9. Show when pages were updated. AI answers favour fresher sources. In a University of Toronto study, the median page GPT-4o cited for consumer-electronics queries was 79.8 days old, against 130.4 days for Google’s results. Put a visible “Updated” date on pricing, comparison and guide pages, and only change it when the content changes.

10. Keep pages fast and mobile-friendly. Same as for SEO. Slow pages and broken mobile layouts hurt indexing and users alike.

11. Link to your key pages. Pricing, comparison and alternatives pages should be linked from your navigation or from relevant guides. Pages buried four clicks deep get crawled less.

Nice to have: watch AI crawler activity

12. Check your logs monthly. See which AI crawlers visit, which pages they fetch and what status codes they get:

grep -E "OAI-SearchBot|PerplexityBot|Claude-SearchBot|bingbot" access.log | awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -30

Lots of 200s on key pages means you’re fine. 403s point to a bot block, 404s to broken links. On Vercel or Cloudflare, filter request logs by user agent instead.

What you can safely skip

Schema still has normal SEO uses, such as rich results. Just don’t expect it to change what ChatGPT cites. The llms.txt evidence is laid out in does llms.txt work?

How to run the checklist in one day

Once the technical side is clean, the work moves to content and mentions: publishing pages that answer buyer prompts and getting named on the third-party sites AI trusts. That’s covered in our ChatGPT citation playbook.

Platform notes: where each check usually breaks

The same twelve checks fail in different places depending on how your site is built. Start with the row that matches your stack:

Help centres and docs often live on a separate platform (Zendesk, Intercom, GitBook). They’re frequently cited, because they answer specific questions, so run checks 1 to 4 on that subdomain too.

A ticket you can hand to your developer

If you’re a marketer and someone else owns the code, paste this into a ticket. It’s specific enough to be done without a meeting:

Title: Make key pages readable by AI search crawlers

Pages: /pricing, /compare/*, /alternatives/*, /blog/* (top 10 by traffic)

1. robots.txt: allow OAI-SearchBot, Claude-SearchBot, PerplexityBot (keep current rules for others)
2. CDN/firewall: allow verified AI search crawlers; confirm with
   curl -A "OAI-SearchBot" <url> returns 200
3. Rendering: main text, pricing tables and FAQs must appear in the
   initial HTML (view-source or curl), not only after JavaScript runs
4. Indexing: no noindex/nosnippet on these pages; all in sitemap.xml
5. Redirects: 301 any 404ing URLs found in the crawl report (attached)

Done when: curl as each bot returns 200 and the page text is in the response.

How to know it worked

Technical fixes don’t show up as a ranking jump. Look for these signs over the following four to six weeks:

• Crawler logs: AI search bots fetching your key pages with 200 responses, more often than before.
• Bing Webmaster Tools: key pages indexed, and Copilot citations appearing in the AI Performance report.
• Prompt checks: run your top 20 buyer prompts in ChatGPT and Perplexity monthly and note when your pages start appearing as sources. Our guide to tracking brand mentions in ChatGPT has a free spreadsheet method.
• Referral traffic: visits from chatgpt.com and perplexity.ai in your analytics.

If crawlers are fetching your pages and you’re still not cited, the problem has moved from access to content and authority. That’s good news: it means the technical foundation is doing its job.

Common mistakes

• Blocking every AI bot at once. Blocking training crawlers is a choice. Blocking search crawlers takes you out of AI answers.
• Trusting robots.txt alone. A CDN setting can still block crawlers. Test with a real request.
• Checking only the homepage. Pricing, comparison and guide pages are the ones that get cited.
• Spending a week on llms.txt and schema. Do the must-dos first; they’re what decides access.
• Never rechecking. Redesigns, plugins and new CDN rules quietly undo fixes. Add the checks to your release process.