How to Optimize Your Website for AI Search: The Technical Checklist
The AI search technical checklist: which crawlers to allow, the Cloudflare change, JavaScript rendering, and meta tags that pull pages from AI answers.
Tanissh Amit
TL;DR: Most of the technical work behind AI search visibility is ordinary SEO. Google says AI Overviews and AI Mode have no additional technical requirements beyond indexing and snippet eligibility, and Bing says Copilot runs on the same crawling and indexing foundation as Bing search. The AI-specific work is short and easy to miss: allow each vendor's search crawler separately from its training crawler, confirm your CDN or firewall is not blocking them, serve content in raw HTML, and remove directives such as nosnippet, NOARCHIVE and NOCACHE from pages you want cited. llms.txt and special AI schema are not on the list.
To optimize for AI search at the technical level, make sure AI search crawlers can reach your pages, read them without running JavaScript, and use them without a directive telling the engine not to. Google's documentation states that a page must be indexed and eligible to show with a snippet to appear in AI Overviews or AI Mode, and that there are no additional technical requirements. This checklist covers the few places where AI engines depart from that baseline, taken from each engine's own documentation.
I run this list before any content work starts, because an engine cannot cite a page it cannot fetch.
Most of the AI search checklist is SEO you should already have
An AI search crawler is a bot an AI company uses to fetch and index web pages so its assistant can cite them in answers. A training crawler is a different bot that collects content to build future models. The distinction matters because the major AI companies now run them separately, and blocking one does not block the other.
The foundation is shared. Google lists the same fundamentals for AI features as for Search: crawling allowed in robots.txt and by any CDN or hosting infrastructure, internal links, a good page experience, important content in text form, and structured data that matches the visible page. Bing's Webmaster Guidelines say Bing and Copilot rely on the same core crawling, indexing and ranking foundation as traditional search.
I set out where SEO and GEO share this layer and where they split in SEO and GEO: do you need both. What follows is the part a standard technical SEO audit does not check.
Which AI crawlers should you allow, and which can you block?
Allow every search crawler you want citations from. Block training crawlers only if you have a content-licensing reason to. OpenAI, Anthropic and Perplexity each document separate bots for search indexing, model training and user-triggered fetches, and each is controlled independently in robots.txt.
| Company | Search crawler | Training crawler | User-triggered fetcher | Does robots.txt apply to the fetcher? |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User | OpenAI says robots.txt rules may not apply |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User | Anthropic says its bots honour robots.txt |
| Perplexity | PerplexityBot | None listed; Perplexity says PerplexityBot is not used for training | Perplexity-User | Perplexity says it generally ignores robots.txt |
| Googlebot (Search, AI Overviews, AI Mode) | Google-Extended token (Gemini training and grounding) | Not covered here | Not applicable |
The consequences are specific. OpenAI's crawler documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, while disallowing GPTBot only signals that content should not be used for training. OpenAI says a robots.txt change takes about 24 hours to reach its search systems.
Perplexity states that PerplexityBot is not used to crawl content for AI foundation models. Blocking PerplexityBot costs you Perplexity search visibility without protecting anything from training. Anthropic says that disabling Claude-SearchBot prevents your content being indexed for search, which may reduce your visibility in Claude's search results.
Google-Extended works differently. Google documents it as a robots.txt control token with no user agent of its own, used to manage Gemini training and grounding, and states it has no effect on inclusion or ranking in Google Search.
A robots.txt that keeps all three search crawlers open while opting out of OpenAI training looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
Blocking a training crawler is a licensing decision. Blocking a search crawler is a visibility decision. Treat them as two decisions.
Is your CDN or firewall blocking AI search crawlers?
Your CDN or firewall can block AI crawlers even when robots.txt allows them, and robots.txt will not tell you. Google's AI features documentation names this layer directly, asking site owners to ensure crawling is allowed "by any CDN or hosting infrastructure." OpenAI and Perplexity both recommend allowing requests from their published IP ranges, not only allowing their user agents.
Anthropic adds a warning in the other direction. It says blocking its IP addresses may not work correctly or persistently guarantee an opt-out, because it stops Anthropic reading your robots.txt. Robots.txt is the control the AI companies document; IP rules are a blunter tool.
Cloudflare is the setting to check first. On 1 July 2026, Cloudflare announced that it now sorts AI traffic into Search, Agent and Training categories. From 15 September 2026, new domains onboarding to Cloudflare block Training and Agent bots by default on pages that display ads, while Search stays allowed.
The larger change affects existing sites. Cloudflare says that from 15 September 2026, multi-purpose crawlers such as Googlebot, Applebot and Bingbot will be blocked for customers who have chosen to block Training, including through the older "Block AI bots" setting. If anyone on your team switched that setting on to stop AI training, check it now: Cloudflare says owners can opt out of the change in their Security settings.
Can AI crawlers read content that loads with JavaScript?
Treat content that only appears after JavaScript runs as invisible to AI crawlers other than Googlebot. Vercel and MERJ's 2024 analysis of crawler traffic on Vercel's network found that none of the major AI crawlers rendered JavaScript, covering OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot as well as Anthropic's crawler.
The same analysis counted 569 million GPTBot fetches in one month against 4.5 billion for Googlebot. It also found ChatGPT spent 34.82% of its fetches on 404 pages, against 8.22% for Googlebot. Broken URLs and stale redirects cost more with AI crawlers than with Google, so clean sitemaps and correct redirects belong on this list too.
Google says it can process JavaScript content as long as it is not blocked, though it calls JavaScript framework sites more complex to work with. Bing's guidelines go further, telling site owners to avoid hiding critical content behind client-side rendering and warning that content which cannot be reliably rendered may not be selected for grounding results.
The Vercel data is two years old and no newer primary study exists, so test your own pages rather than rely on it. Open a page, choose view source rather than the browser's inspect panel, and search for a sentence from the main content. If the sentence is not in the source, a crawler that does not run JavaScript cannot read it. Server-side rendering is the fix I treat as non-negotiable on any client-rendered site.
Which meta tags remove pages from AI answers?
Several standard robots directives now control whether AI engines can use a page, and Bing documents effects that most SEO audits do not look for. A directive left on a template can remove or truncate every page built from it.
| Directive | Effect on Google AI Overviews and AI Mode | Effect on Bing and Copilot |
|---|---|---|
| noindex | Removes the page from Search, including AI features | Removes the URL from Bing search, Copilot and grounding results |
| nosnippet | Prevents the page being used as a direct input to AI Overviews and AI Mode | May limit Copilot citation quality |
| max-snippet | Limits how much content can be used as a direct input | Not documented |
| data-nosnippet | Not documented for AI features | May limit Copilot citation quality |
| NOARCHIVE | Not documented for AI features | Prevents content being used in Copilot responses and grounding |
| NOCACHE | Not documented for AI features | Limits Copilot to the URL, title and snippet |
The Google effects come from its robots meta tag specifications, which state that nosnippet and max-snippet apply to AI Overviews and AI Mode. The Bing effects come from Bing's Webmaster Guidelines, which recommend the data-snippet attribute if you want to control exactly which text Bing can display or cite.
Bing documents AI effects for NOARCHIVE and NOCACHE that Google's AI features documentation does not mention. NOARCHIVE, originally a way to stop cached copies, now keeps a page out of Copilot's answers.
Which Search Console and Bing settings control AI visibility?
Two platform settings sit outside your site's code and still decide eligibility. Google's July 2026 guidance says a site must be included in Search generative AI features in Search Console to be eligible, in addition to the normal Search requirements.
The Search generative AI control rolled out to every website on 31 August 2026 and defaults to include. The risk is inheritance: a URL-prefix or subdomain property follows its closest configured parent, so an exclusion set at domain level applies silently below it. Google says exclusions take effect within one to two days, and excluded sites receive no traffic or impressions from those features.
For Copilot, verify the site in Bing Webmaster Tools and set up IndexNow. Microsoft says IndexNow helps AI systems reference the most current version of a page, and that Bing Places for Business keeps local details eligible for AI-generated responses.
What can you skip for AI search?
For Google Search, skip llms.txt, content chunking and special AI schema. Google's optimization guide says you can ignore chunking and unnecessary AI text files like llms.txt, and that structured data is not required for generative AI search, with no special schema.org markup to add. Bing says structured data may support clearer grounding but does not guarantee visibility.
I implement llms.txt for clients because it does no harm, but it does not belong near the top of a priority list. I covered the rest of what Google says to ignore in how to use AI for SEO.
What order should you fix AI search technical issues in?
Confirm indexing in Google and Bing. Check that priority pages are indexed and snippet-eligible in both. Bing's guidelines tie Copilot to Bing's own crawling and indexing, so Google rankings do not carry over.
Allow the search crawlers in robots.txt. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot. Decide on GPTBot, ClaudeBot and Google-Extended separately, as a licensing question.
Check the CDN and firewall. Review Cloudflare's AI traffic settings against the 15 September 2026 change and allow OpenAI's and Perplexity's published IP ranges. A block at the edge overrides an allow in robots.txt.
Test raw HTML on every template. Confirm in view source that main content, titles and navigation are in the initial HTML. Move anything missing to server-side rendering.
Audit robots directives at template level. Remove nosnippet, low max-snippet values, NOARCHIVE and NOCACHE from templates whose pages you want cited. Keep noindex only where a page should leave every engine.
Check the Search Console generative AI control. Confirm the domain property is set to include. Then check that no URL-prefix or subdomain property overrides it.
Set up Bing Webmaster Tools and IndexNow. Submit changes as they happen so Copilot cites the current page. Keep sitemaps to canonical URLs with accurate lastmod dates.
Keep entity details consistent. Bing's guidelines ask for people, organisations, products and locations to be named clearly and consistently. Use one version of your business name, services and locations everywhere.
Measure after the fixes. Search Console's generative AI performance report shows AI Overviews and AI Mode impressions by page, and Bing's AI Performance report shows Copilot citations. For crawler activity, I use Microsoft Clarity's bot reports.
Frequently asked questions
- Should I block GPTBot?
- Block GPTBot only if you do not want OpenAI to use your content for model training, because blocking it does not remove you from ChatGPT search. OpenAI documents GPTBot and OAI-SearchBot as independent settings: GPTBot collects content for training, while OAI-SearchBot surfaces sites in ChatGPT's search answers. A site can disallow GPTBot and allow OAI-SearchBot, and it will still be eligible to appear in ChatGPT search. The same split applies at Anthropic, where ClaudeBot handles training and Claude-SearchBot handles search indexing.
- Does blocking AI training crawlers hurt my visibility in AI search?
- No, provided you block only the training crawlers and leave the search crawlers open. OpenAI, Anthropic and Perplexity each separate search indexing from training. The risk comes from blanket rules. A robots.txt that disallows every AI user agent, or a firewall setting that blocks all AI bots, removes you from AI search along with training. Cloudflare's rules from 15 September 2026 extend a Training block to multi-purpose crawlers such as Googlebot and Bingbot unless you opt out.
- Do I need llms.txt to show up in AI search?
- No. Google's July 2026 guidance says you can ignore unnecessary AI text files like llms.txt for Google Search, which covers AI Overviews and AI Mode. I still add llms.txt for clients because it does no harm, but it does not decide whether an engine can use your pages. Crawler access, raw HTML and clean robots directives do. Fix those first, and treat llms.txt as the last item on the list rather than the first. Anyone selling it as your route into AI Overviews is selling something Google has said it ignores.
- How do I check whether AI crawlers can access my site?
- Check three layers: robots.txt, the CDN or firewall, and the HTML itself. Read your robots.txt for rules naming OAI-SearchBot, Claude-SearchBot, PerplexityBot or a blanket disallow. Check your CDN's bot settings, especially Cloudflare's AI traffic categories. Then open a key page in view source and confirm the main content is present without JavaScript. Server logs or Microsoft Clarity's bot activity report will show whether AI crawlers are actually reaching your pages. If a crawler is allowed everywhere and still never appears in the logs, look at the CDN first.
- How long does it take for AI search to pick up a robots.txt change?
- About a day for the AI search crawlers, and a few days for Google's AI features. OpenAI says it can take around 24 hours for its search systems to reflect a robots.txt update, and Perplexity says changes can take up to 24 hours. For Google, changes to the Search generative AI control take effect within one to two days, though Google says some content can take longer because of caching. Allow for that lag before judging whether a fix worked.
Find out what is stopping you being cited
If you want to know whether any of these technical issues is costing you citations today, I run a free AI visibility audit. It shows where your business appears across ChatGPT, Gemini, Perplexity, Copilot and Google AI Overviews, and what is stopping your pages being cited.
[1][2][3][4][5][6][7][8][9][10][11][12][13]
Sources
Tagged