AI search guide

10 November 2026

Is your robots.txt quietly blocking you from AI search? What changed and how to check

Every website has a small text file called robots.txt. It sits at yoursite.ie/robots.txt and tells visiting bots where they’re allowed to go. For years, most small businesses never thought about it, because the only bot that mattered was Google’s.

That has changed. Each AI company now runs its own crawlers, and most have split them into two jobs: some collect content to train AI models, and others fetch pages so the AI can find, cite and link to you in answers. Blocking the first is a reasonable privacy choice. Blocking the second makes you invisible.

The bots that decide whether AI search can see you

AI productSearch crawler — allow this to be foundTraining crawler — your choice
ChatGPT searchOAI-SearchBotGPTBot
PerplexityPerplexityBot (not used for training, per Perplexity)—
ClaudeClaude-SearchBotClaudeBot
Microsoft CopilotBingbot (Copilot uses Bing’s index)—
Google AI Overviews / AI ModeGooglebotGoogle-Extended (Gemini training)
Apple Siri / SpotlightApplebotApplebot-Extended

OpenAI says plainly that sites which opt out of OAI-SearchBot “will not be shown in ChatGPT search answers” (OpenAI). Perplexity recommends allowing PerplexityBot “to ensure your site appears in search results” (Perplexity). Anthropic warns that disabling Claude-SearchBot “may reduce your site’s visibility” (Anthropic).

One common mix-up: blocking Google-Extended does not remove you from Google’s AI Overviews. Those are part of Google Search and follow normal Googlebot rules (Google Search Central).

Three ways businesses block AI search by accident

  1. The “block everything except Google” file. Someone once wrote User-agent: * with Disallow: / and an exception for Googlebot. That locks out every AI search bot.
  2. A firewall setting. Cloudflare started blocking AI crawlers by default on new domains in July 2025 (Cloudflare). Security plugins can do the same thing. Your robots.txt can say “welcome” while the firewall says “no”.
  3. A copy-pasted “block AI” list. Lists shared online often lump search bots and training bots together.

A sensible starting point

If you want to appear in AI search but keep your content out of AI training, this is a common setup (check with your web developer before changing anything):

# Search and answer crawlers — allow so AI can find and cite you
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Applebot
Allow: /
# Training crawlers — your choice. Delete these lines to allow training.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://www.yoursite.ie/sitemap.xml

Changes can take around 24 hours to be picked up, according to both OpenAI and Perplexity.

How to check yours in 30 seconds

Type your domain followed by /robots.txt into a browser. If you see Disallow: / under User-agent: * and no Allow lines for the bots above, you have a problem. If the file looks fine but you use Cloudflare or a security plugin, check its bot settings too, because robots.txt alone won’t tell you the whole story.