Plain English · No jargon

Is your robots.txt blocking search engines?

Robots.txt is a small text file that tells visiting bots where they're allowed to go. One wrong line — usually copied from somewhere, or left behind by an old developer — can quietly block every search engine from your whole site. Here's how to tell if that's happened to you.

Audit my site free Buy a report — from €29

What robots.txt does — and doesn't

It's a set of directions for crawlers: which parts of the site they may look at. It doesn't hide anything from a determined visitor — it only asks well-behaved bots to keep out. Block the wrong thing, and you ask Google itself to ignore you.

The one line that hides a whole site

"User-agent: *" followed by "Disallow: /" means "every bot, stay out of everything." It's the most common accidental block in the business — usually a leftover from when the site was under construction, left live by mistake.

A partial block can be just as bad

Blocking only certain folders seems safe until you realise your sitemap, your stylesheets, or your key service pages live inside one of them. A block you can't remember making is a block you should verify.

The AI-search twist

New AI search crawlers — ChatGPT's OAI-SearchBot, PerplexityBot, Claude-SearchBot and the rest — read robots.txt too. A file written to keep only Google in now also keeps every AI assistant out. What used to be harmless is now the new way to be invisible.

How to check yours

Type your domain followed by /robots.txt and read it — or run an audit that parses the file and reports exactly which engines you're blocking, in plain English, plus the noindex tags doing the same job page by page.

See exactly what your robots.txt is blocking — run the free snapshot.

Audit my site free Buy a report — from €29