Is your robots.txt blocking search engines?
Robots.txt is a small text file that tells visiting bots where they're allowed to go. One wrong line — usually copied from somewhere, or left behind by an old developer — can quietly block every search engine from your whole site. Here's how to tell if that's happened to you.
What robots.txt does — and doesn't
It's a set of directions for crawlers: which parts of the site they may look at. It doesn't hide anything from a determined visitor — it only asks well-behaved bots to keep out. Block the wrong thing, and you ask Google itself to ignore you.
The one line that hides a whole site
"User-agent: *" followed by "Disallow: /" means "every bot, stay out of everything." It's the most common accidental block in the business — usually a leftover from when the site was under construction, left live by mistake.
A partial block can be just as bad
Blocking only certain folders seems safe until you realise your sitemap, your stylesheets, or your key service pages live inside one of them. A block you can't remember making is a block you should verify.
The AI-search twist
New AI search crawlers — ChatGPT's OAI-SearchBot, PerplexityBot, Claude-SearchBot and the rest — read robots.txt too. A file written to keep only Google in now also keeps every AI assistant out. What used to be harmless is now the new way to be invisible.
How to check yours
Type your domain followed by /robots.txt and read it — or run an audit that parses the file and reports exactly which engines you're blocking, in plain English, plus the noindex tags doing the same job page by page.
More on choosing an audit
Why isn't my website showing up on Google? · How much does an SEO audit cost? · What does an SEO audit actually check? · Free vs paid SEO audit — what's the real difference? · Does my small business really need SEO? · SEO vs Google Ads — which is right for a small business? · Local SEO: how to show up when people search for your service nearby · What's a page title — and why should you care? · A simple website check-up for small-business owners · How long does SEO take to work? · Your customers are asking AI. Is your website part of the answer? · A website audit that tells you what's wrong — and how to fix it · Check your website — see the grade and the top issues · Connect Google Search Console to your Gold report · Why isn't my website indexed by Google? · Why did my website traffic drop? · Core Web Vitals, explained in plain English · Why is my website slow? · Broken links: what they cost you and how to find them · Duplicate content: what it is and why it hurts · Is your website mobile friendly? · Redirect loops and chains: what they are and how to fix them · Missing titles & meta descriptions: the invisible leak · Was my website hacked? Signs and what to do · Thin content: why your pages aren't ranking · Pages missing from your sitemap: why it matters · The SEO mistakes WordPress, Shopify, Wix and Squarespace each make