Technical SEO
Robots.txt
Robots.txt is a plain text file at the root of a domain that tells crawlers which areas of the site they may or may not access.
It controls crawling, not indexing, so a page blocked here can still appear in results if it is linked elsewhere. To keep a page out of the index entirely, use a noindex tag instead. You can view any site's file by adding /robots.txt to its domain.
The file is small enough to review by hand and dangerous enough to deserve it. The single most expensive mistake in SEO is shipping a staging Disallow rule to production, which blocks crawling site-wide and can erase visibility before anyone notices. Two habits prevent it. Check the live file after every deploy that touches infrastructure, and set up an alert on its contents, so a change tells you instead of waiting to be found. Search Console will report a spike in blocked pages, but that report arrives days later, which is long enough to matter.
Example
A relaunch goes live carrying one line from staging: "Disallow: /". Nothing breaks visibly, the site works for every visitor, but crawlers are locked out sitewide. Rankings start sliding within days, and the error is found a week later during a routine check. The rollback takes one minute; the recovery takes most of two months.
Frequently asked questions
- Does robots.txt stop a page appearing in Google?
- No. Robots.txt stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it, usually with no description. To keep a page out of results, allow crawling and use a noindex meta tag instead.
- Should I block AI crawlers in robots.txt?
- It depends on your goal. Blocking GPTBot, ClaudeBot or PerplexityBot prevents your content being used, but it also removes you from AI-generated answers where competitors will still appear. For most marketing sites, visibility in those answers is worth more than the content protection.
Related terms
Working on Robots.txt? See how our SEO audit can help.
Explore SEO audit