Technical SEO

Robots.txt

Robots.txt is a plain text file at the root of a domain that tells crawlers which areas of the site they may or may not access.

It controls crawling, not indexing, so a page blocked here can still appear in results if it is linked elsewhere. To keep a page out of the index entirely, use a noindex tag instead. You can view any site's file by adding /robots.txt to its domain.