What is a robots.txt file?▼
robots.txt is a text file placed at the root of your website that tells search engine crawlers which pages or sections they can or cannot access. It's part of the Robots Exclusion Protocol.
Where should I place robots.txt?▼
The robots.txt file must be placed at the root of your domain, e.g., https://example.com/robots.txt. It won't work if placed in a subdirectory.
Does robots.txt block pages from Google?▼
robots.txt tells crawlers not to crawl certain pages, but it doesn't prevent them from being indexed. If other sites link to a blocked page, Google may still index the URL. Use a 'noindex' meta tag to fully prevent indexing.
What is the Crawl-delay directive?▼
Crawl-delay tells crawlers to wait a specified number of seconds between requests. Google ignores this directive (use Search Console instead), but Bing and Yandex respect it.
Should I include a Sitemap in robots.txt?▼
Yes! Including a Sitemap directive in robots.txt helps search engines discover your sitemap. Format: Sitemap: https://example.com/sitemap.xml