Page index
- What robots.txt is and what it does
What a robots.txt file is, which programs read it, what it can't do, and whether a small website needs one.
- Where robots.txt goes and how to check it is there
The file has to sit at the top level of your site. Here's where that is, which addresses it covers, and how to confirm crawlers can reach it.
- Robots.txt syntax: user-agent, disallow and allow
What each line in a robots.txt file means, how groups work, how paths and wildcards match, and a short example you can adapt.
- Robots.txt is not noindex: blocking a crawl versus hiding a page
Blocking a page in robots.txt doesn't reliably remove it from Google. Here's how blocking a crawl differs from noindex, and which one suits what.
- Robots.txt mistakes that can hide a whole site
One stray slash or a server error can stop crawlers reading your site. These are the robots.txt slip-ups to check first when pages vanish from search.
- How to test a robots.txt file
Check that your robots.txt file is public, that its rules read the way you meant, and that Google has picked up your latest change.
- Adding a sitemap line to robots.txt
How to add a Sitemap line to robots.txt, what it needs to look like, and how it compares with submitting your sitemap in Search Console.