Plain notes on robots.txt for small websites
Robots.txt, explained for small websites
Robots.txt is a small text file that tells search crawlers where they can and can't go on your site. These notes cover what it does, how to write the rules, how to test them, and the slips that hide pages from search. Each page weighs the options side by side.
All the robots.txt notes
What robots.txt is and what it does
Where robots.txt goes and how to check it is there
Robots.txt syntax: user-agent, disallow and allow
Robots.txt is not noindex: blocking a crawl versus hiding a page
Robots.txt mistakes that can hide a whole site
How to test a robots.txt file
Adding a sitemap line to robots.txt