Skip to content

Page index

  1. What robots.txt is and what it does

    What a robots.txt file is, which programs read it, what it can't do, and whether a small website needs one.

  2. Where robots.txt goes and how to check it is there

    The file has to sit at the top level of your site. Here's where that is, which addresses it covers, and how to confirm crawlers can reach it.

  3. Robots.txt syntax: user-agent, disallow and allow

    What each line in a robots.txt file means, how groups work, how paths and wildcards match, and a short example you can adapt.

  4. Robots.txt is not noindex: blocking a crawl versus hiding a page

    Blocking a page in robots.txt doesn't reliably remove it from Google. Here's how blocking a crawl differs from noindex, and which one suits what.

  5. Robots.txt mistakes that can hide a whole site

    One stray slash or a server error can stop crawlers reading your site. These are the robots.txt slip-ups to check first when pages vanish from search.

  6. How to test a robots.txt file

    Check that your robots.txt file is public, that its rules read the way you meant, and that Google has picked up your latest change.

  7. Adding a sitemap line to robots.txt

    How to add a Sitemap line to robots.txt, what it needs to look like, and how it compares with submitting your sitemap in Search Console.