A robots.txt file is a short text file that tells search engine crawlers which parts of your site they can visit.1 A crawler is an automated program that follows links from page to page and collects what it finds.
That's the whole job. It's a set of requests to visiting programs, nothing more.
Who reads it
Search engines are the usual readers. Google's own crawler is called Googlebot. Before crawling a site, Google's crawlers download the file and read it to see which parts may be crawled.2
The file sits at the root of your site, so for www.example.com it lives at www.example.com/robots.txt.3
What it is not
Two jobs people expect from it are jobs it can't do.
| What people expect | What actually happens |
|---|---|
| It keeps a page out of Google | Google says not to use it to hide pages. A blocked page's address can still show up in results.1 |
| It keeps people out | The rules aren't a form of access authorization.4 |
| Every crawler obeys it | Google says it can't enforce behaviour, and some crawlers might not follow it.1 |
The second row matters if you're tempted to list a private folder. Malicious bots can use the file as a directory of which pages to visit.5 So a line that names a secret folder can point straight at it. If something is private, protect it with a password instead.
The protocol also relies on voluntary compliance, which Wikipedia says has been true since the standard was developed in 1994.5 Polite crawlers follow it. Others don't have to.
Does a small site need one?
Not always. Unless the file says otherwise, all files are implicitly allowed for crawling.3 So a site with no file isn't shut out of search.
A file earns its place when you want to do one of these:
- Steer well-behaved crawlers away from pages that add nothing, such as internal search results.
- Point crawlers to your sitemap, which Adding a sitemap line to robots.txt covers.
- Avoid overloading your site with requests, which Google gives as the main use.1
If none of those apply, leaving the file off is a fair choice. A file that blocks nothing and no file at all give the same result.
If you use a website builder
Some builders manage the file for you. Google notes that if you use a CMS such as Wix or Blogger, you might not need to (or be able to) edit the file directly.1 Your builder may offer a search settings page instead. Check your own provider's help pages before you go looking for the file.
To check whether your site has a file, see Where robots.txt goes and how to check it is there. For the lines inside it, see Robots.txt syntax.
Sources used
- Introduction to robots.txt, Google Search Central
- How Google interprets the robots.txt specification, Google Search Central
- How to write and submit a robots.txt file, Google Search Central
- RFC 9309: Robots Exclusion Protocol, IETF (RFC Editor)
- robots.txt, Wikipedia