Put the file at the top level of your site and name it robots.txt. For https://www.example.com/ the address is https://www.example.com/robots.txt.1 Crawlers don't look for it in subfolders.1
The rules for the file itself
- The file must be named robots.txt, and your site can have only one.2
- It can't sit in a subfolder. An address such as https://example.com/pages/robots.txt doesn't work.2 Crawlers don't check for robots.txt files in subdirectories.1
- The address is case-sensitive, so keep the name in lower case.1
- It must be a plain text file saved as UTF-8. Google says not to use a word processor, because those can add unexpected characters such as curly quotes.2
A basic text editor is enough. Google lists Notepad and TextEdit among the editors that can make a valid file.2 When the save box asks about encoding, pick UTF-8.2
One file covers one address, not your whole name
This catches people out. The rules in a file apply only to the host, protocol and port where it is hosted.1 Here's what that means in practice.
| Where the file lives | Covers | Doesn't cover |
|---|---|---|
| https://example.com/robots.txt | https://example.com/ and every folder under it | https://other.example.com/ and http://example.com/1 |
| https://www.example.com/robots.txt | https://www.example.com/ | https://example.com/1 |
So if your site answers on both the www and non-www address, check which one you've set as the main one. A file on the other may not do what you think. If you run a blog on a subdomain, it needs a file of its own.
Getting the file onto the site
How you upload depends on your site and server. Google says there's no single tool for it, and suggests asking your hosting company or searching its documentation.2 Follow your own provider's help pages.
Some hosts and builders handle the file for you, or give you a settings page instead. If you can't reach the root of your site, Google suggests another blocking method, such as meta tags.2
Before you change an existing file, copy what's there first. Google's update page suggests downloading a copy, either by opening the address and pasting the text into a new file, or with a tool like curl.3 Keep that copy somewhere safe. If a change goes wrong, you can put the old version back.
Check it is really there
Open a private browsing window and type the file's address. If you see the contents of your file, it's publicly accessible and you're ready to test the markup.2 [fig:check-flow]
What the crawler sees matters too. Google reads the status code your server sends back.
| What your server says | What Google does |
|---|---|
| Success (2xx) | Reads the file as sent.1 |
| Not found or similar (4xx, except 429) | Treats it as if no valid file existed, so no crawl limits.1 |
| Server error (5xx) | Stops crawling the site for the first 12 hours while it keeps trying to fetch the file.1 |
A missing file means Google assumes there are no crawl limits. A file that fails with a server error is a different matter, so if the address shows an error page rather than your rules, fix that first.
For how to check the rules themselves, see How to test a robots.txt file.
Sources used
- How Google interprets the robots.txt specification, Google Search Central
- How to write and submit a robots.txt file, Google Search Central
- Update your robots.txt file, Google Search Central