If your pages vanished after you touched robots.txt, look for a rule that blocks too much. Most of these slips are a single character.
A slash that blocks everything
A disallow line with just a slash matches the root and any lower-level URL.1 That's the whole site.
User-agent: *
Disallow: /
Google's own examples use this pattern to block all crawlers except AdsBot.2 It's meant for a site that isn't ready, or that should stay closed to crawlers. It's the wrong file for a live site.
Compare it with two lookalikes:
| Line | What it does |
|---|---|
| Disallow: / | Blocks the whole site.1 |
| Disallow: (nothing after it) | A rule without a path is ignored, so nothing is blocked.1 |
| Allow: /$ | Matches only the root. Lower-level pages are allowed for crawling.1 |
Only the first one shuts crawlers out of everything. Read your own file line by line and ask which of these it resembles.
Leftovers from a test copy
If a file was written for a test copy of your site, it may block everything. If someone else built or rebuilt your site, ask whether the file came from a test copy. Then open your live robots.txt and look for a bare disallow slash.
Blocking things the page needs
Google says to leave resource files alone when the page is harder for its crawler to understand without them.3 Blocking whole folders of scripts or styles can stop a page being read properly.
Rules that don't mean what you think
- Case. Rules are case-sensitive, so /Photos/ and /photos/ are different paths.2
- A named group replaces the general one. Only one group is valid for a particular crawler.1 If you add a group for Googlebot, it ignores your group for *.
- A second file. A site can have only one robots.txt file.2 If your builder or a plugin makes one, don't upload a second.
- Wrong editor. A word processor can add unexpected characters such as curly quotes.2 Use a plain text editor.
Blocked pages with a noindex tag
Adding a disallow line for a page that also has noindex stops the crawler reading the tag. The page can then stay in results.4 The noindex comparison shows how the two fight each other.
A file that returns a server error
Even a correct file can cause trouble if the server can't deliver it. If Google finds the file but can't fetch it, crawling of your site pauses for the first 12 hours, and Google keeps retrying the file.1 Check the address in a browser, as in Where robots.txt goes.
Why a fix isn't instant
Google generally caches the file for up to 24 hours.1 So after you fix a bad rule, give it time before you decide the fix failed. Then run the checks on the testing page.
Before any change, note the existing file's contents first. If your host or website builder manages the file, follow its help pages for your own account. Don't hand your login to anyone to "fix" it.
Sources used
- How Google interprets the robots.txt specification, Google Search Central
- How to write and submit a robots.txt file, Google Search Central
- Introduction to robots.txt, Google Search Central
- Block Search indexing with noindex, Google Search Central