Testing has two halves. First, can anyone fetch the file? Second, do the rules block what you meant and nothing else? Google's guide to creating the file lists testing as the last of four steps.1
Before you save: read it like a crawler
Take a copy first. Google's update page lists downloading the live file as the starting point, and the curl tool is one way to do it.2 Leave that copy unedited.
Then, for each rule you've added, answer three questions on paper:
- Which crawler is this group for?
- Which paths does each disallow line match?
- Is there a page you want found that one of those lines would catch?
The last question is the one that saves sites. Check your most important pages by hand against every line. The syntax page explains how paths match.
Save the file as UTF-8 in a plain text editor.2 Google ignores invalid lines, so check that each line reads the way you meant.3
After you upload: three checks
| Check | What it tells you | Who it suits |
|---|---|---|
| Open the address in a private window | Whether the file is public. If you see your rules, you're ready to test the markup.1 | Everyone |
| The robots.txt report in Search Console | How Google read the file. It only works for files already accessible on your site.1 | Owners with Search Console set up |
| Google's open source robots.txt library | Lets developers test files locally on their own computer.1 | People comfortable with code |
Most small site owners need only the first two. The third is for developers.
Test the pages, not just the file
A clean file can still hide a page you care about. Pick a few key addresses, such as your home page, a service page and a recent article, and compare each with your rules.
For a page that's meant to be out of results, check it a second way. Google's URL Inspection tool shows the HTML that Googlebot received while crawling the page.4 That helps you confirm a noindex tag is really visible. A page blocked in robots.txt can't show it, as this page explains.
Waiting for Google
Google's crawlers notice a changed file and update the cached copy every 24 hours.2 The cache can last longer if Google can't refresh it.3 If you need it sooner, Search Console's robots.txt report has a Request a recrawl function.2
If the file shows an error when you open it, fix that before anything else. A missing file is read as no restrictions.3 For the status codes Google reacts to, see the file location page.
When a test fails
Put back the copy you saved, then change one rule at a time. Testing after each change tells you which line caused the trouble. If your site's host or builder controls the file, use its help pages, and keep your account details to yourself.
Sources used
- How to write and submit a robots.txt file, Google Search Central
- Update your robots.txt file, Google Search Central
- How Google interprets the robots.txt specification, Google Search Central
- Block Search indexing with noindex, Google Search Central