Skip to content

Robots.txt is not noindex: blocking a crawl versus hiding a page

Reviewed on 20 November 2026

No, blocking a page in robots.txt won't reliably make it disappear. Google says not to use the file as a means to hide your web pages.1 The two tools do different jobs, and mixing them up is a common reason a page stays in search.

Crawling and indexing are different steps

Crawling means a program visits the page. Indexing means the search engine stores it so it can appear in results. Robots.txt speaks to the first step. Noindex speaks to the second.

If Google can't crawl a page, it can't index the content, but it may still index the address and show it without a snippet.2 Google adds that a blocked page's address can still appear in results, with no description.1 That happens when other sites link to the page.

Side by side

Robots.txt disallow Noindex tag or header
What it controls Whether a crawler visits the page Whether the page is kept in search results
Where it lives One file at the top of the site On each page, or in a response header
Page can still be listed Yes, as a bare address.1 Google will drop the page entirely once it sees the rule.3
Suits Cutting down pointless crawling Keeping a page out of results

Noindex is a rule set with either a meta tag or an HTTP response header.3 Both have the same effect, so choose the one that's easier on your site.3 Google doesn't support putting noindex inside the robots.txt file.3

The trap: using both on one page

This is the mistake behind many "why is it still in Google?" questions. For noindex to work, the page can't be blocked by robots.txt. If it is, the crawler will never see the noindex rule, and the page can still appear in search results.3

Why a blocked page can't be noindexedPage has a noindex tagRobots.txt blocks the pageCrawler can't open the pageTag is never seen
The crawler never reaches the tag, so the page can stay listed.

So pick one for each page.

  • Want it out of results? Leave it crawlable and add noindex, or put it behind a password. Google names both.1
  • Just want less crawling? A disallow rule is the right tool. Don't expect it to remove anything.

If a blocked page is already showing

Google's advice for a blocked page that appears in results is to remove the robots.txt entry that blocks it. If you want to hide the page completely, use another method.1 After the block is gone, the crawler can see your noindex rule. To unblock a page from Google, you must edit your robots.txt file.3

Then wait. A page may keep showing because Google hasn't crawled it since you added the rule. Depending on the page, it may take months for Googlebot to revisit.3

If you use a website builder

You might not have access to the raw file or the page code. Google suggests searching your provider's help pages for how to change a page's visibility in search engines.1 Follow their steps for your own account, and don't share your login details with anyone who offers to do it for you.

Not sure which pages your rules block? Work through the testing steps.

Sources used

  1. Introduction to robots.txt, Google Search Central
  2. How Google interprets the robots.txt specification, Google Search Central
  3. Block Search indexing with noindex, Google Search Central