Skip to content

Robots.txt syntax: user-agent, disallow and allow

Reviewed on 20 November 2026

A robots.txt file is a list of short lines. Each is a field, a colon and a value.1 Spaces are optional. They're recommended, though, because they make the file easier to read.1

Four fields matter for a small site: user-agent, disallow, allow and sitemap. Google also says other fields such as crawl-delay aren't supported.1 The sitemap line has its own page.

A group, line by line

Lines come in groups. Each group begins with a user-agent line that says who the rules are for. Then it gives which folders or files that agent can access and which it can't.2

# Keep all crawlers out of the cart pages
User-agent: *
Disallow: /cart/
Allow: /cart/help.html

The # character starts a comment, and comments are ignored.2 Here's what each line does.

Line Meaning
User-agent: * The rules below are for every crawler, apart from the AdsBot crawlers, which must be named.2
Disallow: /cart/ Don't crawl anything in the /cart/ folder. A folder path should end in a slash.2
Allow: /cart/help.html An exception. Allow is used to override a disallow for a page inside a blocked folder.2
How one group is builtUser-agent: who the rules are forDisallow: paths to skipAllow: exceptions inside those paths
Each group starts with a user-agent line and then lists its rules.

Paths are relative to the root of your site, and they must start with a / character.2 They're also case-sensitive. A rule for /file.asp doesn't cover /FILE.asp.2

The default is open

Nothing is blocked until a rule says so. A crawler can crawl any page or folder that a disallow rule doesn't block.2 A disallow line with nothing after it is ignored, so it doesn't mean "block everything".1

Which group wins

Only one group is valid for a particular crawler.1 Google picks the one with the most specific user-agent match, and groups for the generic * aren't combined with groups for a named crawler.1 Order doesn't decide it.

Say you write one group for Googlebot and another for *. Googlebot follows only its own. A rule you meant for everyone won't reach it unless you repeat that rule inside the Googlebot group. Easy to miss.

When two rules clash

Google uses the most specific rule, judged by the length of the path. If rules conflict, including ones with wildcards, Google uses the least restrictive.1

Rules Result for /page
Allow: /p and Disallow: / Allowed, because /p is more specific.1
Allow: /page and Disallow: /page Allowed, because the least restrictive rule wins.1

Other search engines may treat clashes differently. Google says crawlers might interpret rules differently, so check each one's own documentation.3

Two wildcards

Google, Bing and other major search engines support a limited form of wildcards.1 The star means any run of characters, including none.1 The dollar sign means the end of the URL.1

User-agent: *
Disallow: /*.pdf$

That blocks addresses ending in .pdf. Without the dollar sign, it would also match addresses with .pdf in the middle.1

A lone /$ is worth knowing about. It matches only the root, so any lower-level URL stays open for crawling.1 Use it with care.

Lines that don't match any known rule are ignored.2 So a typo in a field name leaves you with a line that does nothing. Crawlers are requested to honour these rules, not forced to.4

For why a disallow line isn't a way to hide a page, read Robots.txt is not noindex.

Sources used

  1. How Google interprets the robots.txt specification, Google Search Central
  2. How to write and submit a robots.txt file, Google Search Central
  3. Introduction to robots.txt, Google Search Central
  4. RFC 9309: Robots Exclusion Protocol, IETF (RFC Editor)