Yes, you can point to your sitemap from robots.txt, and it takes one line. A sitemap is a file that lists the pages on your site. The robots.txt line is one of several ways to tell Google where it is.
What the line looks like
Google's example is this:
Sitemap: https://example.com/my_sitemap.xml
You can insert it anywhere in the file, and Google says it will find the sitemap the next time it crawls your robots.txt file.1 Google, Bing and other major search engines support the field.2
A few rules keep the line working:
- Use the full address. It must be a fully qualified URL, including the protocol and host.2
- Match your real address. Google doesn't assume or check http, https, www and non-www alternates.3
- Add more than one if you need to. You can specify multiple sitemap lines, with no limit.1
- The sitemap can live on another host from the robots.txt file.2
The line isn't tied to a particular crawler. It's not part of any group, so it doesn't matter where it sits relative to your user-agent lines.2 Putting it at the bottom keeps the file tidy.
Robots.txt line or Search Console?
Both work, and they aren't exclusive.
| Sitemap line in robots.txt | Submit in Search Console | |
|---|---|---|
| Effort | Edit one text file | Add it in the Sitemaps report |
| What you see afterwards | Nothing extra | When Googlebot accessed the sitemap and any processing errors.1 |
| Who can read it | Any crawler that supports the field.2 | Google, through your own account |
| Suits | Sites that want one place for crawlers to look | Owners who want feedback on the sitemap |
Google calls submitting a sitemap merely a hint. It doesn't guarantee that Google will download the sitemap or use it for crawling URLs.1 So don't expect the line to force pages into search.
If your site already makes a sitemap
If you use a CMS such as WordPress, Wix or Blogger, it's likely your CMS has already made a sitemap available to search engines.1 Open your robots.txt before you add a line, so you can see what's already in it.
Look up your provider's help pages for where its sitemap lives. If you do edit the file, note what's in it first, as the testing page suggests.
Getting the address right
Check the sitemap's real address by opening it in your browser first. If you see your list of pages, copy the address from the browser bar into the line. Because Google doesn't check the other versions of your address, a line with the wrong version points at nothing useful.3
If your site has no sitemap yet, Google describes three ways to make one. Let your CMS generate it, build it by hand for a short list of pages, or have your site software generate it for a bigger site.1
Keep it consistent
A sitemap lists the pages you want found. A disallow rule says a crawler shouldn't visit a page. Check that you aren't listing pages in the sitemap that you've also blocked. The mistakes page has more on rules that block too much.
Anyone can open your robots.txt, so never list private folders in it.
Sources used
- Build and submit a sitemap, Google Search Central
- How Google interprets the robots.txt specification, Google Search Central
- How to write and submit a robots.txt file, Google Search Central