Can domains be used in robots.txt?

user_5818406 0
Viewed 180

We have a development server at dev.example.com that is being indexed by Google. We are using AWS Lightsail to duplicate the development server to our production environment in totality — the same robots.txt file is used on both dev.example.com and example.com.

Google's robots.txt documentation doesn't explicitly state whether root domains can be defined. Can I implement domain specific rules to the robots.txt file? For example, is this acceptable:

User-agent: *
Disallow: https://dev.example.com/

User-agent: *
Allow: https://example.com/

Sitemap: https://example.com/sitemap.xml

To add, this can be resolved through .htaccess rewrite engine — my question is specifically about robots.txt.

robots.txt
2 Answers

No, you can't specify domain in robots.txt. Disallow: https://dev.example.com/ is not valid. Page 6 of the robots.txt exclusion standard says that a disallow line should contain a "path" as opposed to a full URL including the domain.

Each host name (domain or subdomain) has its own robots.txt file. So to prevent Googlebot from crawling http://dev.example.com/ you would need to serve https://dev.example.com/robots.txt with the content:

User-agent: *
Disallow: /

At the same time you would need to serve a different file from http://example.com/, perhaps with the content:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

If the same code base powers both your dev and production servers, you will need to conditionalize the content of robots.txt based on whether it is running in production or not.

Alternately, you could allow Googlebot to crawl both, but include <link rel=canonical href=...> tags in every page that point to the URL for the page on the live site. See How to use rel='canonical' properly

user_11453880

Listing full domains in robots.txt is not allowed according to Google's Create a robots.txt documentation:

"A directory or page, relative to the root domain, that you don't want the user agent to crawl. If the rule refers to a page, it must be the full page name as shown in the browser. It must start with a / character and if it refers to a directory, it must end with the / mark."

user_58184060
Related