Meta tag vs robots.txt

Viewed 19706
  1. Is it better to use meta tags* or the robots.txt file for informing spiders/crawlers to include or exclude a page?

  2. Are there any issues in using both the meta tags and the robots.txt?

*Eg: <#META name="robots" content="index, follow">

11 Answers

Here is my knowledge about them. I am talking about their work area. Both we can use for blocking content.

The difference between both is:

  • Meta Robot can block a single page with some piece of the code paste in the header of the website. By using the meta robot tag we tell the search engine for which function we are using meta tag.
  • In Robots.txt file you can block the whole website.

Here is the example of meta robot:

<meta name="robots" content="index, follow"> 
<meta name="robots" CONTENT="all">
<meta name="robots" content="noindex, follow">
<meta name="robots" content="noindex, nofollow">
<meta name="robots" content="index, nofollow" />
<meta name="robots" content="noindex, nofollow" />

Here is the example of Robots.txt file:

Allowing crawlers to crawl all website

user-agent: *
Allow:
Disallow:

Disallowing crawlers to crawl all website

user-agent: *
Allow:
Disallow:/

Is it better to use meta tags* or the robots.txt file for informing spiders/crawlers to include or exclude a page?

Answer: Both are important to use, they are used for different purposes. Robots file is used to include or exclude pages or root files from spider's index. While, Meta tags are used analyse a website page that defines about it's niche & content within the page.

Are there any issues in using both the meta tags and the robots.txt?

Answer: Both should be implemented to sites so that search engine spiders/crawlers can index or de-index the site urls.

Read more here about working of a search engine spiders >>https://www.playbuzz.com/alexhuber10/how-search-and-spider-engines-work

Related