What if I leave the user agent empty in the robots.txt file.
It is basically like that: User-agent: *
Disallow: /specific page
But What if it is like:
User-agent:
Disallow: /specific page
1 Answers
User-agent: * Specifies instructions for all robots of all search engines simultaneously
Or if you want to specify for a specific search engine, use User-agent: Googlebot
So if you do not specify an asterisk, there is a chance that search bots will not understand what the designations after
Related