Do you have any suggestions on how to disallow these kinds of URLs, that has "-shp" in the end with robots.txt without blocking the product page?
I want to disallow these:
/product/liver-health-formula-shp
/product/blood-sugar-formula-shp
etc.
I want to allow:
/product/liver-health-formula
/product/blood-sugar-formula
etc.
Google and other search engines support wild cards and "ends with" $. So this will work:
User-Agent: *
Disallow: /*-shp$
Most robots are not advanced enough to understand that, but Google and Bing will honor it.
However, preventing a URL from getting crawled using robots.txt won't always prevent it from being indexed. In rare cases, Google can index URLs it doesn't crawl if it thinks they would be important for users. This typically happens when a URL gets a few external links. Then Google indexes just the URL and external anchor text for it without knowing what content it contains. If you are more concerned with preventing indexing than preventing crawling, you should use meta noindex tags instead of robots.txt.