How to detect amazon sitemap

Viewed 3818

I am trying to scrape some products from amazon.com, but it I can't find it in its robots.txt

I tried

amazon.com/sitemap.xml
amazon.com/sitemap.xml.gz
amazon.com/sitemap1.xml.gz
amazon.com/sitemap1.xml

all turn-up nothing

I also tried sitemap detector such like

https://seositecheckup.com/tools/sitemap-test

The result shows Amazon doesn't have a sitemap. Is that true? or I didn't have the correct approach.

1 Answers

Look at robots.txt, you will see a sitemap link at bottom with access denied. This ressources may be accessible only to robots (specific user-agent, IP...).

Related