We are currently trying to scrape article ratings on the Amazon.co.uk website using the Scrapy Framework. We use LTE dongles that we use as proxies. We change the IP and the user agent as soon as the bans/captchas come.
On a small level it works without problems and we rarely get banned/captchas and the IP refresh shows its effect.
When we start to scrape larger amounts of Asins we get bans/captchas again IMMEDIATELY after rotating the IP.
What could be the reasons for this?