scrapy mega noob here.
When I try to scrapy shell a website like for example:
scrapy shell https://shop.coles.com.au/a/a-vic-metro-oakleigh/product/gasmate-cartridge-butane
I get the following messages:
...
[scrapy.core.engine] INFO: Spider opened
[scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://shop.coles.com.au/a/a-vic-metro-oakleigh/product/gasmate-cartridge-butane> (failed 1 times): 502 Bad Gateway
[scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://shop.coles.com.au/a/a-vic-metro-oakleigh/product/gasmate-cartridge-butane> (failed 2 times): 502 Bad Gateway
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://shop.coles.com.au/a/a-vic-metro-oakleigh/product/gasmate-cartridge-butane> (referer: None)
...
Then when I try to see what the response.body has:
In [1]: print(response.body)
b'<html><body><script>var $j=\'c\';$3=\'c\';$q=\'c\';$G=\'f\';$s=\'c\';$F=\'c\';$X=\'c\';$t=\'c\';$H=\'c\';$e=\'=\';$D=\'c\';$g=\'c\';$8=\'c\';$A=\'c\';$6=\'c\';$O=\'=\';$P=\'c\';$U=\'5\';$4=\'6\';$y=\'c\';$v=\'c\';$u=\'c\';$b=\'c\';$V=\'b\';$r=\'5\';$2=\'6\';$Q=\'f\';$R=\'c\';$5=\'c\';$9=\'c\';$c=\'c\';$S=\'c\';$l=\'c\';$k=\'c\';$m=\'_\';$M=\'5\';$N=\'c\';$C=\'c\';$d=\'c\';$J=\'b\';$E=\'5\';$1=\'6\';$i=\'f\';document.cookie=(!4?$j:"")+(!""?$3:"")+(!4?$q:"")+(!4?$G:"")+(!({})?$s:"")+(!NaN?$F:"")+(!NaN?$X:"")+(![]?$t:"")+(!0?$H:"")+(![]?$e:"")+(!4?$D:"")+(!""?$g:"")+(!""?$8:"")+(![]?$A:"")+(!NaN?$6:"")+(!NaN?$O:"")+(![]?$P:"")+(!0?$U:"")+(!({})?$4:"")+(!4?$y:"")+(!""?$v:"")+(!0?$u:"")+(!0?$b:"")+(!""?$V:"")+(!0?$r:"")+(!0?$2:"")+(!""?$Q:"")+(!0?$R:"")+(!NaN?$5:"")+(!""?$9:"")+(!NaN?$c:"")+(!""?$S:"")+(!""?$l:"")+(!NaN?$k:"")+(!0?$m:"")+(!0?$M:"")+(!""?$N:"")+(!NaN?$C:"")+(!NaN?$d:"")+(!NaN?$J:"")+(!""?$E:"")+(!""?$1:"")+(!0?$i:"")+\'; path=/\';window.location.href=window.location.href;</script></body></html'
Which is not the website HTML, I can check in a Browser that the HTML of https://shop.coles.com.au/a/a-vic-metro-oakleigh/product/gasmate-cartridge-butane is totally different, therefore I know I'm being redirected to somewhere.
Question is how and why this is happening? and most importantly how to avoid it?
Additional info: I'm using a proxy service that will use random proxies each time I use Scrapy shell from a pool of over 20.000. It's also worth noting that I've been scraping this webpage for quite a long time before this issue started.