I'm currently crawling the site, but prices seem to be delayed by an average of 5 minutes. When I check in the browser, the prices seem up-to-date, but the bot can see the current prices after an average of 5 minutes.
I also return random numbers in the url.
The method I am currently trying for bypass:
headers={
'Cache-Control': 'no-cache, max-age=0',
'Expires': 'Sun, 19 Nov 1978 05:00:00 GMT',
'accept-language': 'tr-TR,tr;q=0.9,en-US;q=0.8,en;q=0.7,fr;q=0.6,zh-CN;q=0.5,zh-TW;q=0.4,zh;q=0.3',
'strict-transport-security': 'max-age=31536000',
'age': '0',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
I could not detect any blocking on the site. Which method should I follow so that I can get the current prices simultaneously?
Thanks in advance for your help.