I have an experimental web crawler and I noticed that it cannot read some pages, for example on some particular domains curl says it failed after following 50 redirects but wget reads that same domain just fine:
curl 'netflix.com' -L -o 'output.txt'
Result:
curl: (47) Maximum (50) redirects followed
No data in output.txt file.
While this command works fine:
wget netflix.com
Any ideas on what can cause this? I doubt that remote server handles requests based on the two different user agents.