What could be the reason why scrapy-splash times out and what is the possible fix for this?
I have read through different suggestions on GitHub, Stackoverflow, but None seems to provide a definite answer. my question is similar to this Splash freezes with "Timing out client: IPv4Address"
I get this output when the web scraper times out.
2022-09-13 16:54:53 \[sitename\] ERROR: <twisted.python.failure.Failure twisted.internet.error.TimeoutError: User timeout caused connection failure: Getting [http://localhost:8050/execute](http://localhost:8050/execute) took longer than 180.0 seconds..>
I have tried increasing the timeout when starting splash and also setting a higher timeout in splashRequest args.
docker run -it -p 8050:8050 scrapinghub/splash --max-timeout 3600
From docker log.
2022-09-13 15:52:40.837139 [-] Timing out client: IPv4Address(type='TCP', host='172.17.0.1', port=40138)
2022-09-13 15:52:43.691866 [-] Timing out client: IPv4Address(type='TCP', host='172.17.0.1', port=40132)
2022-09-13 15:52:44.234285 [-] Timing out client: IPv4Address(type='TCP', host='172.17.0.1', port=40146)
2022-09-13 15:52:52.609186 [-] Timing out client: IPv4Address(type='TCP', host='172.17.0.1', port=40126)
2022-09-13 15:52:54.869906 [-] Timing out client: IPv4Address(type='TCP', host='172.17.0.1', port=40104)
2022-09-13 15:52:57.208869 [-] Timing out client: IPv4Address(type='TCP', host='172.17.0.1', port=40148)
2022-09-13 15:52:57.299684 [-] Timing out client: IPv4Address(type='TCP', host='172.17.0.1', port=36082)
2022-09-13 15:53:00.609067 [-] Timing out client: IPv4Address(type='TCP', host='172.17.0.1', port=40110)