Splash freezes with "Timing out client: IPv4Address"

user_8551679 0
Viewed 604

I am running scrapy-splash for scraping data from one website.

Regularly ( randomly) splash freezes with next logs:

[36msplash-service_1        |[0m 2020-07-16 08:49:35.119333 [-] "172.31.0.4" - - [16/Jul/2020:08:49:34 +0000] "POST /execute HTTP/1.1" 200 266018 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"
[36msplash-service_1        |[0m 2020-07-16 08:50:10.012973 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51970)
[36msplash-service_1        |[0m 2020-07-16 08:50:10.858080 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51978)
[36msplash-service_1        |[0m 2020-07-16 08:50:16.873014 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51974)
[36msplash-service_1        |[0m 2020-07-16 08:50:17.547947 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51966)
[36msplash-service_1        |[0m 2020-07-16 08:50:18.037436 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51976)
[36msplash-service_1        |[0m 2020-07-16 08:50:29.064655 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51932)
[36msplash-service_1        |[0m 2020-07-16 08:50:35.119997 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51968)

How can I get the reason of that? Why it might stuck?

P.S I run it with args={"lua_source": self.lua_script_navigate, "timeout":60000}

scrapy-splash
1 Answers

Refer to Splash's HTTP API documention of the argument timeout:

timeout : float : optional

A timeout (in seconds) for the render (defaults to 30).

By default, maximum allowed value for the timeout is 90 seconds. To override it start Splash with --max-timeout command line option. For example, here Splash is configured to allow timeouts up to 5 minutes:

$ docker run -it -p 8050:8050 scrapinghub/splash --max-timeout 300

If you has not started splash with --max-timeout, your lua_script is aborted after 30s even when setting a higher timeout in args.

user_20538080
Related