What are the possible reasons that an h2o service might stop responding to an http REST API request?
We are using the R CRAN package, and after a fair period of time, the h2o server stops responding. We have captured a variety of logs, but there is no obvious error or reason for this. There is also a matter of consistency, which I will detail after the logs. The h2o.logging() log shows:
Time: 2019-01-07 11:13:19.262
GET http://localhost:54321/3/Jobs/$03017f00000132d4ffffffff$_936500deb000be7364a7e2ce61d5451e
postBody:
curlError: FALSE
curlErrorMessage:
httpStatusCode: 200
httpStatusMessage: OK
millis: 3513
{"__meta":{"data":"removed as it is not relevant"}}
------------------------------------------------------------
Time: 2019-01-07 11:13:25.013
GET http://localhost:54321/3/Jobs/$03017f00000132d4ffffffff$_936500deb000be7364a7e2ce61d5451e
postBody:
curlError: TRUE
curlErrorMessage: Failed to connect to localhost port 54321: Connection refused
httpStatusCode: -1
httpStatusMessage:
millis: 88616
The main log shows it was working just before:
Connection successful!
R is connected to the H2O cluster:
H2O cluster uptime: 2 minutes 9 seconds
H2O cluster timezone: Etc/UTC
H2O data parsing timezone: UTC
H2O cluster version: 3.20.0.8
H2O cluster version age: 3 months and 17 days !!!
H2O cluster name: H2O_started_from_R_root_ttz747
H2O cluster total nodes: 1
H2O cluster total memory: 255.99 GB
H2O cluster total cores: 2
H2O cluster allowed cores: 2
H2O cluster healthy: TRUE
H2O Connection ip: localhost
H2O Connection port: 54321
H2O Connection proxy: NA
H2O Internal Security: FALSE
H2O API Extensions: XGBoost, Algos, AutoML, Core V3, Core V4
R Version: R version 3.5.1 (2018-07-02)
|======================================================================| 100%
|======================================================================| 100%
Error in .h2o.__checkConnectionHealth() :
H2O connection has been severed. Cannot connect to instance at http://localhost:54321/
Failed to connect to localhost port 54321: Connection refused
Calls: <Anonymous> -> .h2o.__remoteSend -> .h2o.__checkConnectionHealth
The interesting thing about this is that it is 100% repeatable on machines that have this problem (the majority), but we also have two machines that don't show this problem (at least don't seem to, we can't rule out they never have, but they usually run it okay).
I've seen that h2o never shuts itself down on its own, so that is unlikely. We stopped any parallelism in the R script, and that didn't change anything.
We therefore need suggestions on how to identify the problem here, and of course fix it.