What could stop h2o responding to our R script?

Viewed 697

What are the possible reasons that an h2o service might stop responding to an http REST API request?

We are using the R CRAN package, and after a fair period of time, the h2o server stops responding. We have captured a variety of logs, but there is no obvious error or reason for this. There is also a matter of consistency, which I will detail after the logs. The h2o.logging() log shows:

Time:     2019-01-07 11:13:19.262

GET       http://localhost:54321/3/Jobs/$03017f00000132d4ffffffff$_936500deb000be7364a7e2ce61d5451e
postBody:

curlError:         FALSE
curlErrorMessage:
httpStatusCode:    200
httpStatusMessage: OK
millis:            3513

{"__meta":{"data":"removed as it is not relevant"}}

------------------------------------------------------------

Time:     2019-01-07 11:13:25.013

GET       http://localhost:54321/3/Jobs/$03017f00000132d4ffffffff$_936500deb000be7364a7e2ce61d5451e
postBody:

curlError:         TRUE
curlErrorMessage:  Failed to connect to localhost port 54321: Connection refused
httpStatusCode:    -1
httpStatusMessage:
millis:            88616

The main log shows it was working just before:

Connection successful!

R is connected to the H2O cluster:
    H2O cluster uptime:         2 minutes 9 seconds
    H2O cluster timezone:       Etc/UTC
    H2O data parsing timezone:  UTC
    H2O cluster version:        3.20.0.8
    H2O cluster version age:    3 months and 17 days !!!
    H2O cluster name:           H2O_started_from_R_root_ttz747
    H2O cluster total nodes:    1
    H2O cluster total memory:   255.99 GB
    H2O cluster total cores:    2
    H2O cluster allowed cores:  2
    H2O cluster healthy:        TRUE
    H2O Connection ip:          localhost
    H2O Connection port:        54321
    H2O Connection proxy:       NA
    H2O Internal Security:      FALSE
    H2O API Extensions:         XGBoost, Algos, AutoML, Core V3, Core V4
    R Version:                  R version 3.5.1 (2018-07-02)

  |======================================================================| 100%
  |======================================================================| 100%
Error in .h2o.__checkConnectionHealth() :
  H2O connection has been severed. Cannot connect to instance at http://localhost:54321/
Failed to connect to localhost port 54321: Connection refused
Calls: <Anonymous> -> .h2o.__remoteSend -> .h2o.__checkConnectionHealth

The interesting thing about this is that it is 100% repeatable on machines that have this problem (the majority), but we also have two machines that don't show this problem (at least don't seem to, we can't rule out they never have, but they usually run it okay).

I've seen that h2o never shuts itself down on its own, so that is unlikely. We stopped any parallelism in the R script, and that didn't change anything.

We therefore need suggestions on how to identify the problem here, and of course fix it.

1 Answers

It looks like the h2o connection 'open' too many files beyond the default limit set on the machine and that causes the crash.

The limit on open file descriptors can be checked with:

ulimit -n

And later the soft limit can be modified to a larger valuer:

ulimit -Sn 5000
Related