I followed the directions to create an H2o cluster in K8s here. The headless service description that I deployed is:
apiVersion: v1
kind: Service
metadata:
name: h2o-service
namespace: dasi-ml-elr-dev
spec:
type: ClusterIP
clusterIP: None
selector:
app: h2o-k8s
ports:
- protocol: TCP
port: 54321
and the StatefulSet description that I deployed is:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: h2o-stateful-set
namespace: dasi-ml-elr-dev
spec:
serviceName: h2o-service
podManagementPolicy: "Parallel"
replicas: 3
selector:
matchLabels:
app: h2o-k8s
template:
metadata:
labels:
app: h2o-k8s
spec:
containers:
- name: h2o-k8s
image: h2oai/h2o-open-source-k8s:latest
resources:
limits:
cpu: "4000m"
memory: "8Gi"
requests:
cpu: "4000m"
memory: "8Gi"
ports:
- containerPort: 54321
protocol: TCP
readinessProbe:
httpGet:
path: /kubernetes/isLeaderNode
port: 8081
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 1
env:
- name: H2O_KUBERNETES_SERVICE_DNS
value: h2o-service.dasi-ml-elr-dev.svc.cluster.local
- name: H2O_NODE_LOOKUP_TIMEOUT
value: '180'
- name: H2O_NODE_EXPECTED_COUNT
value: '3'
- name: H2O_KUBERNETES_API_PORT
value: '8081'
I can see that the pods in the cluster are running:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
h2o-stateful-set-0 1/1 Running 0 5h27m
h2o-stateful-set-1 1/1 Running 0 5h27m
h2o-stateful-set-2 1/1 Running 0 5h27m
Although the logs from each pod include "Using the following pods to form H2O cluster: [192.168.38.24,192.168.32.78,192.168.16.159]" they also all include "Created cluster of size 1, leader node IP is 'h2o-stateful-set-1.h2o-service.dasi-ml-elr-dev.svc.cluster.local/192.168.32.78'" or "Created cluster of size 1, leader node IP is 'h2o-stateful-set-2.h2o-service.dasi-ml-elr-dev.svc.cluster.local/192.168.38.24'" or "Created cluster of size 1, leader node IP is '/192.168.16.159'" depending on which pod's log I'm looking at. Each pod appears to think that it's the leader. Does anyone know why and how to fix this? Here is a full log example from one of the pods.
$ kubectl logs h2o-stateful-set-1
15:00:35.214 [main] WARN hex.tree.xgboost.util.NativeLibrary - Cannot load library from path lib/linux_64/libxgboost4j_gpu.so
15:00:35.217 [main] WARN hex.tree.xgboost.util.NativeLibrary - Failed to load library from both native path and jar!
15:00:35.218 [main] INFO hex.tree.xgboost.util.NativeLibraryLoaderChain - Cannot load library: xgboost4j_gpu (lib/linux_64/libxgboost4j_gpu.so)
15:00:35.242 [main] INFO hex.tree.xgboost.util.NativeLibrary - Loaded library from lib/linux_64/libxgboost4j_minimal.so (/tmp/libxgboost4j_minimal11832558774653271011.so)
15:00:35.256 [main] INFO water.k8s.H2OCluster - Starting Kubernetes-related REST API services
15:00:35.273 [main] INFO water.k8s.H2OCluster - Kubernetes REST API services successfully started.
15:00:35.273 [main] INFO water.k8s.H2OCluster - Initializing H2O Kubernetes cluster
15:00:35.274 [main] INFO water.k8s.H2OCluster - Timeout contraint: 180 seconds.
15:00:35.274 [main] INFO water.k8s.H2OCluster - Cluster size constraint: 3 nodes.
15:00:35.289 [main] INFO water.k8s.lookup.KubernetesDnsLookup - Timeout for node discovery is set to 180 seconds.
15:00:35.289 [main] INFO water.k8s.lookup.KubernetesDnsLookup - Desired cluster size is set to 3 nodes.
15:00:35.346 [main] INFO water.k8s.lookup.KubernetesDnsLookup - New H2O pod with DNS record 'h2o-stateful-set-0.h2o-service.dasi-ml-elr-dev.svc.cluster.local./192.168.16.159' discovered.
15:00:40.448 [main] INFO water.k8s.lookup.KubernetesDnsLookup - New H2O pod with DNS record 'h2o-stateful-set-1.h2o-service.dasi-ml-elr-dev.svc.cluster.local./192.168.32.78' discovered.
15:02:30.513 [main] INFO water.k8s.lookup.KubernetesDnsLookup - New H2O pod with DNS record 'h2o-stateful-set-2.h2o-service.dasi-ml-elr-dev.svc.cluster.local./192.168.38.24' discovered.
15:02:31.515 [main] INFO water.k8s.H2OCluster - **Using the following pods to form H2O cluster: [192.168.38.24,192.168.32.78,192.168.16.159]**
2022-08-05 15:02:31.587:INFO::main: Logging initialized @117689ms to org.eclipse.jetty.util.log.StdErrLog
08-05 15:02:31.778 192.168.32.78:54321 1 main INFO water.default: Dynamically loaded 'water.k8s.KubernetesEmbeddedConfigProvider' as AbstractEmbeddedH2OConfigProvider.
08-05 15:02:31.779 192.168.32.78:54321 1 main INFO water.default: ----- H2O started -----
08-05 15:02:31.780 192.168.32.78:54321 1 main INFO water.default: Build git branch: rel-zumbo
08-05 15:02:31.780 192.168.32.78:54321 1 main INFO water.default: Build git hash: 0c87fbeb537d68e7d140dd25fdedc4bb5f02f402
08-05 15:02:31.780 192.168.32.78:54321 1 main INFO water.default: Build git describe: jenkins-3.36.1.3-23-g0c87fbe
08-05 15:02:31.781 192.168.32.78:54321 1 main INFO water.default: Build project version: 3.36.1.4
08-05 15:02:31.781 192.168.32.78:54321 1 main INFO water.default: Build age: 1 day
08-05 15:02:31.781 192.168.32.78:54321 1 main INFO water.default: Built by: 'jenkins'
08-05 15:02:31.781 192.168.32.78:54321 1 main INFO water.default: Built on: '2022-08-03 17:21:00'
08-05 15:02:31.782 192.168.32.78:54321 1 main INFO water.default: Found H2O Core extensions: [XGBoost, KrbStandalone, Infogram]
08-05 15:02:31.782 192.168.32.78:54321 1 main INFO water.default: Processed H2O arguments: []
08-05 15:02:31.782 192.168.32.78:54321 1 main INFO water.default: Java availableProcessors: 4
08-05 15:02:31.782 192.168.32.78:54321 1 main INFO water.default: Java heap totalMemory: 130.0 MB
08-05 15:02:31.783 192.168.32.78:54321 1 main INFO water.default: Java heap maxMemory: 4.00 GB
08-05 15:02:31.783 192.168.32.78:54321 1 main INFO water.default: Java version: Java 11.0.16 (from Red Hat, Inc.)
08-05 15:02:31.783 192.168.32.78:54321 1 main INFO water.default: JVM launch parameters: [-XX:+UseContainerSupport, -XX:MaxRAMPercentage=50]
08-05 15:02:31.783 192.168.32.78:54321 1 main INFO water.default: JVM process id: 1@h2o-stateful-set-1
08-05 15:02:31.784 192.168.32.78:54321 1 main INFO water.default: OS version: Linux 4.18.0-240.22.1.el8_3.x86_64 (amd64)
08-05 15:02:31.784 192.168.32.78:54321 1 main INFO water.default: Machine physical memory: 8.00 GB
08-05 15:02:31.784 192.168.32.78:54321 1 main INFO water.default: Machine locale: en_US
08-05 15:02:31.784 192.168.32.78:54321 1 main INFO water.default: X-h2o-cluster-id: 1659711634049
08-05 15:02:31.785 192.168.32.78:54321 1 main INFO water.default: User name: 'root'
08-05 15:02:31.785 192.168.32.78:54321 1 main INFO water.default: IPv6 stack selected: false
08-05 15:02:31.785 192.168.32.78:54321 1 main INFO water.default: Possible IP Address: eth0 (eth0), 192.168.32.78
08-05 15:02:31.785 192.168.32.78:54321 1 main INFO water.default: Possible IP Address: lo (lo), 127.0.0.1
08-05 15:02:31.786 192.168.32.78:54321 1 main INFO water.default: H2O node running in unencrypted mode.
08-05 15:02:31.787 192.168.32.78:54321 1 main INFO water.default: Internal communication uses port: 54322
08-05 15:02:31.787 192.168.32.78:54321 1 main INFO water.default: Listening for HTTP and REST traffic on http://192.168.32.78:54321/
08-05 15:02:31.788 192.168.32.78:54321 1 main INFO water.default: H2O cloud name: 'root' on /192.168.32.78:54321, static configuration based on -flatfile null
08-05 15:02:31.788 192.168.32.78:54321 1 main INFO water.default: If you have trouble connecting, try SSH tunneling from your local machine (e.g., via port 55555):
08-05 15:02:31.789 192.168.32.78:54321 1 main INFO water.default: 1. Open a terminal and run 'ssh -L 55555:localhost:54321 root@192.168.32.78'
08-05 15:02:31.789 192.168.32.78:54321 1 main INFO water.default: 2. Point your browser to http://localhost:55555
08-05 15:02:32.406 192.168.32.78:54321 1 main INFO water.default: Kerberos not configured
08-05 15:02:32.408 192.168.32.78:54321 1 main INFO water.default: Log dir: '/tmp/h2o-root/h2ologs'
08-05 15:02:32.408 192.168.32.78:54321 1 main INFO water.default: Cur dir: '/'
08-05 15:02:32.419 192.168.32.78:54321 1 main INFO water.default: Subsystem for distributed import from HTTP/HTTPS successfully initialized
08-05 15:02:32.420 192.168.32.78:54321 1 main INFO water.default: HDFS subsystem successfully initialized
08-05 15:02:32.423 192.168.32.78:54321 1 main INFO water.default: S3 subsystem successfully initialized
08-05 15:02:32.433 192.168.32.78:54321 1 main INFO water.default: GCS subsystem successfully initialized
08-05 15:02:32.434 192.168.32.78:54321 1 main INFO water.default: Flow dir: '/root/h2oflows'
08-05 15:02:32.443 192.168.32.78:54321 1 main INFO water.default: Cloud of size 1 formed [/192.168.32.78:54321]
08-05 15:02:32.443 192.168.32.78:54321 1 main INFO water.default: Created cluster of size 1, leader node IP is 'h2o-stateful-set-1.h2o-service.dasi-ml-elr-dev.svc.cluster.local/192.168.32.78'
08-05 15:02:32.451 192.168.32.78:54321 1 main INFO water.default: Registered parsers: [GUESS, ARFF, XLS, SVMLight, AVRO, PARQUET, CSV]
08-05 15:02:32.468 192.168.32.78:54321 1 main INFO water.default: XGBoost extension initialized
08-05 15:02:32.469 192.168.32.78:54321 1 main INFO water.default: KrbStandalone extension initialized
08-05 15:02:32.469 192.168.32.78:54321 1 main INFO water.default: Infogram extension initialized
08-05 15:02:32.469 192.168.32.78:54321 1 main INFO water.default: Registered 3 core extensions in: 1143ms
08-05 15:02:32.470 192.168.32.78:54321 1 main INFO water.default: Registered H2O core extensions: [XGBoost, KrbStandalone, Infogram]
08-05 15:02:32.470 192.168.32.78:54321 1 main INFO water.default: Registered: 1 auth extensions in: 116272ms
08-05 15:02:32.470 192.168.32.78:54321 1 main INFO water.default: Registered Auth extensions: [LeaderNodeRequestFilter]
08-05 15:02:32.476 192.168.32.78:54321 1 main INFO hex.tree.xgboost.XGBoostExtension: Found XGBoost backend with library: xgboost4j_minimal
08-05 15:02:32.476 192.168.32.78:54321 1 main WARN hex.tree.xgboost.XGBoostExtension: Your system supports only minimal version of XGBoost (no GPUs, no multithreading)!
08-05 15:02:32.726 192.168.32.78:54321 1 main INFO water.default: Registered: 270 REST APIs in: 256ms
08-05 15:02:32.727 192.168.32.78:54321 1 main INFO water.default: Registered REST API extensions: [XGBoost, Amazon S3, Algos, Infogram, AutoML, Core V3, TargetEncoder, Core V4]
08-05 15:02:32.826 192.168.32.78:54321 1 main INFO water.default: Registered: 318 schemas in 98ms
08-05 15:02:32.827 192.168.32.78:54321 1 main INFO water.default: H2O started in 118774ms
08-05 15:02:32.827 192.168.32.78:54321 1 main INFO water.default:
08-05 15:02:32.828 192.168.32.78:54321 1 main INFO water.default: Open H2O Flow in your web browser: http://192.168.32.78:54321
08-05 15:02:32.828 192.168.32.78:54321 1 main INFO water.default: