I'm deploying distributed MinIO on Kubernetes cluster. I created a k8s cluster of 2 nodes (physical machine): master has machine name ip19 and a worker node has machine name ip38 (added to cluster by command kubectl join).
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip38 Ready <none> 23h v1.22.2 192.168.1.38 <none> Ubuntu 18.04.5 LTS 4.15.0-156-generic docker://19.3.9
ip19 Ready control-plane,master 5d4h v1.22.2 192.168.1.19 <none> Ubuntu 20.04.2 LTS 5.4.0-86-generic docker://20.10.8
MinIO use the MinIO DirectCSI to automatically provision Persistent Volumes from locally attached drives to support the generated PVC. I had followed direct-csi installation document to install.
$ kubectl direct-csi install --crd
Flag --crd has been deprecated, Will be removed in version 1.5 or greater
I1013 08:08:02.437460 2161586 install.go:99] 'direct.csi.min.io' namespace created
W1013 08:08:02.469568 2161586 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
I1013 08:08:02.500269 2161586 install.go:119] 'direct.csi.min.io' rbac roles created
I1013 08:08:02.531709 2161586 install.go:137] crds successfully registered
I1013 08:08:11.706289 2161586 install.go:146] 'direct.csi.min.io' conversion deployment created
I1013 08:08:14.752083 2161586 register.go:124] successfully updated conversion webhook settings in 'directcsidrives.direct.csi.min.io' CRD
I1013 08:08:14.783354 2161586 register.go:124] successfully updated conversion webhook settings in 'directcsivolumes.direct.csi.min.io' CRD
I1013 08:08:14.798965 2161586 install.go:159] 'direct.csi.min.io' csidriver created
I1013 08:08:15.833652 2161586 request.go:668] Waited for 1.030910794s due to client-side throttling, not priority and fairness, request: GET:https://192.168.1.19:6443/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s
I1013 08:08:16.293940 2161586 install.go:168] 'direct.csi.min.io' storageclass created
I1013 08:08:16.305892 2161586 install.go:177] 'direct.csi.min.io' service created
I1013 08:08:16.330584 2161586 install.go:186] 'direct.csi.min.io' daemonset created
I1013 08:08:22.615619 2161586 install.go:195] 'direct.csi.min.io' deployment created
$ kubectl get all --namespace direct-csi-min-io
NAME READY STATUS RESTARTS AGE
pod/direct-csi-min-io-45lnj 3/3 Running 21 (2m17s ago) 116m
pod/direct-csi-min-io-79b9667549-429b7 2/2 Running 0 115m
pod/direct-csi-min-io-79b9667549-spbwx 2/2 Running 0 115m
pod/direct-csi-min-io-79b9667549-xzwzg 2/2 Running 0 115m
pod/direct-csi-min-io-z8d2q 3/3 Running 21 (2m17s ago) 116m
pod/directcsi-conversion-webhook-5ffbc9745b-4qh42 1/1 Running 0 116m
pod/directcsi-conversion-webhook-5ffbc9745b-647ht 1/1 Running 0 116m
pod/directcsi-conversion-webhook-5ffbc9745b-v95xm 1/1 Running 0 116m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/direct-csi-min-io ClusterIP 10.108.90.115 <none> 12345/TCP 116m
service/directcsi-conversion-webhook ClusterIP 10.96.66.192 <none> 443/TCP 116m
service/directcsi-validation-controller ClusterIP 10.110.11.236 <none> 443/TCP 115m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/direct-csi-min-io 2 2 2 2 2 <none> 116m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/direct-csi-min-io 3/3 3 3 115m
deployment.apps/directcsi-conversion-webhook 3/3 3 3 116m
NAME DESIRED CURRENT READY AGE
replicaset.apps/direct-csi-min-io-79b9667549 3 3 3 115m
replicaset.apps/directcsi-conversion-webhook-5ffbc9745b 3 3 3 116m
And my problem is I only see drives in master node (machine name ip19) but not see drives in machine ip38.
$ kubectl direct-csi drives ls
DRIVE CAPACITY ALLOCATED FILESYSTEM VOLUMES NODE ACCESS-TIER STATUS
/dev/sda2 1.0 GiB 493 MiB ext4 - ip19 - Available
/dev/sdb 1.8 TiB 1.8 TiB - - ip19 - Available
/dev/sdc 1.8 TiB 30 GiB ext4 - ip19 - Available* failed to format drive ; error while formatting: exit status 1 output: mkfs.xfs: cannot open /dev/sdc/41404a36-7eca-41ff-9d04-38f1548fc97a: No such device or address
/dev/sdd 1.8 TiB 1.8 TiB - - ip19 - Available
/dev/sde 1.8 TiB 1.8 TiB - - ip19 - Available
/dev/sdf 3.6 TiB 3.6 TiB - - ip19 - Available
/dev/sdg 3.6 TiB 3.6 TiB - - ip19 - Available
/dev/sdh 3.6 TiB 3.6 TiB - - ip19 - Available
How can I see all drives in all nodes of k8s cluster?