when i restart a pod with 'kubectl delete -n minio pod minio-3', kubelet shows "Volume is already attached by pod minio/minio-3". it seems like that the volume is attached to a old one. How can I make it work.
[root@control01 ~]# kubectl get pod -n minio
NAME READY STATUS RESTARTS AGE
minio-0 0/1 ContainerCreating 0 62m
minio-1 1/1 Running 0 128m
minio-2 1/1 Running 1 6d7h
minio-3 0/1 ContainerCreating 0 96m
[root@control12 ~]# cat /var/log/messages |grep 'Sep 13'
Sep 13 16:48:10 control12 kubelet: E0913 16:48:10.189343 40141 nestedpendingoperations.go:270] Operation for "\"flexvolume-ceph.rook.io/rook-ceph/e51e0fbd-f09f-430a-8b47-2ca1dbdfdc2e-pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9\" (\"e51e0fbd-f09f-430a-8b47-2ca1dbdfdc2e\")" failed. No retries permitted until 2022-09-13 16:50:12.189281072 +0800 CST m=+5554.416909175 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9\" (UniqueName: \"flexvolume-ceph.rook.io/rook-ceph/e51e0fbd-f09f-430a-8b47-2ca1dbdfdc2e-pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9\") pod \"minio-3\" (UID: \"e51e0fbd-f09f-430a-8b47-2ca1dbdfdc2e\") : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9 for pod minio/minio-3. Volume is already attached by pod minio/minio-3. Status Pending"
[root@control01 ~]# kubectl describe pod -n minio minio-3
Name: minio-3
Namespace: minio
Priority: 0
Node: control12/192.168.1.112
Start Time: Tue, 13 Sep 2022 15:18:28 +0800
Labels: app=minio
controller-revision-hash=minio-95c8c444c
statefulset.kubernetes.io/pod-name=minio-3
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/minio
Containers:
minio:
Container ID:
Image: minio/minio:RELEASE.2021-04-06T23-11-00Z
Image ID:
Port: 9000/TCP
Host Port: 0/TCP
Args:
server
http://minio-{0...3}.minio.minio.svc.cluster.local/minio/data
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
...
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 52m (x13 over 93m) kubelet, control12 Unable to attach or mount volumes: unmounted volumes=[minio-data], unattached volumes=[minio-data tz-config default-token-np5x5]: timed out waiting for the condition
Warning FailedMount 11m (x50 over 97m) kubelet, control12 MountVolume.SetUp failed for volume "pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9 for pod minio/minio-3. Volume is already attached by pod minio/minio-3. Status Pending
Warning FailedMount 6m36s (x7 over 95m) kubelet, control12 Unable to attach or mount volumes: unmounted volumes=[minio-data], unattached volumes=[tz-config default-token-np5x5 minio-data]: timed out waiting for the condition
Warning FailedMount 2m1s (x9 over 86m) kubelet, control12 Unable to attach or mount volumes: unmounted volumes=[minio-data], unattached volumes=[default-token-np5x5 minio-data tz-config]: timed out waiting for the condition