What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.]
1.Simulate disk removal
echo "scsi remove-single-device 0:0:8:0" > /proc/scsi/scsi
2.Simulate inserting a disk
echo "scsi add-single-device 0:0:8:0" > /proc/scsi/scsi
happend:
- Pods with mounted PV volumes cannot use the PV volumes, and the CSI did not detect them.
- After restarting the pod, the LV volume cannot be mounted to the pod.
3.You can restore using lvchange an and lvchange ay when the LV is not being used by a pod
What did you expect to happen:
Hope to automatically detect when the disk is removed and inserted for automatic recovery
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)
I0402 06:53:16.075089 1 grpc.go:72] GRPC call: /csi.v1.Node/NodeUnpublishVolume requests {"target_path":"/var/lib/kubelet/pods/768ea25c-8bb4-45cc-90a4-30a6a9cebed8/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount","volume_id":"pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17"}
I0402 06:53:16.148001 1 mount.go:128] umount done pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 path /var/lib/kubelet/pods/768ea25c-8bb4-45cc-90a4-30a6a9cebed8/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount
I0402 06:53:16.150122 1 agent.go:309] hostpath: volume pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 path: /var/lib/kubelet/pods/768ea25c-8bb4-45cc-90a4-30a6a9cebed8/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount has been unmounted.
I0402 06:53:16.150188 1 grpc.go:81] GRPC response: {}
I0402 06:53:17.608025 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"idmes-es-node3-0","csi.storage.k8s.io/pod.namespace":"elastic","csi.storage.k8s.io/pod.uid":"af8a4d3d-05b0-4cfa-be7b-3a1731705382","csi.storage.k8s.io/serviceAccount.name":"default","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"node0003-es-0","storage.kubernetes.io/csiProvisionerIdentity":"1774949368192-4826-local.csi.openebs.io"},"volume_id":"pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17"}
I0402 06:53:17.625491 1 mount_linux.go:367] Disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17]
E0402 06:53:19.799780 1 mount_linux.go:373] format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io~csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: 86c23e87-6cb9-489d-b68b-59e79fd515d0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:19.799819 1 mount.go:73] lvm: failed to mount volume /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 [ext4] to /var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount, error format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: 86c23e87-6cb9-489d-b68b-59e79fd515d0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:19.799855 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io~csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: 86c23e87-6cb9-489d-b68b-59e79fd515d0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
I0402 06:53:20.333912 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"idmes-es-node3-0","csi.storage.k8s.io/pod.namespace":"elastic","csi.storage.k8s.io/pod.uid":"af8a4d3d-05b0-4cfa-be7b-3a1731705382","csi.storage.k8s.io/serviceAccount.name":"default","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"node0003-es-0","storage.kubernetes.io/csiProvisionerIdentity":"1774949368192-4826-local.csi.openebs.io"},"volume_id":"pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17"}
I0402 06:53:20.343232 1 mount_linux.go:367] Disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17]
E0402 06:53:24.704607 1 mount_linux.go:373] format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: f1d5261e-4370-4c09-a493-c81bcb3a7b48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:24.704679 1 mount.go:73] lvm: failed to mount volume /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 [ext4] to /var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount, error format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: f1d5261e-4370-4c09-a493-c81bcb3a7b48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:24.704777 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io~csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: f1d5261e-4370-4c09-a493-c81bcb3a7b48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
kubectl logs -f openebs-lvm-localpv-controller-7b6d6b4665-fk78q -n openebs -c openebs-lvm-plugin
kubectl logs -f openebs-lvm-localpv-node-[xxxx] -n openebs -c openebs-lvm-plugin
kubectl get pods -n openebs
kubectl get lvmvol -A -o yaml
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
- LVM Driver version
- Kubernetes version (use
kubectl version):
- Kubernetes installer & version:
- Cloud provider or hardware configuration:
- OS (e.g. from
/etc/os-release):
What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.]
1.Simulate disk removal
echo "scsi remove-single-device 0:0:8:0" > /proc/scsi/scsi
2.Simulate inserting a disk
echo "scsi add-single-device 0:0:8:0" > /proc/scsi/scsi
happend:
3.You can restore using lvchange an and lvchange ay when the LV is not being used by a pod
What did you expect to happen:
Hope to automatically detect when the disk is removed and inserted for automatic recovery
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)
I0402 06:53:16.075089 1 grpc.go:72] GRPC call: /csi.v1.Node/NodeUnpublishVolume requests {"target_path":"/var/lib/kubelet/pods/768ea25c-8bb4-45cc-90a4-30a6a9cebed8/volumes/kubernetes.io
csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount","volume_id":"pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17"}csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mountI0402 06:53:16.148001 1 mount.go:128] umount done pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 path /var/lib/kubelet/pods/768ea25c-8bb4-45cc-90a4-30a6a9cebed8/volumes/kubernetes.io
I0402 06:53:16.150122 1 agent.go:309] hostpath: volume pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 path: /var/lib/kubelet/pods/768ea25c-8bb4-45cc-90a4-30a6a9cebed8/volumes/kubernetes.io
csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount has been unmounted.csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"idmes-es-node3-0","csi.storage.k8s.io/pod.namespace":"elastic","csi.storage.k8s.io/pod.uid":"af8a4d3d-05b0-4cfa-be7b-3a1731705382","csi.storage.k8s.io/serviceAccount.name":"default","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"node0003-es-0","storage.kubernetes.io/csiProvisionerIdentity":"1774949368192-4826-local.csi.openebs.io"},"volume_id":"pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17"}I0402 06:53:16.150188 1 grpc.go:81] GRPC response: {}
I0402 06:53:17.608025 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io
I0402 06:53:17.625491 1 mount_linux.go:367] Disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17]
E0402 06:53:19.799780 1 mount_linux.go:373] format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io~csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: 86c23e87-6cb9-489d-b68b-59e79fd515d0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:19.799819 1 mount.go:73] lvm: failed to mount volume /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 [ext4] to /var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io
csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount, error format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: 86c23e87-6cb9-489d-b68b-59e79fd515d0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:19.799855 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io~csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: 86c23e87-6cb9-489d-b68b-59e79fd515d0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
I0402 06:53:20.333912 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io
csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"idmes-es-node3-0","csi.storage.k8s.io/pod.namespace":"elastic","csi.storage.k8s.io/pod.uid":"af8a4d3d-05b0-4cfa-be7b-3a1731705382","csi.storage.k8s.io/serviceAccount.name":"default","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"node0003-es-0","storage.kubernetes.io/csiProvisionerIdentity":"1774949368192-4826-local.csi.openebs.io"},"volume_id":"pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17"}csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)I0402 06:53:20.343232 1 mount_linux.go:367] Disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17]
E0402 06:53:24.704607 1 mount_linux.go:373] format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: f1d5261e-4370-4c09-a493-c81bcb3a7b48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:24.704679 1 mount.go:73] lvm: failed to mount volume /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 [ext4] to /var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io
csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount, error format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: f1d5261e-4370-4c09-a493-c81bcb3a7b48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:24.704777 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io~csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: f1d5261e-4370-4c09-a493-c81bcb3a7b48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
kubectl logs -f openebs-lvm-localpv-controller-7b6d6b4665-fk78q -n openebs -c openebs-lvm-pluginkubectl logs -f openebs-lvm-localpv-node-[xxxx] -n openebs -c openebs-lvm-pluginkubectl get pods -n openebskubectl get lvmvol -A -o yamlAnything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
kubectl version):/etc/os-release):