Skip to content

lvm-localpv removing the disk and then reinserting it, the LV volume cannot be used #464

@hardship1

Description

@hardship1

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.]
1.Simulate disk removal
echo "scsi remove-single-device 0:0:8:0" > /proc/scsi/scsi
2.Simulate inserting a disk
echo "scsi add-single-device 0:0:8:0" > /proc/scsi/scsi
happend:

  1. Pods with mounted PV volumes cannot use the PV volumes, and the CSI did not detect them.
  2. After restarting the pod, the LV volume cannot be mounted to the pod.
    3.You can restore using lvchange an and lvchange ay when the LV is not being used by a pod
    What did you expect to happen:
    Hope to automatically detect when the disk is removed and inserted for automatic recovery

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)
I0402 06:53:16.075089 1 grpc.go:72] GRPC call: /csi.v1.Node/NodeUnpublishVolume requests {"target_path":"/var/lib/kubelet/pods/768ea25c-8bb4-45cc-90a4-30a6a9cebed8/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount","volume_id":"pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17"}
I0402 06:53:16.148001 1 mount.go:128] umount done pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 path /var/lib/kubelet/pods/768ea25c-8bb4-45cc-90a4-30a6a9cebed8/volumes/kubernetes.io
csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount
I0402 06:53:16.150122 1 agent.go:309] hostpath: volume pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 path: /var/lib/kubelet/pods/768ea25c-8bb4-45cc-90a4-30a6a9cebed8/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount has been unmounted.
I0402 06:53:16.150188 1 grpc.go:81] GRPC response: {}
I0402 06:53:17.608025 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io
csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"idmes-es-node3-0","csi.storage.k8s.io/pod.namespace":"elastic","csi.storage.k8s.io/pod.uid":"af8a4d3d-05b0-4cfa-be7b-3a1731705382","csi.storage.k8s.io/serviceAccount.name":"default","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"node0003-es-0","storage.kubernetes.io/csiProvisionerIdentity":"1774949368192-4826-local.csi.openebs.io"},"volume_id":"pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17"}
I0402 06:53:17.625491 1 mount_linux.go:367] Disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17]
E0402 06:53:19.799780 1 mount_linux.go:373] format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io~csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: 86c23e87-6cb9-489d-b68b-59e79fd515d0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:19.799819 1 mount.go:73] lvm: failed to mount volume /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 [ext4] to /var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount, error format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: 86c23e87-6cb9-489d-b68b-59e79fd515d0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:19.799855 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io~csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: 86c23e87-6cb9-489d-b68b-59e79fd515d0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
I0402 06:53:20.333912 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"idmes-es-node3-0","csi.storage.k8s.io/pod.namespace":"elastic","csi.storage.k8s.io/pod.uid":"af8a4d3d-05b0-4cfa-be7b-3a1731705382","csi.storage.k8s.io/serviceAccount.name":"default","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"node0003-es-0","storage.kubernetes.io/csiProvisionerIdentity":"1774949368192-4826-local.csi.openebs.io"},"volume_id":"pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17"}
I0402 06:53:20.343232 1 mount_linux.go:367] Disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17]
E0402 06:53:24.704607 1 mount_linux.go:373] format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io
csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: f1d5261e-4370-4c09-a493-c81bcb3a7b48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:24.704679 1 mount.go:73] lvm: failed to mount volume /dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17 [ext4] to /var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount, error format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.iocsi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: f1d5261e-4370-4c09-a493-c81bcb3a7b48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system
)
E0402 06:53:24.704777 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/node0003-es-0/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17" failed: type:("ext4") target:("/var/lib/kubelet/pods/af8a4d3d-05b0-4cfa-be7b-3a1731705382/volumes/kubernetes.io~csi/pvc-648fadc5-0f9a-40c0-8f26-01b961e56d17/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.47.0 (5-Feb-2023)
Warning: could not erase sector 2: I/O error
Creating filesystem with 24903680 4k blocks and 6225920 inodes
Filesystem UUID: f1d5261e-4370-4c09-a493-c81bcb3a7b48
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Warning: could not read block 0: I/O error
Warning: could not erase sector 0: I/O error
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: I/O error while writing out and closing file system

  • kubectl logs -f openebs-lvm-localpv-controller-7b6d6b4665-fk78q -n openebs -c openebs-lvm-plugin
  • kubectl logs -f openebs-lvm-localpv-node-[xxxx] -n openebs -c openebs-lvm-plugin
  • kubectl get pods -n openebs
  • kubectl get lvmvol -A -o yaml

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • LVM Driver version
  • Kubernetes version (use kubectl version):
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions