PV reservation with Local PV ZFS?

[fabriziogiudici]

Hello. I'm working to evaluate OpenEBS as an improved way of managing volumes in a Kubernetes project that presently uses native storage supplied by the cloud provider. My first step is to integrate Local PV ZFS; when and if it will be consolidated I'll try Replicated PV Mayastor.

I followed the instructions and had the system up and running with no big troubles, but I have two issues that so far I wasn't able to fix:

1. The created persistent volumes have the name pvc-__UUID__ and I'd like to have a custom name, as it happens in my current system.
2. Durability works in the sense that I can reboot and still have my data there (I've configured the StorageClass to have reclaimPolicy: Retain), but if I undeploy/redeploy a package that is using a volume (or if I rebuild the system from scratch) new empty persistent volumes are created each time; the old ones are still in the zpool, but unused.

This to me is really a problem since I can't resync the volumes from a backup every time I re-create the cluster, as it would take several minutes. Of course in production undeploying / rebuilding the system is something that I seldom do, but in the test environment it might happen a few times in a day when there is intense development.

Below the configuration files I'm using.

This is the StorageClass I've deployed:

apiVersion:     storage.k8s.io/v1
kind:           StorageClass
metadata:
  name:         openebs-zfs-sc
allowVolumeExpansion: true
parameters:
  recordsize:   '128k'
  compression:  'on' 
  dedup:        'on'
  fstype:       'zfs'
  poolname:     'zfspv-pool'
  shared:       'yes'
provisioner:    'zfs.csi.openebs.io'
reclaimPolicy:  'Retain'
allowedTopologies:
- matchLabelExpressions:
  - key:        kubernetes.io/hostname
    values:
      - k8s-hu-t-wo03

This is the PVC:

apiVersion:         v1
kind:               PersistentVolumeClaim
metadata:
  name:             portainer
  namespace:        portainer
  labels:
    app:            portainer
spec:
  storageClassName: openebs-zfs-sc
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage:      512Mi

I've tried to add a volumeName: portainer which, according to the specs, should set the created volume, but the pvc just hangs in Pending state.

I've tried to configure the thing with the usual approach described at https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reserving-a-persistentvolume:

apiVersion:         v1
kind:               PersistentVolumeClaim
metadata:
  name:             portainer
  namespace:        portainer
  labels:
    app:            portainer
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage:      512Mi

and having an explicit PV (as documented also at https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/#pre-provisioned-volumes):

apiVersion:         v1
kind:               PersistentVolume
metadata:
  name:             portainer
  labels:
    app:            portainer
spec:
  storageClassName: openebs-zfs-sc
  claimRef:
    name:           portainer
    namespace:      portainer
  accessModes:
    - ReadWriteOnce
  capacity:
    storage:        512Mi
  persistentVolumeReclaimPolicy: Retain # redundant
  volumeMode:       Filesystem
  csi:
    driver:         zfs.csi.openebs.io
    fsType:         zfs
    volumeHandle:   portainer

A pv with the desided name appears and it seems bound to the claim:

kubectl describe pv portainer
Name:            portainer
Labels:          app=portainer
Annotations:     <none>
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    openebs-zfs-sc
Status:          Bound
Claim:           portainer/portainer
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        512Mi
Node Affinity:   <none>
Message:         
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            zfs.csi.openebs.io
    FSType:            zfs
    VolumeHandle:      portainer
    ReadOnly:          false
    VolumeAttributes:  <none>
Events:                <none>

Also the pvc seems ok:

kubectl describe pvc portainer -n portainer
Name:          portainer
Namespace:     portainer
StorageClass:  
Status:        Bound
Volume:        portainer
Labels:        app=portainer
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       portainer-756d949455-9k88r
Events:        <none>

But the pod can't mount it:

kubectl describe pod portainer-756d949455-9k88r -n portainer
...
Events:
  Type     Reason       Age                  From     Message
  ----     ------       ----                 ----     -------
  Warning  FailedMount  2m6s (x45 over 77m)  kubelet  MountVolume.SetUp failed for volume "portainer" : rpc error: code = Internal desc = zfsvolumes.zfs.openebs.io "portainer" not found

And actually it doesn't appear in the zfs list on the node where the zpool is accessible.

I've read https://github.com/openebs/openebs/issues/479, but the proposed solution is cumbersome and not easily automatable. I have playbooks capable to create the infrastructure and deploy k8s from scratch without operator intervention and I want them to stay.

Please:

1. let me know whether I've done something wrong or whether OpenEBS Local PV doesn't support pv reservation.
2. whether there's a workaround for my problem.

Thanks.