What steps did you take and what happened:
I followed the various steps from velero and this repo documentations to create a remote snapshot location to a minio instance running on a machine outside the cluster.
I deployed a couple of services and tried to do a backup, unfortunately I am facing an error. The backup reports as Partially Failed and the logs shows that certain volumes couldn't be backed up with the following error:
time="2022-10-13T12:42:00Z" level=warning msg="Epoll wait failed : interrupted system call" backup=velero/test3 cmd=/plugins/velero-blockstore-openebs logSource="/go/src/github.com/openebs/velero-plugin/pkg/clouduploader/server.go
:302" pluginName=velero-blockstore-openebs
<... can be 50+ similar log entries and then ...>
time="2022-10-13T12:42:00Z" level=info msg="1 errors encountered backup up item" backup=velero/test3 logSource="pkg/backup/backup.go:413" name=registry-server-7d5466494d-84fj5
time="2022-10-13T12:42:00Z" level=error msg="Error backing up item" backup=velero/test3 error="error taking snapshot of volume: rpc error: code = Unavailable desc = error reading from server: EOF" logSource="pkg/backup/backup.go:4
17" name=registry-server-7d5466494d-84fj5
What did you expect to happen:
I expect the backup to be successful
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)
kubectl logs deployment/velero -n velero -> here
kubectl logs deployment/maya-apiserver -n openebs -> N/A
velero backup describe <backupname>
Name: test3
Namespace: velero
Labels: velero.io/storage-location=local-backup
Annotations: velero.io/source-cluster-k8s-gitversion=v1.25.2
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=25
Phase: PartiallyFailed (run `velero backup logs test3` for more information)
Errors: 4
Warnings: 165
Namespaces:
Included: *
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Storage Location: local-backup
Velero-Native Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2022-10-13 13:41:29 +0100 BST
Completed: 2022-10-13 13:42:38 +0100 BST
Expiration: 2022-11-12 12:41:28 +0000 GMT
Total items to be backed up: 767
Items backed up: 767
Velero-Native Snapshots: 1 of 5 snapshots completed successfully (specify --details for more information)
CSI Volume Snapshots: <none included>
velero backup logs <backupname> -> here
Anything else you would like to add:
Currently migrating a 25 nodes cluster to k8s. This is the initial setup/test before definitive migration with just 5 nodes, self-hosted microk8s cluster running cStor
Environment:
- Velero version (use
velero version):
Client:
Version: v1.9.2
Git commit: -
Server:
Version: v1.9.2
- Velero features (use
velero client config get features): features: EnableCSI
- Velero-plugin version:
3.3.0
- OpenEBS version:
3.3.0
- Kubernetes version (use
kubectl version): 1.25
- Kubernetes installer & version:
- Cloud provider or hardware configuration: 5 nodes, home made from consumer products, bought over the years so nodes are slightly different
- Intel i7 or i9 from 8th to 10th Gen
- 16GB to 32GB
- 2 nodes have 3x 1TB SSD for cStor
- OS (e.g. from
/etc/os-release):
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
What steps did you take and what happened:
I followed the various steps from velero and this repo documentations to create a remote snapshot location to a minio instance running on a machine outside the cluster.
I deployed a couple of services and tried to do a backup, unfortunately I am facing an error. The backup reports as Partially Failed and the logs shows that certain volumes couldn't be backed up with the following error:
What did you expect to happen:
I expect the backup to be successful
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)
kubectl logs deployment/velero -n velero-> herekubectl logs deployment/maya-apiserver -n openebs-> N/Avelero backup describe <backupname>velero backup logs <backupname>-> hereAnything else you would like to add:
Currently migrating a 25 nodes cluster to k8s. This is the initial setup/test before definitive migration with just 5 nodes, self-hosted microk8s cluster running cStor
Environment:
velero version):velero client config get features):features: EnableCSI3.3.03.3.0kubectl version):1.25/etc/os-release):