Warning: Do not use storj.io as storage location!

[J4NS-R]

storj.io (formerly known as tardigrade) is a decentralized storage provider, similar to Sia, Filecoin, and others. Storj also provides an S3-compatible gateway, which allows velero to use it as a storage location. I've been doing backups to storj for about two years with no issues, until this week.

As it turns out, there is a limitation in the Storj-hosted S3 gateway regarding listing files that only appears once you have a sufficiently large repo. Not sure of the exact cut-off, but in my case my repo is 617GB with 160k files (objects). The error I keep seeing in the logs is this:

velero time="2026-03-11T18:24:04Z" level=debug msg="refresh attempt failed: error refreshing: error loading write epoch: error listing all blobs: ListObjects(V2): listing too many items for gateway-side filtering using arbitrary delimiter/prefix, sleeping 15s before next retry" logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:92" logger name="[epoch-manager]"

Once you hit this point everything breaks. Velero can no longer perform backups or even do repo maintenance (eg deleting expired backups).

This is a known problem with the Storj S3 gateway. It has to do with the fact that storj encrypts object paths. There is an open issue from 2024 to address this, but it doesn't look like it's high on the priority list.

Potential workarounds

• You can self-host your own Storj S3 gateway and specify a custom flag --s3.fully-compatible-listing. I do commend storj for at least allowing this, though of course you need the additional compute and storage available to run this oneself.
• In the past storj provided a native velero plugin, but this has since been archived supposedly because of the S3 gateway (lol)
• Create a new bucket and disable object path encryption. This requires a full download and re-upload.

[kaovilai]

Thanks for the PSA! I wonder if we'd like to doc this or not.

[J4NS-R]

Certainly. Especially since the archived storj plugin is currently listed as a supported backend. Easiest would be to just remove all mention of storj from the docs.