Refactoring proposal based on AWS SDK and presigned URLs

[claustres]

feathers-blob currently relies on a model conforming to the abstract-blob-store interface. Most of the time it appears feathers-blob is used to store data in cloud object storages, e.g. AWS S3 or Google Cloud Storage, thus relying on e.g. https://github.com/jb55/s3-blob-store or https://github.com/maxogden/google-cloud-storage under-the-hood, which do not seem to be maintained anymore. For instance the S3 module uses https://github.com/nathanpeck/s3-upload-stream as a dependency, which is now deprecated for a long time.

Now most cloud providers provides a similar interface for their Object Storage as Amazon S3 does, we wonder if refactoring feathers-blob directly using an up-to-date version of the AWS SDK wouldn't be the most relevant.

Moreover, a lot of issues indicate that file uploading/downloading is still a challenging task for most people who have to understand a lot of concepts like blobs, data uris, multipart support, the different client/server responsibilities, etc. In order to simplify this we could also rely on presigned URLs.

At Kalisio we have already started an effort with something able to replace feathers-blob based on this proposal. It looks like this:

• backend service
  • create operation => generates a presigned URL to be used to PUT the object based on a key
  • get operation => generates a presigned URL to be used to GET the object based on a key
  • upload/download proxy routes to cloud provider in order to avoid CORS problems in constrained environments (required if your client cannot directly access the provider service)
• additional frontend service methods
  • upload =>get signed URL and post data to either the proxy or the provider service directly
  • download => get signed URL and read data from either the proxy or the provider service directly

Let us know what you think about that, notably if feathers-blob handles others use cases that would not benefit from this proposal. Otherwise, any help is welcome to upgrade this module.

[fratzinger]

Oh yes please!!!
We have two instances. One legacy server is running on a windows vm and thus we use fs with feathers-blob. But the newer server runs on aws with AWS S3. We use https://github.com/jb55/s3-blob-store with multer which really is a pain because it uses too much memory. We need to switch to presigned URLs asap. But we also need to support the legacy windows fs-blob-store but I can get around that with app.get/app.set easily.

I'm highly interested in your workflow! If you can share something in the short run it would be highly appreciated! Even maybe privately and maybe we can work out a way to make it more generic and publicly available. I would love to see this baked into feathers-blob somehow!

We have a database table that stores additional information to uploads. In fact we use a public /uploads service that redirects to internal /uploads-blobs and /uploads-db depending on the request. We also use https://github.com/fratzinger/feathers-casl to decide who has access to which uploads and operations.

[DaddyWarbucks]

I like where this is going! And I agree that upload/download is difficult in feathers for all the reasons listed. Feathers is a JSON server and 99% of its tools, ecosystem, community are driven around that so trying to fit in multipart, etc is difficult. And I agree that most users are using some 3rd party service, and anyone not using a third party service is probably familiar enough with these concepts to comfortably roll their own solution. But, with that said, this rewrite may deserve to be a totally new package because it is so different.

I have not used feathers-blob in the past and have used my own implementation with Cloudinary. So my only concern with this new approach would be its tight coupling with AWS. So something more generic would be better IMO. Maybe something like

const service = blobService({
  sign: (params) => {
    cloudninary.sign(params);
    // aws.sign(params);
    // somethingElse.sign(params);
  },
  upload: (signedUrl, base64) => {
    cloudninary.upload(signedUrl, base64);
    // aws.upload(signedUrl, base64);
    // somethingElse.upload(signedUrl, base64);
  },
 download: (signedUrl) => {
    cloudninary.download(signedUrl);
    // aws.download(signedUrl);
    // somethingElse.download(signedUrl);
  }
})

app.use('/uploads', service)

That leaves all the provider setup and config to the user.

[claustres]

@DaddyWarbucks Thanks for feedback. Using the AWS SDK does not mean a tigh coupling to AWS as their S3 API has becoming a defacto standard. We already use the AWS SDK to connect to three different providers on our side: AWS, OVH and Scaleway. Currently OVH does not yet support CORS for instance, so that a proxy is required on your app. Any feedback on AWS SDK compatibility with others cloud providers is welcomed.

I agree that it might be better to start a whole new module. A the present time our implementation is really simple and read the file content as a buffer then send it to the signed URL on the frontend (or backend if using a proxy). So we read the entire content in memory, which is not appropriate for large files, but is probably sufficient to most use cases such as image/docs upload. Any insight on using presigned URL and multipart upload is also welcomed to improve this.

We will soon share a skeleton of our implementation.

[cnouguier]

In the futrue, a potential solution to implement multipart upload is presented in this article https://blog.logrocket.com/multipart-uploads-s3-node-js-react/

[mdartic]

Thanks for your proposal.

Actually, we are using feathers-blob with either a file system or s3-based storage (and we use aws-sdk under the hood too).
We use minio as we host all of our infra with open source softwares.
(if interested, code is available here)

The backend part is quite easy, thanks to this package.
We only need to choose the fs/s3 storage depending the config the user set in our project.

The frontend is not so nice, as I think we don't send data directly to the provider in case of s3, but we send it first to the API.
We don't use presigned URL, it could be easier to do.
We can do better.

We also use a proxy mechanism with nginx + feathers to protect / check the ability the user can download a s3 object.
If the user have the authorization from the API, we let the user access the s3 object.

We think we can do better, the proxy routes upload/download are an interesting way, and clearly more robust / secure than our nginx implementation.

If the package could still be compatible with a file system, that would be nice !

[claustres]

I think that willing to create a module handling all use cases will probably fail due to the underlying complexity. Relying on presigned URLs is really a different beast than relying on a fs storage. The whole workflow is actually different.

We plan to share soon a feathers-s3 module specifically focused on this to avoid any confusion. It could then be possible to still use feathers-blob beside to store data in a local file whenever required.

[claustres]

Maybe an option to support filesystem-based storage with the same API is to use https://github.com/minio/minio. Any feedback on this tool would be useful (does it work fine with the AWS SDK, does it support presigned URLs, etc. ?).

[bil-ash]

I think that willing to create a module handling all use cases will probably fail due to the underlying complexity. Relying on presigned URLs is really a different beast than relying on a fs storage. The whole workflow is actually different.

We plan to share soon a feathers-s3 module specifically focused on this to avoid any confusion. It could then be possible to still use feathers-blob beside to store data in a local file whenever required.

Looking forward to feathers-s3 because current feathers-blob is indeed problematic. I plan to use minio on a storage VPS(which is not used much now) because minio does support presigned URLs https://min.io/docs/minio/linux/integrations/presigned-put-upload-via-browser.html

[fratzinger]

I've started working on https://github.com/Artesa/feathers-file-stream

It's not designed to be using presigned urls at all but it uses streams.

Prerequisites

I'm curious, if you're not working with presigned urls yet, how do you handle files with feathers-blob?
I use express with multer v1 with memoryStorage which is the first bottleneck. Even if I use diskStorage, then feathers-blob needs the data as Buffer or uri. Both are in memory, I guess.

Do you know a way not to store the files in memory?

@artesa/feathers-file-stream

Yesterday I learned about multer@next which creates a ReadableStream for the files. Using a stream reduces the memory consumption. But how to work with the stream on the feathers layer (hooks) and pass it to s3 or elsewhere. @artesa/feathers-file-stream does exactly that. It takes a stream and uploads it. Currently fs and @aws-sdk/client are supported.

It's not bound to multer. Really anything that produces a stream can work. I would love to see a koa version or even a socket.io version to be transport independent. I also would love to see a minio service.

It's at really early stage and not even published to npm yet. But seeing this valuable discussion here I'm more than curious what you peeps think about.

Unless it's not using presigned urls, do you think this is a thing worth investigating more? Or am I going the wrong way?

[claustres]

Dear community and contributors to this issue @fratzinger @bil-ash @mdartic @DaddyWarbucks, we are pleased to introduce you to https://github.com/kalisio/feathers-s3. Before releasing a first version we welcome feedback from anyone willing to test.

We hope it will provide a more flexible and reliable way to manage upload/download, of possibly large files, in a Feathers app.

[bil-ash]

Great work. Now waiting to get a big storage VPS as part of black friday deals,set up minio on it and then use feathers-s3

[claustres]

@bil-ash Would be great to have a feedback to somewhat "replace" filesystem storage with minio + feathers-s3.