S3 Express support

[lucianpls]

S3 Express is a new-ish s3 type of bucket which has better latency than the normal S3 buckets. As far as I can tell, there are some limitations and differences in REST API that make it incompatible with s3backer.

I was able to make it work only in readOnly mode, with a volume copied from a normal bucket, using this s3backer command:

s3backer --accessEC2IAM=$ROLE --readOnly --baseURL=https://s3express-usw2-az1.us-west-2.amazonaws.com/ --vhost --region=us-west-2 $EXPRESS_BUCKET /s3backer/

When trying to create a new s3backer volume or even mount read-write, using the same parameters, it doesn't seem to be able to write anything, despite the instance role having full s3 and s3express access, with the PUT requests failing with a 501 Not Implemented error. From the --debug-http output:

...
> PUT /s3btest/s3backer-mounted HTTP/1.1
Host: s3xbucket--usw2-az1--x-s3.s3express-usw2-az1.us-west-2.amazonaws.com
User-Agent: s3backer/2.1.3/2.1.3-5-gd3d24a8
Accept: */*
Content-Type: text/plain
Content-MD5: nk7Dv0/ECUDwR/Cwri4DIw==
x-amz-meta-s3backer-mount-token: 3d880e52
x-amz-acl: private
...
Content-Length: 60

* We are completely uploaded and fine
< HTTP/1.1 501 Not Implemented
< content-type: application/xml
< server: AmazonS3
...

[archiecobbs]

This is one likely obstacle:

New authentication model – The new CreateSession function returns a session token that grants access to a specific bucket for five minutes. You must include this token in the requests that you make to other S3 API functions that operate on the bucket or the objects in it, with the exception of CopyObject, which requires IAM credentials. The newest versions of the AWS SDKs handle session creation automatically.

Ref: https://aws.amazon.com/s3/storage-classes/express-one-zone/

[lucianpls]

The readOnly s3backer mount works fine on S3 Express, without the CreateSession.
In most of the documents I've read, the CreateSession is "recommended", not required, seems to have to do with a simpler (faster) authentication, per connection.

[archiecobbs]

Hmm, ok thanks. Then I'm not sure how to interpret 501 Not Implemented...

[lucianpls]

@archiecobbs
I got it working, two changes to the request headers are needed for basic functionality, without the CreateSession.

• Storage Class : EXPRESS_ONEZONE
• Content-MD5 is not supported, I commented out the code that set that header