On Swift, when storing "large" amounts of data at once, s3backer chokes on requesting non-existent block files

[zilti]

s3backer version 2.1.3, FreeBSD 14.2, using a Swift object storage container.

When storing larger amounts of data at once, s3backer runs into a problem that locks up the entire block device. It tries to request a block file that does not exist (yet). Swift replies with a 503 to that:

Mar  3 20:58:36 lyrion s3backer[12982]: rec'd 503 response: GET https://s3.pub1.infomaniak.cloud/datastore/000dc3b1
Mar  3 20:59:07 lyrion s3backer[12982]: giving up on: GET https://s3.pub1.infomaniak.cloud/datastore/000dc3b1

It seems that the block file is not meant to exist - or at least contain data - in the first place. My "workaround" currently is that when I encounter this error, I manually upload a 1-byte file with the requested name, wait for this error:

Mar  3 21:01:42 lyrion s3backer[12982]: read of block 000dc3b1 returned 1 != 1048576 bytes

and then delete it again. After that, s3backer properly creates and fills the missing block file.

[archiecobbs]

This seems like a server-side (i.e., Swift) bug to me. The error being returned is 503 Service Unavailable.

Any 5xx error is by definition the server's fault, not the client's fault.

The correct response to s3backer requesting a block that doesn't exist yet would be 404 Not Found.

What am I missing?

[zilti]

Yes, that is true, it is weird that it returns a 503 and not a 404 for something not found. Then again those APIs are sometimes very weird in what they return, so I didn't question it much beyond thinking "huh, odd".

So I guess you are missing nothing; I will contact my hoster's support about if this is a configuration issue on their end. But I think it is also odd that s3backer is trying to - and insisting on - requesting a block file it did not create, I suppose.

I will keep this updated when I get replies from my hoster about this.

[archiecobbs]

So I guess you are missing nothing; I will contact my hoster's support about if this is a configuration issue on their end.

Good luck, hopefully they can help.

But I think it is also odd that s3backer is trying to - and insisting on - requesting a block file it did not create, I suppose.

It's not necessarily wrong, because s3backer doesn't always know whether a block exists or not (it depends on the situation). When it doesn't know, it does a GET and sees if anything comes back. Then it knows, at least for a while :)

[zilti]

As expected, they're really dragging their feet and take their time to even reply. I got this gem in one reply after giving them the entire URL s3backer tries to access:

Unfortunately, I'm not sure we can change the S3 storage behavior. If you don't specify the full path, it's normal to get a 503 error because the server can't process the request.

I'm now waiting for them to explain what an actual full path is supposed to be. Something else I am wondering though: when s3backer gets a 503, I can manually create the object, wait until s3backer sees it (which triggers a ton of errors in the logs), then delete that 1-byte object again. After that, s3backer can create it. Is there a GET request in between again that expects (and surprisingly gets) a 404?

Because if not, I was wondering if it might be a stopgap to recompile s3backer with HTTP_NOT_FOUND set to 503... but I don't really feel comfortable with that idea.

[archiecobbs]

If you don't specify the full path, it's normal to get a 503 error because the server can't process the request.

There shouldn't be any examples where s3backer is not "sending a full path", although to be honest I'm not even sure what that is supposed to mean. Maybe you could ask them to provide an example.

You could also remind them that you're not asking them to change their S3 storage behavior, rather, you're asking for them to make it behave correctly :) A 503 error is by definition their fault.

Something else I am wondering though: when s3backer gets a 503, I can manually create the object, wait until s3backer sees it (which triggers a ton of errors in the logs), then delete that 1-byte object again. After that, s3backer can create it.

It sounds like you are in effect working around their bug.

I was wondering if it might be a stopgap to recompile s3backer with HTTP_NOT_FOUND set to 503... but I don't really feel comfortable with that idea.

You wouldn't want to do that, but you could hack s3backer to treat 503 the same way as 404 Not Found, like this:

diff --git a/http_io.c b/http_io.c
index 6bfd71d..40b0925 100644
--- a/http_io.c
+++ b/http_io.c
@@ -2414,7 +2414,7 @@ http_io_perform_io(struct http_io_private *priv, struct http_io *io, http_io_cur
 
         // In the case of a DELETE, treat an HTTP_NOT_FOUND error as successful
         if (curl_code == CURLE_HTTP_RETURNED_ERROR
-          && http_code == HTTP_NOT_FOUND
+          && (http_code == HTTP_NOT_FOUND || http_code == 503)
           && strcmp(io->method, HTTP_DELETE) == 0)
             curl_code = 0;
 
@@ -2494,6 +2494,7 @@ http_io_perform_io(struct http_io_private *priv, struct http_io *io, http_io_cur
         case CURLE_HTTP_RETURNED_ERROR:                 // special handling for some specific HTTP codes
             switch (http_code) {
             case HTTP_NOT_FOUND:
+            case 503:
                 if (config->debug)
                     (*config->log)(LOG_DEBUG, "rec'd %ld response: %s %s", http_code, io->method, io->url);
                 http_io_free_error_payload(io);