Method for correlating S3 block to filesystem object?

[Lord-Dimwit-Flathead-the-Excessive]

Occasionally, our S3 backend at Wasabi chokes and we get:

Feb 01 09:52:59 mema nbdkit[40310]: s3backer: rec'd 500 response: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:53:06 mema nbdkit[40310]: s3backer: rec'd 500 response: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:53:14 mema nbdkit[40310]: s3backer: rec'd 500 response: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:53:21 mema nbdkit[40310]: s3backer: rec'd 500 response: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:53:30 mema nbdkit[40310]: s3backer: rec'd 500 response: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:53:40 mema nbdkit[40310]: s3backer: rec'd 500 response: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:53:54 mema nbdkit[40310]: s3backer: rec'd 500 response: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:54:14 mema nbdkit[40310]: s3backer: rec'd 500 response: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:54:26 mema nbdkit[40310]: s3backer: rec'd 500 response: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:54:26 mema nbdkit[40310]: s3backer: giving up on: GET https://s3.us-central-1.wasabisys.com/ke-central/mema/musketeers/00215f2d
Feb 01 09:54:26 mema nbdkit[40310]: s3backer.13: error writing block 00215f2d: Input/output error

This is obviously Wasabi's problem and I have contacted them regarding it. But I am wondering if there is any way to identify the filesystem object the S3 block belongs to so we can manually repair the data?

[archiecobbs]

But I am wondering if there is any way to identify the filesystem object the S3 block belongs to so we can manually repair the data?

First let's acknowledge an assumption underlying this question: Whatever Wasabi's problem is, it has something to do with the specific S3 object 00215f2d. I guess you've determined that by seeing it happen repeatedly with that single block...?

Secondly, how to map a disk block to a filesystem file (if that's even what it is) is a filesystem question. What filesystem are you using? For ext3 apparently you could use fsck -l .. see google for more info.

[Lord-Dimwit-Flathead-the-Excessive]

Yes. S3Backer diligently retried the GET ten times over 90 seconds before it gave up according to the log. I'd be interested if there was a way to adjust the retry interval.

I am testing BTRFS right now (incredible incremental transfer capabilities) but no matter.

What I want to know is if the S3 block "name" equates to the sequential block number on the network block device (NBD). That seems obvious to me now but when I posted I was a little flustered. Can you confirm?

[archiecobbs]

I'd be interested if there was a way to adjust the retry interval.

The retry interface uses exponential backoff. You can control the initial retry interval and the total amount of time spent retrying using these flags:

     --initialRetryPause=MILLIS
             Specify the initial pause time in milliseconds before the first
             retry attempt after failed HTTP operations.  Failures include
             network failures and timeouts, HTTP errors, and reads of stale
             data (i.e., MD5 mismatch); s3backer will make multiple retry at-
             tempts using an exponential backoff algorithm, starting with this
             initial retry pause time.  Default value is 200ms.  See also
             --maxRetryPause.

     --maxRetryPause=MILLIS
             Specify the total amount of time in milliseconds s3backer should
             pause when retrying failed HTTP operations before giving up.
             Failures include network failures and timeouts, HTTP errors, and
             reads of stale data (i.e., MD5 mismatch); s3backer will make mul-
             tiple retry attempts using an exponential backoff algorithm, up
             to this maximum total retry pause time.  This value does not in-
             clude the time it takes to perform the HTTP operations themselves
             (use --timeout for that).  Default value is 30000 (30 seconds).
             See also --initialRetryPause.

What I want to know is if the S3 block "name" equates to the sequential block number on the network block device (NBD)

The names of the S3 objects are just the block numbers expressed in hexadecimal, starting from the beginning of the device.

So e.g. if you are using a 256k block size, then the byte at offset 1,234,567,890 in the block device would be contained in S3 object 00001265 at byte offset 131,794.

[Lord-Dimwit-Flathead-the-Excessive]

Thanks, Archie. You're a prince!