async behaviour of block cache

[truxton]

I have a large block cache (1TB)
I would like the interaction between s3backer and the upper filesystem to be as
asynchronous as possible from the underlying S3 http traffic, but I cant seem
to find the parameters to achieve this. If I'm keeping the filesystem use small,
everything should easily fit in the block cache, but I end up blocking on network
traffic. I use --blockCacheNoVerify. I can increase --blockCacheWriteDelay
but as soon as dirty blocks start to get written to S3, the filesystem IO
is drastically reduced. Ideally file IO would only start to block on the
network once the block cache has filled up. Am I hitting a limitation,
or am I missing something?

Thanks,

-Truxton

[archiecobbs]

A few hot takes...

• The block cache network I/O does run entirely asynchronously from upper filesystem interactions
• A journaling upper filesystem can make things slow (see --blockCacheNumProtected)
• What happens if you set --blockCacheMaxDirty and --blockCacheWriteDelay to very high values?

What are all of the parameters/flags you are passing to s3backer on the command line?

[truxton]

Thanks for the help, and thanks for writing s3backer!
It's good to know that you expect the block cache to run asynchronously.

I'm using a "m7a.xlarge" EC2 instance on AWS (4 cores, 16 GB ram)

The upper level filesystem is zfs, I am using this s3backer config :
--size=512t
--blockSize=512k
--accessFile=/usr/local/etc/s3b_passwd
--listBlocks
--listBlocksThreads=24
--ssl
--blockCacheSize=1956000
--blockCacheNumProtected=32768
--blockCacheFile=/mnt/s3b_cache/cachefile.dat
--blockCacheWriteDelay=128000
--blockCacheThreads=16
--blockCacheNoVerify
--blockCacheFileAdvise
--blockHashPrefix
--timeout=120

and invoking s3backer on the command line like this :
s3backer -f --configFile=/usr/local/etc/s3backer-options.zdev3.conf mybucket/zdev3 /mnt/s3b

since I do not specify --blockCacheMaxDirty I assume the default is 0
which means unlimited.

starting s3backer on a fresh s3 bucket looks like this :

s3backer: auto-detecting block size and total file size...
s3backer: auto-detection failed; using configured block size 512k and file size 512t
2025-10-16 10:29:31 NOTICE: creating new cache file "/mnt/s3b_cache/cachefile.dat" with capacity 1956000 blocks
2025-10-16 10:29:31 INFO: reading meta-data from cache file "/mnt/s3b_cache/cachefile.dat"
2025-10-16 10:29:31 INFO: loaded cache file "/mnt/s3b_cache/cachefile.dat" with 1956000 free and 0 used blocks (max index 0)
2025-10-16 10:29:31 INFO: established new mount token 0x656aa46e
2025-10-16 10:29:31 INFO: s3backer process 1867 for /mnt/s3b started
2025-10-16 10:29:31 INFO: starting non-zero block survey
2025-10-16 10:29:31 INFO: mounting /mnt/s3b
2025-10-16 10:29:32 INFO: non-zero block survey completed (0 non-zero blocks reported)

I create a fresh filesystem like this :

losetup /dev/loop3 /mnt/s3b/file

zpool create -O atime=off -O compression=off -O xattr=sa -O sync=disabled -O encryption=off mypool /dev/loop3

zfs create mypool/shared

Writing to the local disk is relatively fast :

dd if=/dev/urandom bs=1M count=2048 of=/root/random.dat conv=fsync

2147483648 bytes (2.1 GB, 2.0 GiB) copied, 17.5025 s, 123 MB/s

copying that 2GB file to the upper layer filesystem before any dirty
block cache is pushed to s3 is also relatively fast (assuming overhead for fuse)

dd if=/root/random.dat of=/mypool/shared/random1.dat

2147483648 bytes (2.1 GB, 2.0 GiB) copied, 84.2736 s, 25.5 MB/s

But then once the dirty blocks from the first copy are being
sent over the network, the next copy is slower :

dd if=/root/random.dat of=/mypool/shared/random2.dat

2147483648 bytes (2.1 GB, 2.0 GiB) copied, 187.528 s, 11.5 MB/s

It's curious to note that 3.5G instead of 4G is being used by
the block cache :

df -h /mnt/s3b_cache/

Filesystem Size Used Avail Use% Mounted on
/dev/nvme4n1 1007G 3.4G 953G 1% /mnt/s3b_cache

ls -lh /mnt/s3b_cache/cachefile.dat

-rw-r--r-- 1 root root 3.5G Oct 16 14:01 /mnt/s3b_cache/cachefile.dat

When more activity is happening, but we are still well below
using the 1TB of block cache, then file IO speeds are as
low as 1 MB/sec.

And also I note that the s3backer process gets up to about 95% cpu
(a few percent per fuse worker thread that all adds up to 95%)

The s3backer stats look like this when the second copy of 2GB
is active :

http_normal_blocks_read 0
http_normal_blocks_written 2479
http_zero_blocks_read 4017
http_zero_blocks_written 0
http_empty_blocks_read 0
http_empty_blocks_written 0
http_gets 24
http_puts 2480
http_deletes 0
http_avg_get_time 0.060 sec
http_avg_put_time 0.251 sec
http_avg_delete_time 0.000 sec
http_unauthorized 0
http_forbidden 0
http_stale 0
http_redirect 0
http_verified 0
http_mismatch 0
http_5xx_error 0
http_4xx_error 0
http_3xx_error 0
http_other_error 0
http_canceled_writes 0
http_num_retries 0
http_total_retry_delay 0.000 sec
curl_handle_reuse_ratio 0.9963
curl_timeouts 0
curl_connect_failed 0
curl_host_unknown 0
curl_out_of_memory 0
curl_other_error 0
block_cache_current_size 4017 blocks
block_cache_initial_size 0 blocks
block_cache_dirty_ratio 0.00078630
block_cache_read_hits 0
block_cache_read_misses 0
block_cache_read_hit_ratio 0.00000000
block_cache_write_hits 31835
block_cache_write_misses 4017
block_cache_write_hit_ratio 0.88795604
block_cache_verified 0
block_cache_mismatch 0
zero_block_cache_size 1073737807 blocks
zero_block_cache_read_hits 74
zero_block_cache_write_hits 4
out_of_memory_errors 0

I'm considering using bcache on top of s3backer for yet another layer of async cache.

Thanks,

-Truxton

[archiecobbs]

I'm not sure what is causing slowness. I assume your uplink is fast. You could do packet traces to measure how long it takes a single block to be transferred by writing one 512k block at a time and watching the data go by (it will all be encrypted of course but you can see the large packets go by and time their ACK's from Amazon).

You are only using --blockCacheThreads=16, so at most 16 * 512K = 8MB of data can be in flight at a given time. If you increase that (e.g., to 128) does that affect things?

I'm just guessing at this point.

[truxton]

Reporting back after lots of testing.
using a faster VM made a positive difference.
using a larger s3 block size helped a lot (--blockSize=8192k)
using bcache or lvm cache on top of s3backer did not help.
using more (or fewer) blockCacheThreads did not make much difference.
Are some blockCacheThreads used for read requests, and some are used for writes?
Perhaps if the thread pool were saturated with one type of request (read or write),
then the other type of request might starve?
Thanks,
-Truxton

[archiecobbs]

using a larger s3 block size helped a lot (--blockSize=8192k)

This makes some sense I suppose, more aggregation, less granular I/O operations.

Are some blockCacheThreads used for read requests, and some are used for writes? Perhaps if the thread pool were saturated with one type of request (read or write), then the other type of request might starve?

Starvation like that shouldn't be possible. The block cache threads do mostly writes; the only reads they do are for read-ahead in FUSE mode (which you can disable with --readAhead=0).

Other than read-ahead, when a block needs to read, the kernel thread requesting the read is the thread that performs the read, so no thread from the block cache thread pool is needed.

[abzmr]

Are some blockCacheThreads used for read requests, and some are used for writes? Perhaps if the thread pool were saturated with one type of request (read or write), then the other type of request might starve?

Starvation like that shouldn't be possible. The block cache threads do mostly writes; the only reads they do are for read-ahead in FUSE mode (which you can disable with --readAhead=0).

Actually, block cache threads also do reads for the first partial blockwrite into any unused block. Since your stats show a high number of http_zero_blocks_read, this will probably be the case. So starvation is quite likely. Double blocksize helps since it halves the number of first partial blockwrites, and so on.

More details in #240 (comment) and same issue as #201 .

[archiecobbs]

Actually, block cache threads also do reads for the first partial blockwrite into any unused block.

Thanks for pointing that out. I should have clarified I was referring to "reads" and "writes" from the kernel's perspective, not the network's perspective. When the block sizes are not the same then kernel writes can trigger network reads.