Split cache file for 2 (write / write + read)

[anatolii2makaroff]

Hi,

I have an idea to use a split cache with two separate files:

1. write only cache
2. write + read cache

I’d like to use preserve mode for the write cache to ensure high availability and replicate it to different nodes (using SDS), while keeping the read cache local to each node.

This way, I can keep the write cache as small as possible—just enough to store all dirty and unflushed data safely—and increase the read cache size with minimal cost. If a node with the read cache goes down, I’m okay with reloading the data from S3.

Does this feature make sense? Thanks!

[archiecobbs]

Interesting idea but obviously would add some complexity.

But I'm unclear on exactly what you're talking about. What is a "node" here? Are we talking about a single s3backer process or multiple?

[anatolii2makaroff]

Yeah, it's a single s3backer process, but the underlying storage has multiple nodes. In case the node with s3backer goes down, I can spin it up s3backer on another node without losing data (write cache file) and repopulate the read cache. In that case, it could potentially save me a lot of space/money on the underlying storage, since I only need to replicate the write cache.

[archiecobbs]

OK thanks. I have more questions :)

So you are saying you have some filesystem in which you could have two files in the same directory e.g. /var/cache/rwcache.bin and /var/cache/rocache.bin and somehow configure the underlying filesystem to synchronize the first one across a cluster with full consistency and redundancy on fsync(2) but not synchronize the second one? Is this "preserve mode"? What filesystem does that?

Or is some independent process doing the synchronization? If so, isn't there still a race condition?

[anatolii2makaroff]

Actually, I have mounted two different disks/block devices on the same host.

For example:

/dev/nvme1n1 → /var/cache/rwcache/. The underlying block device here is SDS, such as EBS or Ceph, which support RAID and can store data in a distributed manner.
/dev/nvme0n1 → /var/cache/rocache/. This might be an ephemeral local NVMe disk, for example, without the requirement for replication.

Write operations

                       -------->   /var/cache/rwcache/disk1 (sds, data distributed thru SDS cluster but size is minimal)
  client write  ----> |
                       -------->   /var/cache/rocache/disk1 (local, we can afford loose it, but size might be much bigger without extra pay for redundancy)

Read operations

  client read  -------->   /var/cache/rocache/disk1

So, it could bring some space and cost benefits. However, it would also introduce write amplification, since each write operation would result in two writes to different files.

That said, for many read-heavy workloads, it might still be beneficial.

[archiecobbs]

OK got it - thanks for the explanation.

[anatolii2makaroff]

@archiecobbs After some research and playing with nbdkit I found maybe more better solution for that.
We can use nbdkit-cache-filter https://libguestfs.org/nbdkit-cache-filter.1.html to point to new cache location so it will be used for reads cache-on-read=true

I checked and looks like it works.

[archiecobbs]

Interesting... nice demonstration of the flexibility of NBD.

[anatolii2makaroff]

yeah, good tool.
Only one thing — maybe add a param to the s3baker to allow set nbd filters lists?
There is nbd-param, and I'm able to pass extra NBD params, but I can't set the filters there, right?

[archiecobbs]

I'm able to pass extra NBD params, but I can't set the filters there, right?

You should be able to stack filters using --nbd-flags flag and configure them using --nbd-param. Those two flags pass their arguments as-is to nbdkit(1) either before or after the PLUGIN parameter.

E.g. s3backer --nbd-flags --filter=foobar --nbd-param foobar_conf=42, etc.