How does raidz1 space efficiency vary with the number of drives due to stripe overhead?

[zwimer]

According to this post https://forums.truenas.com/t/the-problem-with-raidz/1589 the amount of space taken up by a file on raidz1 is affected by the number of drives in an array- with 3 and 5 being an optimal amount for a raidz1.

Specifically, the relevant snippets here are:

RAIDZ1 with 3 drives
With 3 drives, we get a stripe that is 3 drives wide.
Each stripe has two 4k data sectors (8k) and one 4k parity sector.
For a volblock of 16k, we need two stripes, because one stripe stores 8k data and two stripes will store the needed 16k (16k/8k).
Each stripe has two 4k data sectors, two stripes are in total 16k.
Each stripe has one 4k parity sector, two stripes are in total 8k.
That gets us to 24k in total to store 16k.
24k is 6 sectors and that can be divided by 2 so there is no padding needed.
Storage efficiency is 66.66%, as expected.

RAIDZ1 with 4 drives
With 4 drives, we get a stripe 4 drives wide.
Each stripe has three 4k data sectors (12k) and one 4k parity sector.
For a volblock of 16k, we need 1.33 stripes (16k/12k).
The first stripe has three 4k data sectors, in total 12k.
The first stripe also has one 4k sector for parity.
The second stripe has one 4k data sector.
The second stripe also has one 4k sector for parity.
In total, we have four 4k data sectors and two 4k parity sectors.
That gets us to 24k in total to store 16k.
24k is 6 sectors and that can be divided by 2 so there is no padding needed.
We expected a storage efficiency of 75%, but only got 66.66%!

This implies to me that when storing large files, users of a 4 drive array would only have at most 72.72/75 = 96.96% of their storage available; roughly a 3% overhead per file. Small files would be disproportionally affected by this overhead and have a much larger number, but most large files over 100MB should roughly have about that 3% overhead per file.

Other calculations I've seen show that the limit of overhead tended to as file size increases is / is a multiple of recordsize is:
100 * (parity_diff_per_record * ashift) / recordsize is the percent of extra bytes a file will take on a 4 disk raid1z vs 5 disk.

Any help clarifying this would be appreciated.

[gmelikov]

There's a useful info and great table with some calcs for different raidz configurations: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/RAIDZ.html

In short - yes, recordsize and ashift sizes matter, so is specific raidz configuration.

If you want to calculate each byte - ZFS reserves some space too https://openzfs.github.io/openzfs-docs/man/master/4/zfs.4.html#spa_slop_shift