Special vdev for metadata and small blocks on raidz not just mirror

[rcfa]

Maybe I’m wrong, but I’m under the impression that currently special vdevs for metadata and small blocks/files can only be of type stripe or mirror. Someone elsewhere explained that historically the feature was implemented to improve draid performance, and of course, if one specifically adds SSDs to a spinning disk array to improve performance, one would use the fastest possible way of doing that, and that’s of course not a raidzN setup. Perfectly reasonable.

However, I come from a completely different angle: I have a SSD RAIDZ2 pool and a spinning disk RAIDZ2 pool, and there is no plan to add additional devices, for space and cost reasons. However, since in both cases metadata is stored on a RAIDZ vdev anyway, it still would speed up things if the spinning disks had their metadata on the SSDs. Similarly, having two pools and having to manually decide which data goes where brings additional overhead.
If I could create a pool where both the SSDs and the spinning disks were RAIDZ2, but the SSDs were used as special vdev for small files/blocks and metadata, then most file ops would be as fast as an SSD RAIDZ2 even on the spinning disks, and I could organize the file system logically, rather than by what files I want on the spinners vs. on the SSDs.

So, coming from that angle, where everything is RAIDZ anyway, and it’s just a question if things end up on slow or fast hardware, having a special vdev as raidz2 suddenly does make sense.

Am I missing something here? If not, how big would be the effort to actually get this implemented, given that both raidz2 logic is fully implemented and special vdev logic is implemented. It would almost seem, recommended practices aside, once a vdev is created, the zpool should be agnostic to how it’s being used. After all, nothing would stop me (except common sense) to use a mirror of spinning disk drives as special vdev for an SSD RAIDZ2 pool. 😆

[tonyhutter]

Maybe I’m wrong, but I’m under the impression that currently special vdevs for metadata and small blocks/files can only be of type stripe or mirror.

Special devices can be any type, including raidz2 or draid. They just have to match the redundancy level of the pool. You can verify the behavior by making a test pool of sparse files:

$ truncate -s 100M file{1..20}
$ sudo ./zpool create tank draid ./file{1..10} special draid ./file{11..20}
$

[amotin]

Before ZFS 2.4 special was artificially restricted to be only mirrors. It was relaxed just recently.

[rcfa]

Before ZFS 2.4? I thought we’re at 2.3.4? So is this going to be relaxed in an upcoming release? Or did you mean 2.3.4, since you used "was" rather than "will be"?

[amotin]

2.4 is now in RC1 stage, testers are welcome. Technically it would be trivial to merge it to 2.3.x, since the limitation is completely artificial, but it is quite a big user-visible functionality change.

[rcfa]

Before ZFS 2.4 special was artificially restricted to be only mirrors. It was relaxed just recently.

Since you say it was artificially restricted, was that restriction only in the options to create such pools, or in the rest of the code, too?
In other words: were I to create a pool with ZFS 2.4RC1 that were to use a RAIDZ2 special vdev, could it be mounted by current stable releases of ZFS (e.g. as being used in TrueNAS (2.3.4)), or would the pool be recognized as incompatible?

[amotin]

Yes, it was limited only on creation, so should be backwards-compatible. But same time 2.4 has few other pool features added, which are not compatible, that you may need them explicitly disabled on creation if you wish 2.3 compatibility.

[rcfa]

Another thing that’s not quite clear to me: if a special vdev is full, will it simply overflow to the regular devices, or will I experience something akin to a regular file system running out of inodes even though there is still plenty of unused disk space?
Conversely, if the regular devices are full, will things bigger than the small files limit end up on the special vdev, or will the pool be full, even though there would still be space on the special vdev?

[amotin]

Currently special vdevs overflow to normal, but not backwards. Dedup vdevs first overflow to special and then to normal.

[rcfa]

So then I could err on the side of writing too much on the special vdev, and if it turns out to fill up too quickly, I could change the parameters and use the new zfs rewrite functionality to rebalance things, right?

[amotin]

Right. But IIRC special vdev also has an additional mechanism to disable special_small_blocks once reaching certain percent of capacity, so overflow might be not too trivial to trigger, unless significantly fragmented.

[PrivatePuffin]

Interesting question, but why would you even want a single raidz2 ssd metadata vdev?!

I get it in large setups where you can have 4 or 5 of those in a huge array.

But smallblocks/metadata is mostly intended for 4k-16k R/W, which is iops limited not throughput limited.
So why use a throughput-optimised iops-inefficient setup like raidz2 for metadata/smallblocks?

You much-rather want the iops of multiple ssds instead of your current iops of 1 ssd for the whole raidz2.
Even a single mirror gives already double(!) read iops for the data likely accessed on metadata/smallblocks ssds.

[amotin]

There may be different use cases. Setting special_small_blocks property high (16MB in ZFS 2.4) allows to put the whole dataset to a special vdev, if better performance is needed, or zero if otherwise. Use of zfs rewrite allows to migrate things post-factum. And I hope we'll be able to extend it further. But doing things like that might need more capacity, where mirror might be too expensive. Though I'd think of a couple 3-wide RAIDZ1 vs one 6-wide RAIDZ2, though there can be options.

[PrivatePuffin]

I'm aware with the other use cases, I just dont think that using special-small-blocks for throughput preferential use cases Is the correct way to go.

Setting special_small_blocks property high (16MB in ZFS 2.4) allows to put the whole dataset to a special vdev

I'm aware of that (though its same-or-bigger than record size, not necessarily 16MB)

[rcfa]

This reminds me of an old joke: a catholic and a protestant priest sit in a train compartment, both reading their prayer books. After a while the catholic priest pulls out his pipe, stuffs, it lights it up, and continues his prayer smoking.
The protestant priest after a while can’t hold back and asks: "Excuse me, it’s none of my business, but I’m curious. I’m also a pipe smoker, but when I asked my bishop if it’s OK to smoke my pipe while praying, the answer was, that NO, nothing should distract from prayer, and that thus smoking while praying isn’t allowed. Is that different with catholics?"
The catholic priest answers: "Interesting. You see, I asked if it’s OK to pray while smoking, and my bishop said it’s always OK to pray."

In short: If you create a special vdev to speed things up, then a RAIDZ configuration isn’t the ideal configuration.
However, if you have both an SSD based RAIDZ for "regular" files and an HDD based RAIDZ for large media files, archives, etc. but you don’t want the headaches that come with having to structure your file system around that split, and would prefer to have a single file system that’s organized logically rather than by storage medium, then it makes perfect sense.
The idea is NOT to speed things up, because in either case the SSDs will end up in a RAIDZ configuration, but to simplify the file system layout and as a side effect, also speed up a little bit the HDD based RAIDZ by having its metadata operations sped up to SSD RAIDZ levels.

Just like with the priest joke: it depends from where you start to decide if it’s OK or not.

In my case, I don’t have slots or money to waste of SSDs simply to speed things up. I have two RAIDZ (SSD and HDD), and if I can simplify and unify and have a marginal speed up on some operations (HDD), then it’s already worth it.

[PrivatePuffin]

In my case, I don’t have slots or money to waste of SSDs simply to speed things up. I have two RAIDZ (SSD and HDD), and if I can simplify and unify and have a marginal speed up on some operations (HDD), then it’s already worth it.

I dont your whole rambling about filesystem structures, both mirror and raidz2 metadatavdevs would function exactly the same from an end-user or structuring perspective.

I guess you missed a clue on that.
My question has nothing to do with "preachings" but rather the inherent performance profiles of different kinds of drives.

I don’t have slots or money to waste of SSDs simply to speed things up

Yet you added a raidz metadata vdev... to speed things up...
negating your whole argument.

So I'm going to safely assuming you just didn't understand the underlaying point/question.

[rcfa]

You can pretend to not understand. I ABSOLUTELY CANNOT AFFORD TO LOSE ANY CAPACITY. Get it now?
I need the MAXIMUM CAPACITY from my SSDs and HDDs, while having RAIDZ2 redundancy.
I'm NOT GOING TO WASTE CAPACITY for stupid mirrors.
The fact that my HDDs RAIDZ2 gets sped up for certain operation is a FRINGE BENEFIT, NOT A GOAL.
The GOAL is to have ONE file system, NOT TWO, which is what I would have if I had a separate SSD and HDD based RAIDZ2 file system. A special vdev for small files and meta data, which I dump on the RAIDZ2 SSDs merged in effect with the RAIDZ2 HDDs creates the equivalent of a single file system with a "hot zone".
Not sure what's so hard to get about that, except for people who have money and storage capacity to burn.

[rcfa]

Have one more question: in the past, if the main pool was a mirrored pool and the special vdev was (as was the only option) mirrored, the special vdev could be detached, but not if the main pool was a raidzN.
Now, if both main pool and special vdev are raidz2, can the special vdev be detached, just like it’s possible if both are mirrored?
In other words: does detaching only work in the mirror case, or in any case where main pool and special vdev are of the same type?

If a raidz2 special vdev can be deteached if the main pool is raidz2, do they need to have the same device count (e.g. 8 devices each), or does it also work if e.g. the main pool consists of 5 HDD, and the special vdev consists of 7 SSD?

[amotin]

IIRC no device removal was ever implemented for anything involving RAIDZ. With RAIDZ to RAIDZ it should probably be less hopeless than from mirror to RAIDZ, but the vdev topology would have to be the same indeed.

[rcfa]

OK, thanks.
Wonder if the recently added rewrite mechanism couldn’t be employed/modified to that effect…

Does AnyRAID support detaching devices, or is that strictly a plain mirror device thing?

[rcfa]

Looks like the TrueNAS nightlies are already using ZFS2.4RC3. Given that I just migrated all my data from the SSD pool to the HDD pool, I can now nuke the SSD pool and add the freed SSDs as special vdev to the existing pool.

If I do that, will the special vdev simply inherit/adopt the pool’s settings (like encryption)?

What’s the precise command for adding the ssds?

zpool add <pool_name> special raidz2 <ssd_device1> <ssd_device2> …<ssd_deviceN> 
zfs set special_small_blocks=16M <pool_name>
zfs rewrite -rvx /mnt/pool_name

I figure I can see how the distribution of files sizes looks like after a while, and if need be adjust/rewrite. But since I have plenty of space on the special vdev and in essence only want to ban large media files to the HDDs, I think the above should work...

[amotin]

You should better add vdevs via the WebUI to properly partition them. Nightlies UI don't allow RAIDZ2 specials yet, but soon will be.

[rcfa]

You should better add vdevs via the WebUI to properly partition them. Nightlies UI don't allow RAIDZ2 specials yet, but soon will be.

Thanks! Just messing around, learning about it, etc.

Which brings me to a question your answer raised: what sort of partitioning are you talking about? What does the UI do differently from what I did at the CLI?

I'm curious such that if i have other systems with ZFS pools, I'd create them in a way that if need be they are maximally compatible...

[rcfa]

One thing that might need some work is how available capacity is reported. So say I have set the size for small files to 16MB, and I have like 37TB worth of SSD and 80TB worth of HDD (capcities after raidz2 configuration), with the former being used as metadata and small files storage for files up to 16MB. The capcity of the pool is shown as 80TB, and as I keep adding (currently almost exclsively small files, that will change down the road), it looks like I'm taking up no space.
I fully understand why this is the case, but long-term a better solution might be needed, e.g. dynamically adding the amount taken up by small files on the spcial vdev to the total capacity reported, and to the capacity used.

[amotin]

I am thinking about adding some pool properties to report each individual allocation class, since now it can only be seen in per-vdev stats in zpool list -v. But I am not sure how to better report total space usage once special class will play bigger role. Sure it is trivial to just add all capacities together, but then user might see that there is some free space (on a special vdev), while normal vdev is full and we currently have no overflow from normal class back to special.

[rcfa]

Maybe report all used space, but report free space only from the regular pool?
Ideally eventually this will evolve into a true "fusion drive" where overflow can go both ways...