Is mixing logical sector size in the same PV safe?

[FilipeMaia]

It's unclear to me if it's safe to mix different sector sizes (512 and 4096) in the same PV.
I have a VG made up of multiple mdadm RAID6 arrays. All the arrays have sector size of 512. I was adding another array and realized it has a sector size of 4096 (I got the appropriate error message during the vgextend).
I'm using a the VG to create a thin volume where I have an xfs filesystem with 4096 as sector size:

         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
data     =                       bsize=4096   blocks=432181084160, imaxpct=1
         =                       sunit=256    swidth=8192 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Is it safe to enable allow_mixed_block_sizes and combine the array? What situations are safe and which ones are not?

[zkabelac]

It's NOT safe - so there is a reason why this option is 'disabled'.
If you are experienced user and you know all associated risks - it can possibly work, but mostly in a a way you would use 'LVs' on block size A and other LVs on block size B. Combining them into single LV is NOT recommended and for some filesystems (XFS) it may cause fundamental troubles.

i.e. write atomicity, journal logic when FS was formatted to 512b and should use 4K sectors...

Basically a lot of problems - so lvm2 by default goes with disallowed creation of such mixed VG - if you are willing to spend your time and test & benchmark around - it might eventually be working with proper settings through the whole stack.

[FilipeMaia]

If that is the case, what happens when a PV changes logical sector size (for example when an mdadm array with 512 sectors gets a disk with 4k sectors added to it)?
Does LVM directly work with logical sectors, or is it the filesystems above it that can cause problems?

[zkabelac]

Due to the fact that this 'protection' to VG creation was added much much later (years) - when users already started to report issues with mixed PVs - we cannot simply disable such VG usage. There are users where the mixed LVs are not cause problem(s) (at least not known problems) - for them it works - so 'mixed' VGs are left as is.

lvm2 could possibly start to report some sort of warnings when such LVs are activated - and user would need to 'vgchange' something to disable warning - however so far it's very rare problem.

Also lvm2 is not protecting 'change' of PVs under the hood - as it's expected such fancy stuff is being done by skilled admin - so he likely knows what's best for his system.

[FilipeMaia]

Ok. I suspect the PV change under the hood might become more common as 4Kn drives proliferate.
The logical sector size change can be unintended. For example, if I have an mdadm RAID5 made of 512e disks, and have a disk loss, if I repair it with a 4Kn disk the array sector size will change. mdadm handles this automatically with no warning whatsoever, so the user likely is not aware the sector size changed, e.g.:

truncate -s 1G /tmp/loop512a
truncate -s 1G /tmp/loop512b
truncate -s 1G /tmp/loop512c
truncate -s 1G /tmp/loop4Ka
losetup --sector-size 512  --direct-io=on /dev/loop0  /tmp/loop512a
losetup --sector-size 512  --direct-io=on /dev/loop1  /tmp/loop512b
losetup --sector-size 512  --direct-io=on /dev/loop2  /tmp/loop512c
losetup --sector-size 4096  --direct-io=on /dev/loop3  /tmp/loop4Ka
mdadm --create /dev/md2 --level=5 --raid-devices=3 /dev/loop[0-2]
# blockdev returns 512
blockdev --getss /dev/md2
mdadm /dev/md2 -a /dev/loop3
mdadm /dev/md2 -f /dev/loop2
# blockdev still returns 512
blockdev --getss /dev/md2
mdadm -S /dev/md2
mdadm -A /dev/md2 /dev/loop0 /dev/loop1 /dev/loop3
# blockdev now returns 4096
blockdev --getss /dev/md2

[zkabelac]

To handle such cases - we would need to introduce another lvm2 metadata value that would store user's confirmation that he is aware the he combines 4K & 512B devices and that may cause troubles (like we are mostly aware of them when using XFS - for ext4 this is usually fine).

There are probably other device geometry info - where it's combination may cause trouble - i.e. handling of discard - where mixing i.e. hdd & ssd/nvme may have side effects....

[FilipeMaia]

Instead of trying to control the logical sector size, wouldn't it be possible to make lvm indifferent to sector size (e.g. make lvm always use a 4K sector size, assuming all reasonable sector sizes divide 4K)?

[FilipeMaia]

I suspect in my case it's safe to mix sector sizes as the filesystem already has 4096 as the sector size (it would definitely not be safe if it were 512). At least my tests with loop devices and XFS didn't show any problem. But I won't risk it for now.