System won't boot with systemd message : PANIC: zfs: accessing past end of object

[jjb2016]

Hi All - hopefully I've come to the right place for help with this issue. I have a laptop running Arch Linux on ZFS. It's been running absolutely rock solid for a couple of years at least. Last time I used it was a couple of days ago, but yesterday when I tried to boot it up the boot process got stuck with a systemd message ...

PANIC: zfs: accessing past end of object 46/42303 (size=46080 access=45660+4096)

Here are couple of images during the boot process ...

I don't know what this means or how to troubleshoot it.

I have booted the machine from a USB attached disk running Manjaro, with same linux kernel and zfs version that the defunct Arch Linux OS has installed, and I did a scrub of the zfsroot pool - but it found no errors.

The first image shows no pools available to import, but I am able to import the pool using zpool import -f -o readonly=on -o failmode=continue -N -F zfsroot. After doing this I can see ...

sudo zpool import                                                                                                                    
  pool: zfsroot
    id: 6910647570180683164
 state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
	the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

	zfsroot                                       ONLINE
	  mirror-0                                    ONLINE
	    ata-SSD_512GB_AA20231025512G213271-part2  ONLINE
	    ata-Crucial_CT275MX300SSD1_16291347C045   ONLINE

ChatGPT suggested that I run zdb -bcsv zfsroot. I did this and couldn't see any obvious errors - but then I'm not quite sure what to look for. I'm happy to re-run that and post the output here if that would help.

I have also run a MemTest and system memory passed.

Linux kernel version: 6.12.48-1-lts
ZFS version:zfs-2.3.4-1

Any help troubleshooting this would be much appreciated!

SOLVED:

After reading this similar issue thread I was able to resolve my issue with these steps:

1. I booted the machine up from a USB attached drive running Manjaro linux, with zfs 2.3.4 installed.
2. I was able to import my zfsroot pool with zpool import -f -d /dev/disk/by-id -R /mnt zfsroot -N.
3. Then tried mounting my datasets one by one until I found the one that caused the PANIC, which was my zfsroot/DATA/home dataset (mountpoint /home).
4. I snapshotted the dataset and then replicated it to a new dataset with zfs send -v zfsroot/DATA/home@defunct | zfs recv -fv zfsroot/DATA/home-new. This completed with no issues. I did not have to change any of the new datasets properties away form the defaults.
5. Set the old defunct dataset mountpoint to none (actually used zfs inherit mountpoint zfsroot/DATA/home to inherit none from parent dataset).
6. Set the mountpoint for the new dataset with zfs set mountpoint=/home zfsroot/DATA/home-new.
7. Exported the pool with zpool export zfsroot
8. Rebooted successfully back into original Arch Linux OS on the zfsroot pool - hurray!

So ... I still don't know what happened to the original zfsroot/DATA/home dataset, or how, but I still have it to try and troubleshoot it. Will leave that for another day!

[IvanVolosyuk]

Do you have ECC ram? This kind of error indicates of inconsistency on disk metadata which can be caused by bitflips before data have been written to disk.

You can try setting zfs_recover zfs module parameter to keep going, but it will not heal the error as it is hard to know what is the correct value. With faulty ram zpool scrub can actually do more damage, so be careful.

/*
 * zfs_recover can be set to nonzero to attempt to recover from
 * otherwise-fatal errors, typically caused by on-disk corruption.  When
 * set, calls to zfs_panic_recover() will turn into warning messages.
 * This should only be used as a last resort, as it typically results
 * in leaked space, or worse.
 */
int zfs_recover = B_FALSE;

You may want to try to import read-only first as it generally safer.

[jjb2016]

@IvanVolosyuk I have added some additional information to my original post, if that helps.

[IvanVolosyuk]

I think it should be zfs.zfs_recover=1 kernel argument.

[jjb2016]

I have tried both zfs_recover=1 and zfs.zfs_recover=1 but it made no difference. At the beginning of the boot process, the pool is not being imported and then later on I get the PANIC: zfs: ... fun message. Maybe there is some issue with my EFI boot partition / data. I'll inverstigate that for now.

[jjb2016]

@IvanVolosyuk - I fixed my system, but the cause remains a mystery. Thankfully the fix was relatively easy. I've updated my original post with details.

[IvanVolosyuk]

Interesting, the crash happens on mount attempt of the corrupted original home dataset, but it doesn't prevent it to be snapshoted and sent. You probably don't want to delete it though as it may make the double allocation to become freed/allocated inconsistency. This corruption can still bite you in the future IMHO as it probably still there, just doesn't cause crashes now.

[IvanVolosyuk]

Now I'm getting the same issue on openzfs 2.3.4 and I do have ECC ram. For me the system is still somewhat functional. #17829 (comment)

[jjb2016]

Oh dear! That's not good. Well, it's been approx 3 weeks since I fixed the issue in my case (by replicating the broken dataset, see details in my original post) and the issue has not reoccurred. So far so good.