Mirrored pool corrupt during the cable jerk + throwing BSOD (SYSTEM_SERVICE_EXCEPTION) on import afterwards

[sskras]

System information

Type	Version/Name
Distribution Name	Windows 10
Distribution Version	21H2
Kernel Version	10.0.19044.3086
Architecture	x86-64
OpenZFS Version	2.3.1 rc14

Describe the problem you're observing

I have a pool of two mirrored HDDs packed into USB-C enclosure named ppool (as in Portable Pool). It's seen as following:

  ...
  pool: ppool
    id: 7387830995368866575
 state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

        ppool                    ONLINE
          mirror-0               ONLINE
            Harddisk1Partition1  ONLINE
            Harddisk2Partition1  ONLINE
          indirect-1             ONLINE
          indirect-2             ONLINE

After some cable incident and the involved Windows restart, now it throws a SYSTEM_SERVICE_EXCEPTION BSOD every time I try to import the pool:

The bugcheck was: 0x0000003b (0x0000000080000003, 0xfffff806599ef1d0, 0xfffffd09ca05e410, 0x0000000000000000).

Unsure if a BSOD (the same or any other) happened during the incident / restart itself.

Describe how to reproduce the problem

No idea. I was using it on Mint 21 (based on Ubuntu 22.04, OpenZFS v2.1.5) for 10 months or so, and have written more than 4T successfully into it.

3 months ago I wrote several tens of gigabytes on Windows 10 (OpenZFS v2.3.1 rc13) just fine too.

After some glitch/jerk of the USB connection, my Windows laptop lost access to the pool, and after physical disconnection, reboot and the reconnection it then failed to import the pool (even blue-screened after a while).

I left the drives attached to my laptop and rebooted Windows, only to find that it tries to import the pool and so gets BSOD again.

After several iterations of the reboot loop I disconnected the pool drives from my Windows laptop, and tried to import it back on my Linux laptop – please see the openzfs#17968 issue in the upstream repo.

Upgrade from rc13 to rc14 didn't seem to change the outcome.

Include any warning/errors/backtraces from the system logs

This Mega.nz folder includes two MEMORY.DMP files from the BSOD (each being circa 1 GiB in size) compressed as .7z files:

• OpenZFS-2.3.1-rc14-import-crash-v1–MEMORY.7z (102.4 MB)

• OpenZFS-2.3.1-rc14-import-crash-v2–MEMORY.7z (98.2 MB)

As analyzed by @lundman in #534 (comment) :

07 ffffa308`7e17ec80 fffff806`531af1d1     nt!KiBreakpointTrap+0x310
08 ffffa308`7e17ee10 fffff806`534c4904     OpenZFS!spl_panic+0x71 [C:\src\openzfs\module\os\windows\spl\spl-err.c @ 87] 
09 ffffa308`7e17ee70 fffff806`533232ec     OpenZFS!vdev_indirect_mapping_open+0x94 [C:\src\openzfs\module\zfs\vdev_indirect_mapping.c @ 348] 
0a ffffa308`7e17ef40 fffff806`532595bc     OpenZFS!spa_remove_init+0x42c [C:\src\openzfs\module\zfs\vdev_removal.c @ 609] 
0b ffffa308`7e17eff0 fffff806`5325866e     OpenZFS!spa_ld_open_indirect_vdev_metadata+0x2c [C:\src\openzfs\module\zfs\spa.c @ 4556] 
0c ffffa308`7e17f040 fffff806`5324c2d5     OpenZFS!spa_load_impl+0x21e [C:\src\openzfs\module\zfs\spa.c @ 5475] 
0d ffffa308`7e17f0c0 fffff806`53258382     OpenZFS!spa_load+0x95 [C:\src\openzfs\module\zfs\spa.c @ 3486] 
0e ffffa308`7e17f130 fffff806`5324b93c     OpenZFS!spa_load_retry+0x92 [C:\src\openzfs\module\zfs\spa.c @ 5719] 
0f ffffa308`7e17f170 fffff806`5324b061     OpenZFS!spa_load_best+0x37c [C:\src\openzfs\module\zfs\spa.c @ 5800] 
10 ffffa308`7e17f220 fffff806`531f7dbf     OpenZFS!spa_import+0x301 [C:\src\openzfs\module\zfs\spa.c @ 6824] 
11 ffffa308`7e17f300 fffff806`531eed8d     OpenZFS!zfs_ioc_pool_import+0x13f [C:\src\openzfs\module\zfs\zfs_ioctl.c @ 1554] 
12 ffffa308`7e17f370 fffff806`53510a5d     OpenZFS!zfsdev_ioctl_common+0x79d [C:\src\openzfs\module\zfs\zfs_ioctl.c @ 8122] 
13 ffffa308`7e17f450 fffff806`535

08 ffffa308`7e17ee10 fffff806`534c4904     OpenZFS!spl_panic+0x71 [C:\src\openzfs\module\os\windows\spl\spl-err.c @ 87] 
09 ffffa308`7e17ee70 fffff806`533232ec     OpenZFS!vdev_indirect_mapping_open+0x94 [C:\src\openzfs\module\zfs\vdev_indirect_mapping.c @ 348] 
0a ffffa308`7e17ef40 fffff806`532595bc     OpenZFS!spa_remove_init+0x42c [C:\src\openzfs\module\zfs\vdev_removal.c @ 609] 
0b ffffa308`7e17eff0 fffff806`5325866e     OpenZFS!spa_ld_open_indirect_vdev_metadata+0x2c [C:\src\openzfs\module\zfs\spa.c @ 4556] 

[lundman]

Didn't we already fix vdev_indirect_mapping_open() ?

[lundman]

Oh we did not - sorry, hard to remember the various issues. And upstream did not really help any.

We could give you a knob to disable vdev_indirect_mapping_open, and related panic - could be you can open it read-only and copy some data off. On the gamble that the device was removed early, before much data was copied on it.

Of course, right now, I am waiting to get my new EV certificate, so I can not codesign a build today.

[lundman]

OK so I did this;

Create a pool with vdev removal active.

• zpool create BROKEN PHYSICALDRIVE2 PHYSICALDRIVE3
• zpool remove BROKEN PHYSICALDRIVE3
• cp README.md /BROKEN/README.md # ok, actually E:/ but let's pretend to be posix
• zpool export BROKEN

Now, add a call to panic at the same place your panic triggers in vdev_indirect_mapping_open()

Now discover zdb.exe does not work anymore, due to unrelated changes.

Fix zdb.exe, confirm it dies in the same way:

$ out/build/x64-Debug/cmd/zdb/zdb.exe -ivvve BROKEN
libzpool: PANIC: simulated panic

Add new tunable to let us pretend the mapping table is empty. This will generally go
very badly if the mapping table is actually used. But, if you happened to have had removed the
device early on - perhaps when setting up the pool - there could be a chance some
data will copy off it.

Test the new tunable:

$ out/build/x64-Debug/cmd/zdb/zdb.exe -ivvve BROKEN
libzpool: PANIC: simulated panic

$ out/build/x64-Debug/cmd/zdb/zdb.exe -o zfs_indirect_open_disable=1  -ivvve BROKEN
Dataset mos [META], ID 0, cr_txg 4, 312K, 49 objects
Dataset BROKEN [ZPL], ID 54, cr_txg 1, 124K, 9 objects
MOS object 83 (uint32) leaked
Verified large_blocks feature refcount of 0 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
Verified edonr feature refcount of 0 is correct
Verified userobj_accounting feature refcount of 1 is correct
Verified encryption feature refcount of 0 is correct
Verified project_quota feature refcount of 1 is correct
Verified redaction_bookmarks feature refcount of 0 is correct
Verified redacted_datasets feature refcount of 0 is correct
Verified bookmark_written feature refcount of 0 is correct
Verified livelist feature refcount of 0 is correct
Verified zstd_compress feature refcount of 0 is correct
Verified zilsaxattr feature refcount of 0 is correct
Verified blake3 feature refcount of 0 is correct
Verified redaction_list_spill feature refcount of 0 is correct
Verified longname feature refcount of 0 is correct
Verified large_microzap feature refcount of 0 is correct
Verified physical_rewrite feature refcount of 0 is correct

Neat, with a bit of luck, see if we can list root, 34 is root ID. The slash on "BROKEN/" is important. "BROKEN" != "BROKEN/"

$ out/build/x64-Debug/cmd/zdb/zdb.exe -o zfs_indirect_open_disable=1 -e -dvvv BROKEN/ 34

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
        34    1   128K    512     8K     512    512  100.00  ZFS directory

                README.md = 4 (type: Regular File)

So README.md is objID 4.

Of course, we knew the path, we could have queried the path directly:

$ out/build/x64-Debug/cmd/zdb/zdb.exe -e -O BROKEN/ README.md
    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         4    1   128K  2.50K    12K     512  2.50K  100.00  ZFS plain file

Yep, README.md is objID 4.

Let's try recovering the file;

$ mkdir recovery

# By filename
$ out/build/x64-Debug/cmd/zdb/zdb.exe -o zfs_indirect_open_disable=1 -e -r BROKEN/ README.md recovery/README.md

# Or by id
$ out/build/x64-Debug/cmd/zdb/zdb.exe -o zfs_indirect_open_disable=1 -e -r -O BROKEN/ 4 recovery/README.md

$ ls -l recovery
-rw-r--r-- 1 WDKRemoteUser 197121 2263 Mar  4 14:39 README.md

If you want to give it a go, note I am not able to codesign zdb.exe at the moment, should be in a day or two.

zdb-tunable.zip