zfs-windows-2.4.1rc5

[lundman]

Synced up with upstream for openzfs-2.4.1 -

This is a very large list of commits:

https://github.com/openzfsonwindows/openzfs/commits/zfs-Windows-2.4.1-release/?since=20260116

---

This discussion was created from the release zfs-windows-2.4.1rc1.

[lundman]

OK, even though there isn't any Windows specific changes in this build, it would be nice to go over and check what Issue is still current.

I would like to hear back about the memory leak,
I need to look at the icons not showing.
Junctions can't be deleted.

Champion for your issue that you want looked at :)

[Armin-windows]

Surprise in the morning, a new version.
Tried it right away, integrated with my 4xHDD raidz1 pool.
When writing to the pool, Windows crashes immediately and the system freezes.
Unable to reboot, “Page fault in Nonpaged Area (0x50)”.
After restoring the backup, pool ok.
Writing to the pool ok.
Reinstalled new driver 2.4.1, rebooted.
Same problem as before, integrating and reading the pool ok, but crashes immediately when writing.

[lundman]

What a bummer, did you get a memory.dmp at all? Also, restoring from backup sounds quite extreme, you can remove the openzfs.sys driver in advanced boot cmdline
https://github.com/openzfsonwindows/openzfs/wiki/Windows-boot-loop

[Armin-windows]

sorry no memory Dump because reboot not possible.

[lundman]

Actually, we should probably disabled auto-import this early on, which is most likely why it panicked again on boot. Either by commenting import out in the zedlet, or disabling zed service.
Oh, or you can import not using the zpool.cache file

[Armin-windows]

After some trial and error:
After importing tank raidz with rc1 and rc2, reading is OK, but writing freezes the system. Boots again after deleting openzfs.sys. Dump file was created with 0 bytes, so it's useless.
I then created a pool with a single drive using rc2.
During creation, the Explorer window goes dark and no longer displays anything.
After exporting the pool, the Explorer window disappears and everything is displayed again after reopening it.
Then import all, the pool is integrated and displayed in Explorer.
"Copying files to the pool with one drive works.
Copying files to the Raidz pool immediately freezes.
Reading from the Raidz pool is fine."

cbuf was created after explorer Windows goes dark.
cbuf.txt

[lundman]

Ah that's a good clue, I have not tried raidz locally, and they did make some changes over there.

[lundman]

Might also be worth looking at what implementation raidz is using, and changing to generic

[Armin-windows]

P.s. I just created a second pool with another disk, and it worked perfectly. Maybe the Explorer window going dark was just a coincidence.

[lundman]

Created a 3 disk raidz without issue, my vdev_raidz_impl only has original or scalar, so no issues there.

[Armin-windows]

Okay, I've recreated this on my end.
Created pool raidz1 with 3 drives. OK.
Copied data to the pool. OK.
Encrypted pool: zfs create -o encryption=aes-256-gcm -o keyformat=passphrase -o keylocation=file:///s:/drivekey.bin tank/encrypted. OK.
Assigned drive letters: zfs set driveletter=m tank/encrypted. OK.
Disabled drive letters for Tank: zfs set driveletter=off tank.
Immediately froze the system.

022526-74359-01.dmp

[lundman]

so turn off driveletter for the pool, so it is to go in C:? or just tank/encrypted?

[lundman]

022526-74359-01.dmp

00 ffff8085`32b3f5e0 fffff806`6a095d7d     OpenZFS!spl_panic+0x70 [C:\src\openzfs\module\os\windows\spl\spl-err.c @ 86] 
01 ffff8085`32b3f640 fffff806`6a08f30f     OpenZFS!zio_write_compress+0x240d [C:\src\openzfs\module\zfs\zio.c @ 2113] 
02 (Inline Function) --------`--------     OpenZFS!__zio_execute+0x304 [C:\src\openzfs\module\zfs\zio.c @ 2483] 
03 ffff8085`32b3fa00 fffff806`69ef4da1     OpenZFS!zio_execute+0x31f [C:\src\openzfs\module\zfs\zio.c @ 2394] 
04 ffff8085`32b3fa80 fffff806`b4c8a75a     OpenZFS!taskq_thread+0x501 [C:\src\openzfs\module\os\windows\spl\spl-taskq.c @ 2083] 
05 ffff8085`32b3fb30 fffff806`b4eabe84     nt!PspSystemThreadStartup+0x5a
06 ffff8085`32b3fb80 00000000`00000000     nt!KiStartSystemThread+0x34

Ah yep, there is a new zstd, and it was a hassle to get to compile. It also enabled intrinsics so cpu type will matter. Upstream disables instrinsics, so maybe we should too.

[lundman]

Ah nope, its that upstream went to "assume enums are unsigned" - so we need to #ifdef around ALL of them

[lundman]

db0e328

Thanks, the dump file was the key.

[Armin-windows]

The drive letter should simply be invisible because the encrypted part has its own.

I found another dump file, maybe it will help.

Even after recreating a RAIDZ or mirror pool,encrypted or not, the computer eventually just stops working when the pool is mounted.
ResourceTimeout-20260225-0942.dmp

[Armin-windows]

RC3 pool import ok.
pool writing .....freeze.
Uploading 022626-25187-01.dmp…

[lundman]

ResourceTimeout-20260225-0942.dmp

Two threads ffffe40952f020c0 <*> ffffe40952cfc080

Thread A ffffe40952f020c0 

 : nt!KiSwapContext+0x76
 : nt!KiSwapThread+0x6d4
 : nt!KiCommitThreadWait+0x39d
 : nt!KeWaitForMultipleObjects+0x5bd
 : OpenZFS!spl_cv_wait+0x10c [C:\src\openzfs\module\os\windows\spl\spl-condvar.c @ 130] 
 : 0xdeadbeef`deadbeef

Thread B ffffe40952cfc080 

 : nt!KiSwapContext+0x76
 : nt!KiSwapThread+0x6d4
 : nt!KiCommitThreadWait+0x39d
 : nt!KeWaitForSingleObject+0x859
 : nt!ExpWaitForResource+0x72
 : nt!ExpAcquireResourceSharedLite+0xcea
 : nt!ExAcquireResourceSharedLite+0x36
 : OpenZFS!rw_enter+0x43 [C:\src\openzfs\module\os\windows\spl\spl-rwlock.c @ 100] 
 : ffffe409`4a7eb000 ffffe409`4ac29008 ffffe409`4ac29218 00000002`37f76da8 : OpenZFS!spl_mutex_exit+0xf8 [C:\src\openzfs\module\os\windows\spl\spl-mutex.c @ 167] 
  : OpenZFS!zfs_sticky_remove_access+0x50 [C:\src\openzfs\module\os\windows\zfs\zfs_dir.c @ 1234] 


.thread ffffe40952cfc080  # ThreadB

 : OpenZFS!rw_enter+0x43 [C:\src\openzfs\module\os\windows\spl\spl-rwlock.c @ 100] 

0: kd> dt rwlp
Local var @ 0xffff808537f76cb8 Type krwlock*
0xffffe409`4ac29218 
   +0x000 rw_lock          : _ERESOURCE
   +0x068 rw_owner         : 0xffffe409`52f020c0 Void  # thread A 
   +0x070 rw_readers       : 0n0
   +0x074 rw_pad           : 0n305419896

So ThreadB is waiting in rw_enter(), which is held by ThreadA. 


0: kd> .thread 0xffffe409`52f020c0 # ThreadA

Implicit thread is now ffffe409`52f020c0

0: kd> k
  *** Stack trace for last set context - .thread/.cxr resets it
 # Child-SP          RetAddr               Call Site
00 ffff8085`371dde20 fffff806`b4a5a514     nt!KiSwapContext+0x76
01 ffff8085`371ddf60 fffff806`b4ae8a3d     nt!KiSwapThread+0x6d4
02 ffff8085`371ddff0 fffff806`b4ae575d     nt!KiCommitThreadWait+0x39d
03 ffff8085`371de080 fffff806`69f10b0c     nt!KeWaitForMultipleObjects+0x5bd
04 ffff8085`371de180 deadbeef`deadbeef     OpenZFS!spl_cv_wait+0x10c [C:\src\openzfs\module\os\windows\spl\spl-condvar.c @ 130] 
05 ffff8085`371de188 ffffe409`52f020c0     0xdeadbeef`deadbeef
06 ffff8085`371de190 ffffcf09`b983df18     0xffffe409`52f020c0
07 ffff8085`371de198 ffff8085`371de7b0     0xffffcf09`b983df18
08 ffff8085`371de1a0 ffff8085`371de700     0xffff8085`371de7b0
09 ffff8085`371de1a8 00000000`00000000     0xffff8085`371de700

0: kd> dt cvp
Local var @ 0xffff8085371de1d8 Type cv*
0xffffe409`4ab68c60 
   +0x000 cv_kevent        : [2] _KEVENT
   +0x030 cv_waiters_count : ??
   +0x034 cv_initialised   : ??
Memory read error ffffe4094ab68c94

Not enough information in the .DMP file to see the values there.

[Armin-windows]

In 2.4.1, regardless of the version and regardless of the type of pool (single disk or RAID), immediate crash upon the first write attempt.
after crash no dmp file.

I'm back to rc14.
When copying Files to a new encrypted pool after 20 minutes ... crash.
030726-24078-01.dmp

[lundman]

First one:

04 ffff9086cd5be9a0 0001808000000000 OpenZFS!spl_vsnprintf+0x429 [C:\src\openzfs\module\os\windows\spl\spl-ddi.c @ 764]

So that is interesting, and quite likely a place for a bug. I can't see what called it, tho..

Second is.. almost identical, but different filename. Non-latin characters in both. Curious.

I do wish you could increase the size of the your dumps so we can see exactly where it crashes. The minis are just too lean..

OK I think i see the issue - will make a build once I get my new EV certificate

[Armin-windows]

How can i increase the size of the dumpfile?
In my setting is used full memory dump.

[lundman]

Ah hmm, looks ok.

C:\Windows\Minidump*.dmp usually means small dump
C:\Windows\MEMORY.DMP usually means kernel/complete/automatic/active

so maybe window does whatever it does

[Armin-windows]

got a new dumpfile.

030926-27562-01.dmp
030926-28031-01.dmp

maybe a problem with usb drives?

[lundman]

Hard to tell'

k
# Child-SP          RetAddr               Call Site
00 fffffb0f`4bc87318 fffff806`bc58708f     nt!KeBugCheckEx
01 fffffb0f`4bc87320 fffff806`6f8caf82     nt!IoBuildPartialMdl+0x12f
02 fffffb0f`4bc87360 fffff806`6f8c9939     USBSTOR!USBSTOR_DataTransfer+0x602
03 fffffb0f`4bc87420 fffff806`bc5d9424     USBSTOR!USBSTOR_CbwCompletion+0x219
04 fffffb0f`4bc874a0 fffff806`bc5d9083     nt!IopfCompleteRequest+0x374
05 fffffb0f`4bc87620 fffff806`4ec559fe     nt!IofCompleteRequest+0x13
06 (Inline Function) --------`--------     Wdf01000!FxIrp::CompleteRequest+0x13 [minkernel\wdf\framework\shared\inc\private\km\FxIrpKm.hpp @ 75] 
07 fffffb0f`4bc87650 fffff806`4ec846a7     Wdf01000!FxRequest::CompleteInternal+0x13e [minkernel\wdf\framework\shared\core\fxrequest.cpp @ 869] 
08 (Inline Function) --------`--------     Wdf01000!FxRequest::Complete+0x35 [minkernel\wdf\framework\shared\inc\private\common\FxRequest.hpp @ 803] 
09 fffffb0f`4bc876e0 fffff806`6d8a5ed0     Wdf01000!imp_WdfRequestComplete+0xa7 [minkernel\wdf\framework\shared\core\fxrequestapi.cpp @ 436] 
0a fffffb0f`4bc87760 fffff806`6d893334     USBXHCI!Bulk_Transfer_CompleteCancelable+0x1a0
0b fffffb0f`4bc877f0 fffff806`6d892e30     USBXHCI!Bulk_ProcessTransferEventWithED1+0x4e4
0c fffffb0f`4bc878d0 fffff806`6d89278e     USBXHCI!Bulk_EP_TransferEventHandler+0x10
0d fffffb0f`4bc87900 fffff806`6d8b5e44     USBXHCI!Endpoint_TransferEventHandler+0x8e
0e fffffb0f`4bc87960 fffff806`6d8b580c     USBXHCI!Interrupter_DeferredWorkProcessor+0x624
0f fffffb0f`4bc87a60 fffff806`4ec7bcb1     USBXHCI!Interrupter_WdfEvtInterruptDpc+0xc
10 (Inline Function) --------`--------     Wdf01000!FxInterrupt::DpcHandler+0x69 [minkernel\wdf\framework\shared\irphandlers\pnp\km\interruptobjectkm.cpp @ 75] 
11 fffffb0f`4bc87a90 fffff806`bc408e12     Wdf01000!FxInterrupt::_InterruptDpcThunk+0xb1 [minkernel\wdf\framework\shared\irphandlers\pnp\km\interruptobjectkm.cpp @ 405] 
12 fffffb0f`4bc87ae0 fffff806`bc405c4e     nt!KiExecuteAllDpcs+0x692
13 fffffb0f`4bc87d20 fffff806`bc8b2ad5     nt!KiRetireDpcList+0x43e
14 fffffb0f`4bc87fb0 fffff806`bc8b2a7f     nt!KxSwapStacksAndRetireDpcList+0x5
15 fffffb0f`528eefe0 fffff806`bc446bc5     nt!KiPlatformSwapStacksAndCallReturn
16 fffffb0f`528eeff0 fffff806`bc8b22ab     nt!KiDispatchInterrupt+0x65
17 fffffb0f`528ef020 fffff806`bc8acf44     nt!KiDpcInterruptBypass+0x1b
18 fffffb0f`528ef050 fffff806`bc462f54     nt!KiInterruptDispatch+0x64
19 fffffb0f`528ef1e0 fffff806`bc4601f2     nt!ExAllocateHeapPool+0x2cf4
1a fffffb0f`528ef470 fffff806`bcd7218d     nt!ExpAllocatePoolWithTagFromNode+0x52
1b fffffb0f`528ef4b0 fffff806`bcd720b4     nt!ExAllocatePool2+0x9d
1c fffffb0f`528ef560 fffff806`6e9f2349     nt!ExAllocatePoolWithTag+0xa4
1d fffffb0f`528ef5a0 00000000`00000000     AscRegistryFilter+0x2349

nothing definitely OpenZFS but we cant rule it out either

[Armin-windows]

with rc3 no pool write possible for me.
i returned my system to rc14 and all seems ok but after 20 minutes writing data to pool my system freeze.
after reboot see dmp file.

[xzr6d]

Just wanted to report that I attempted an install of rc3 on Windows Sandbox, but it resulted in a crash and immediate termination of the sandbox right after the installer finished.

Because Windows Sandbox is ephemeral, the container wiped itself immediately upon crashing, so I wasn't able to grab the dump files before they disappeared. I will try a Hyper-V install later and it should throw a BSOD. That way I'll actually be able to capture the memory dump.

After I gather that data I will edit this post to provide a follow-up with the crash details.

Anyway, ZFS on Windows is a cool project. I'm looking forward to reading up more and hopefully contributing in the future!

Recording.2026-03-10.123416.mp4

[lundman]

presumably hypervisor and windows kernel debugger can play nice, and you can just connect to it when crashed instead of hoping for memory.dmp

[xzr6d]

It did not have an earlier version of OpenZFS for Windows installed. It was a completely fresh Windows 11 Hyper-V environment.

Interestingly, on my most recent install attempt, it threw an IRQL_NOT_LESS_OR_EQUAL BSOD instead of the PAGE_FAULT one, and it actually managed to successfully write a MEMORY.DMP.

I dumped it in WinDbg, but it looks like delayed memory corruption. The stack trace shows the system dying down in the kernel ntkrnlmp.exe during routine processor scheduling tasks nt!KiAddThreadToPrcbQueue and nt!KiIdleLoop.

BUGCHECK_CODE:  a
BUGCHECK_P1: 8
BUGCHECK_P2: 2
BUGCHECK_P3: 0
BUGCHECK_P4: fffff803369385e1

PROCESS_NAME:  System
IP_IN_PAGED_CODE: 
nt!KiAddThreadToPrcbQueue+191
fffff803`369385e1 4c397108        cmp     qword ptr [rcx+8],r14

STACK_TEXT:  
ffffe409`1625ec48 fffff803`36a2bd29     : 00000000`0000000a 00000000`00000008 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
ffffe409`1625ec50 fffff803`36a27189     : ffffe501`ff191180 fffff803`36807a13 00000000`00000000 00000000`00000002 : nt!KiBugCheckDispatch+0x69
ffffe409`1625ed90 fffff803`369385e1     : ffffe501`ff191180 ffffc08d`e534f080 ffffe501`ff191180 ffffc08d`ec140001 : nt!KiPageFault+0x489
ffffe409`1625ef20 fffff803`36807d13     : ffffe501`ff191180 00000000`00000002 ffffe501`ff191180 ffffe409`1625f070 : nt!KiAddThreadToPrcbQueue+0x191
ffffe409`1625ef70 fffff803`36807674     : ffffe501`ff191180 00000000`00000000 ffffe501`ff191180 ffffe501`ff191180 : nt!KiGroupSchedulingGenerationEnd+0x2bb
ffffe409`1625f170 fffff803`36935453     : 0002df82`0002df9a 00000000`00000000 ffffe501`ff191180 ffffe501`ff191100 : nt!KiGroupSchedulingQuantumEnd+0xdc
ffffe409`1625f350 fffff803`36a1ae14     : 00000000`00000de8 ffffc08d`ec140080 ffffe501`ff191180 00000000`00000000 : nt!KiQuantumEnd+0x5e3
ffffe409`1625f5c0 00000000`00000000     : ffffe409`16260000 ffffe409`16259000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0xb4

[lundman]

Hmm disappointing little information there, can you see if you have a OpenZFS!cbuf to dump as well? I usually use

.writemem C:\Users\<your Windows username>\Desktop\cbuf.txt poi(OpenZFS!cbuf) L100000

There is a OpenZVOL!cbuf as well, but that is loaded later.

[xzr6d]

I took your advice to try and catch the cbuf live, so I set up a live kernel debugger session via a named pipe between my VM and WinDbg on the host.

The install of 2.4.1rc3 actually completed successfully without crashing unlike my previous attempts. However, upon the first reboot the system failed to boot and went straight into a boot loop, eventually dropping into Windows Startup Repair.

Because it's crashing so early in the boot process the system isn't writing a MEMORY.DMP to the disk at all.

I did manage to catch the crash live over the KD pipe, but the Hyper-V serial pipe desynced and dropped the connection at the exact moment the Bugcheck occurred, throwing endless GetContextState failed, 0xD0000001 errors. Because WinDbg lost its grip on the processor context, I couldn't run .writemem to pull the OpenZFS!cbuf or OpenZVOL!cbuf live.

I did manage to capture the exact console output right before the KD connection died. It’s a Bugcheck 0x50 occurring immediately after OpenZVOL attempts its DriverEntry.

ZFS: Loaded module zfswin-2.4.1rc3, ZFS pool version 5000, ZFS filesystem version 5
OpenZFS: Started
OpenZVOL: DriverEntry
Created /Device/StopUnload
Created /Device/OpenZVOL
ZFS: Loaded module zfswin-2.4.1rc3, ZFS pool version 5000, ZFS filesystem version 5
KDTARGET: Refreshing KD connection

*** Fatal System Error: 0x00000050
                       (0xFFFFF8034BC0E498,0x0000000000000002,0xFFFFF8032E661D4D,0x0000000000000002)

Break instruction exception - code 80000003 (first chance)

I will continue hammering away at this to see if I can get you some better data to work off of.

[lundman]

Just double checking, when I set up target VM, Install SDK/WDK then run WDK Test Target Setup x64-x64_en-us.msi, and tell VS to deploy to it. It installs a driver just above the ethernet drive, so it can handle reboot:
https://github.com/openzfsonwindows/openzfs/tree/development/module/os/windows

Is that what you have done?

Can you also remove OpenZVOL.sys for now - in case it is related to that old issue

[lundman]

Good News Everyone, I have received the new EV certificate, so we are back in business.