Free-threaded Python support

[adamjstewart]

I would like to test my library with the latest free-threaded Python 3.13t and 3.14t with the GIL disabled. However, netCDF4 does not yet provide wheels for this platform. Even if free-threaded mode is not yet fully supported or taken advantage of, publishing wheels for this platform will allow other upstream authors to test their own code in this new mode.

[dopplershift]

Is there benefit to publishing these when the netcdf c library is not thread safe and calling netcdf functions from multiple threads is likely to lead to segfaults?

[adamjstewart]

@rgommers would know the answer to that better than me.

[rgommers]

Yes, there is quite a bit of benefit, especially for a package like this with a hard-to-obtain build dependency. It can be useful for a couple of different reasons:

1. It makes it easier to start testing netcdf4-python itself. By default, the GIL will be re-enabled for cp314t wheels (conservative, safe), and end users can opt in to free-threading by setting the PYTHON_GIL=0 env var.
2. It makes other packages that depend on this one much easier to install. I suspect most users who want to install netcdf4-python under a free-threaded interpreter aren't going to load NetCDF4 files in parallel, they just want regular single-threaded file loading to work so they can use free-threading with their own code without having to build netcdf4-python from source.

@adamjstewart falls in category (2) I suspect: he wants to work on torchgeo's free-threading support, so only needs netcdf4-python to install easily.

[dopplershift]

Seems reasonable. Just wanted to make sure expectations were in line with reality.

[ocefpaf]

Aren't we providing a footgun in 2? Sure, those who know will use it in a single thread, but the number of issues due to the fact that netcdf-c is not thread safe is already high. My guess is that those would only increase if it is provided a a wheel for the free-threaded Python.

[rgommers]

Aren't we providing a footgun in 2?

No, not really, it only makes install easier. Note that if you just add a wheel without making any changes to make the extension modules thread-safe (e.g., add locks around netcdf-c access), then the GIL will be re-enabled on import so that would be no more and no less unsafe than a default (with-GIL) build when used from multiple threads (of course, no faster either).

Only after an audit and testing for thread-safety are done do you then want to update the extension module initialization to keep the GIL disabled at runtime.

---

I think this package is in a very similar position to h5py, which also has to deal with the underlying C library (libhdf5) not being thread-safe when used with Python-level threading. To do so, it has internal locking. Does netcdf-python have internal locks already?

[jswhit]

no netcdf4-python does not have internal locks

[rgommers]

Then it might make sense indeed to consider uploading cp314t wheels (cp313t is experimental and not useful) while leaving the GIL enabled. It's not super difficult for downstream packages to build from source, but figuring out the sudo apt install libhdf5-dev libnetcdf-dev equivalent on all platforms is enough of a pain that it'd be helpful with downstream development work.

[ngoldbaum]

Also worth considering that the library isn't thread-safe on the GIL-enabled build either. I just ran unittest-ft using the GIL-enabled build of 3.14.2, and I immediately hit a segfault inside libhdf5:

goldbaum at Nathans-MacBook-Pro in ~/Documents/netcdf4-python on master
± lldb $(pyenv which python) -- $(pyenv which unittest-ft)
error: module importing failed: Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cpython_lldb'
(lldb) target create "/Users/goldbaum/.pyenv/versions/3.14.2/bin/python"
Current executable set to '/Users/goldbaum/.pyenv/versions/3.14.2/bin/python' (arm64).
(lldb) settings set -- target.run-args  "/Users/goldbaum/.pyenv/versions/3.14.2/bin/unittest-ft"
(lldb) r
Process 65101 launched: '/Users/goldbaum/.pyenv/versions/3.14.2/bin/python' (arm64)
Process 65101 stopped
* thread #5, name = 'ThreadPoolExecutor-0_1', stop reason = EXC_BAD_ACCESS (code=1, address=0x685f72657662696c)
    frame #0: 0x00000001016d9a3c libhdf5.310.5.1.dylib`H5FL_fac_malloc + 24
libhdf5.310.5.1.dylib`H5FL_fac_malloc:
->  0x1016d9a3c <+24>: ldr    x8, [x0]
    0x1016d9a40 <+28>: str    x8, [x19, #0x18]
    0x1016d9a44 <+32>: ldr    w8, [x19, #0x8]
    0x1016d9a48 <+36>: sub    w8, w8, #0x1
  thread #6, name = 'ThreadPoolExecutor-0_2', stop reason = EXC_BAD_ACCESS (code=1, address=0x685f72657662696c)
    frame #0: 0x00000001016d9a3c libhdf5.310.5.1.dylib`H5FL_fac_malloc + 24
libhdf5.310.5.1.dylib`H5FL_fac_malloc:
->  0x1016d9a3c <+24>: ldr    x8, [x0]
    0x1016d9a40 <+28>: str    x8, [x19, #0x18]
    0x1016d9a44 <+32>: ldr    w8, [x19, #0x8]
    0x1016d9a48 <+36>: sub    w8, w8, #0x1
  thread #7, name = 'ThreadPoolExecutor-0_3', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x00000001017c2a10 libhdf5.310.5.1.dylib`H5SL_search + 248
libhdf5.310.5.1.dylib`H5SL_search:
->  0x1017c2a10 <+248>: ldr    x24, [x8, x20, lsl #3]
    0x1017c2a14 <+252>: cbz    x24, 0x1017c2a48 ; <+304>
    0x1017c2a18 <+256>: ldr    w8, [x24, #0x20]
    0x1017c2a1c <+260>: cmp    w8, w21
Target 0: (python) stopped.

so these crashes are possible on the GIL-enabled build too. People probably don't hit them very often because multithreaded programming with the GIL is difficult. Now that free-threaded Python is a thing, it's probably time to start thinking about protecting users from these crashes on both builds.

[ngoldbaum]

Unidata/netcdf-c#382 seems related

[dopplershift]

Should note that the GIL is intentionally not held on netCDF C API calls (to allow other non-netcdf threads to proceed), so any kind of multithreading use of the netcdf4-python API can cause segfaults.

[jswhit]

free-threaded wheels now on pypi

[adamjstewart]

Great! From my perspective, this issue can be closed. But if you want me to keep it open so we have a place to track progress on native free-threaded support, let me know.