Move from pybind11 to nanobind + raw C API

[JCGoran]

From an internal discussion:

In NMODL we use pybind11 to achieve 2 things:

a) making a Python wrapper for (some of the) C++ API
b) embedding the interpreter

For b) we actually just call our hand-made Python script for solving systems of equations using SymPy, and nothing else. Since in NEURON we are using nanobind for a), I think it would make sense to just port a) to nanobind, and use the Python C API directly for b) (since nanobind does not support embedding the interpreter). The benefits of this switch are two-fold:

• once we integrate NMODL into NEURON, we don’t need to keep track of two libraries, but only one (nanobind is anyway smaller and more performant)
• pybind11 does not support the Python limited C API (which has guaranteed ABI stability), which means that, until we remove pybind11, we need to build wheels for every single minor version of Python. On the other hand, nanobind supports the limited API since 3.12, so at least the Python wheels would be more or less future-proof (so we’d only need to build wheels on one Python version, i.e. the oldest one supported)

[Helveg]

we need to build wheels for every single minor version of Python

Crossposting here as well, but building wheels for minor versions is not as big of a burden as you would imagine with a dedicated wheel building tool like cibuildwheel, it repeats your build across every minor version, interpreter, architecture, platform, ... and at the end of the CI run you have a "wheelhouse" directory with hundreds of juicy wheels to upload to pypi :)

[JCGoran]

we need to build wheels for every single minor version of Python

Crossposting here as well, but building wheels for minor versions is not as big of a burden as you would imagine with a dedicated wheel building tool like cibuildwheel, it repeats your build across every minor version, interpreter, architecture, platform, ... and at the end of the CI run you have a "wheelhouse" directory with hundreds of juicy wheels to upload to pypi :)

For building NMODL wheels this is fine, however, we're currently trying to integrate it in NEURON (see neuronsimulator/nrn#3308 and associated PRs), which is already using nanobind, so there are several issues with the status quo:

• we are using 2 Python wrapper libraries, pybind11 and nanobind, with very similar functionality (in fact, nanobind was created by the main dev of pybind11 as a smaller and faster replacement for it). Using just one would simplify the codebase, as well as probably reduce build times (according to Compile time investigation of NMODL #299 and Can we split pyast.cpp into multiple files? #260, pybind11 parts of NMODL make up a large part of the build time)
• while NMODL itself can use cibuildwheel, NEURON does not support it yet (one of the medium-term objectives is to move NEURON's build system towards supporting cibuildwheel, but the current build system is rather complex so progress is a bit slow)
• using the stable ABI should (keyword is "should" 😅) allow us to support newer minor versions of Python automatically, enabling us to dedicate more time towards the development of other, more impactful features (of course, Python support is very important, but if we can avoid having to dig in the cPython API for each release, that'd be great)
• to add to the above point, there's also the fact that we are spending part of the (IMHO unnecessary) CI budget building wheels, as well as bloating PyPI by having to upload what is basically the same thing for each minor version of Python (the latter is more of a problem for nightly, since we hit the 10 GB limit multiple times, and needed to purge older versions)

Overall, while the current situation with the build system works, there's definitely room for improvement.