Using pyo3 on Windows is very difficult

[Ralith]

Recording this for reference by other interested users and in the hopes of motivating future work to streamline things.

Fixups

pyo3-build-config

cargo_env = ["CARGO_PKG_VERSION"]

[[buildscript]]
[buildscript.gen_srcs]

pyo3-macros-backend, pyo3-macros

cargo_env = ["CARGO_PKG_VERSION"]

pyo3-ffi, pyo3

cargo_env = ["CARGO_PKG_VERSION"]

[[buildscript]]
[buildscript.rustc_flags]

Thoughts:

• Why are cargo env vars opt-in? Surely these could be provided by default.
• Reverse-engineering fixup capabilities and syntax from source was difficult. Documentation here would be great.
• Some way to share fixups across the ecosystem could save a lot of pain in the long run.

Target

First attempt:

rust_library(
    name = "py-extension",
    srcs = [...],
    supports_python_dlopen = True,
    link_style = "static",
    deps = ["//third-party:pyo3"]
)

This allows pyo3 itself to build, but trying to build :py-extension[cdylib] (it was difficult to discover that [cdylib] exists; this seems to be wholly undocumented) fails with the literal error LINK : fatal error LNK1181: cannot open input file 'pythonXY.lib'. That arises from blocks in pyo3 that contain lines like this:

#[cfg_attr(windows, link(name = "pythonXY"))]
extern "C" { ... }

pythonXY here is not the name of a real library. Instead the pyo3-ffi build script is supposed to alias it to whatever python version you're targeting at build time by printing something like cargo:rustc-link-lib=pythonXY:python3. reindeer drops this type of output silently.

We can correct the library name by adding a rustc_flags = ["-lpythonXY:python3"] fixup or similar, but this just results in the linker failing to find python3.lib instead. In the cargo build, pyo3 auto-detects the system python's library path (cargo build --verbose shows -L "native=C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.2544.0_x64__qbz5n2kfra8p0\libs").

My first idea was to try to add a prebuilt_cxx_library rule for python3.lib and mark that as a dependency. However, this doesn't remove python3.lib from the linker command line, so the error remains! Further, prebuilt_cxx_library doesn't seem to like import libraries regardless for some reason. Instead, I defined a filegroup named after and containing only the python3.lib copied from my install, then added linker_flags = ["/LIBPATH:$(location :python3.lib)"] to the rust_library. With this, finally, the extension was able to build as a DLL, though I haven't yet had time to see if it works.

[Ralith]

pyo3 also has an optional feature generate-import-lib which uses https://github.com/PyO3/python3-dll-a to construct the import lib on demand in pyo3-ffi's build script. However, this "generate a static library at build-time, but not from C/C++ sources" behavior doesn't seem to be expressible in fixups, and the free "python3.lib" appearing in linker args remains an issue.