Release CFFI 2.0.0

[ngoldbaum]

I'm opening this now since 3.14.0rc2 came out last week. We should have one more release candidate in September and then the final release in October. It would be nice to have a CFFI release available sometime in September, since that will unblock work in reverse dependencies of cryptography and pynacl.

I'll defer to @nitzmahone on what the broader blockers on a final release are, but here's what I think needs to happen:

• Ensure manylinux2014 wheels are built: Explicitly specify manylinux2014 in wheel building config #184
• Add a top-level thread safety page to the docs
• Clean up the Windows testing situation. Fix this commented out line and make this step unconditionally run the same tests on all operating systems.

Anything else?

#126 alludes to a need for more multithreaded testing and better multithreaded test coverage. I'd appreciate more detail on what @nitzmahone had in mind there if he considers it a blocker.

I'd also like to make the Py_mod_gil setting configurable. Right now we unconditionally set it to Py_MOD_GIL_NOT_USED, which is true of the wrapper, but not necessarily the wrapped C code. This requires spitballing an API to control that at build time. I don't think this is a blocker since it can be worked around but I'm also happy to work on this now if you'd like.

I'm motivated to work on this but I'm going to need a little guidance on some of these and on whether there are any other blockers. For now I'm going to work on the thread safety docs and the windows testing situation.

[nitzmahone]

I'd be thrilled if all of that was in for a September 2.0.0 - I probably won't have a ton of time to pitch in on implementation before then, but will definitely commit to reviews and release work as needed.

I had similar thoughts about coming up with a nicer way to expose a module init opt in/out for GIL behavior, but didn't want to hold up the initial pre-release trying to design it. IIRC I successfully hacked together a throwaway attempt with a completely manual call to it from a user CFFI module back in a 3.13 pre-release, but it feels like something that should be a little more declarative and less "everyone needs to reinvent this wheel".

I've got mixed feelings on iOS wheels- I haven't spent any time poking at the PR that was opened for it, but if cibuildwheel supports it with near-native performance on a pretty vanilla GHA Mac aarch64 runner (including actually running at least part of the test suite, not just a YOLO cross-compile as many of the early MacOS aarch64 things were doing), I'm okay with it. If it's going to require a bunch of non-standard tooling/config/setup/emulation though, or will result in completely untested artifacts, I'd rather defer that until it doesn't.

I don't think significant new multi-threaded/free-threaded test coverage needs to be a blocker for a 2.0.0. I've wanted to build a stress test around thread-agile generation, use, and collection of ephemeral types to make sure they're all properly cleaned up, and thread-agile handoffs and collection of CFFI wrapper objects to ensure the refcounting is sound. We're getting a little of that for free by running the existing test suite multi-threaded, but I wouldn't call what I remember seeing a stress test by any means.

[ngoldbaum]

Thanks for the quick feedback! I removed the iOS wheels checkbox since it's not a blocker.

[ngoldbaum]

I opened #188 to add some new thread safety documentation.

[ngoldbaum]

I opened #189, but I think re-enabling full Windows CI is going to need something along the lines of #137. @mattip do you think we can resurrect #137 or does that approach have issues?

[mattip]

I think we can resurrect #137. The approach is sound, the devil is in the small details. Some documentation around how tests are structured might help, i.e. there is this comment

cffi0 checks the verify() call that was in cffi 0.x.y, while cffi1 checks the new compile() call.

and there is that thing about cleaning out the test directories that seems fragile