[Python][C++] Python 3.15 Builds Fail due to PyWeakref_GET_OBJECT Removal

[p-vdp]

Describe the bug, including details regarding any error messages, version, and platform.

Python v3.15.0a1 is out, and after building it from source I'm trying to build pyarrow from apache-arrow-22.0.0-rc0.

I'm following the same steps to build pyarrow that I've used successfully on Python 3.14.0.

Here's the error output:

234.8       /root/builds/arrow/python/pyarrow/src/arrow/python/extension_type.cc:
234.8       In member function ‘PyObject* arrow::py::PyExtensionType::GetInstance()
234.8       const’:
234.8       /root/builds/arrow/python/pyarrow/src/arrow/python/extension_type.cc:167:20:
234.8       error: ‘PyWeakref_GET_OBJECT’ was not declared in this scope
234.8         167 |   PyObject* inst = PyWeakref_GET_OBJECT(type_instance_.obj());
234.8             |                    ^~~~~~~~~~~~~~~~~~~~
234.8       gmake[2]: *** [CMakeFiles/arrow_python.dir/build.make:160:
234.8       CMakeFiles/arrow_python.dir/pyarrow/src/arrow/python/extension_type.cc.o]
234.8       Error 1
234.8       gmake[2]: *** Waiting for unfinished jobs....
234.8       gmake[1]: *** [CMakeFiles/Makefile2:126:
234.8       CMakeFiles/arrow_python.dir/all] Error 2
234.8       gmake: *** [Makefile:136: all] Error 2
234.8       error: command '/usr/bin/cmake' failed with exit code 2

I'm running this in Docker on Ubuntu 24.04, it's available as the next image here.

Component(s)

Python, C++

[p-vdp]

Still getting this with rc1:

247.4       In member function ‘PyObject* arrow::py::PyExtensionType::GetInstance()
247.4       const’:
247.4       /root/builds/arrow/python/pyarrow/src/arrow/python/extension_type.cc:167:20:
247.4       error: ‘PyWeakref_GET_OBJECT’ was not declared in this scope
247.4         167 |   PyObject* inst = PyWeakref_GET_OBJECT(type_instance_.obj());
247.4             |                    ^~~~~~~~~~~~~~~~~~~~

[p-vdp]

PyWeakref_GET_OBJECT has been deprecated since Python 3.13 and is being removed in 3.15: https://docs.python.org/3/c-api/weakref.html#c.PyWeakref_GetObject

arrow/python/pyarrow/src/arrow/python/extension_type.cc

Line 167 in 7df396e

PyObject* inst = PyWeakref_GET_OBJECT(type_instance_.obj());

[AlenkaF]

Thank you for opening up the issue @p-vdp !
We do not yet support Python 3.15 but it's very helpful to get this kind of feedback early.

The substitute for PyWeakref_GET_OBJECT will be PyWeakref_GetRef which was added in 3.13. We will need a solution that works for all supported Python versions (currently 3.10 to 3.14).

[paultiq]

Would you consider an #if block specific for 3.15+ like the following, leaving <=3.14 alone.

* This built/ran/passed tests on my local system.

Other approaches:

• c-api-pending-removal-in-3.15 suggests we could use: pythoncapi-compat, but that seems like unnecessary complexity.
• Cython added a backport: https://github.com/cython/cython/blob/master/Cython/Includes/cpython/weakref.pxd. This has one difference in PyWeakref_GET_OBJECT: it checks for inst == NULL

diff --git a/python/pyarrow/src/arrow/python/extension_type.cc b/python/pyarrow/src/arrow/python/extension_type.cc
index 8439ecf858..a2110f9799 100644
--- a/python/pyarrow/src/arrow/python/extension_type.cc
+++ b/python/pyarrow/src/arrow/python/extension_type.cc
@@ -164,6 +164,22 @@ PyObject* PyExtensionType::GetInstance() const {
     return nullptr;
   }
   ARROW_DCHECK(PyWeakref_CheckRef(type_instance_.obj()));
+#if PY_VERSION_HEX >= 0x030F0000  // Python 3.15+
+  PyObject* inst = NULL;
+  int result = PyWeakref_GetRef(type_instance_.obj(), &inst);
+  if (result < 0) {
+    // Error
+    return nullptr;
+  } else if (result == 0) {
+    // Reconstruct - XXX cache again? 
+    return DeserializeExtInstance(type_class_.obj(), storage_type_, serialized_);
+  } else {
+    // Cached instance still alive
+    return inst;
+  }
+#else
+  // PyWeakref_GET_OBJECT removed in 3.15
   PyObject* inst = PyWeakref_GET_OBJECT(type_instance_.obj());
   if (inst != Py_None) {
     // Cached instance still alive
     Py_INCREF(inst);
     return inst;
   } else {
     // Must reconstruct from serialized form
     // XXX cache again?
     return DeserializeExtInstance(type_class_.obj(), storage_type_, serialized_);
   }
+#endif
 }

[raulcd]

In my opinion we should use pythoncapi-compat as it is already vendored in the project and used for other compatibility functions like PyDict_GetItemStringRef or PyList_GetItemRef.

arrow/python/pyarrow/src/arrow/python/vendored/pythoncapi_compat.h

Lines 561 to 584 in ff089cc

	// gh-105927 added PyWeakref_GetRef() to Python 3.13.0a1
	#if PY_VERSION_HEX < 0x030D0000
	static inline int PyWeakref_GetRef(PyObject *ref, PyObject **pobj)
	{
	PyObject *obj;
	if (ref != NULL && !PyWeakref_Check(ref)) {
	*pobj = NULL;
	PyErr_SetString(PyExc_TypeError, "expected a weakref");
	return -1;
	}
	obj = PyWeakref_GetObject(ref);
	if (obj == NULL) {
	// SystemError if ref is NULL
	*pobj = NULL;
	return -1;
	}
	if (obj == Py_None) {
	*pobj = NULL;
	return 0;
	}
	*pobj = Py_NewRef(obj);
	return (*pobj != NULL);
	}
	#endif

[paultiq]

That works too!

In that case, it'd be:

diff --git a/python/pyarrow/src/arrow/python/extension_type.cc b/python/pyarrow/src/arrow/python/extension_type.cc
+ #include "arrow/python/vendored/pythoncapi_compat.h"
...
@@ -164,10 +165,13 @@ PyObject* PyExtensionType::GetInstance() const {
     return nullptr;
   }
   ARROW_DCHECK(PyWeakref_CheckRef(type_instance_.obj()));
-  PyObject* inst = PyWeakref_GET_OBJECT(type_instance_.obj());
-  if (inst != Py_None) {
+  PyObject* inst = NULL;
+  int result = PyWeakref_GetRef(type_instance_.obj(), &inst);
+  if (result < 0) {
+    // Error
+    return nullptr;
+  } else if (result > 0) {
     // Cached instance still alive
-    Py_INCREF(inst);
     return inst;
   } else {
    // Must reconstruct from serialized form
    // XXX cache again?
    return DeserializeExtInstance(type_class_.obj(), storage_type_, serialized_);
  }
 }

[raulcd]

@paultiq do you want to push a PR? I am happy to follow up with testing/review

[paultiq]

PR opened.

I'd suggest: Changing issue title to "Python 3.15 Builds Fail due to PyWeakref_GET_OBJECT Removal"? There's likely many other steps involved for 3.15 support.

In terms of next steps, once PR lands, would it make sense to file a separate issue for adding Python 3.15 to nightlys?