python: segfault when using Arc<dyn T> in Object where T is an exported trait

[joe-p]

Edit: The solution is to make the trait being passed as Arc<dyn T> foreign. Keeping the issue open because a better error message would be nice

I have defined the following Object

#[derive(uniffi::Object, Clone)]
pub struct ABIDynamicArray {
    element_type: Arc<dyn ABIType>,
}

impl FfiToRustABIType for ABIDynamicArray {
    fn to_rust_abi_type(&self) -> RustABIType {
        eprintln!("DEBUGPRINT[43]: dynamic_array.rs:34 (before let cloned = self.clone();)");
        let cloned = self.clone();
        eprintln!("DEBUGPRINT[44]: dynamic_array.rs:35 (after let cloned = self.clone();)");

        eprintln!("DEBUGPRINT[46]: dynamic_array.rs:38 (before cloned.into())");
        let res = cloned.into();
        eprintln!("DEBUGPRINT[47]: dynamic_array.rs:39 (after cloned.into())");
        res
    }
}

#[uniffi::export]
impl ABIType for ABIDynamicArray {
    fn decoode(&self, data: &[u8]) -> ABIValue {
        let rust_abi_type = self.to_rust_abi_type();
        ABIValue::from(rust_abi_type.decode(data).unwrap())
    }

    fn encode(&self, value: ABIValue) -> Vec<u8> {
        let rust_abi_type = self.to_rust_abi_type();
        rust_abi_type.encode(&value.into()).unwrap()
    }
}

#[uniffi::export]
impl ABIDynamicArray {
    #[uniffi::constructor]
    pub fn new(element_type: Arc<dyn ABIType>) -> Self {
        ABIDynamicArray { element_type }
    }
}

Attempting to use the python bindings:

def test_abi_bool_array():
    abi_arr = AbiDynamicArray(element_type=AbiBool())

    arr = AbiValue(array=[AbiValue(bool=True)])
    encoded = abi_arr.encode(arr)

    assert encoded == b'\x000180'

I get the following segfault. One strange thing to note us that sometimes it segfaults on self.clone() and sometimes it segfaults on cloned.into(). This seems like a problem with erroneously dropping a reference in the binding somewhere in a non-deterministic way.

lldb -- "$(poetry run which python)" -X dev -m pytest -s -k array
(lldb) target create "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_uti
ls/.venv/bin/python"
Current executable set to '/Users/joe/git/algorandfoundation/algokit-core/packages/python/algoki
t_utils/.venv/bin/python' (arm64).
(lldb) settings set -- target.run-args  "-X" "dev" "-m" "pytest" "-s" "-k" "array"
(lldb) run
Process 7124 launched: '/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_u
tils/.venv/bin/python' (arm64)
/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.1
3/site-packages/pytest_asyncio/plugin.py:211: PytestDeprecationWarning: The configuration option
 "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future
 versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function sc
ope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the 
future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
============================= test session starts ==============================
platform darwin -- Python 3.13.2, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils
configfile: pyproject.toml
plugins: asyncio-1.1.0
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_sc
ope=function
collected 3 items / 2 deselected / 1 selected                                  

tests/test_utils.py DEBUGPRINT[43]: dynamic_array.rs:34 (before let cloned = self.clone();)
Process 7124 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000102d3e37c libalgokit_utils_ffi.dylib`_$LT$algokit_utils_ffi..abi..abi_typ
e..dynamic_array..ABIDynamicArray$u20$as$u20$algokit_utils_ffi..abi..abi_type..FfiToRustABIType$
GT$::to_rust_abi_type::h7cd0d5bd89393e60 + 68
libalgokit_utils_ffi.dylib`_$LT$algokit_utils_ffi..abi..abi_type..dynamic_array..ABIDynamicArray
$u20$as$u20$algokit_utils_ffi..abi..abi_type..FfiToRustABIType$GT$::to_rust_abi_type::h7cd0d5bd8
9393e60:
->  0x102d3e37c <+68>: ldadd  x22, x8, [x20]
    0x102d3e380 <+72>: tbnz   x8, #0x3f, 0x102d3e3d0 ; <+152>
    0x102d3e384 <+76>: ldr    x21, [x21, #0x8]
    0x102d3e388 <+80>: stp    x20, x21, [sp]
Target 0: (python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000102d3e37c libalgokit_utils_ffi.dylib`_$LT$algokit_utils_ffi..abi..abi_typ
e..dynamic_array..ABIDynamicArray$u20$as$u20$algokit_utils_ffi..abi..abi_type..FfiToRustABIType$
GT$::to_rust_abi_type::h7cd0d5bd89393e60 + 68
    frame #1: 0x0000000102d3edec libalgokit_utils_ffi.dylib`_$LT$algokit_utils_ffi..abi..abi_typ
e..dynamic_array..ABIDynamicArray$u20$as$u20$algokit_utils_ffi..abi..abi_type..ABIType$GT$::enco
de::h000991ce3b96c180 + 36
    frame #2: 0x0000000102d22628 libalgokit_utils_ffi.dylib`uniffi_core::ffi::rustcalls::rust_ca
ll::h521bb2588921a417 + 412
    frame #3: 0x0000000102d3f018 libalgokit_utils_ffi.dylib`uniffi_algokit_utils_ffi_fn_method_a
bidynamicarray_encode + 44
    frame #4: 0x00000001016c0050 libpython3.13.dylib`ffi_call_SYSV + 80
    frame #5: 0x00000001016beae4 libpython3.13.dylib`ffi_call_int + 1452
    frame #6: 0x00000001016be52c libpython3.13.dylib`ffi_call + 52
    frame #7: 0x0000000101d27620 libpython3.13.dylib`_call_function_pointer + 248
    frame #8: 0x0000000101d27194 libpython3.13.dylib`_ctypes_callproc + 724
    frame #9: 0x0000000101d24768 libpython3.13.dylib`PyCFuncPtr_call + 332
    frame #10: 0x000000010137760c libpython3.13.dylib`_PyEval_EvalFrameDefault + 55044
    frame #11: 0x0000000101466d84 libpython3.13.dylib`_PyObject_Call_Prepend + 296
    frame #12: 0x0000000101466798 libpython3.13.dylib`slot_tp_call + 216
    frame #13: 0x000000010137985c libpython3.13.dylib`_PyEval_EvalFrameDefault + 63828
    frame #14: 0x0000000101466d84 libpython3.13.dylib`_PyObject_Call_Prepend + 296
    frame #15: 0x0000000101466798 libpython3.13.dylib`slot_tp_call + 216
    frame #16: 0x000000010137760c libpython3.13.dylib`_PyEval_EvalFrameDefault + 55044
    frame #17: 0x0000000101466d84 libpython3.13.dylib`_PyObject_Call_Prepend + 296
    frame #18: 0x0000000101466798 libpython3.13.dylib`slot_tp_call + 216
    frame #19: 0x000000010137985c libpython3.13.dylib`_PyEval_EvalFrameDefault + 63828
    frame #20: 0x0000000101466d84 libpython3.13.dylib`_PyObject_Call_Prepend + 296
    frame #21: 0x0000000101466798 libpython3.13.dylib`slot_tp_call + 216
    frame #22: 0x000000010137985c libpython3.13.dylib`_PyEval_EvalFrameDefault + 63828
    frame #23: 0x0000000101466d84 libpython3.13.dylib`_PyObject_Call_Prepend + 296
    frame #24: 0x0000000101466798 libpython3.13.dylib`slot_tp_call + 216
    frame #25: 0x000000010137985c libpython3.13.dylib`_PyEval_EvalFrameDefault + 63828
    frame #26: 0x00000001014640c8 libpython3.13.dylib`PyEval_EvalCode + 132
    frame #27: 0x0000000101463bd8 libpython3.13.dylib`builtin_exec + 388
    frame #28: 0x0000000101600160 libpython3.13.dylib`cfunction_vectorcall_FASTCALL_KEYWORDS.llv
m.12452703498012727648 + 88
    frame #29: 0x0000000101374908 libpython3.13.dylib`_PyEval_EvalFrameDefault + 43520
    frame #30: 0x000000010154edcc libpython3.13.dylib`pymain_run_module + 232
    frame #31: 0x000000010154dfc4 libpython3.13.dylib`Py_RunMain + 488
    frame #32: 0x00000001014a758c libpython3.13.dylib`pymain_main + 468
    frame #33: 0x00000001014a73ac libpython3.13.dylib`Py_BytesMain + 36
    frame #34: 0x0000000190582b98 dyld`start + 6076
(lldb) 

[bendk]

What UniFFI version is this? Also, can you share your text fixture so I can test out the segfaults?

[joe-p]

This is on the latest main commit, a4cbb09.

The repo + branch to reproduce is here: https://github.com/algorandfoundation/algokit-core/tree/debug/abi_type_segfault

In the pytest file I've included the paths to where the corresponding Rust types are implemented: https://github.com/algorandfoundation/algokit-core/blob/debug/abi_type_segfault/packages/python/algokit_utils/tests/test_utils.py#L119-L120

To build lib + generate bindings: cargo pkg utils py
To run tests: poetry run pytest in packages/python/algokit_utils

This is a fairly big repo but this specific bug only touches a small part of it. I don't think you'll need anything else than the above commands but let me know if you got any questions.

Edit: Also should note that this branch is a bit different than the snippets in the report, but it's the same bug being reproduced

[bendk]

Sorry for taking a while to get back to you on this one, I got distracted. I had some issues setting up your test fixture, can you try running it with the uniffi/ffi-trace flag enabled? That should give some good low-level details on what's happening with these pointers.

[joe-p]

All good! This currently is not a blocker for us.

tests/test_utils.py::test_abi_bool_encoding calling: uniffi_algokit_utils_ffi_fn_constructor_abivalue_bool
success: uniffi_algokit_utils_ffi_fn_constructor_abivalue_bool
lower: ABIValue 0x6000033fc370
clone: ABIValue (Handle(0x6000033fc370))
calling: uniffi_algokit_utils_ffi_fn_method_abivalue_encoded_bytes
lift: ABIValue (Handle(0x6000033fc370))
success: uniffi_algokit_utils_ffi_fn_method_abivalue_encoded_bytes
clone: ABIValue (Handle(0x6000033fc370))
calling: uniffi_algokit_utils_ffi_fn_method_abivalue_get_bool
lift: ABIValue (Handle(0x6000033fc370))
success: uniffi_algokit_utils_ffi_fn_method_abivalue_get_bool
free: ABIValue (Handle(0x6000033fc370))
PASSED
tests/test_utils.py::test_abi_bool_from_bytes calling: uniffi_algokit_utils_ffi_fn_constructor_abibool_new
success: uniffi_algokit_utils_ffi_fn_constructor_abibool_new
lower: ABIBool 0x6000015f0210
clone: ABIBool (Handle(0x6000015f0210))
calling: uniffi_algokit_utils_ffi_fn_constructor_abivalue_from_bytes
success: uniffi_algokit_utils_ffi_fn_constructor_abivalue_from_bytes
DEBUGPRINT[67]: abi_value.rs:39 (before let rust_abi_type =)
Fatal Python error: Segmentation fault

Current thread 0x0000000204abe140 (most recent call first):
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/algokit_utils/algokit_utils_ffi.py", line 295
 in _uniffi_rust_call_with_error
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/algokit_utils/algokit_utils_ffi.py", line 231
7 in from_bytes
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/tests/test_utils.py", line 121 in test_abi_bo
ol_from_bytes
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/py
thon.py", line 157 in pytest_pyfunc_call
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ca
llers.py", line 121 in _multicall
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ma
nager.py", line 120 in _hookexec
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ho
oks.py", line 512 in __call__
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/py
thon.py", line 1671 in runtest
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ru
nner.py", line 178 in pytest_runtest_call
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ca
llers.py", line 121 in _multicall
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ma
nager.py", line 120 in _hookexec
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ho
oks.py", line 512 in __call__
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ru
nner.py", line 246 in <lambda>
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ru
nner.py", line 344 in from_call
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ru
nner.py", line 245 in call_and_report
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ru
nner.py", line 136 in runtestprotocol
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ru
nner.py", line 117 in pytest_runtest_protocol
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ca
llers.py", line 121 in _multicall
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ma
nager.py", line 120 in _hookexec
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ho
oks.py", line 512 in __call__
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ma
in.py", line 367 in pytest_runtestloop
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ca
llers.py", line 121 in _multicall
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ma
nager.py", line 120 in _hookexec
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ho
oks.py", line 512 in __call__
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ma
in.py", line 343 in _main
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ma
in.py", line 289 in wrap_session
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/ma
in.py", line 336 in pytest_cmdline_main
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ca
llers.py", line 121 in _multicall
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ma
nager.py", line 120 in _hookexec
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/pluggy/_ho
oks.py", line 512 in __call__
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/co
nfig/__init__.py", line 175 in main
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/lib/python3.13/site-packages/_pytest/co
nfig/__init__.py", line 201 in console_main
  File "/Users/joe/git/algorandfoundation/algokit-core/packages/python/algokit_utils/.venv/bin/pytest", line 8 in <module>

Extension modules: msgpack._cmsgpack, _cffi_backend, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.pack
ages.chardet.md (total: 5)
zsh: segmentation fault  poetry run pytest -sv

[bendk]

Ahh, I think I finally get what's going on here:

• AbiBool is an object that implements the AbiType trait
• AbiBool::new() is returning a concrete AbiBool type
• You're then passing it to the AbiValue::from_bytes constructor, which expects a dyn AbiType.
• Logically, the Python code should coerce AbiBool -> dyn AbiType, but we don't currently support this
  • can you double check this for me? Calling abi_bool.get_string() fails right?
• Somehow we're still allowing the FFI call to go through, at that point we're calling Arc::from_raw with the wrong pointer type and this will cause a failure sooner or later.

The ideal solution would be to support this type coercion somehow (#2542). In the meantime we should catch this as a type error and prevent the segfault.

[joe-p]

Logically, the Python code should coerce AbiBool -> dyn AbiType, but we don't currently support this

Ah yep that's it! I can confirm this was the problem. Marking the trait as foreign now results in the tests passing.

I'll leave this issue open since I think having a better error is an actionable item for it, but feel free to close and make another issue specifically for the error if you deem fit.

I do have some questions on the implications of with_foreign as a solution for languages other than python, but will use #2542 for that

[bendk]

Marking the trait as foreign now results in the tests passing

I think that's only working because of a weird interaction. Now that there's with_foreign support, it means Python objects can implement the trait and what you're ending up with is Rust making method calls on a trait that's backed by a Python object that calls back into Rust to implement the methods. So each time you make a call you're round-tripping things in an odd way.

I think it should work, it's just an extra FFI round-trip. The other option would be to make an explicit to_abi_type() method to convert between AbiBool and dyn AbiType.

[joe-p]

Yeah that makes sense. For us the extra overhead of the FFI boundary crosses are not critical.

Another use-case of passing foreign traits around where you actually would want the round trip is testing. We create a foreign trait that our object implements on the Rust side and then use that trait in our Rust-based testing harness. This allows us to test the full trip of raising/lowering for each language with a single Rust implementation and passing the FFI-generated object to Rust as a foreign trait

[mhammond]

The other option would be to make an explicit to_abi_type() method to convert between AbiBool and dyn AbiType.

That's what I was thinking too - some methods for explicit ABI "type casts". I haven't thought it through. I'm mildly surprised mypy doesn't get upset here though?

Another use-case of passing foreign traits around where you actually would want the round trip is testing.

I think that works today OK so long as the types all agree. The scenario above is when the types differ even though we know there exists a conversion between them?

[joe-p]

I think that works today OK so long as the types all agree. The scenario above is when the types differ even though we know there exists a conversion between them?

The scenario here is a object that implements a Rust trait. When the trait is foreign it can be passed back to Rust but it will round-trip every FFI call.

I'm mildly surprised mypy doesn't get upset here though?

In this example, on the Python side of things we have AbiBool which is an object that implements the AbiType trait.

The generated python code for AbiBool is

class AbiBool(AbiBoolProtocol, AbiType):

Thus any method taking AbiType can also take a AbiBool