Skip to content

Conversation

colinmarc
Copy link
Contributor

This fixes a very niche build problem: depending on datafusion-python as an rlib, and then attempting to build on macOS.

Because of the presence of an 'include' configuration in Cargo.toml, everything else is excluded - as well as build.rs, which means the necessary pyo3 linker flags aren't set when building.

I verified this fixes my build issue using a git dependency on datafusion-python:

datafusion-python = { git = "https://github.com/apache/datafusion-python", tag = "48.0.0" }

Whereas just using the crates.io version doesn't build:

    error: linking with `cc` failed: exit status: 1
        |
        = note:  "cc" "-Wl,-exported_symbols_list" "-Wl,/var/folders/b5/5n9p78x14yn8z4ky6fxtdp280000gn/T/rustc7RNcRS/list"
      "/var/folders/b5/5n9p78x14yn8z4ky6fxtdp280000gn/T/rustc7RNcRS/symbols.o" "<17 object files omitted>"
      "/Users/colinmarc/dev/bpln/all-events/target/release/deps/{libpyo3_log-bd50c2e7d1a9be99.rlib,libarc_swap-ad5cf4ebc23fa86d.rlib,libpyo3_async_runtimes-eba16526ca814ba0.rlib,libdatafusion_ffi-93bcb82d1826af3c.rlib,libsemver-69712b2f6934f52b.rlib,libasync_ffi-109cfda0a79f6f46.rlib,libdatafusion_proto-98671f6b4c156261.rlib,libdatafusion_proto_common-c828cce4df9cba68.rlib,libpbjson-e8c3fb6b43fe6a5d.rlib,libbase64-aabdebdc40eac9d2.rlib,libprost-492503f16444bcb4.rlib,libabi_stable-7f2a8da44cc2ad72.rlib,libconst_panic-04abeecc901c5835.rlib,librepr_offset-aec98a1518dda497.rlib,libtstr-e05c375dcbbdf609.rlib,libgenerational_arena-b27652eb370a1900.rlib,liblibloading-c9cfdbf7935ce071.rlib,libcrossbeam_channel-b3c1336cf04f651d.rlib,libabi_stable_shared-4a32241e1181b964.rlib,libcore_extensions-6be478146b530258.rlib,libdatafusion-68a5090fb1c58149.rlib,libdatafusion_functions_table-a785450daa220237.rlib,libdatafusion_functions_window-5c8ad3b057064cc3.rlib,libdatafusion_functions_nested-67c018d9521f14b2.rlib,libdatafusion_optimizer-52cbf43a83dc754a.rlib,libdatafusion_catalog_listing-7240cba6b8bb5026.rlib,libdatafusion_datasource_parquet-1d0a14c6642ff3d7.rlib,libdatafusion_physical_optimizer-07e0639703d8adc2.rlib,libdatafusion_datasource_avro-4b527ad125549f8f.rlib,libdatafusion_datasource_json-819f00f9c2f06d50.rlib,libdatafusion_datasource_csv-65d7c49418d40a12.rlib,libdatafusion_sql-c3b0e476dda0b9bd.rlib,libdatafusion_catalog-2958831fff8d6b8e.rlib,libdatafusion_datasource-a3b673b920baa280.rlib,libglob-5ed07e1ef250c612.rlib,libdatafusion_session-2b1ac2b9b5f38399.rlib,libasync_compression-8b5063c331ce847f.rlib,libbzip2-4f06fb5ed9de7d54.rlib,libdatafusion_physical_plan-2c5fe747d192870e.rlib,libdatafusion_common_runtime-5f702cc8d21973d0.rlib,libdatafusion_functions_aggregate-6b7b6f62fc4cf9c4.rlib,libdatafusion_physical_expr-1224453ebb9347b0.rlib,libpetgraph-d52f52b88facb45f.rlib,libfixedbitset-178cc05d890be25d.rlib,libdatafusion_functions-41d93652653ef031.rlib,libhex-b6c5adf57041980f.rlib,libsha2-c9b63e0c039043f3.rlib,libblake3-b47f6144315c4d77.rlib,libconstant_time_eq-42fd34e10c578200.rlib,libarrayvec-f9543f3f5ae864ab.rlib,libarrayref-d31fc091e2d6ee0e.rlib,libblake2-f4bf0046e980e172.rlib,libdatafusion_execution-c55886d381b3d34d.rlib,libtempfile-a1b2a5a4c563dc3b.rlib,libfastrand-36f87d3ad77c6fd5.rlib,librustix-aa2aaf9c3cec78aa.rlib,liberrno-79ccaecae0769ee1.rlib,libdashmap-6146eda158b025d0.rlib,libcrossbeam_utils-93015497c3e214cc.rlib,libdatafusion_expr-387d959328307058.rlib,libdatafusion_functions_aggregate_common-d90b999257ee962c.rlib,libdatafusion_functions_window_common-57329e93b7b40877.rlib,libdatafusion_doc-4d6559444419a12e.rlib,libdatafusion_physical_expr_common-d485d08f09369c57.rlib,libdatafusion_expr_common-5ef60fe07d8ee1b6.rlib,libdatafusion_common-934a95f38743bfaf.rlib,libhashbrown-80c15c43a5999630.rlib,libparquet-6c7cbfd00f9488c6.rlib,libbrotli-7ab64b7a0bc1c6e8.rlib,libbrotli_decompressor-ee61c12d1794f7c3.rlib,liballoc_stdlib-0497de1978d9f033.rlib,liballoc_no_stdlib-19d7b7eef48a3ff3.rlib,libflate2-9cafbdaee45ba358.rlib,libminiz_oxide-5a7d86077f7b9bd8.rlib,libadler2-5fc4e824e4457afb.rlib,liblibz_rs_sys-ed17e8a28580c3a2.rlib,libzlib_rs-d96186838fec8b4a.rlib,libobject_store-6edbb4329d27bd6c.rlib,libhumantime-5e0e4d17e02e6915.rlib,libthiserror-effe5e3ac7db3b8f.rlib,libwalkdir-9576fe72c80a1302.rlib,libsame_file-1fb48a6cc3f84c12.rlib,libparking_lot-1c1bc5ffa45d3ab6.rlib,libparking_lot_core-29fe49a404d7f533.rlib,liblock_api-a8dec283b0b7515c.rlib,libscopeguard-c1c5590a3fd72fb5.rlib,librustls_pemfile-432c5a98bc793fc7.rlib,librand-9516c305f7708ff0.rlib,librand_chacha-de4fd04099850197.rlib,librand_core-c5592f420480d246.rlib,libquick_xml-489731081c9f4e25.rlib,libmd5-ecdd0a59f23205fe.rlib,libitertools-13b9d0655b2429f3.rlib,libeither-20f8714b75d621d9.rlib,libreqwest-5e3084d5cec20c6e.rlib,libserde_urlencoded-5042827945b6a31b.rlib,libhyper_rustls-224c761b353653e2.rlib,libwebpki_roots-7026cf1d449d14b4.rlib,librustls_native_certs-d906b1ed148a3439.rlib,libsecurity_framework-2087cf7c1037269c.rlib,libcore_foundation-6b0653818f6f8a46.rlib,libsecurity_framework_sys-3d967eaa6d5db32a.rlib,libtokio_rustls-283618c8ca1d0dba.rlib,librustls-f7941ad045182529.rlib,libwebpki-94a5de63cdded17c.rlib,libring-e7703330386703aa.rlib,libuntrusted-c90860e50999b2a9.rlib,librustls_pki_types-0041ed224af372d3.rlib,libzeroize-9022ecac0a5e4acb.rlib,libtower_http-30f42cd6330d5ccd.rlib,libiri_string-0347e74622b08586.rlib,libtower-9503d8b3450d06d1.rlib,libtower_layer-f2e5f5dfcab61a2e.rlib,libhyper_util-1528424faa1a1748.rlib,libipnet-59b76d180a7ed2c3.rlib,libtower_service-2b92735bb08efeaa.rlib,libhyper-6e0744af25ac9bb9.rlib,libwant-79f9ecf12d0e2f65.rlib,libtry_lock-d5b41966a8a8def6.rlib,libhttparse-ef4be4395f90b1ab.rlib,libh2-dfc8aeaf325014af.rlib,libtracing-edd79e603cfb2c6e.rlib,libtracing_core-adf72d8e51cfce6a.rlib,libatomic_waker-834bc70eb5a5ee4c.rlib,libtokio_util-072198e6abe236af.rlib,libhttp_body_util-b3bc1a1aaae7b453.rlib,libhttp_body-75975b120e974f59.rlib,liburl-e285a33fec72a732.rlib,libidna-14cae32c7750eb8b.rlib,libutf8_iter-86aeeea8b517fd05.rlib,libidna_adapter-01f5c27feba1aa00.rlib,libicu_properties-de77c15a0df5c0e0.rlib,libicu_properties_data-5ca75e326589e0ed.rlib,libicu_normalizer-26c421f9ad902b44.rlib,libsmallvec-249d3f4ca8754cd8.rlib,libicu_normalizer_data-832c02018fb20fdc.rlib,libicu_collections-d9edb28e1775c80c.rlib,libpotential_utf-caba7d3c0f82f7bd.rlib,libicu_provider-677e08df1f065d0d.rlib,libicu_locale_core-c93769ac58b78da0.rlib,libtinystr-b4c0d3e234080f71.rlib,liblitemap-7434bcab13789da1.rlib,libwriteable-3595c6a4727da580.rlib,libzerovec-4d93715f0ed3cf04.rlib,libzerotrie-20ffe62c782ed7db.rlib,libyoke-2397f33fa66e22e1.rlib,libstable_deref_trait-1ad5d0b738b52bb9.rlib,libzerofrom-68088c4d8a6d3565.rlib,libform_urlencoded-8511452e00f9e778.rlib,libpercent_encoding-30f7b203e5794502.rlib,libhttp-45aadb77363e617b.rlib,libfnv-e07de106c7b4c98c.rlib,libsync_wrapper-068989de3d3e9fb3.rlib,libtokio-fa259681e99ba4f4.rlib,libsocket2-aae2154fbabf005f.rlib,libmio-261f0c46b28be8f2.rlib,libfutures-c5d96c7fcbbc14e1.rlib,libfutures_executor-67bf974f0b56d822.rlib,libfutures_util-1c3fe680cafecb70.rlib,libfutures_io-f4e11ec4606227af.rlib,libslab-4f719ac61e3db7c6.rlib,libfutures_channel-0cd37d2768807004.rlib,libpin_project_lite-d6b4c3abaa6ba590.rlib,libfutures_sink-0c37eddd73a396c5.rlib,libfutures_task-227a24be170cf210.rlib,libpin_utils-7565c1c1b0d23345.rlib,libfutures_core-73cf6b1d709bf5f9.rlib,libthrift-3009cb079a023204.rlib,libordered_float-40d7d306ca8c64ca.rlib,libthreadpool-07469e391275dec2.rlib,libnum_cpus-eb9b30df06252e68.rlib,libinteger_encoding-aaa7155d90d30c4f.rlib,libbyteorder-99912f0ff4e26454.rlib,libapache_avro-3204a2ccbb43cc6c.rlib,librand-c85d250afe322256.rlib,librand_chacha-f0f4cae7e338eae2.rlib,libppv_lite86-97de4abe00feffa7.rlib,librand_core-81fd3c91bb54801c.rlib,libgetrandom-d93181f1ee66e022.rlib,libzstd-fafe94de6cb95623.rlib,libzstd_safe-82265c9d926be9f7.rlib,libzstd_sys-6576136844b57447.rlib,libsnap-8b39be63489ddb57.rlib,libstrum-4fa9c7db083a41ed.rlib,libserde_bytes-a1c48fa0a9358c6f.rlib,libtyped_builder-8773f0b700ae2c31.rlib,libthiserror-b114ffb56e089b69.rlib,libregex_lite-1cd5459a9075a5b9.rlib,libdigest-7d57b25a08b94040.rlib,libsubtle-fba2f0087773609b.rlib,libconst_oid-6fc9dbda5666a297.rlib,libblock_buffer-21f68a0774c20711.rlib,libcrypto_common-9d5b7d24ca0d1cc9.rlib,libgeneric_array-c52cfd15b4c81270.rlib,libtypenum-dbc94d9ac6ecc909.rlib,libuuid-f7ee5ffe9fc05ec2.rlib,libxz2-fc787732f1ebc2b1.rlib,liblzma_sys-bb85373e4479f82a.rlib,libbzip2-d914a72e469559f1.rlib,libbzip2_sys-d2d139929083af2d.rlib,liblibflate-8b21986f86eb0661.rlib,libdary_heap-f4a7bbd38e9aefc5.rlib,libadler32-7c8e6a0369083f50.rlib,liblibflate_lz77-13c7bc3f6a43dba7.rlib,librle_decode_fast-ccfd9a82bdfba950.rlib,libcore2-d3ea4253b3ae26d0.rlib,libbigdecimal-30e928dee9fc5700.rlib,liblibm-82228d2d19a02ab8.rlib,libcrc32fast-f8aca6d2b09ed99b.rlib,libarrow-8a1f7ce8c9dd14f7.rlib,libarrow_row-84eb08d4cd961ffd.rlib,libarrow_pyarrow-5622f52f998232d6.rlib,libarrow_json-4b0a30102174a6de.rlib,libsimdutf8-46b6d81e66194e68.rlib,libindexmap-21d88474d1f1486d.rlib,libarrow_ipc-85d999ef949f9f5a.rlib,liblz4_flex-b12f51b3c2452d9a.rlib,libtwox_hash-1d2f8dbc720fa8f7.rlib,libflatbuffers-16321d00244e270f.rlib,libarrow_csv-05870c3b279a0398.rlib,libcsv-86a9275ba06878ec.rlib,libcsv_core-7faf0ea49beed3df.rlib,libarrow_string-c8f4d5c67fe11223.rlib,libregex-a26b2741ad471a02.rlib,libregex_automata-63be80d6e35fd497.rlib,libaho_corasick-ebe662c7d81718e6.rlib,libregex_syntax-2fd5791d4664d0bb.rlib,libarrow_cast-3ef77d16a9989f53.rlib,libatoi-898e3c475248cd16.rlib,libbase64-93cc24d04722c6b2.rlib,libcomfy_table-b31f844dae7a2715.rlib,libunicode_segmentation-08caeb8119b6a428.rlib,libunicode_width-819d17f485789e82.rlib,liblexical_core-6b29bf7482d4ac1a.rlib,liblexical_write_float-95a6204f3a06f52c.rlib,liblexical_write_integer-d38178122b23ea98.rlib,liblexical_parse_float-b2b676c7cae99aec.rlib,liblexical_parse_integer-bf0edbbc6c845c6c.rlib,liblexical_util-dff9ce4f975beea9.rlib,libstatic_assertions-b5177fa3f151f88a.rlib,libarrow_arith-8f71231d05b60d21.rlib,libarrow_ord-b8bbf587e071d183.rlib,libarrow_select-149bab3643fafe47.rlib,libarrow_array-cac0bf1b95860c27.rlib,libchrono_tz-7976adf5e08a5faf.rlib,libphf-e4bcc5492e1638c4.rlib,libphf_shared-29089568c7a98d73.rlib,libsiphasher-5e073432ba80b0ec.rlib,libahash-76010dfde000bb47.rlib,libgetrandom-a8a2fba13b7c3926.rlib,libzerocopy-93ff1ffbd9c920f9.rlib,libhashbrown-c09e8c50d4e4de7e.rlib,libfoldhash-8add571487c208a9.rlib,libequivalent-eaa0685fbf36c459.rlib,liballocator_api2-bf6b28bb29149b15.rlib,libchrono-16a492c8d6535c05.rlib,libiana_time_zone-9838eca935254022.rlib,libcore_foundation_sys-c7bdeb0be8f778de.rlib,libarrow_data-ad1393f0813db00b.rlib,libarrow_schema-b2763afed1d75869.rlib,libserde_json-d8d7def69aaeab54.rlib,libmemchr-010e50dec96889d4.rlib,libitoa-9897d6f33baa8efe.rlib,libryu-89145247fa865d8b.rlib,libbitflags-0b752ee354c1714f.rlib,libarrow_buffer-df745097521deb5d.rlib,libbytes-e9a5beea09e177f1.rlib,libhalf-1d5a96a0d129616c.rlib,libnum-1191270fef722447.rlib,libnum_iter-864a6772fe1c3728.rlib,libnum_rational-42bf3d6e0977e09b.rlib,libnum_complex-04cb518d1a34e126.rlib,libnum_bigint-f614862af97b5053.rlib,libserde-379e177ca80553ea.rlib,libnum_integer-9ea6283701072edd.rlib,libnum_traits-73dfdd8565e95ecb.rlib,libsqlparser-dae7132ab45c41eb.rlib,librecursive-ebdee60bb2e2bbbb.rlib,libstacker-b6c1c942456734b9.rlib,libpsm-d45da3fd3c987ab9.rlib,liblog-f848965e9d671f27.rlib,libpyo3-90b98f762f1a9d3b.rlib,libcfg_if-34bf03306b1dc143.rlib,libmemoffset-b977bfb2f9030c4a.rlib,libonce_cell-068f7909549c15e0.rlib,libpyo3_ffi-75cd3589555f4be7.rlib,libunindent-47c92a8720c117c0.rlib,libmimalloc-72a66481084f7fdc.rlib,liblibmimalloc_sys-ad4f99036bc921e8.rlib,liblibc-0caa3c72f476e2c6.rlib}.rlib"
      "<sysroot>/lib/rustlib/aarch64-apple-darwin/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,liballoc-*,librustc_std_workspace_core-*,libcore-*,libcompiler_builtins-*}.rlib"
      "-framework" "Security" "-framework" "CoreFoundation" "-liconv" "-lSystem" "-lc" "-lm" "-arch" "arm64" "-mmacosx-version-min=11.0.0" "-L"
      "/Users/colinmarc/dev/bpln/all-events/target/release/build/bzip2-sys-ce2535ad003224bd/out/lib" "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/lzma-sys-e3b54090af14827e/out"
      "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/zstd-sys-89467c99fe98418d/out" "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/ring-6c070c10dae4d85c/out"
      "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/psm-478f69e03f9c498e/out" "-L" "/Users/colinmarc/dev/bpln/all-events/target/release/build/blake3-b9ff590313ad109f/out" "-L"
      "/Users/colinmarc/dev/bpln/all-events/target/release/build/libmimalloc-sys-b64c9c1ed3774578/out" "-o" "/Users/colinmarc/dev/bpln/all-events/target/release/deps/libdatafusion_python-62a9be6a9f738ad3.dylib"
      "-Wl,-dead_strip" "-dynamiclib" "-nodefaultlibs"
        = note: some arguments are omitted. use `--verbose` to show all linker arguments
        = note: ld: warning: object file (/Users/colinmarc/dev/bpln/all-events/target/release/deps/libblake3-b47f6144315c4d77.rlib[5](a1edd97dd51cd48d-blake3_neon.o)) was built for newer 'macOS' version (15.5)
      than being linked (11.0)
                Undefined symbols for architecture arm64:
                  "_PyBaseObject_Type", referenced from:
                      <snip>
                      ...
                ld: symbol(s) not found for architecture arm64
                clang: error: linker command failed with exit code 1 (use -v to see invocation)


      error: could not compile `datafusion-python` (lib) due to 1 previous error
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO MACOSX_DEPLOYMENT_TARGET="11.0" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.12-64bit"
      PYO3_PYTHON="/Users/colinmarc/.cache/uv/builds-v0/.tmppBVRox/bin/python" PYTHON_SYS_EXECUTABLE="/Users/colinmarc/.cache/uv/builds-v0/.tmppBVRox/bin/python" "cargo" "rustc" "--features"
      "pyo3/extension-module" "--features" "python" "--message-format" "json-render-diagnostics" "--manifest-path" "/Users/colinmarc/dev/bpln/all-events/crates/bpln-df-extensions/Cargo.toml" "--release" "--lib"
      "--crate-type" "cdylib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/bauplan_datafusion_extensions.abi3.so"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/Users/colinmarc/.cache/uv/builds-v0/.tmppBVRox/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

Note that you can verify build.rs is missing this way:

curl -fsSL https://crates.io/api/v1/crates/datafusion-python/48.0.0/download -o - | tar tf - | grep build.rs

This fixes a very niche build problem: depending on datafusion-python as
an rlib, and then attempting to build on macOS.

Because of the presence of an 'include' configuration in Cargo.toml,
everything else is excluded - including build.rs, which means the
necessary pyo3 linker flags aren't set.
Copy link
Member

@timsaucer timsaucer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the contribution!

@timsaucer timsaucer merged commit 0806ce5 into apache:main Aug 25, 2025
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants