fix(local_runtime): Improve local_runtime usability in macos / windows #3148

laramiel · 2025-08-06T04:03:42Z

local_runtime fails to handle many variations of python install on windows and MacOS, such as:

LDLIBRARY on MacOS may refer to a file under PYTHONFRAMEWORKPREFIX, not LIBDIR
LDLIBRARY on Windows refers to pythonXY.dll, not the linkable pythonXY.lib
LIBDIR may not be correctly set on Windows.
On windows, interpreter_path needs to be normalized. Other paths also require this.
SHLIB_SUFFIX does not indicate the correct suffix.

For examples, see

See: https://docs.python.org/3/extending/windows.html

In order to resolve this the shared library resolution has been moved into get_local_runtime_info.py, which now does the following:

Constructs a list of paths to search based on LIBDIR, LIBPL, PYTHONFRAMEWORKPREFIX, and the executable directory.
Constructs a list of libraries to search based on INSTSONAME, LDLIBRARY, pythonXY.lib, etc.
Checks to see which files exist, partitioning the result into a list of "dynamic_libraries" and "static_libraries"

On windows and macos, since SHLIB_SUFFIX does not always indicate the filenames needed searching, this has been removed from local_runtime_repo_setup and replaced with an explicit file.

On windows the interpreter_path and other search paths are now normalized ( \ converted to /).

Additional logging added to local_runtime_repo.

Example get_local_runtime_info.py outputs:

rules_python:local_runtime_repo(@@local_python3) INFO: GetPythonInfo result:
  INSTSONAME: Python.framework/Versions/3.12/Python
  LDLIBRARY: Python.framework/Versions/3.12/Python
  LIBDIR: /Library/Frameworks/Python.framework/Versions/3.12/lib
  MULTIARCH: darwin
  PY3LIBRARY: <empty string>
  PYTHONFRAMEWORKPREFIX: /Library/Frameworks
  base_executable: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12
  implementation_name: cpython
  include: /Library/Frameworks/Python.framework/Versions/3.12/include/python3.12
  major: 3
  micro: 10
  minor: 12

Here we see the MacOS framework issue, namely LDLIBRARY is not under LIBDIR, but directly under frameworks.
The new code adds PYTHONFRAMEWORKPREFIX to the search path, so the library is found.

rules_python:local_runtime_repo(@@local_python3)  INFO: GetPythonInfo result:
  INSTSONAME: None
  LDLIBRARY: python313.dll
  LIBDIR: T:\build_temp\home\python_4379bdec28bdff81a567a01c9b8cf10e3856c8c966e4fe53945bedea6338b416\tools\libs
  MULTIARCH: None
  PY3LIBRARY: python313.lib
  base_executable: T:\build_temp\home\python_4379bdec28bdff81a567a01c9b8cf10e3856c8c966e4fe53945bedea6338b416\tools\python.exe
  implementation_name: cpython
  include: T:\build_temp\home\python_4379bdec28bdff81a567a01c9b8cf10e3856c8c966e4fe53945bedea6338b416\tools\Include
  major: 3
  micro: 1
  minor: 13

Here we see that under windows the python313.dll is reported; python313.lib is added via PY3LIBRARY.

Fixes #3055

local_runtime fails to handle many variations of python install on windows and MacOS, such as: * LDLIBRARY on MacOS may refer to a file under PYTHONFRAMEWORKPREFIX, not LIBDIR * LDLIBRARY on Windows refers to pythonXY.dll, not the linkable pythonXY.lib * LIBDIR may not be correctly set on Windows. * On windows, interpreter_path needs to be normalized. Other paths also require this. * SHLIB_SUFFIX does not indicate the correct suffix. For examples, see See: bazel-contrib#3055 See: https://docs.python.org/3/extending/windows.html Example get_local_runtime_info.py outputs: rules_python:local_runtime_repo(@@local_python3) INFO: GetPythonInfo result: INSTSONAME: Python.framework/Versions/3.12/Python LDLIBRARY: Python.framework/Versions/3.12/Python LIBDIR: /Library/Frameworks/Python.framework/Versions/3.12/lib MULTIARCH: darwin PY3LIBRARY: <empty string> PYTHONFRAMEWORKPREFIX: /Library/Frameworks base_executable: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12 implementation_name: cpython include: /Library/Frameworks/Python.framework/Versions/3.12/include/python3.12 major: 3 micro: 10 minor: 12 rules_python:local_runtime_repo(@@local_python3) INFO: GetPythonInfo result: INSTSONAME: None LDLIBRARY: python313.dll LIBDIR: T:\build_temp\home\python_4379bdec28bdff81a567a01c9b8cf10e3856c8c966e4fe53945bedea6338b416\tools\libs MULTIARCH: None PY3LIBRARY: python313.lib base_executable: T:\build_temp\home\python_4379bdec28bdff81a567a01c9b8cf10e3856c8c966e4fe53945bedea6338b416\tools\python.exe implementation_name: cpython include: T:\build_temp\home\python_4379bdec28bdff81a567a01c9b8cf10e3856c8c966e4fe53945bedea6338b416\tools\Include major: 3 micro: 1 minor: 13 On windows and macos, since SHLIB_SUFFIX does not always indicate the filenames needed searching, this has been removed from local_runtime_repo_setup and replaced with an explicit list of files. In addition, target libraries are searched in multiple locations to handle variations in macos and windows file locations. On windows the interpreter_path and other search paths are now normalized ( \ converted to /). On windows the get_local_runtime_info now returns the pythonXY.lib library Additional logging added to local_runtime_repo. :1

…l to get_local_runtime_info.py

…nylinux.

laramiel · 2025-08-07T01:02:25Z

After integrating this some more with tensorstore, I've reworked the library lookup to handle manylinux and other linux installs with no dynamic libraries exposed.

aignas

Consider also referencing the issue in the changelog.

This PR is very exciting, but I gave it a quick glance only since it is reasonably big.

I would love to ask the local_python example also be made to work with bzlmod - all of the improvements should work there too.

aignas · 2025-08-06T07:36:20Z

python/private/local_runtime_repo.bzl

        implementation_name = info["implementation_name"],
        os = "@platforms//os:{}".format(repo_utils.get_platforms_os_name(rctx)),
-    ))
+    )
+    logger.debug("BUILD.bazel\n{}".format(build_bazel))


Do you need to have it? You can always do bazel build <repo>//:BUILD.bazel to see the file. I think this can be removed and the changes reordering the writing of the files could be also reverted for a smaller diff.

I can do that, however this is logged at debug level and given my multi-target ci it's helpful to print it in the log.

python/private/get_local_runtime_info.py

CHANGELOG.md

rickeylev

Oddly, I can edit and commit files via the ui, but can't push to laramiel/main on the command line. Is there something special setup with your repo or branch? I was going to move the example over into local toolchains, but it was a bit hard to refactor that via the ui alone.

rickeylev · 2025-08-08T04:36:29Z

examples/local_python/WORKSPACE

The integration tests for local toolchains are in tests/integration/local_toolchains. It should already have workspace and bzlmod stuff setup. I think the only thing to do is remove the target_compatible_with constraint that makes it only run on linux/mac.

rickeylev · 2025-08-08T04:42:45Z

python/private/get_local_runtime_info.py

+    # on the system - some systems like version suffix, others don't.""
+    #
+    # A typical INSTSONAME is 'libpython3.8.so.1.0' on Linux, or
+    # 'Python.framework/Versions/3.9/Python' on MacOS.


Just verifying: On mac, INSTSONAME might be a relative path to a dir, not the filename?

rickeylev · 2025-08-08T04:44:54Z

python/private/get_local_runtime_info.py

+
+    # Set the prefix and suffix to construct the library name used for linking.
+    if _IS_DARWIN:
+        # SHLIB_SUFFIX may be ".so"; always override on darwin to be ".dynlib"


Why always override? Because dylib is the idiomatic extension? Or for another reason? Whatever the case, please include it as a comment.

rickeylev · 2025-08-08T04:45:44Z

python/private/get_local_runtime_info.py

+        # SHLIB_SUFFIX on windows is ".dll"; however the compiler needs to
+        # link with the ".lib".


Do you have a reference for this? If so, please include a link. If not, oh well.

rickeylev · 2025-08-08T04:48:58Z

python/private/get_local_runtime_info.py

+    config_vars = sysconfig.get_config_vars()
+
+    # VERSION is X.Y in Linux/macOS and XY in Windows.
+    if not config_vars.get("VERSION"):


In what case is VERSION missing? I'd think this would always be present. I known, please comment. If this is just defense, that's fine too; just comment its for defense.

laramiel requested review from rickeylev and aignas as code owners August 6, 2025 04:03

laramiel added 8 commits August 6, 2025 00:18

Correct _expand_incompatible_template

b9cbc67

Update changelog

965a6a9

Formatting + minor cleanups

8b61915

format WORKSPACE

6ac66c9

Move the complex logic to resolve the python shared libraries from bz…

f4c0fed

…l to get_local_runtime_info.py

Fix formatting and method docs

9283a3f

Better handle missing interface_library values. This can happen on ma…

e8997c1

…nylinux.

Format local_runtime_repo_setup.bzl

655ea0f

aignas reviewed Aug 7, 2025

View reviewed changes

laramiel and others added 6 commits August 6, 2025 19:47

reformat get_local_runtime_info.py and main.py

e4eea87

Update CHANGELOG and minor comments.

87d5391

Minor tweaks to library name genration

09bfee9

expand changelog description

dda37d6

Merge branch 'main' into main

d7b2888

name loop var, add non-uv libs finding comment

0df9bf1

rickeylev requested changes Aug 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(local_runtime): Improve local_runtime usability in macos / windows #3148

fix(local_runtime): Improve local_runtime usability in macos / windows #3148

Uh oh!

laramiel commented Aug 6, 2025 •

edited by rickeylev

Loading

Uh oh!

laramiel commented Aug 7, 2025

Uh oh!

aignas left a comment

Uh oh!

aignas Aug 6, 2025

Uh oh!

laramiel Aug 7, 2025

Uh oh!

Uh oh!

Uh oh!

rickeylev left a comment

Uh oh!

rickeylev Aug 8, 2025

Uh oh!

rickeylev Aug 8, 2025

Uh oh!

rickeylev Aug 8, 2025

Uh oh!

rickeylev Aug 8, 2025

Uh oh!

rickeylev Aug 8, 2025

Uh oh!

Uh oh!

		# SHLIB_SUFFIX on windows is ".dll"; however the compiler needs to
		# link with the ".lib".

Uh oh!

fix(local_runtime): Improve local_runtime usability in macos / windows #3148

Are you sure you want to change the base?

fix(local_runtime): Improve local_runtime usability in macos / windows #3148

Uh oh!

Conversation

laramiel commented Aug 6, 2025 • edited by rickeylev Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laramiel commented Aug 7, 2025

Uh oh!

aignas left a comment

Choose a reason for hiding this comment

Uh oh!

aignas Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

laramiel Aug 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rickeylev left a comment

Choose a reason for hiding this comment

Uh oh!

rickeylev Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

laramiel commented Aug 6, 2025 •

edited by rickeylev

Loading