fix: PyPI cross-platform resolves when dependencies have platform markers #2656
+91
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…solving environment in case dependencies were added
savingoyal
reviewed
Oct 24, 2025
saikonen
commented
Oct 28, 2025
metaflow/plugins/pypi/pip.py
Outdated
| res["hash"] = vcs_info["commit_id"] | ||
| return res | ||
|
|
||
| from packaging.requirements import Requirement |
Collaborator
Author
There was a problem hiding this comment.
note implicit dependency on packaging. this is at least limited to only the solve step, but still not desirable
Collaborator
There was a problem hiding this comment.
we can vendor this lib
Contributor
There was a problem hiding this comment.
packaging is already vendored.
Contributor
|
So are you also re-exploring the patching approach? |
Collaborator
Author
|
closing in favour of #2322 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current PyPI implementation can fail to produce a working environment in the case where the resolving machine is of a different architecture than the deployment target. Such a case is when any of the packages requested employs platform markers in their dependencies, as these depending on the value these can be ignored by the machine resolving the environment, but end up being required by the one executing the code.
PyTorch is one such library, where it specifies CUDA dependencies with the following constraints
Resolving the environment on a machine not matching the above, all CUDA dependencies would end up not being shipped in the code package.
During execution though, torch would expect to find these, as the markers match the platform. This would lead to errors during runtime unless CUDA was provided by other means, f.ex. by the Docker base image being used.