-
-
Notifications
You must be signed in to change notification settings - Fork 615
fix(pypi): pull fewer wheels with experimental_index_url #3058
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Created this as WIP to signal that I am working on this and to invite feedback. The CI should be mostly happy but there are a few TODOs before we can fully merge and I want to test it a little bit more.
TODO cleanup after this PR is proven to not break things:
|
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
The CI failure is a flake. |
dc50a12
to
855a673
Compare
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
This would be ready to be reviewed, but because it is stacked on two PRs, let's get those out of the way first. |
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.
Stacked on bazel-contrib#3058 Since the parent PR is simplifying the code a great deal, we can remove the unused code and its tests. Hopefully this makes the whole codebase easier to contribute to. There is probably more cleanup that can be done, but since the CI is now green, it's a good start.
Stacked on bazel-contrib#3058 Since the parent PR is simplifying the code a great deal, we can remove the unused code and its tests. Hopefully this makes the whole codebase easier to contribute to. There is probably more cleanup that can be done, but since the CI is now green, it's a good start.
I hit this problem with this change (I don't support windows and use
|
Thank you, I'll take a look later. Marking this as a draft for now. |
@keith, #3070 should fix your issue, I think. The toolchain pulling in |
Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747
Before this PR we would be constructing slightly different environments when the `env_marker_setting` is doing it in the analysis phase and when we are doing it in the repo phase due to how the defaults are handled. In this change we simply reuse the same select statements and add an extra helper that is allowing us to process that. Work towards bazel-contrib#2949 Prep for bazel-contrib#3058
This will be needed when we start selecting wheels entirely in the bzlmod extension evaluation phase (bazel-contrib#3058). This adds a few unit tests to just ensure that we conform to the spec even though the code is very simple. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849
Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747
89e58de
to
891af92
Compare
DO NOT MERGE: stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747
DO NOT MERGE: stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747
Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747
Before this PR we would be constructing slightly different environments when the `env_marker_setting` is doing it in the analysis phase and when we are doing it in the repo phase due to how the defaults are handled. In this change we simply reuse the same select statements and add an extra helper that is allowing us to process that. Work towards bazel-contrib#2949 Prep for bazel-contrib#3058
This will be needed when we start selecting wheels entirely in the bzlmod extension evaluation phase (bazel-contrib#3058). This adds a few unit tests to just ensure that we conform to the spec even though the code is very simple. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849
DO NOT MERGE: stacked on bazel-contrib#3110 This PR only implements the selection algorithm where instead of selecting all wheels that are compatible with the set of target platforms, we select a single wheel that is most specialized for a particular *single* target platform. What is more, compared to the existing algorithm it does not assume a particular list of supported platforms and just fully implements the spec. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849
Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747
891af92
to
0b8596d
Compare
DO NOT MERGE: stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747
DO NOT MERGE: stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747
Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747
Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747
Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747
Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747
0b8596d
to
c757256
Compare
Before this PR the configuration for platforms would be built non-incrementally and was somewhat special, making it harder to override particular attributes of the already configured ones.
Before this we would pull all of the wheels that the user target configuration would be compatible with and that meant that it was not customizable. This also meant that there were a lot of footguns in the configuration where the select statements were not really foolproof. With this PR we select only those sources that need to be for the declared configurations. Freethreaded support is done by defining extra freethreaded platforms using the new builder API (bazel-contrib#3063). This is also changing the default platforms to be only the fully supported platforms. This makes the testing easier and avoids us running into compatibility issues during the roll out. Work towards bazel-contrib#2747 Fixes bazel-contrib#2759 Fixes bazel-contrib#2849
c757256
to
e2d2dd0
Compare
DO NOT MERGE: stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747
Before this PR we would be constructing slightly different environments when the `env_marker_setting` is doing it in the analysis phase and when we are doing it in the repo phase due to how the defaults are handled. In this change we simply reuse the same select statements and add an extra helper that is allowing us to process that. Work towards #2949 Prep for #3058 Co-authored-by: Richard Levasseur <[email protected]>
This will be needed when we start selecting wheels entirely in the bzlmod extension evaluation phase (bazel-contrib#3058). This adds a few unit tests to just ensure that we conform to the spec even though the code is very simple. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849
DO NOT MERGE: stacked on #3112.
Before this we would pull all of the wheels that the user target configuration
would be compatible with and that meant that it was not customizable. This also
meant that there were a lot of footguns in the configuration where the select
statements were not really foolproof.
With this PR we select only those sources that need to be for the declared
configurations.
Freethreaded support should be done by defining extra freethreaded platforms
using the new builder API. It is done as a followup in #3063.
This is also changing the default platforms to be only the fully supported
platforms. This makes the testing easier and avoids us running into
compatibility issues during the rollout.
Work towards #2747
Fixes #2759
Fixes #2849