-
Notifications
You must be signed in to change notification settings - Fork 2.8k
kind jobs: stop relying on implicit SKIP=[Serial] #35594
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
All jobs using e2e-k8s.sh with PARALLEL=true automatically skipped serial tests even though technically they now could run in the job: https://github.com/kubernetes-sigs/kind/blame/d1eecc46e30cac9d35cd32dc52677ef75ec22e18/hack/ci/e2e-k8s.sh#L226-L234 kubernetes-sigs/kind#4015 is changing that mandatory skip for jobs using LABEL_FILTER because it may be desirable to include serial tests, depending on the job. It's also better to be explicit about it in each job's LABEL_FILTER to avoid confusion and potential mistakes (not running tests that were expected to run). To prepare for that change, jobs get updated based on the following principles: - If a presubmit runs infrequently and is only invoked to test certain aspects, then including serial tests is desirable to get full test coverage of that aspect. Example: pull-kubernetes-e2e-kind-beta-features - If a presubmit runs always, serial tests should be excluded. Example: pull-kubernetes-e2e-kind - Periodic jobs should always run all supported tests, including the serial ones. Example: ci-kubernetes-e2e-kind - Canary jobs match the behavior of the non-canary variant. Example: pull-kubernetes-e2e-kind-canary To make it more obvious where the upcoming e2e-k8s.sh will change test selection, "Includes serial tests for the sake of completeness." comments get added. Those are not true *right now*, but will be once the script is changed.
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: pohly The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/hold Needs to be reviewed together with kubernetes-sigs/kind#4015. This PR then needs to be merged first. |
/assign @BenTheElder @dims: this changes some of your jobs. Do you agree that including serial tests makes sense for them? @aojea: same with your network jobs. You expressed a desired to exclude serial tests from presubmits, so I leaned in that direction for them. |
I think that's done for jobs not using LABEL_FILTER yet. There is a I was worried that that tests like The same cannot be said for other tests like |
This mirrors the presubmit counterparts that were added in kubernetes#35534 and tested in kubernetes/kubernetes#134250. I can help monitor the existing alpha/beta features jobs. Those need to be renamed in testgrid to avoid confusion. Other than that they remain unchanged for now. Potential future work: - Changes related to serial and/or slow jobs. Serial tests are excluded implicitly by e2e-k8s.sh because the jobs enables PARALLEL (kubernetes#35594). Slow jobs are disabled in LABEL_FILTER because that is what the existing periodics did. We might be able to run them because as long as they overlap with other tests there shouldn't be much impact on overall job duration (same applies to presubmits!). Scheduling of slow tests may be relevant (onsi/ginkgo#1599). - Release informing/blocking. The existing jobs are release informing. alpha-beta-features shouldn't be because breaking alpha tests is not something that the release team should have to deal with. Instead, the new jobs should get promoted once they are known to be stable. beta-features can remain release informing, tests for beta features (even if off-by-default) need to be stable. - Decision about "enabled-conformance". The conformance jobs got included because it was suggested on Slack. They run a subset of the tests run by their "enabled" counterparts. It remains to be seen whether having two jobs instead of one really provides a better release signal.
All jobs using e2e-k8s.sh with PARALLEL=true automatically skipped serial tests even though technically they now could run in the job: https://github.com/kubernetes-sigs/kind/blame/d1eecc46e30cac9d35cd32dc52677ef75ec22e18/hack/ci/e2e-k8s.sh#L226-L234
kubernetes-sigs/kind#4015 is changing that mandatory skip for jobs using LABEL_FILTER because it may be desirable to include serial tests, depending on the job. It's also better to be explicit about it in each job's LABEL_FILTER to avoid confusion and potential mistakes (not running tests that were expected to run).
To prepare for that change, jobs get updated based on the following principles:
is debatable). Example: ci-kubernetes-e2e-kind-1-34
To make it more obvious where the upcoming e2e-k8s.sh will change test selection, "Includes serial tests for the sake of completeness." comments get added. Those are not true right now, but will be once the script is changed.