diff --git a/.github/ISSUE_TEMPLATE/cut-release.md b/.github/ISSUE_TEMPLATE/cut-release.md index 7367da64895..e5091d062cd 100644 --- a/.github/ISSUE_TEMPLATE/cut-release.md +++ b/.github/ISSUE_TEMPLATE/cut-release.md @@ -87,8 +87,8 @@ Help? Ring @release-managers on slack! - [ ] Notify #release-management on Slack: - [ ] Update [`schedule.yaml` file](https://github.com/kubernetes/website/blob/main/data/releases/schedule.yaml) with the latest release using [`schedule-builder`](https://github.com/kubernetes/release/tree/master/cmd/schedule-builder) (_patch releases only_) - [ ] Collect metrics and add them to the `Release steps` table below through `krel history --branch release-1.xx --date-from yyyy-mm-dd` (current date) - - + + ## Release Tools Version diff --git a/.github/ISSUE_TEMPLATE/post-release-branch-creation.md b/.github/ISSUE_TEMPLATE/post-release-branch-creation.md new file mode 100644 index 00000000000..7ed6642b016 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/post-release-branch-creation.md @@ -0,0 +1,78 @@ +--- +name: Post Release Branch Creation Tasks +about: Tasks to perform after the rc.0 is cut and the upcoming release branch is created +title: Post Release Branch Creation Tasks for v1.x.y-rc.0 +labels: sig/release, area/release-eng +--- + +## As a follow up on: + +## Tasks + + + +- [ ] Create a thread in #release-management: +- [ ] (optional) Remove jobs for EOL versions, **only** if agreed upon with Release Managers + +- [ ] Update [`milestone_applier` rules](https://github.com/kubernetes/test-infra/blob/master/config/prow/plugins.yaml) +- [ ] Update [`kubekins-e2e-v2/variants.yaml`](https://github.com/kubernetes/test-infra/blob/master/images/kubekins-e2e-v2/variants.yaml) with the new version config +- [ ] Rotate configuration of release branch jobs in kubernetes/test-infra in particular `releng/test_config.yaml` for the upcoming release +- [ ] Run test generation script, configure the new release dashboards and send a PR with both tests and dashboards config +- [ ] Add a new variant for the `kube-cross` image (`kubernetes/release` repository) and ensure the image is built and pushed properly +- [ ] Add new variants for `k8s-cloud-builder` and `k8s-ci-builder` images (`kubernetes/release` repository) and ensure images are built and pushed properly +- [ ] Update references in `kubernetes/kubernetes` with the new kube-cross image (only after all images are pushed/promoted) +- [ ] Update publishing-bot rules to include the new release branch +- [ ] Ensure that a new [performance tests](https://github.com/kubernetes/perf-tests/) branch is created by SIG Scalability maintainers +- [ ] Inform stakeholders about the post branch creation tasks being completed +- [ ] Monitor the new release dashboard with the Release Signal Lead for at least 48 hours - mainly for missing or misconfigured jobs + +## Action Items + + + +## Open Questions + + + +/milestone +/assign +/cc @kubernetes/release-managers @kubernetes/release-team-release-signal +/priority important-soon +/kind documentation diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index d282f6948b2..4a357ac1981 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -1,5 +1,38 @@ # Cutting a Kubernetes release + +- [Cutting a Kubernetes release](#cutting-a-kubernetes-release) + - [Prerequisites](#prerequisites) + - [Green Release Signal (pre-releases only)](#green-release-signal-pre-releases-only) + - [Access to GCP](#access-to-gcp) + - [Install latest software (every time)](#install-latest-software-every-time) + - [Download or update `Go` to the latest available stable version:](#download-or-update-go-to-the-latest-available-stable-version) + - [Download or update the `gcloud` CLI to interact with GCP.](#download-or-update-the-gcloud-cli-to-interact-with-gcp) + - [Download or update `krel`](#download-or-update-krel) + - [Download or update the latest `kpromo` tool](#download-or-update-the-latest-kpromo-tool) + - [Download schedule-builder](#download-schedule-builder) + - [Configure GitHub Personal Access Token](#configure-github-personal-access-token) + - [1. Release cut issue](#1-release-cut-issue) + - [2. Create a thread on the `#release-management` Slack channel](#2-create-a-thread-on-the-release-management-slack-channel) + - [3. Generate testgrid screenshots](#3-generate-testgrid-screenshots) + - [4. Check publishing-bot status](#4-check-publishing-bot-status) + - [5. Mock stage and Mock release](#5-mock-stage-and-mock-release) + - [6. No-mock stage](#6-no-mock-stage) + - [7. Image promotion](#7-image-promotion) + - [Merge promo PR](#merge-promo-pr) + - [Wait on image promotion job](#wait-on-image-promotion-job) + - [8. No-mock release](#8-no-mock-release) + - [9. Notify public dev Google group mailinglist](#9-notify-public-dev-google-group-mailinglist) + - [Manually create release HTML announcements](#manually-create-release-html-announcements) + - [Legacy Sendgrid method:](#legacy-sendgrid-method) + - [10. Post release tasks](#10-post-release-tasks) + - [\[RC.0 only\] Considerations and post branch creation release tasks](#rc0-only-considerations-and-post-branch-creation-release-tasks) + - [Next Release Branch Creation](#next-release-branch-creation) + - [Post branch creation release tasks](#post-branch-creation-release-tasks) + - [\[Stable only\] Code Thaw](#stable-only-code-thaw) + - [\[Patch only\] Update schedule on k/website](#patch-only-update-schedule-on-kwebsite) + - [Cleanup](#cleanup) + A step by step guide for cutting Kubernetes patch releases. At a high-level: - Maintain GitHub release cut issue @@ -15,13 +48,17 @@ A step by step guide for cutting Kubernetes patch releases. At a high-level: ## Prerequisites -### Green Release Signal +### Green Release Signal (pre-releases only) On the same day of the release, a green signal must've been given in the #release-management Slack channel. If in doubt, double check with the current Release Signal Team Lead. You can find the complete list of release signal team members at this link (substitute `1.xx` with the current release version): `https://github.com/kubernetes/sig-release/blob/master/releases/release-1.xx/release-team.md` +> [!NOTE] +Ensure that there are no patch releases in progress, coordinating with @release-managers. +These are typically scheduled on different days of the week, so there is usually no need to plan around them, but since they can sometimes overlap with other release activities, it's good to double-check. + ### Access to GCP You must be a member of [k8s-infra-release-editors](https://github.com/kubernetes/k8s.io/blob/main/groups/sig-release/groups.yaml) on GitHub. @@ -117,10 +154,18 @@ krel version #### Download or update the latest `kpromo` tool -Run the following command ([source](https://github.com/kubernetes-sigs/promo-tools/blob/main/docs/promotion-pull-requests.md#preparing-environment)): +Run the following command ([source](https://github.com/kubernetes-sigs/promo-tools/blob/main/docs/promotion-pull-requests.md#preparing-environment)) to get the latest release of `kpromo`: + +``` +go install sigs.k8s.io/promo-tools/v4/cmd/kpromo@main +``` + +or to build the latest version directly from a target branch: ``` -go install sigs.k8s.io/promo-tools/v4/cmd/kpromo@latest +git clone https://github.com/kubernetes-sigs/promo-tools +git pull origin +make kpromo ``` Validate with: @@ -248,6 +293,9 @@ If the issue is open you must stop the release process and inform #release-manag ## 5. Mock stage and Mock release +> [!WARNING] +Before cutting `alpha.1` ideally some days before, ensure that @release-managers have performed the propedeutic tasks for the alpha cut (e.g. setting up the new OBS project) + Mock stages and mock releases are non-destructive and can be reran upon failure. To begin the mock stage, run the following `krel stage` command (replace the stage with the appropriate "type"). Run the following command from within the release repo directory. @@ -477,12 +525,38 @@ krel history --branch release-1.33 --date-from 2025-04-23 ## 10. Post release tasks -### [RC.0 only] Post rc.0 release tasks +### [RC.0 only] Considerations and post branch creation release tasks + +Remember that before launching the `nomock release` command for an rc.0, you need to ensure that the image promo job for the next alpha.0 has been completed successfully. -See [here](post-rc0-release-tasks.md) for the complete list of post rc.0 release tasks. +#### Next Release Branch Creation + +> [!IMPORTANT] +> The new release branch is created in the `nomock` staging phase and pushed to the repository during the `nomock` release phase of an rc.0 cut. + +During a `rc.0` release our release tooling creates a new release branch named `release-X.Y`, where `X.Y.0` is the version of the upcoming release. +Additionally, the `rc.0` release automatically triggers an `alpha.0` build for the subsequent release (e.g. for `v1.34.0-rc.0`, `v1.35.0-alpha.0` is created automatically). + +Behind the scenes `krel` is executing a `git branch create` command and `git push`. + +At the same time Prow’s [`branchprotector`](https://git.k8s.io/test-infra/prow/cmd/branchprotector/README.md) runs every hour at 54 minutes past the hour and automatically adds [branch protection](https://help.github.com/articles/about-protected-branches/) to any new branch in the `kubernetes/kubernetes` repo, including the newly created one. +No need to manually create the branch protection rule. + +However, it is important to ensure that the branch is protected. We had cases where the branch was not protected and this was noticed very late. + +> [!NOTE] +This means that the staging step will take about twice as long, as it will stage both versions `vX.Y.0-rc.0` and `vX.{Y+1}.0-alpha.0`. +The release step will also be extended, but not substantially longer in time. + +#### Post branch creation release tasks + +See [here](post-rc0-release-tasks.md) for the complete list of post branch creation release tasks. Such list resides in a different document to mainain this one in a bite-sized SRE style format. +> [!WARNING] +You will not be able to cut an rc.1 or any other cut against the new branch until the post branch creation tasks (post rc.0) are complete. + ### [Stable only] Code Thaw Code thaw means you need to lift the code freeze, [here](https://github.com/kubernetes/sig-release/blob/master/release-engineering/role-handbooks/branch-manager.md#code-thaw) diff --git a/release-engineering/handbooks/post-release-branch-creation.md b/release-engineering/handbooks/post-release-branch-creation.md new file mode 100644 index 00000000000..e96812505a6 --- /dev/null +++ b/release-engineering/handbooks/post-release-branch-creation.md @@ -0,0 +1,398 @@ +# Post release branch creation + + +- [Post release branch creation](#post-release-branch-creation) + - [Checklist](#checklist) + - [Remove EOL version jobs from test-infra (optional)](#remove-eol-version-jobs-from-test-infra-optional) + - [Update milestone applier rules and check milestone requirements](#update-milestone-applier-rules-and-check-milestone-requirements) + - [Update Kubekins-e2e v2 variants](#update-kubekins-e2e-v2-variants) + - [Update release branch jobs in kubernetes/test-infra for the new release and create the dashboards](#update-release-branch-jobs-in-kubernetestest-infra-for-the-new-release-and-create-the-dashboards) + - [Configure the release dashboards](#configure-the-release-dashboards) + - [Run test generation script](#run-test-generation-script) + - [Submit the PR for release branch jobs and dashboards in kubernetes/test-infra](#submit-the-pr-for-release-branch-jobs-and-dashboards-in-kubernetestest-infra) + - [Add new variant for kube-cross image](#add-new-variant-for-kube-cross-image) + - [Update k8s-cloud-builder and k8s-ci-builder](#update-k8s-cloud-builder-and-k8s-ci-builder) + - [Update `kubernetes/kubernetes` references for the `kube-cross` image](#update-kuberneteskubernetes-references-for-the-kube-cross-image) + - [Update publishing bot rules](#update-publishing-bot-rules) + - [Create Performance Tests Branch](#create-performance-tests-branch) + - [Check and eventually improve release scripts (optional)](#check-and-eventually-improve-release-scripts-optional) + - [Additional Resources](#additional-resources) + - [Notes](#notes) + +This document details the tasks that need to be executed after cutting a Kubernetes rc.0 release. These tasks ensure proper configuration for the new release branch and testing infrastructure. +They must be executed after the `nomock release` is completed, and the release branch is created, as stated in the [branch creation chapter](k8s-release-cut.md#next-release-branch-creation) of the release cut handbook. + +PR can be created beforehand (and this is recommended in order to get reviews in a timely manner) but you got to remember to put a `/hold` on all the PRs, they have to be lifted only once the `nomock` release phase is done and the branch is created. + +> [!WARNING] +It is essential to follow these steps to maintain the integrity of the release process and ensure that all necessary components are updated accordingly, the examples provided are for illustrative purposes and need to be adapted to the specific release version, so please replace `1.30`, `1.33` and `1.34` with the actual version(s) you are deprecating, currently working on or preparing grounds for. + +## Checklist + +Open a new issue using [this template](https://github.com/kubernetes/sig-release/issues/new?template=post-release-branch-creation.md). + +### Remove EOL version jobs from test-infra (optional) + +Consider removing jobs for versions that are going EOL. This is sometimes done before adding new jobs to avoid overwhelming the Prow cluster. + +Removing EOL jobs involves different steps, including deleting old configs from `config/jobs/kubernetes/sig-release/release-branch-jobs/` e.g. 1.29 EOL jobs removed in [this PR](https://github.com/kubernetes/test-infra/pull/34672). +There could also be some other jobs living outside `config/jobs/kubernetes/sig-release/release-branch-jobs` that also needs to be removed. + +You should also remove the unused EOL jobs from the `kubekins-e2e-v2/variants.yaml` file ([here](https://github.com/kubernetes/test-infra/blob/master/images/kubekins-e2e-v2/variants.yaml)), for example: + +```bash +# Example PR: https://github.com/kubernetes/test-infra/pull/34674 +# Changes would involve removing the EOL version from: images/kubekins-e2e-v2/variants.yaml + +# Diff example: + +- '1.30': +- CONFIG: '1.30' +- GO_VERSION: 1.23.6 +- K8S_RELEASE: latest-1.30 +- BAZEL_VERSION: 3.4.1 +``` + +> [!NOTE] +This step may alternatively be performed as part of a patch release process. It's mandatory to consult with the release engineering team regarding the timing of this step. + +### Update milestone applier rules and check milestone requirements + +Create a PR to update the milestone applier rules to include the new version and remove the oldest version. + +> [!WARNING] +Only remove the oldest version if it is already EOL and the release branch jobs have been already removed. + +```yaml +# Example PR: https://github.com/kubernetes/test-infra/pull/34650 +# Edit this file: config/prow/plugins.yaml + +# Add the new version and remove the oldest version in the kubernetes/kubernetes section - if the oldest version is EOL: +milestone_applier: + kubernetes/kubernetes: + master: v1.33 + release-1.33: v1.33 + release-1.32: v1.32 + ... + # remove oldest version (e.g., release-1.29: v1.29) +``` + +> [!NOTE] +Look out for the code freeze config and ensure excluded and included branches include the newly created release branch `release-1.xy`. + +### Update Kubekins-e2e v2 variants + +Create a PR to update the `kubekins-e2e-v2/variants.yaml` file with the new version's configuration. +File can be found [here](https://github.com/kubernetes/test-infra/blob/master/images/kubekins-e2e-v2/variants.yaml). + +> [!WARNING] +Remember to use the appropriate (updated) Go version for the release, coordinate with #release-management and @release-managers to ensure the correct version is used. + +```yaml +# Example PR: https://github.com/kubernetes/test-infra/pull/34651 +# Edit this file: images/kubekins-e2e-v2/variants.yaml + +# Add the new version configuration as follows: +variants: + '1.34': + CONFIG: '1.34' + GO_VERSION: 1.24.5 # Use appropriate Go version for the release + K8S_RELEASE: latest-1.34 + BAZEL_VERSION: 3.4.1 +``` + +Before proceding with the next step, wait for the `post-test-infra-push-kubekins-e2e` postsubmit to finish. You can check the status on [the Prow Status page](https://prow.k8s.io/?job=post-test-infra-push-kubekins-e2e). + +### Update release branch jobs in kubernetes/test-infra for the new release and create the dashboards + +> [!CAUTION] +Follow the guidelines below very carefully during the update process. + +- Do not remove old jobs while adding new jobs in this phase, just do not. Handle it before or after the post branch creation tasks, or let release engineering take care of this. +- Do not segregate PRs, just separate auto-generated files from manually updated ones in two (or more) clearly documented commits. +- Be super careful about `releng/test_config.yaml` epecially when commenting out `stable4` +- Do not hesitate to remove broken jobs, but let the interested SIG(s) know about it so they can re-add it. + +Updating the release branch jobs in `kubernetes/test-infra` for the new release version involves some steps, it is unpredictable work and requires manual intervention as not every job is generated. +Some of them are added manually and live outside of the generated job tree. + +First of all you need to modify the `releng/test_config.yaml` file ([here](https://github.com/kubernetes/test-infra/blob/master/releng/test_config.yaml)) to: +- Update version references +- Rotate stable job configurations sequentially (n -> n+1) - _note: you should put config of stable1 to stable2, then repeat it until you get to the end._ +- Add new version as beta +- Update configuration for all test suites + +Just shift "args" but not touch interval, sigowners or any other field. +Remember that args are bound to a set of tests for a specific release, while intervals and everything else is bound to a release "status": stable1/2/3/4 or beta. An example of how to correctly rotate the jobs can be found [here](https://github.com/kubernetes/test-infra/pull/34668/commits/819c9d253ab873aff6626a5eaf7635560f7b769e). + +Keep in mind that jobs order might be different, e.g. the first stable1 job and and the first stable2 job might not be the same type of the job, so remember to pay particular attention where you're copying from and to. + +> [!WARNING] +Consider what has been pointed out in [this issue](https://github.com/kubernetes/test-infra/issues/34675). +When updating to new release image variants, you’ll need to update the jobs to use these new images. However, keep in mind that you can only update a job after the new image tag is available. If you attempt to update before the new image exists, the job may fail. +This is related to how the tags are managed and you should consider using `latest` images instead or you will find yourself opening PRs like [this](https://github.com/kubernetes/test-infra/pull/35288) to fix the image tags for some jobs. + +Remember to update all configs before running the [generation script](#run-test-generation-script) for the upcoming release branch jobs and to verify that the jobs are generated correctly. + +#### Configure the release dashboards + +After configuring the jobs you can now configure the [release dashboards](https://github.com/kubernetes/test-infra/blob/master/config/testgrids/kubernetes/sig-release/config.yaml), example [commit](https://github.com/kubernetes/test-infra/commit/31fb8f2b5c4458af675e37765dfebd128da19971), remembering to: + +- Remove the deprecated release sig-release-1.xx-{blocking,informing} dashboards +- Add the new dashboards for the current release e.g., sig-release-1.xx-{blocking,informing} + +> [!NOTE] +Comparing the new jobs with previous version(s) might help to identify any missing jobs or misconfigurations. + +#### Run test generation script + +After updating the configurations you can run the following command from the root of your `test-infra` fork to generate the updated jobs configurations, remember to use the correct architecture and OS for your environment, e.g., `GOARCH=arm64 GOOS=darwin` for macOS on ARM: + +```bash +GOARCH=arm64 GOOS=darwin make -C releng prepare-release-branch +``` + +Breaking down what the above command does: + +- Navigates to the releng directory (-C releng) +- Executes the `prepare-release-branch target` from the `Makefile` in that directory +- Ensures Python 3 requirements are installed +- Executes the `run.sh` script in the releng directory, passing `prepare-release-branch` as an argument +- Sets error handling (errexit, nounset, pipefail) +- Determines the repository root directory +- Sources Go setup script `setup-go.sh` +- Builds the binary and stores them in the `_bin` directory +- Runs `config-rotator` +- Runs `config-forker` +- Runs the Python script `prepare_release_branch.py` inside a Python container +- Passes the built tools and another Python script `generate_tests.py` as arguments to the main script + +#### Submit the PR for release branch jobs and dashboards in kubernetes/test-infra + +You can finally issue a new PR as [this example](https://github.com/kubernetes/test-infra/pull/34668/files) one. + +After this PR is merged, you will be pairing with the Release Signal lead, checking that the new dashboards are working by visiting the TestGrid pages for the new dashboards and verifying that all the necessary jobs show up correctly. This task should be considered completed within 48 hours after the dashboards are created. + +Additionally update/fix the following scripts, if you've found any issues with or a way to improve them (they are fragile): +- `hack/run-in-python-container.sh` +- `releng/run.sh` +- `releng/prepare_release_branch.py` + +### Add new variant for kube-cross image + +> [!Note] +This work on the kube-cross image could have been done as part of the Golang bumps, but it is worth mentioning here as it is a common step. + +Before updating the builder images, you need to update the kube-cross image which they depend on: + +1. Create a PR to update the kube-cross variants.yaml file: + - Update the [dependencies.yaml](https://github.com/kubernetes/release/blob/master/dependencies.yaml) file + - Edit [images/build/cross/variants.yaml](https://github.com/kubernetes/release/blob/master/images/build/cross/variants.yaml) + - Add a new variant with the appropriate Go version for the current release + + + Example PR: https://github.com/kubernetes/release/pull/3870/files + + You also have to edit the dependencies.yaml to update "Kubernetes version (next candidate.0)" to the upcoming minor. + +> [!WARNING] +Bumping the stable version to the n+1 version of Kubernetes is done only when the official release is out. + +```yaml + - name: "Kubernetes version (stable.0)" # Update after the stable marker has been updated to stable.0 + version: v1.34.0 + refPaths: + - path: images/build/cross/Makefile + match: KUBERNETES_VERSION\ \?=\ v((([0-9]+)\.([0-9]+)\.([0-9]+)(?:-([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?)(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?) + - path: images/build/cross/variants.yaml + match: "KUBERNETES_VERSION: 'v((([0-9]+)\\.([0-9]+)\\.([0-9]+)(?:-([0-9a-zA-Z-]+(?:\\.[0-9a-zA-Z-]+)*))?)(?:\\+([0-9a-zA-Z-]+(?:\\.[0-9a-zA-Z-]+)*))?)'" + + # Update after the stable marker has been updated to stable.0 + - name: "Kubernetes version (next candidate.0)" + version: v1.35.0 + refPaths: + - path: images/build/cross/variants.yaml + match: "KUBERNETES_VERSION: 'v((([0-9]+)\\.([0-9]+)\\.([0-9]+)(?:-([0-9a-zA-Z-]+(?:\\.[0-9a-zA-Z-]+)*))?)(?:\\+([0-9a-zA-Z-]+(?:\\.[0-9a-zA-Z-]+)*))?)'" +``` + +2. Wait for the Prow job to complete: + - Monitor the [post-release-push-image-kube-cross](https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-release-push-image-kube-cross) job + +3. Create an image promotion PR: + - This promotes the image from staging to production + - Example PR: https://github.com/kubernetes/k8s.io/pull/8307/files + +### Update k8s-cloud-builder and k8s-ci-builder + +> [!NOTE] +Part of this work on k8s-cloud-builder and k8s-ci-builder could have been done as part of the Golang bumps, but it is worth mentioning here as it is a common step. +In any case you should always update the k8s-cloud-builder and k8s-ci-builder images to use the new kube-cross image. + +Once the updated kube-cross image is available: + +1. Update the k8s-cloud-builder variants.yaml file: + - Update dependencies.yaml + - Edit [images/k8s-cloud-builder/variants.yaml](https://github.com/kubernetes/release/blob/master/images/k8s-cloud-builder/variants.yaml) + - Add a new variant that references the new kube-cross image + + ```yaml + v1.XX-cross1.XX.Y-Z: # Example addition to variants.yaml + CONFIG: 'cross1.XX' + KUBE_CROSS_VERSION: 'v1.XX.y-go1.XX.Y-Z' # Match with the kube-cross image you created + ``` + +The k8s-ci-builder needs to be updated in a similar fashion: + +2. Update the k8s-ci-builder variants.yaml file: + - Update dependencies.yaml + - Edit [images/releng/k8s-ci-builder/variants.yaml](https://github.com/kubernetes/release/blob/master/images/releng/k8s-ci-builder/variants.yaml) + - Add a new variant with the appropriate Go version that matches the release + + ```yaml + '1.34': + CONFIG: '1.34' + GO_VERSION: '1.24.5' + GO_VERSION_TOOLING: '1.24.5' + OS_CODENAME: 'bullseye' + ``` + +3. Submit your PR and wait for review and merge + - Example PR (that also included the k8s-ci-builder Image update): https://github.com/kubernetes/release/pull/4065/files + +4. Monitor the build job in Prow: + - After merge, the image will be automatically built + +After the PR is merged, make sure the relevant ProwJobs are green. + +You can verify the images were built successfully by either pulling the images or using `crane` to check their digest. + +Or you can use `gcloud` to list the tags of the images: + + ```bash + # Check the kube-cross image + gcloud container images list-tags gcr.io/k8s-staging-build-image/kube-cross + + # Check the k8s-cloud-builder image + gcloud container images list-tags gcr.io/k8s-staging-releng/k8s-cloud-builder + + # Check the k8s-ci-builder image + gcloud container images list-tags gcr.io/k8s-staging-releng/k8s-ci-builder + ``` + +### Update `kubernetes/kubernetes` references for the `kube-cross` image + +Once the new builder images are available: + +Create a PR in kubernetes/kubernetes to reference the new kube-cross image in `build/build-image/cross/VERSION` ([here](https://github.com/kubernetes/kubernetes/blob/master/build/build-image/cross/VERSION)): + - Example PR: https://github.com/kubernetes/kubernetes/pull/132897/files + +### Update publishing bot rules + +The Kubernetes Publishing Bot is responsible for: +- ensuring that the master and release branches in the staging repositories are in-sync with the appropriate branches in `kubernetes/kubernetes` +- creating tags in the staging repositories for each Kubernetes release + +Update the publishing bot rules to include the new version. Use the update-rules CLI tool to generate the necessary changes as described [here](https://github.com/kubernetes/publishing-bot/blob/master/cmd/update-rules/README.md). + +You can install the tool on Linux with a simple `make build` command, similarly on MacOS by specifying your OS `GOOS=darwin make build`. +By default the binary will be located in `_output/update-rules`. This is the preferred way to run the tool. + +You can also use the Docker image to run the tool, which is useful if you don't want to build it locally (but you can also do that with make `make build-image`). The live Docker image is available at `gcr.io/k8s-staging-publishing-bot/publishing-bot:latest` and you can invoke the containerized tool as follows: + +```bash +docker run -t gcr.io/k8s-staging-publishing-bot/publishing-bot:latest /update-rules -branch release-1.34 -go 1.24.5 -rules ~/kubernetes/staging/publishing/rules.yaml -o /tmp/rules.yaml +``` + +It's recommended to set the release branch, the Go version and the current publishing rules from kubernetes/kubernetes master branch as environment variables before running the command: + +```bash + # set release branch + export K8S_REL_BRANCH=release-1.34 + # https://github.com/kubernetes/release/blob/master/images/build/cross/variants.yaml#L7 + export GO_VERSION=1.24.5 + export CURRENT_K8S_MASTER_RULES_FILE=/kubernetes/staging/publishing/rules.yaml +``` + +> [!WARNING] +Remember to use the appropriate (updated) Go version for the release, coordinate with #release-management or @release-managers to ensure the correct version is used. + +Basically you need to run the following command after having the `update-rules` tool installed. + +```bash +_output/update-rules -branch ${K8S_REL_BRANCH} -go ${GO_VERSION} -rules ${CURRENT_K8S_MASTER_RULES_FILE} -o /tmp/rules.yaml +``` + +Or, if you ever need to manually edit the rules file, you can do so, as follows: + +```yaml +# Example PR: https://github.com/kubernetes/kubernetes/pull/131250 +# Edit this file: staging/publishing/rules.yaml + +# Repeat this new section for each repository following this pattern: +- name: release-1.34 + go: 1.24.5 # Remember to use the appropriate Go version for the release + dependencies: + - repository: apimachinery + branch: release-1.34 + - repository: api + branch: release-1.34 + source: + branch: release-1.34 + dirs: + - staging/src/k8s.io/[repository-name] +``` + +> [!WARNING] +You should never really have to manually do this. If you have problems with the update-rules tool, those should be addressed as soon as possible. + +Once the rules are updated you can submit a PR similar to [this one](https://github.com/kubernetes/kubernetes/pull/131250), to the publishing-bot repository. + +### Create Performance Tests Branch + +Ensure a performance tests branch is created for the new version: + +```bash +# Example: https://github.com/kubernetes/perf-tests/issues/3290 + +# A maintainer from SIG Scalability should create: +https://github.com/kubernetes/perf-tests/tree/release-1.xx + +# Example for 1.34: +https://github.com/kubernetes/perf-tests/tree/release-1.34 + +# Verify the branch is working in CI: +https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-kubemark-500-gce-1-xx + +# Example for 1.34: +https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-kubemark-500-gce-1-34 +``` + +Reach out to SIG Scalability to ensure a new branch is cut in the [sigs.k8s.io/perf-tests](https://github.com/kubernetes/perf-tests/) repo. + +### Check and eventually improve release scripts (optional) + +If needed, fix error messages or anomalies found in any of the release scripts you've run during the post-rc.0 tasks. + +```bash +# Example: Fixing line 307 of this script: +# http://github.com/kubernetes/release/blob/9a9572f7c48f637de8499a201fb8e3ff52f8d4ba/pkg/gcp/gcb/gcb.go#L307 + +# This would help avoid having a "release job" message for nomock stage commands +``` + +Generally speaking, update scripts and documentation as needed to ensure they are up-to-date and reflect the current release process. + +## Additional Resources + +- [Release Manager Handbook](https://github.com/kubernetes/sig-release/blob/master/release-engineering/role-handbooks/branch-manager.md) +- [Example 1.34.0-rc.0 Release Cut Issue](https://github.com/kubernetes/sig-release/issues/2824) +- [Example of post branch creation tasks issue for 1.34.0-rc.0](https://github.com/kubernetes/sig-release/issues/2826) _this also contains an example of each PR linkedin in the body of the issue_ +- [Slack Discussion Thread for 1.33.0-rc.0](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1744125003875769) - _do not rely on the Slack thread being long lived, if it got archived or the channel got deleted, you should just rely on the docs and the PRs linked in this document_ + +## Notes + +- The order of operations is important. Generally, update configuration files first, then generate files, check them and push on your fork before opening a PR. +- Some tasks may be release-specific. Always check with the rest of release engineering and inform them if you're unsure about any step. Also remember to keep the Release Team informed as they are responsible for the success of the release as a whole. +- After completing these tasks, verify that CI jobs are running properly for the new release branch and that the dashboards reflect the new version. + +--- diff --git a/release-engineering/role-handbooks/branch-manager.md b/release-engineering/role-handbooks/branch-manager.md index 078bd9b8e53..21a4d40c26b 100644 --- a/release-engineering/role-handbooks/branch-manager.md +++ b/release-engineering/role-handbooks/branch-manager.md @@ -31,18 +31,9 @@ - [Security fixes](#security-fixes) - [Announcing Security Fixes](#announcing-security-fixes) - [Release Validation](#release-validation) - - [Post-release Activities](#post-release-activities) - - [Update kubekins-e2e variants](#update-kubekins-e2e-variants) - - [Cut next alpha](#cut-next-alpha) - [Branch Management](#branch-management) - - [Branch Creation](#release-branch-creation) - - [Post release branch creation tasks](#post-release-branch-creation-tasks) - - [Update test-infra configurations](#update-test-infra-configurations) - - [Update milestone appliers](#update-milestone-appliers) - - [Update milestone requirements](#update-milestone-requirements) - - [Update e2e variants](#update-e2e-variants) - - [Generate release branch jobs](#generate-release-branch-jobs) - - [Update publishing-bot rules](#update-publishing-bot-rules) + - [Release Branch Creation](#release-branch-creation) + - [Post new release branch creation tasks](#post-new-release-branch-creation-tasks) - [Configure Merge Automation](#configure-merge-automation) - [Tide](#tide) - [Code Freeze](#code-freeze) @@ -503,173 +494,17 @@ The following are some ways to determine if the release process was successful: 3. CHANGELOG-X.Y.md is automatically loaded into the kubernetes/kubernetes repo: [https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md) - -### Post-release Activities - -> [!IMPORTANT] -> These activities are needed after `rc.0` releases only - -#### Update kubekins-e2e variants - -Set the `K8S_RELEASE` marker for the current release variant to `stable-x.y` which was currently `latest-x.y` in the [`variants.yaml` file for `kubekins-e2e`](https://github.com/kubernetes/test-infra/blob/fa43d4a7a6c88c0dedd0db83b250cec485b60736/images/kubekins-e2e/variants.yaml). ([reference PR review comment](https://github.com/kubernetes/test-infra/pull/13870#discussion_r313628808)) - -#### Cut next alpha - -Recall that an alpha.0 of the next minor release was created during [release branch creation](#release-branch-creation). - -The previously created alpha.0 is now several commits behind `master`. -As an example, see the [comparison between the `v1.18.0-alpha.0` (after 1.17 branch creation) and `v1.18.0-alpha.1` (after 1.17.0 release) tags](https://github.com/kubernetes/kubernetes/compare/v1.18.0-alpha.0...v1.18.0-alpha.1). - -To assist downstream consumers of Kubernetes, a new alpha should be cut to bring our next release tag to the tip of `master`. - -Begin the release process with: - -```shell -krel stage -``` - -Proceed with the [alpha release steps](#alpha-releases). - -Example `1.18.0-alpha.1` release issue template: https://github.com/kubernetes/sig-release/issues/928 - -#### Branching perf-tests - -Reach out to SIG Scalability to ensure a new branch is cut in [sigs.k8s.io/perf-tests](https://github.com/kubernetes/perf-tests/) repository. - -In case the release branch is missing - we will end up with an issue we had for 1.30, see [#124119](https://github.com/kubernetes/kubernetes/issues/124119) - ## Branch Management This section discusses the methods in managing commits on the `release-x.y` branch. -### Update test-infra configurations - -Before getting started, Branch Managers should: - -- Fork the [test-infra repository](https://github.com/kubernetes/test-infra) -- Clone their fork of `kubernetes/test-infra`: - -```shell -git clone git@github.com:/test-infra.git -``` - -- (optional) [Install Bazel](https://docs.bazel.build/versions/master/install.html) or run Bazel inside a container - - Running Bazel in a container is recommended over installing Bazel locally, as Bazel has many dependencies - ### Release Branch Creation -> [!IMPORTANT] -> The branch is created in the nomock staging phase and is pushed to the repository during the nomock release phase. - -During a `rc.0` release our release tooling creates a new release branch named `release-x.0`, where `x` is the major version of the next release. - -Behind the scenes `krel` is executing a `git branch create` command and `git push`. - -At the same time Prow’s [`branchprotector`](https://git.k8s.io/test-infra/prow/cmd/branchprotector/README.md) runs every hour at 54 minutes past the hour and automatically adds [branch protection](https://help.github.com/articles/about-protected-branches/) to any new branch in the `kubernetes/kubernetes` repo, including the newly created one. -No need to manually create the branch protection rule. - -New release branch creation (for example: `release-1.18`) also automatically triggers an alpha.0 build for the subsequent release (for example: [`v1.19.0-alpha.0`](https://github.com/kubernetes/kubernetes/releases/tag/v1.19.0-alpha.0)). - -> [!NOTE] -This means that the staging step will take about twice as long, as it will stage both versions `v1.18.0-rc.0` and `v1.19.0-alpha.0`. -The release step will also be extended, but not substantially longer in time. - -#### Post release branch creation tasks - -During the mock stage runs and before the nomock release job is started, run through the following tasks. - -> [!WARNING] -Rembember to put a `/hold` on all the PRs, they have to be lifted only once the nomock release phase is done and the branch is created. - -- [Update test-infra configurations](#update-test-infra-configurations) - - [Update milestone appliers](#update-milestone-appliers) - - [Update milestone requirements](#update-milestone-requirements) (this should have been already done as part of the Code Freeze routine) - - [Update e2e variants](#update-e2e-variants) - -After the rc.0 cut is done and you announced it, proceed with the following: +See [this section](../handbooks/k8s-release-cut.md#next-release-branch-creation) of the release cut handbook for more info. -- [Generate release branch jobs](#generate-release-branch-jobs) -- [Add a new variant for the kube-cross image](#add-a-new-variant-for-the-kube-cross-image) -- [Update publishing-bot rules](#update-publishing-bot-rules) +#### Post new release branch creation tasks -Once the new `release-x.0` branch is created, the [release jobs and dashboards should be generated and merged](#generate-release-branch-jobs). - -#### Update milestone appliers - -The [milestone applier plugin](https://github.com/kubernetes-sigs/prow/blob/main/pkg/plugins/milestoneapplier/milestoneapplier.go) automatically applies a GitHub milestone to pull requests after they have merged. - -This only applies to repos that have the milestone applier configured and for pull requests that do not already have a milestone. - -Update the `milestoneapplier` plugin configs for `kubernetes/kubernetes`: - -- Remove configs for the unsupported release branches, if present - _this is typically the oldest release in the list_ -- Add config for the current release branch - -Here's an [example PR](https://github.com/kubernetes/test-infra/pull/20075). - -#### Update milestone requirements - -> [!NOTE] -If the [code freeze](#code-freeze) was enabled before creating the release branch, there's nothing to do, as the milestone requirements would include the newest release branch already. - -Find the query config for `kubernetes/kubernetes` (in [config.yaml][config.yaml] file) with the code freeze enabled and add the newest release release branch. - -[config.yaml]: https://github.com/kubernetes/test-infra/blob/3a6962d/config/prow/config.yaml - -Here's an [example PR](https://github.com/kubernetes/test-infra/pull/20077). - -#### Update e2e variants - -Images used in Kubernetes e2e tests are generated via our [GCB Builder tool](https://github.com/kubernetes/test-infra/blob/master/images/builder/README.md). - -The `variants.yaml` config file, used in conjunction with the GCB Builder, allows us to target various branches or branch combinations during CI tests. The `K8S_VERSION` variable maps to the version marker `latest-x.y.txt` file viewable in the [`kubernetes-release` GCS bucket](https://gcsweb.k8s.io/gcs/kubernetes-release/release/) for example, [`latest-1.17.txt`](https://dl.k8s.io/release/latest-1.17.txt). - -Update the [variants for the `kubekins-e2e` image](https://github.com/kubernetes/test-infra/blob/master/images/kubekins-e2e/variants.yaml). - -- Remove the release variants for the unsupported releases -- Add an entry for the newest release variant -- Copy the `master` variant, rename it to the newest release and ensure the following: - - The `K8S_RELEASE` marker for `experimental` matches `master` - - The `CONFIG` marker for the newest release matches the variant/release name - - The `K8S_RELEASE` marker for the newest release variant is `latest-x.y` - - The `K8S_RELEASE` marker for every other release variant is `stable-x.y` - -Create a PR with this change and wait for it to be merged ([example PR](https://github.com/kubernetes/test-infra/pull/20076)). - -**Wait for the `test-infra-push-kubekins-e2e` presubmit to finish (you can [check on prow](https://prow.k8s.io/?job=post-test-infra-push-kubekins-e2e)).** - -**`git pull` latest from `master` before continuing.** - -#### Generate release branch jobs - -> [!CAUTION] -To minimize drift with tooling changes, this section has been moved to -[kubernetes/test-infra][release-branch-job-creation]. - -See [kubernetes/test-infra](release-branch-job-creation#release-branch-jobs) for the exact steps to follow. - -#### Add a new variant for the kube-cross image - -Once we have cut the branch, it is a good time to start producing a new kube-cross image. To do this, we first need to add the variant to kube-cross [variants.yaml](https://github.com/kubernetes/release/blob/master/images/build/cross/variants.yaml) file and update the [dependencies.yaml](https://github.com/kubernetes/release/blob/master/dependencies.yaml) file. Here is an [example PR](https://github.com/kubernetes/release/pull/2344/files). - -Once that is merged, wait for this [prow job](https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-release-push-image-kube-cross) to complete and then do an image promotion PR. Here is an [example](https://github.com/kubernetes/k8s.io/pull/4412). - -To create a new k8s-cloud-builder image with the new kube-cross image, a similar PR is created to update the variants.yaml for [k8s-cloud-builder](https://github.com/kubernetes/release/blob/master/images/k8s-cloud-builder/variants.yaml). Here is an older [example pr](https://github.com/kubernetes/release/commit/830da9f3fc51a6b581ea58df82cba4c3b805be99). - -Finally, we can bump the kube-cross builder image in kubernetes/kubernetes. Here is an [example PR](https://github.com/kubernetes/kubernetes/commit/242649ee7474f9d2d11add22e5747a8323221f4d). - -#### Update publishing-bot rules - -The Kubernetes Publishing Bot is responsible for: - -* ensuring that the master and release branches in the staging repositories are in-sync with the appropriate branches in `kubernetes/kubernetes` -* creating tags in the staging repositories for each Kubernetes release - -It's required to create the appropriate publishing-bot rules for the publishing-bot to work with the release branches. Once a new release branch is created in `kubernetes/kubernetes`, the Release Manager needs to update the publishing-bot rules as described in the [`k/publishing-bot` repository](https://git.k8s.io/publishing-bot#updating-rules). This best way to do this is by using the [update-rules](https://github.com/kubernetes/publishing-bot/blob/master/cmd/update-rules/README.md) CLI tool. - -Here's an [example PR](https://github.com/kubernetes/kubernetes/pull/100616), but please generate the changes with the `update-rules` tooling. - -[sig-release-x.y-blocking]: https://testgrid.k8s.io/sig-release-1.17-blocking +Run through the following tasks detailed [here](../handbooks/post-rc0-release-tasks.md) after the `rc.0` release is communicated and therefore the release branch has been created. --- @@ -779,6 +614,9 @@ Example PRs: - [1.33](https://github.com/kubernetes/test-infra/pull/34693) +> [!IMPORTANT] +After Code Thaw is performed, remind @release-managers to perform the propedeutic tasks for the next `alpha.1` cut, such as setting up the new OBS project. + ### Branch Fast Forward We now run the branch fast forward automatically to even the branches.