chore: complete uv migration (CI triggers, Dockerfiles, cleanup and generator fixes). Follow up to #12770

[sh4shv4t]

Description of your changes:

This PR is a follow-up to #12770 and addresses @hbelmiro's review comments on issue #12686 after the revert in #12979.

1. CI trigger coverage for dependency changes

• Updated workflow path filters to include:
• **pyproject.toml
• uv.lock
• Ensures Python-related workflows rerun when dependencies change.
• Added safeguards for lockfile drift (check-uv-lock).

2. Dockerfile migration to uv

• Migrated non-visualization Dockerfiles from requirements.txt-based installs to uv-based installs.
• For workspace-oriented images, this uses:
• install uv
• copy pyproject.toml and uv.lock
• run uv sync --frozen
• For standalone/component images, this uses uv pip install with pinned dependencies.
• Ensures reproducible, dependency-controlled builds across images.

Exception (intentional):

• Dockerfile.visualization continues using requirements.txt
• Reason: standalone module with dependency conflicts (as noted by reviewers)

3. Removal of legacy requirements-based dependency management

Removed many scattered:

• requirements.txt
• requirements.in
• setup.py
• MANIFEST.in
• Introduced workspace-based dependency management using:
• pyproject.toml
• uv.lock

Result:

• Single source of truth for dependencies
• Reduced drift between environments
• Some requirements files are intentionally retained (see Known exceptions)

4. Generator-aware fixes (CI stability)

• Fixed validate-generated-files failures caused by editing generated outputs directly.
• Moved changes to source-of-truth templates:
• tox.mustache
• test-requirements.mustache
• Updated build_kfp_server_api_python_package.sh:
• Preserve requirements.txt for tox compatibility
• Regenerated v2beta1 Python client to align with generation output.

5. Protobuf + generated file drift fixes

• Aligned generated .pb.go outputs with the current generator behavior used in CI.
• Prevents recurring check-diff failures due to generator drift.

6. Docker build fix (visualization)

• Fixed legacy ENV key value to ENV key=value
• Replaced fragile gcloud install invocation with a more stable approach
• Resolves CI Docker build failure

7. CI / tooling consistency improvements

• Updated shared actions and scripts to consistently use uv.
• Standardized Python setup and dependency sync in CI.
• Improved reproducibility and reduced environment-specific issues.

---

Known exceptions / intentional deviations

• Dockerfile.visualization continues using requirements.txt
• Some requirements.txt files are retained:
• Generated API clients (tox compatibility)
• PyTorch components (tooling dependency)

These are temporary and required to avoid breaking:

• OpenAPI generator outputs
• tox-based workflows

---

Notes

• make check-diff now passes after regeneration
• Changes are aligned with generator behavior (no manual edits to generated files)
• CI failures previously observed were due to:
• generator drift
• missing trigger paths
• Dockerfile issues

---

Issue linkage

• Fixes chore(sdk): Migrate Python dependency management to uv workspaces #12686
• Follow-up to chore: migrate build system from pip/setup.py to uv workspaces (#12686) #12770

---

Checklist:

• You have signed off your commits
• The title follows the PR title convention

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign droctothorpe for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[google-oss-prow]

Hi @sh4shv4t. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Copilot]

Pull request overview

Completes the repository-wide migration from requirements*.txt/setup.py workflows to a uv workspace with pyproject.toml + uv.lock, updating CI triggers, Dockerfiles, docs, and generator templates to keep builds reproducible and generated outputs in sync.

Changes:

• Introduces a workspace-level pyproject.toml and updates multiple Python packages to pyproject.toml-based builds, aligning CI/scripts on uv sync / uv run.
• Migrates many Dockerfiles and CI workflows from pip install -r ... to uv-based installs; adds a lockfile drift check workflow.
• Removes legacy requirements/setup artifacts and adjusts generator templates and docs to reflect the new source-of-truth.

Reviewed changes

Copilot reviewed 134 out of 135 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
third_party/ml-metadata/update_version.sh	Updates MLMD version replacement to target Dockerfile instead of requirements files
test/tag_for_hosted.sh	Sync comment updated to point to metadata_writer Dockerfile
test/presubmit-yapf-sdk.sh	Switches formatting presubmit to uv run
test/presubmit-tests-sdk.sh	Runs SDK regression tests via uv run
test/presubmit-tests-sdk-unit.sh	Runs SDK unit tests via uv run
test/presubmit-tests-sdk-client.sh	Runs SDK client tests via uv run
test/presubmit-test-sdk-upgrade.sh	Adjusts SDK upgrade test to build with uv and install wheels via pip
test/presubmit-isort-sdk.sh	Switches import cleanup presubmit to uv run
test/presubmit-docformatter-sdk.sh	Switches docformatter presubmit to uv run
test/presubmit-component-yaml.sh	Removes per-script pip setup; relies on CI env and runs component-yaml test script
test/kfp-kubernetes-native-migration-tests/requirements.txt	Removes standalone requirements file for native migration tests
test/README.md	Updates local test instructions to use uv sync / uv run
sdk/python/setup.py	Removes setuptools setup.py for SDK in favor of pyproject.toml
sdk/python/requirements.txt	Removes compiled requirements lock for SDK
sdk/python/requirements.in	Removes legacy SDK requirements input file
sdk/python/requirements-dev.txt	Removes legacy SDK dev requirements file
sdk/python/requirements-deprecated.txt	Removes deprecated runtime requirements file (inlined elsewhere)
sdk/python/pyproject.toml	Adds SDK pyproject.toml with dependencies/extras and hatch versioning
sdk/python/pre-release-requirements-update.sh	Removes pip-tools based pre-release requirements generation
sdk/python/kfp/version.py	Bumps SDK version to 2.16.0
sdk/python/kfp/client/auth.py	Updates comment to be less requirements-file specific
sdk/python/MANIFEST.in	Removes requirements inclusion from manifest
sdk/Makefile	Builds SDK package with uv build inside generator image
sdk/CONTRIBUTING.md	Updates contributor docs to use uv workflows
samples/tutorials/Data passing in python components/Data passing in python components - Files.py.yaml	Adds regenerated compiled pipeline YAML (kfp 2.16.0)
samples/tutorials/DSL - Control structures/DSL - Control structures.py.yaml	Adds regenerated compiled pipeline YAML (kfp 2.16.0)
pyproject.toml	Adds workspace pyproject.toml defining members and shared extras
proxy/requirements.txt	Removes proxy requirements file
proxy/Dockerfile	Installs requests via uv pip during image build
manifests/kustomize/base/metadata/base/metadata-grpc-deployment.yaml	Updates MLMD sync comment to point to metadata_writer Dockerfile
manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/run_tests.sh	Inlines test deps install instead of requirements-dev.txt
manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/requirements-dev.txt	Removes requirements-dev.txt for profile controller tests
kubernetes_platform/python/setup.py	Removes setuptools setup.py in favor of pyproject.toml
kubernetes_platform/python/requirements.txt	Removes compiled requirements lock for kfp-kubernetes
kubernetes_platform/python/requirements.in	Removes legacy requirements input file
kubernetes_platform/python/requirements-dev.txt	Removes legacy dev requirements file
kubernetes_platform/python/release.sh	Builds via python -m build and avoids importing package for version
kubernetes_platform/python/pyproject.toml	Adds kfp-kubernetes pyproject.toml with hatch versioning
kubernetes_platform/python/pre-release-requirements-update.sh	Removes pip-tools based pre-release requirements generation
kubernetes_platform/python/kfp/kubernetes/init.py	Bumps kfp-kubernetes version to 2.16.0
kubernetes_platform/python/docs/requirements.txt	Removes docs requirements file for kfp-kubernetes docs
kubernetes_platform/python/docs/.readthedocs.yml	Switches RTD install flow to pip installs with extra requirements
kubernetes_platform/python/create_release_branch.sh	Avoids importing package for version check (grep version)
kubernetes_platform/python/RELEASE.md	Updates release docs to mention uv/pip build prerequisites
kubernetes_platform/python/MANIFEST.in	Removes requirements files from manifest
kubernetes_platform/Makefile	Builds kfp-kubernetes via uv build; updates python-dev install
hack/update-requirements.sh	Removes pip-tools docker-based requirements update script
hack/update-all-requirements.sh	Removes script that updated multiple requirements files
docs/sdk/requirements.txt	Removes docs requirements file
docs/sdk/build_docs_locally.sh	Updates local docs build to uv sync --extra docs
components/test_load_all_components.sh	Switches to running loader under uv run
components/kserve/requirements.txt	Removes kserve component requirements file
components/kserve/Dockerfile	Installs component deps via uv pip
components/aws/sagemaker/v2.Dockerfile.dockerignore	Stops ignoring removed requirements files
components/aws/sagemaker/v2.Dockerfile	Installs v2 image deps via uv pip
components/aws/sagemaker/tests/unit_tests/README.md	Inlines dependency install instructions
components/aws/sagemaker/tests/unit_tests/Dockerfile	Installs unit test deps via uv pip
components/aws/sagemaker/tests/integration_tests/requirements.txt	Removes integration test requirements file
components/aws/sagemaker/tests/integration_tests/Dockerfile	Installs integration test deps via uv pip
components/aws/sagemaker/requirements_v2.txt	Removes v2 requirements file
components/aws/sagemaker/requirements.txt	Removes runtime requirements file
components/aws/sagemaker/dev_requirements.txt	Removes dev requirements file
components/aws/sagemaker/TrainingJob/samples/mnist-kmeans-training/requirements.txt	Removes sample requirements file
components/aws/sagemaker/TrainingJob/samples/mnist-kmeans-training/README.md	Inlines pip install command for sample
components/aws/sagemaker/Dockerfile	Installs runtime deps via uv pip
components/aws/sagemaker/CONTRIBUTING.md	Updates contributor instructions to uv-centric setup
components/PyTorch/pytorch-kfp-components/tox.ini	Switches tox deps to requirements files
components/PyTorch/pytorch-kfp-components/test-requirements.txt	Adds pinned-ish test requirements for tox
components/PyTorch/pytorch-kfp-components/requirements.txt	Adds base requirements for tox installs
backend/update_requirements.sh	Replaces pip-tools update with uv lock
backend/src/v2/test/requirements.txt	Removes v2 test requirements file
backend/src/v2/test/components/run_sample.yaml	Inlines deprecated deps install list in sample runner
backend/src/v2/test/Dockerfile	Uses uv sync + editable install for SDK in v2 test image
backend/src/v2/README.md	Updates test deps instructions to use uv
backend/requirements.txt	Removes backend requirements lock file
backend/requirements.in	Removes backend requirements input file
backend/metadata_writer/update_requirements.sh	Replaces pip-tools update with uv lock
backend/metadata_writer/requirements.txt	Removes metadata_writer requirements lock file
backend/metadata_writer/requirements.in	Removes metadata_writer requirements input file
backend/metadata_writer/README.md	Updates dependency editing instructions to workspace pyproject + uv lock
backend/metadata_writer/Dockerfile	Migrates metadata_writer image to uv sync + explicit MLMD install
backend/api/v2beta1/python_http_client_template/tox.mustache	Adds tox template using requirements + test-requirements
backend/api/v2beta1/python_http_client_template/test-requirements.mustache	Adds test requirements template
backend/api/v2beta1/python_http_client/tox.ini	Updates tox config to include test requirements and skip missing interpreters
backend/api/v2beta1/python_http_client/test-requirements.txt	Updates pytest/pytest-cov/pytest-randomly versions
backend/api/v2beta1/python_http_client/setup.py	Removes setup.py in favor of pyproject
backend/api/v2beta1/python_http_client/pyproject.toml	Adds pyproject for generated client
backend/api/v1beta1/python_http_client_template/tox.mustache	Adds tox template using requirements + test-requirements
backend/api/v1beta1/python_http_client_template/test-requirements.mustache	Adds test requirements template
backend/api/v1beta1/python_http_client/tox.ini	Updates tox config to include test requirements and skip missing interpreters
backend/api/v1beta1/python_http_client/test-requirements.txt	Updates pytest/pytest-cov/pytest-randomly versions
backend/api/build_kfp_server_api_python_package.sh	Makes removal robust and adds post-build pyproject restoration for v2beta1
backend/README.md	Updates dependency management docs to uv-based workflow
backend/Dockerfile.visualization	Fixes ENV syntax and gcloud install invocation
backend/Dockerfile	Migrates compiler stage to uv workspace + generates protos during build
api/v2alpha1/python/setup.py	Removes setup.py in favor of pyproject
api/v2alpha1/python/requirements.txt	Removes compiled requirements lock file
api/v2alpha1/python/requirements.in	Removes requirements input file
api/v2alpha1/python/pyproject.toml	Adds pyproject for pipeline spec package
api/v2alpha1/python/MANIFEST.in	Removes requirements inclusion from manifest
api/Makefile	Builds pipeline spec package with uv build inside generator image
RELEASE.md	Updates release workflow steps to pyproject+uv lock model
AGENTS.md	Updates agent/developer workflows to uv commands
.readthedocs.yml	Migrates RTD build to uv sync + commands-based build
.pre-commit-config.yaml	Switches Python formatting hooks to run via uv run
.github/workflows/validate-generated-files.yml	Adds uv-related path triggers and switches to shared setup-python action
.github/workflows/upgrade-test.yml	Adds uv-related path triggers
.github/workflows/sdk-yapf.yml	Adds uv-related path triggers and switches to shared setup-python action
.github/workflows/sdk-upgrade.yml	Adds uv-related path triggers and updates protobuf step usage
.github/workflows/sdk-isort.yml	Adds uv-related path triggers and switches to shared setup-python action
.github/workflows/sdk-docformatter.yml	Adds uv-related path triggers and switches to shared setup-python action
.github/workflows/sdk-component-yaml.yml	Adds uv-related path triggers and simplifies setup steps
.github/workflows/readthedocs-builds.yml	Adds uv-related path triggers and updates docs build commands to uv run
.github/workflows/publish-packages.yml	Builds/checks wheels via uv build and uv run twine
.github/workflows/legacy-v2-api-integration-tests.yml	Adds uv-related path triggers
.github/workflows/kfp-sdk-unit-tests.yml	Adds uv-related path triggers and simplifies setup steps
.github/workflows/kfp-sdk-tests.yml	Adds uv-related path triggers and simplifies setup steps
.github/workflows/kfp-sdk-client-tests.yml	Adds uv-related path triggers and simplifies setup steps
.github/workflows/kfp-kubernetes-native-migration-tests.yaml	Adds uv-related path triggers and switches test runs to uv run
.github/workflows/kfp-kubernetes-library-test.yml	Adds uv-related path triggers and switches tests to uv run
.github/workflows/integration-tests-v1.yml	Adds uv-related path triggers
.github/workflows/gcpc-modules-tests.yml	Adds uv-related path triggers and switches setup/tests to uv-based flow
.github/workflows/e2e-test.yml	Adds uv-related path triggers
.github/workflows/e2e-test-frontend.yml	Adds uv-related path triggers
.github/workflows/docs-freshness.yml	Adds uv-related path triggers
.github/workflows/compiler-tests.yml	Adds uv-related path triggers
.github/workflows/check-uv-lock.yml	Adds new workflow to detect uv.lock drift
.github/workflows/backend-visualization.yml	Adds uv-related path triggers
.github/workflows/api-server-tests.yml	Adds uv-related path triggers
.github/resources/scripts/kfp-readiness/requirements.txt	Removes readiness requirements file
.github/resources/scripts/helper-functions.sh	Inlines readiness deps install instead of requirements file
.github/actions/setup-python/action.yml	Adds composite action to install Python+uv and sync CI deps
.github/actions/protobuf/action.yml	Simplifies protobuf action to rely on uv and install spec wheel via uv
.github/actions/kfp-k8s/action.yml	Switches build/install steps to uv build and uv pip install

Comments suppressed due to low confidence (5)

backend/api/v2beta1/python_http_client/tox.ini:1

• envlist = py27, py3 is incompatible with the updated test dependencies (pytest>=7) and with the package metadata (requires-python = \">=3.9\" in pyproject.toml). This can make tox -e py27 fail if Python 2.7 is present, and it also advertises unsupported environments. Update the envlist to only supported Python versions (e.g., py39, py310, py311) or drop py27 entirely in both this file and the generator templates.
  .readthedocs.yml:1
• make -C api python and make -C kubernetes_platform python invoke Docker-based build steps (per the Makefile diffs), which Read the Docs build environments typically do not support. This is likely to break RTD builds. Prefer non-Docker proto generation steps in RTD (e.g., install protoc and run the generate_proto.py scripts directly), or switch RTD to targets that do not require Docker.
  test/presubmit-yapf-sdk.sh:1
• find sdk/python/kfp/**/*.py -type f is not a reliable find invocation because ** globbing isn’t expanded by find (and may not be enabled in the shell), so this can yield no files or errors depending on the environment. Use a find root directory plus -name '*.py' (e.g., search under sdk/python/kfp). Also, || true can mask real failures (not just 'file modified' exits); if the intent is to ignore only 'changes made' behavior, handle that specific exit code instead of swallowing all errors.
  backend/Dockerfile:1
• Downloading .proto files during the Docker build introduces a supply-chain risk and reduces reproducibility (no hash verification; relies on external availability). Prefer vendoring these protos into the repo and copying them into the image, or at minimum verify downloads via pinned SHA256 checksums and fail closed if verification fails.
  backend/api/build_kfp_server_api_python_package.sh:1
• The TODO references a placeholder issue (issues/XXXX), which makes it hard to track and can confuse future maintainers. Replace with the actual issue link (or create one and reference it), or remove the link if no issue exists.