feat(backend): Patch pipeline spec from APIServer EnvVar

[gmfrasca]

Description of your changes:
Add an env var to API Server that applies/patches all compiled Pipelines with a given configuration.

• Allows a KFP admin to apply a custom specpatch to all incoming pipelines
• Useful for installment-wide configuration, such as workflow ttl
• Take in a JSON string provided via the environment variable COMPILED_PIPELINE_SPEC_PATCH

Testing instructions:

• Test 1:
  • Deploy KFP
  • Verify pipelines still run
  • Verify COMPILED_PIPELINE_SPEC_PATCH env var on the ml-pipeline Deployment exists and is set to empty JSON ('{}')
• Test 2:
  • Deploy KFP.
  • Update COMPILED_PIPELINE_SPEC_PATCH env var to '{"podGC":{"strategy":"OnPodCompletion"}}' on ml-pipeline Deployment.
  • Wait for ml-pipeline pod to redeploy
  • Start new pipeline run
  • Verify pipeline runs, but Pods created are deleted/Garbage Collected as soon as they successfully complete
    • (This is because we have patched in the podGC OnPodCompletion spec)
• Test 3:
  • Deploy KFP
  • Update COMPILED_PIPELINE_SPEC_PATCH env var to some-invalid-json on ml-pipeline Deployment.
  • Attempt to start a new pipeline run
  • Verify Error message is displayed, indicating the Invalid JSON for workflow spec patch

Checklist:

• You have signed off your commits
• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[mprahl]

/cc @VaniHaripriya

[VaniHaripriya]

/lgtm

Verified the changes and they’re working as expected. Thank you for the PR @gmfrasca.

[HumairAK]

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HumairAK

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• backend/OWNERS [HumairAK]
• manifests/kustomize/OWNERS [HumairAK]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment