kubeflow · kaikaila · May 19, 2025 · Jun 25, 2025 · Jul 4, 2025 · May 19, 2025
diff --git a/backend/Makefile b/backend/Makefile
@@ -136,4 +136,12 @@ lint:
 
 .PHONY: format
 format:
-	golangci-lint fmt
+	golangci-lint fmt
+
+.PHONY: start-dev-db
+start-dev-db:
+	@./scripts/start-dev-db.sh
+
+.PHONY: stop-dev-db
+stop-dev-db:
+	@./scripts/stop-dev-db.sh
diff --git a/backend/README.md b/backend/README.md
@@ -8,7 +8,9 @@ Pipelines backend.
 This README will help you set up your coding environment in order to build and run the Kubeflow Pipelines backend. The KFP backend powers the core functionality of the KFP platform, handling API requests, workflow management, and data persistence.
 
 ## Prerequisites
+
 Before you begin, ensure you have:
+
 - [Go installed](https://go.dev/doc/install)
 - Docker or Podman installed (for building container images)
 
@@ -32,6 +34,30 @@ LOCAL_API_SERVER=true go test -v ./backend/test/v2/integration/... -namespace ku
 To run a specific test, you can use the `-run` flag. For example, to run the `TestCacheSingleRun` test in the
 `TestCache` suite, you can use the `-run 'TestCache/TestCacheSingleRun'` flag to the above command.
 
+## Local Database Deployment
+
+You can deploy a development database (PostgreSQL or MySQL) locally using Makefile targets and scripts:
+
+- Default (PostgreSQL):
+  ```bash
+  cd backend
+  make start-dev-db
+  ```
+- Switch to MySQL:
+  ```bash
+  cd backend
+  DB_TYPE=mysql make start-dev-db
+  ```
+- Cleanup:
+  ```bash
+  cd backend
+  make stop-dev-db
+  # Or for MySQL:
+  DB_TYPE=mysql make stop-dev-db
+  ```
+
+This allows you to quickly spin up and tear down a test database for development and CI testing.
+
 ## Build
 
 The API server itself can be built using:
@@ -41,18 +67,22 @@ go build -o /tmp/apiserver backend/src/apiserver/*.go
 ```
 
 The API server image can be built from the root folder of the repo using:
+
 ```
 export API_SERVER_IMAGE=api_server
 docker build -f backend/Dockerfile . --tag $API_SERVER_IMAGE
 ```
+
 ### Deploying the APIServer (from the image you built) on Kubernetes
 
 First, push your image to a registry that is accessible from your Kubernetes cluster.
 
 Then, run:
+
 ```
 kubectl edit deployment.v1.apps/ml-pipeline -n kubeflow
 ```
+
 You'll see the field reference the api server container image (`spec.containers[0].image: gcr.io/ml-pipeline/api-server:<image-version>`).
 Change it to point to your own build, after saving and closing the file, apiserver will restart with your change.
 
@@ -68,10 +98,10 @@ dependencies. To update dependencies, edit [requirements.in](requirements.in)
 and run `./update_requirements.sh` to update and pin the transitive
 dependencies.
 
-
 ### Building conformance tests (WIP)
 
 Run
+
 ```
 docker build . -f backend/Dockerfile.conformance -t <tag>
 ```
@@ -87,16 +117,16 @@ pods on the cluster using the `ml-pipeline` `Service`.
 
 #### Prerequisites
 
-* The [kind CLI](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) is installed.
-* The following ports are available on your localhost: 3000, 3306, 8080, 9000, and 8889. If these are unavailable,
+- The [kind CLI](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) is installed.
+- The following ports are available on your localhost: 3000, 3306, 8080, 9000, and 8889. If these are unavailable,
   modify [kind-config.yaml](../tools/kind/kind-config.yaml) and configure the API server with alternative ports when
   running locally.
-* If using a Mac, you will need to modify the
+- If using a Mac, you will need to modify the
   [Endpoints](../manifests/kustomize/env/dev-kind/forward-local-api-endpoint.yaml) manifest to leverage the bridge
   network interface through Docker/Podman Desktop. See
   [kind #1200](https://github.com/kubernetes-sigs/kind/issues/1200#issuecomment-1304855791) for an example manifest.
-* Optional: VSCode is installed to leverage a sample `launch.json` file.
-  * This relies on dlv: (go install -v github.com/go-delve/delve/cmd/dlv@latest)
+- Optional: VSCode is installed to leverage a sample `launch.json` file.
+  - This relies on dlv: (go install -v github.com/go-delve/delve/cmd/dlv@latest)
 
 #### Provisioning the Cluster
 
@@ -175,24 +205,24 @@ Then you may leverage the following sample `.vscode/launch.json` file to run the
 
 ```json
 {
-    "version": "0.2.0",
-    "configurations": [
-        {
-            "name": "Launch Scheduled Workflow controller (Kind)",
-            "type": "go",
-            "request": "launch",
-            "mode": "debug",
-            "program": "${workspaceFolder}/backend/src/crd/controller/scheduledworkflow",
-            "env": {
-                "CRON_SCHEDULE_TIMEZONE": "UTC"
-            },
-            "args": [
-                "-namespace=kubeflow",
-                "-kubeconfig=${workspaceFolder}/kubeconfig_dev-pipelines-api",
-                "-mlPipelineAPIServerName=localhost"
-            ]
-        }
-    ]
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Launch Scheduled Workflow controller (Kind)",
+      "type": "go",
+      "request": "launch",
+      "mode": "debug",
+      "program": "${workspaceFolder}/backend/src/crd/controller/scheduledworkflow",
+      "env": {
+        "CRON_SCHEDULE_TIMEZONE": "UTC"
+      },
+      "args": [
+        "-namespace=kubeflow",
+        "-kubeconfig=${workspaceFolder}/kubeconfig_dev-pipelines-api",
+        "-mlPipelineAPIServerName=localhost"
+      ]
+    }
+  ]
 }
 ```
 
@@ -231,52 +261,55 @@ You may use the following VS Code `launch.json` file to run the API server which
 command to use Delve and the Driver image to use debug image built previously.
 
 VSCode configuration:
+
 ```json
 {
-    "version": "0.2.0",
-    "configurations": [
-        {
-            "name": "Launch API server (Kind) (Debug Driver)",
-            "type": "go",
-            "request": "launch",
-            "mode": "debug",
-            "program": "${workspaceFolder}/backend/src/apiserver",
-            "env": {
-                "POD_NAMESPACE": "kubeflow",
-                "DBCONFIG_MYSQLCONFIG_HOST": "localhost",
-                "MINIO_SERVICE_SERVICE_HOST": "localhost",
-                "MINIO_SERVICE_SERVICE_PORT": "9000",
-                "METADATA_GRPC_SERVICE_SERVICE_HOST": "localhost",
-                "METADATA_GRPC_SERVICE_SERVICE_PORT": "8080",
-                "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_HOST": "localhost",
-                "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_PORT": "8888",
-                "V2_LAUNCHER_IMAGE": "ghcr.io/kubeflow/kfp-launcher:master",
-                "V2_DRIVER_IMAGE": "kfp-driver:debug",
-                "V2_DRIVER_COMMAND": "dlv exec --listen=:2345 --headless=true --api-version=2 --log /bin/driver --"
-            }
-        }
-    ]
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Launch API server (Kind) (Debug Driver)",
+      "type": "go",
+      "request": "launch",
+      "mode": "debug",
+      "program": "${workspaceFolder}/backend/src/apiserver",
+      "env": {
+        "POD_NAMESPACE": "kubeflow",
+        "DBCONFIG_MYSQLCONFIG_HOST": "localhost",
+        "MINIO_SERVICE_SERVICE_HOST": "localhost",
+        "MINIO_SERVICE_SERVICE_PORT": "9000",
+        "METADATA_GRPC_SERVICE_SERVICE_HOST": "localhost",
+        "METADATA_GRPC_SERVICE_SERVICE_PORT": "8080",
+        "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_HOST": "localhost",
+        "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_PORT": "8888",
+        "V2_LAUNCHER_IMAGE": "ghcr.io/kubeflow/kfp-launcher:master",
+        "V2_DRIVER_IMAGE": "kfp-driver:debug",
+        "V2_DRIVER_COMMAND": "dlv exec --listen=:2345 --headless=true --api-version=2 --log /bin/driver --"
+      }
+    }
+  ]
 }
 ```
 
 GoLand configuration:
+
 1. Create a new Go Build configuration
 2. Set **Run Kind** to Directory and set **Directory** to /backend/src/apiserver absolute path
 3. Set the following environment variables
 
-   | Argument                                     | Value     |
-   |----------------------------------------------|-----------|
-   | POD_NAMESPACE                                | kubeflow  |
-   | DBCONFIG_MYSQLCONFIG_HOST                    | localhost |
-   | MINIO_SERVICE_SERVICE_HOST                   | localhost |
-   | MINIO_SERVICE_SERVICE_PORT                   | 9000      |
-   | METADATA_GRPC_SERVICE_SERVICE_HOST           | localhost |
-   | METADATA_GRPC_SERVICE_SERVICE_PORT           | 8080      |
-   | ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_HOST | localhost |
-   | ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_PORT | 8888      |
-   | V2_LAUNCHER_IMAGE                            | localhost |
-   | V2_DRIVER_IMAGE                              | localhost |
+   | Argument                                     | Value                                                                                       |
+   | -------------------------------------------- | ------------------------------------------------------------------------------------------- |
+   | POD_NAMESPACE                                | kubeflow                                                                                    |
+   | DBCONFIG_MYSQLCONFIG_HOST                    | localhost                                                                                   |
+   | MINIO_SERVICE_SERVICE_HOST                   | localhost                                                                                   |
+   | MINIO_SERVICE_SERVICE_PORT                   | 9000                                                                                        |
+   | METADATA_GRPC_SERVICE_SERVICE_HOST           | localhost                                                                                   |
+   | METADATA_GRPC_SERVICE_SERVICE_PORT           | 8080                                                                                        |
+   | ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_HOST | localhost                                                                                   |
+   | ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_PORT | 8888                                                                                        |
+   | V2_LAUNCHER_IMAGE                            | localhost                                                                                   |
+   | V2_DRIVER_IMAGE                              | localhost                                                                                   |
    | V2_DRIVER_COMMAND                            | dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec /bin/driver -- |
+
 4. Set the following program arguments: --config ./backend/src/apiserver/config -logtostderr=true --sampleconfig ./backend/src/apiserver/config/test_sample_config.json
 
 #### Starting a Remote Debug Session
@@ -300,24 +333,26 @@ Set a breakpoint on the Driver code in VS Code. Then remotely connect to the Del
 Code `launch.json` file:
 
 VSCode configuration:
+
 ```json
 {
-    "version": "0.2.0",
-    "configurations": [
-        {
-            "name": "Connect to remote driver",
-            "type": "go",
-            "request": "attach",
-            "mode": "remote",
-            "remotePath": "/go/src/github.com/kubeflow/pipelines",
-            "port": 2345,
-            "host": "127.0.0.1",
-        }
-    ]
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Connect to remote driver",
+      "type": "go",
+      "request": "attach",
+      "mode": "remote",
+      "remotePath": "/go/src/github.com/kubeflow/pipelines",
+      "port": 2345,
+      "host": "127.0.0.1"
+    }
+  ]
 }
 ```
 
 GoLand configuration:
+
 1. Create a new Go Remote configuration and title it "Delve debug session"
 2. Set **Host** to localhost
 3. Set **Port** to 2345
@@ -347,6 +382,7 @@ kind delete clusters dev-pipelines-api
 ```
 
 ## Contributing
+
 ### Code Style
 
 Backend codebase follows the [Google's Go Style Guide](https://google.github.io/styleguide/go/). Please, take time to get familiar with the [best practices](https://google.github.io/styleguide/go/best-practices). It is not intended to be exhaustive, but it often helps minimizing guesswork among developers and keep codebase uniform and consistent.

diff --git a/backend/helm-charts/postgresql/Chart.yaml b/backend/helm-charts/postgresql/Chart.yaml
@@ -0,0 +1,4 @@
+apiVersion: v2
+name: postgresql
+version: 0.1.0
+description: A Helm chart for local dev PostgreSQL (GSoC testing)
diff --git a/backend/helm-charts/postgresql/README.md b/backend/helm-charts/postgresql/README.md
@@ -0,0 +1,86 @@
+# PostgreSQL Helm Deployment for Kubeflow Pipelines
+
+This directory contains the Helm values file to deploy a PostgreSQL instance for Kubeflow Pipelines using the Bitnami PostgreSQL Chart.
+
+## Prerequisites
+
+- **Helm**: version 3.x installed (`helm version` should show v3).
+- **Kubernetes** cluster with the `kubeflow` namespace created (or adjust commands to match your target namespace).
+
+## Installation Steps
+
+1. **Add the Bitnami chart repository**
+
+   ```bash
+   helm repo add bitnami https://charts.bitnami.com/bitnami
+   ```
+
+2. **Update local chart repository cache**
+
+   ```bash
+   helm repo update
+   ```
+
+3. **Install or upgrade the PostgreSQL release**
+
+   ```bash
+   helm upgrade --install kfp-postgres bitnami/postgresql \
+     --namespace kubeflow \
+     -f helm-charts/postgresql/values.yaml
+   ```
+
+   - `kfp-postgres` is the release name.
+   - `bitnami/postgresql` refers to the community chart.
+   - `-f values.yaml` applies your custom parameters.
+
+## values.yaml Highlights
+
+Customize the following key sections in `values.yaml` as needed:
+
+```yaml
+global:
+  postgresql:
+    postgresqlPassword: <YOUR_PASSWORD> # PostgreSQL superuser password
+    postgresqlDatabase: kubeflow # Default database name
+    postgresqlUsername: kubeflow # Default database user
+persistence:
+  enabled: true
+  size: 10Gi # PVC size
+  storageClass: standard # StorageClass for persistent volume
+service:
+  port: 5432 # PostgreSQL service port
+```
+
+## Verifying Deployment
+
+1. **Check resources**
+
+   ```bash
+   kubectl get pods,svc,pvc -n kubeflow
+   ```
+
+2. **Port-forward and connect with psql**
+   ```bash
+   kubectl port-forward svc/kfp-postgres 5432:5432 -n kubeflow
+   psql postgres://kubeflow:<YOUR_PASSWORD>@localhost:5432/kubeflow
+   ```
+
+## Upgrading & Uninstalling
+
+- **Upgrade with new values**
+
+  ```bash
+  helm upgrade kfp-postgres bitnami/postgresql \
+    --namespace kubeflow \
+    -f helm-charts/postgresql/values.yaml
+  ```
+
+- **Uninstall release**
+  ```bash
+  helm uninstall kfp-postgres --namespace kubeflow
+  ```
+
+## Notes
+
+- You can override individual settings via `--set` flags instead of editing `values.yaml`.
+- To use a different namespace, modify the `--namespace` flag accordingly.
diff --git a/backend/helm-charts/postgresql/values.yaml b/backend/helm-charts/postgresql/values.yaml
@@ -0,0 +1,12 @@
+auth:
+  postgresPassword: kubeflow # PostgreSQL superuser password
+  database: kubeflow # Default database name
+  username: kubeflow # Default database user
+
+persistence:
+  enabled: true
+  size: 10Gi # PVC size
+  storageClass: standard # StorageClass for persistent volume
+
+service:
+  port: 5432 # PostgreSQL service port