diff --git a/backend/Makefile b/backend/Makefile index ae9567222c4..f4660361ffa 100644 --- a/backend/Makefile +++ b/backend/Makefile @@ -136,4 +136,12 @@ lint: .PHONY: format format: - golangci-lint fmt \ No newline at end of file + golangci-lint fmt + +.PHONY: start-dev-db +start-dev-db: + @./scripts/start-dev-db.sh + +.PHONY: stop-dev-db +stop-dev-db: + @./scripts/stop-dev-db.sh diff --git a/backend/README.md b/backend/README.md index c62403cdb0d..45add329726 100644 --- a/backend/README.md +++ b/backend/README.md @@ -8,7 +8,9 @@ Pipelines backend. This README will help you set up your coding environment in order to build and run the Kubeflow Pipelines backend. The KFP backend powers the core functionality of the KFP platform, handling API requests, workflow management, and data persistence. ## Prerequisites + Before you begin, ensure you have: + - [Go installed](https://go.dev/doc/install) - Docker or Podman installed (for building container images) @@ -32,6 +34,30 @@ LOCAL_API_SERVER=true go test -v ./backend/test/v2/integration/... -namespace ku To run a specific test, you can use the `-run` flag. For example, to run the `TestCacheSingleRun` test in the `TestCache` suite, you can use the `-run 'TestCache/TestCacheSingleRun'` flag to the above command. +## Local Database Deployment + +You can deploy a development database (PostgreSQL or MySQL) locally using Makefile targets and scripts: + +- Default (PostgreSQL): + ```bash + cd backend + make start-dev-db + ``` +- Switch to MySQL: + ```bash + cd backend + DB_TYPE=mysql make start-dev-db + ``` +- Cleanup: + ```bash + cd backend + make stop-dev-db + # Or for MySQL: + DB_TYPE=mysql make stop-dev-db + ``` + +This allows you to quickly spin up and tear down a test database for development and CI testing. + ## Build The API server itself can be built using: @@ -41,18 +67,22 @@ go build -o /tmp/apiserver backend/src/apiserver/*.go ``` The API server image can be built from the root folder of the repo using: + ``` export API_SERVER_IMAGE=api_server docker build -f backend/Dockerfile . --tag $API_SERVER_IMAGE ``` + ### Deploying the APIServer (from the image you built) on Kubernetes First, push your image to a registry that is accessible from your Kubernetes cluster. Then, run: + ``` kubectl edit deployment.v1.apps/ml-pipeline -n kubeflow ``` + You'll see the field reference the api server container image (`spec.containers[0].image: gcr.io/ml-pipeline/api-server:`). Change it to point to your own build, after saving and closing the file, apiserver will restart with your change. @@ -68,10 +98,10 @@ dependencies. To update dependencies, edit [requirements.in](requirements.in) and run `./update_requirements.sh` to update and pin the transitive dependencies. - ### Building conformance tests (WIP) Run + ``` docker build . -f backend/Dockerfile.conformance -t ``` @@ -87,16 +117,16 @@ pods on the cluster using the `ml-pipeline` `Service`. #### Prerequisites -* The [kind CLI](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) is installed. -* The following ports are available on your localhost: 3000, 3306, 8080, 9000, and 8889. If these are unavailable, +- The [kind CLI](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) is installed. +- The following ports are available on your localhost: 3000, 3306, 8080, 9000, and 8889. If these are unavailable, modify [kind-config.yaml](../tools/kind/kind-config.yaml) and configure the API server with alternative ports when running locally. -* If using a Mac, you will need to modify the +- If using a Mac, you will need to modify the [Endpoints](../manifests/kustomize/env/dev-kind/forward-local-api-endpoint.yaml) manifest to leverage the bridge network interface through Docker/Podman Desktop. See [kind #1200](https://github.com/kubernetes-sigs/kind/issues/1200#issuecomment-1304855791) for an example manifest. -* Optional: VSCode is installed to leverage a sample `launch.json` file. - * This relies on dlv: (go install -v github.com/go-delve/delve/cmd/dlv@latest) +- Optional: VSCode is installed to leverage a sample `launch.json` file. + - This relies on dlv: (go install -v github.com/go-delve/delve/cmd/dlv@latest) #### Provisioning the Cluster @@ -175,24 +205,24 @@ Then you may leverage the following sample `.vscode/launch.json` file to run the ```json { - "version": "0.2.0", - "configurations": [ - { - "name": "Launch Scheduled Workflow controller (Kind)", - "type": "go", - "request": "launch", - "mode": "debug", - "program": "${workspaceFolder}/backend/src/crd/controller/scheduledworkflow", - "env": { - "CRON_SCHEDULE_TIMEZONE": "UTC" - }, - "args": [ - "-namespace=kubeflow", - "-kubeconfig=${workspaceFolder}/kubeconfig_dev-pipelines-api", - "-mlPipelineAPIServerName=localhost" - ] - } - ] + "version": "0.2.0", + "configurations": [ + { + "name": "Launch Scheduled Workflow controller (Kind)", + "type": "go", + "request": "launch", + "mode": "debug", + "program": "${workspaceFolder}/backend/src/crd/controller/scheduledworkflow", + "env": { + "CRON_SCHEDULE_TIMEZONE": "UTC" + }, + "args": [ + "-namespace=kubeflow", + "-kubeconfig=${workspaceFolder}/kubeconfig_dev-pipelines-api", + "-mlPipelineAPIServerName=localhost" + ] + } + ] } ``` @@ -231,52 +261,55 @@ You may use the following VS Code `launch.json` file to run the API server which command to use Delve and the Driver image to use debug image built previously. VSCode configuration: + ```json { - "version": "0.2.0", - "configurations": [ - { - "name": "Launch API server (Kind) (Debug Driver)", - "type": "go", - "request": "launch", - "mode": "debug", - "program": "${workspaceFolder}/backend/src/apiserver", - "env": { - "POD_NAMESPACE": "kubeflow", - "DBCONFIG_MYSQLCONFIG_HOST": "localhost", - "MINIO_SERVICE_SERVICE_HOST": "localhost", - "MINIO_SERVICE_SERVICE_PORT": "9000", - "METADATA_GRPC_SERVICE_SERVICE_HOST": "localhost", - "METADATA_GRPC_SERVICE_SERVICE_PORT": "8080", - "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_HOST": "localhost", - "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_PORT": "8888", - "V2_LAUNCHER_IMAGE": "ghcr.io/kubeflow/kfp-launcher:master", - "V2_DRIVER_IMAGE": "kfp-driver:debug", - "V2_DRIVER_COMMAND": "dlv exec --listen=:2345 --headless=true --api-version=2 --log /bin/driver --" - } - } - ] + "version": "0.2.0", + "configurations": [ + { + "name": "Launch API server (Kind) (Debug Driver)", + "type": "go", + "request": "launch", + "mode": "debug", + "program": "${workspaceFolder}/backend/src/apiserver", + "env": { + "POD_NAMESPACE": "kubeflow", + "DBCONFIG_MYSQLCONFIG_HOST": "localhost", + "MINIO_SERVICE_SERVICE_HOST": "localhost", + "MINIO_SERVICE_SERVICE_PORT": "9000", + "METADATA_GRPC_SERVICE_SERVICE_HOST": "localhost", + "METADATA_GRPC_SERVICE_SERVICE_PORT": "8080", + "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_HOST": "localhost", + "ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_PORT": "8888", + "V2_LAUNCHER_IMAGE": "ghcr.io/kubeflow/kfp-launcher:master", + "V2_DRIVER_IMAGE": "kfp-driver:debug", + "V2_DRIVER_COMMAND": "dlv exec --listen=:2345 --headless=true --api-version=2 --log /bin/driver --" + } + } + ] } ``` GoLand configuration: + 1. Create a new Go Build configuration 2. Set **Run Kind** to Directory and set **Directory** to /backend/src/apiserver absolute path 3. Set the following environment variables - | Argument | Value | - |----------------------------------------------|-----------| - | POD_NAMESPACE | kubeflow | - | DBCONFIG_MYSQLCONFIG_HOST | localhost | - | MINIO_SERVICE_SERVICE_HOST | localhost | - | MINIO_SERVICE_SERVICE_PORT | 9000 | - | METADATA_GRPC_SERVICE_SERVICE_HOST | localhost | - | METADATA_GRPC_SERVICE_SERVICE_PORT | 8080 | - | ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_HOST | localhost | - | ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_PORT | 8888 | - | V2_LAUNCHER_IMAGE | localhost | - | V2_DRIVER_IMAGE | localhost | + | Argument | Value | + | -------------------------------------------- | ------------------------------------------------------------------------------------------- | + | POD_NAMESPACE | kubeflow | + | DBCONFIG_MYSQLCONFIG_HOST | localhost | + | MINIO_SERVICE_SERVICE_HOST | localhost | + | MINIO_SERVICE_SERVICE_PORT | 9000 | + | METADATA_GRPC_SERVICE_SERVICE_HOST | localhost | + | METADATA_GRPC_SERVICE_SERVICE_PORT | 8080 | + | ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_HOST | localhost | + | ML_PIPELINE_VISUALIZATIONSERVER_SERVICE_PORT | 8888 | + | V2_LAUNCHER_IMAGE | localhost | + | V2_DRIVER_IMAGE | localhost | | V2_DRIVER_COMMAND | dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec /bin/driver -- | + 4. Set the following program arguments: --config ./backend/src/apiserver/config -logtostderr=true --sampleconfig ./backend/src/apiserver/config/test_sample_config.json #### Starting a Remote Debug Session @@ -300,24 +333,26 @@ Set a breakpoint on the Driver code in VS Code. Then remotely connect to the Del Code `launch.json` file: VSCode configuration: + ```json { - "version": "0.2.0", - "configurations": [ - { - "name": "Connect to remote driver", - "type": "go", - "request": "attach", - "mode": "remote", - "remotePath": "/go/src/github.com/kubeflow/pipelines", - "port": 2345, - "host": "127.0.0.1", - } - ] + "version": "0.2.0", + "configurations": [ + { + "name": "Connect to remote driver", + "type": "go", + "request": "attach", + "mode": "remote", + "remotePath": "/go/src/github.com/kubeflow/pipelines", + "port": 2345, + "host": "127.0.0.1" + } + ] } ``` GoLand configuration: + 1. Create a new Go Remote configuration and title it "Delve debug session" 2. Set **Host** to localhost 3. Set **Port** to 2345 @@ -347,6 +382,7 @@ kind delete clusters dev-pipelines-api ``` ## Contributing + ### Code Style Backend codebase follows the [Google's Go Style Guide](https://google.github.io/styleguide/go/). Please, take time to get familiar with the [best practices](https://google.github.io/styleguide/go/best-practices). It is not intended to be exhaustive, but it often helps minimizing guesswork among developers and keep codebase uniform and consistent. diff --git a/backend/helm-charts/postgresql/Chart.yaml b/backend/helm-charts/postgresql/Chart.yaml new file mode 100644 index 00000000000..681f5cdb6d5 --- /dev/null +++ b/backend/helm-charts/postgresql/Chart.yaml @@ -0,0 +1,4 @@ +apiVersion: v2 +name: postgresql +version: 0.1.0 +description: A Helm chart for local dev PostgreSQL (GSoC testing) diff --git a/backend/helm-charts/postgresql/README.md b/backend/helm-charts/postgresql/README.md new file mode 100644 index 00000000000..9c5bbf63a9f --- /dev/null +++ b/backend/helm-charts/postgresql/README.md @@ -0,0 +1,86 @@ +# PostgreSQL Helm Deployment for Kubeflow Pipelines + +This directory contains the Helm values file to deploy a PostgreSQL instance for Kubeflow Pipelines using the Bitnami PostgreSQL Chart. + +## Prerequisites + +- **Helm**: version 3.x installed (`helm version` should show v3). +- **Kubernetes** cluster with the `kubeflow` namespace created (or adjust commands to match your target namespace). + +## Installation Steps + +1. **Add the Bitnami chart repository** + + ```bash + helm repo add bitnami https://charts.bitnami.com/bitnami + ``` + +2. **Update local chart repository cache** + + ```bash + helm repo update + ``` + +3. **Install or upgrade the PostgreSQL release** + + ```bash + helm upgrade --install kfp-postgres bitnami/postgresql \ + --namespace kubeflow \ + -f helm-charts/postgresql/values.yaml + ``` + + - `kfp-postgres` is the release name. + - `bitnami/postgresql` refers to the community chart. + - `-f values.yaml` applies your custom parameters. + +## values.yaml Highlights + +Customize the following key sections in `values.yaml` as needed: + +```yaml +global: + postgresql: + postgresqlPassword: # PostgreSQL superuser password + postgresqlDatabase: kubeflow # Default database name + postgresqlUsername: kubeflow # Default database user +persistence: + enabled: true + size: 10Gi # PVC size + storageClass: standard # StorageClass for persistent volume +service: + port: 5432 # PostgreSQL service port +``` + +## Verifying Deployment + +1. **Check resources** + + ```bash + kubectl get pods,svc,pvc -n kubeflow + ``` + +2. **Port-forward and connect with psql** + ```bash + kubectl port-forward svc/kfp-postgres 5432:5432 -n kubeflow + psql postgres://kubeflow:@localhost:5432/kubeflow + ``` + +## Upgrading & Uninstalling + +- **Upgrade with new values** + + ```bash + helm upgrade kfp-postgres bitnami/postgresql \ + --namespace kubeflow \ + -f helm-charts/postgresql/values.yaml + ``` + +- **Uninstall release** + ```bash + helm uninstall kfp-postgres --namespace kubeflow + ``` + +## Notes + +- You can override individual settings via `--set` flags instead of editing `values.yaml`. +- To use a different namespace, modify the `--namespace` flag accordingly. diff --git a/backend/helm-charts/postgresql/values.yaml b/backend/helm-charts/postgresql/values.yaml new file mode 100644 index 00000000000..98376ba2e59 --- /dev/null +++ b/backend/helm-charts/postgresql/values.yaml @@ -0,0 +1,12 @@ +auth: + postgresPassword: kubeflow # PostgreSQL superuser password + database: kubeflow # Default database name + username: kubeflow # Default database user + +persistence: + enabled: true + size: 10Gi # PVC size + storageClass: standard # StorageClass for persistent volume + +service: + port: 5432 # PostgreSQL service port diff --git a/backend/scripts/start-dev-db.sh b/backend/scripts/start-dev-db.sh new file mode 100755 index 00000000000..17d987014ba --- /dev/null +++ b/backend/scripts/start-dev-db.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# Use DB_TYPE environment variable to choose the database (default: postgres) +DB_TYPE="${DB_TYPE:-postgres}" +# Allow overriding Helm release name; default is dev--db +RELEASE_NAME="${RELEASE_NAME:-dev-${DB_TYPE}-db}" +# Use DB_NAMESPACE to choose the namespace; default is -test +DB_NAMESPACE="${DB_NAMESPACE:-${DB_TYPE}-test}" +# For example: DB_NAMESPACE=kubeflow DB_TYPE=postgres make start-dev-db + +if [ "$DB_TYPE" = "postgres" ]; then + # Use Bitnami PostgreSQL chart for CI-friendly deployment + helm upgrade --install ${RELEASE_NAME} bitnami/postgresql \ + --namespace ${DB_NAMESPACE} --create-namespace \ + -f ./helm-charts/postgresql/values.yaml +else + # Example: deploy MySQL using the Bitnami chart + helm upgrade --install ${RELEASE_NAME} bitnami/mysql \ + --namespace ${DB_NAMESPACE} --create-namespace +fi + +# Waiting for the service to be ready +kubectl -n ${DB_NAMESPACE} wait --for=condition=Ready pod -l app.kubernetes.io/instance=${RELEASE_NAME} --timeout=2m +echo "✅ ${DB_TYPE} is ready in namespace ${DB_NAMESPACE}" diff --git a/backend/scripts/stop-dev-db.sh b/backend/scripts/stop-dev-db.sh new file mode 100755 index 00000000000..652f5db00b1 --- /dev/null +++ b/backend/scripts/stop-dev-db.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +set -euo pipefail + +DB_TYPE="${DB_TYPE:-postgres}" +RELEASE_NAME="dev-${DB_TYPE}-db" +NAMESPACE="${DB_TYPE}-test" + +helm uninstall "${RELEASE_NAME}" --namespace "${NAMESPACE}" || true +kubectl delete namespace "${NAMESPACE}" --ignore-not-found + +echo "✅ Stopped and cleaned up ${DB_TYPE} in namespace ${NAMESPACE}" diff --git a/backend/src/apiserver/client/sql.go b/backend/src/apiserver/client/sql.go index 3a111eb3afc..74a81770bb1 100644 --- a/backend/src/apiserver/client/sql.go +++ b/backend/src/apiserver/client/sql.go @@ -22,11 +22,9 @@ import ( ) const ( - MYSQL_TEXT_FORMAT string = "longtext not null" - MYSQL_TEXT_FORMAT_NULL string = "longtext" - MYSQL_EXIST_ERROR string = "database exists" + //nolint:staticcheck // allow legacy ALL_CAPS constant name + MYSQL_EXIST_ERROR string = "database exists" - PGX_TEXT_FORMAT string = "text" PGX_EXIST_ERROR string = "already exists" ) diff --git a/backend/src/apiserver/client_manager/client_manager.go b/backend/src/apiserver/client_manager/client_manager.go index 0d2f9e5d3a8..6aa56928931 100644 --- a/backend/src/apiserver/client_manager/client_manager.go +++ b/backend/src/apiserver/client_manager/client_manager.go @@ -24,10 +24,8 @@ import ( "time" "github.com/cenkalti/backoff" - "github.com/go-sql-driver/mysql" + mysqlStd "github.com/go-sql-driver/mysql" "github.com/golang/glog" - "github.com/jinzhu/gorm" - _ "github.com/jinzhu/gorm/dialects/sqlite" "github.com/kubeflow/pipelines/backend/src/apiserver/archive" "github.com/kubeflow/pipelines/backend/src/apiserver/auth" "github.com/kubeflow/pipelines/backend/src/apiserver/client" @@ -37,6 +35,9 @@ import ( "github.com/kubeflow/pipelines/backend/src/common/util" k8sapi "github.com/kubeflow/pipelines/backend/src/crd/kubernetes/v2beta1" "github.com/minio/minio-go/v7" + "gorm.io/driver/mysql" + "gorm.io/driver/postgres" + "gorm.io/gorm" "k8s.io/apimachinery/pkg/runtime" "sigs.k8s.io/controller-runtime/pkg/cache" ctrlclient "sigs.k8s.io/controller-runtime/pkg/client" @@ -320,29 +321,80 @@ func (c *ClientManager) Close() { c.db.Close() } +// isAllowedTable returns true only for tables we know we need to alter. +func isAllowedTable(name string) bool { + allowed := map[string]bool{ + "pipeline": true, + "pipeline_versions": true, + } + return allowed[name] +} + // addDisplayNameColumn adds a DisplayName column to the given table with a default value of Name. -// It panics if this fails. -func addDisplayNameColumn(db *gorm.DB, scope *gorm.Scope, quotedTableName string, driverName string) []error { +// It returns an error if this fails. +func addDisplayNameColumn(db *gorm.DB, quotedTableName string, driverName string) error { + if !isAllowedTable(quotedTableName) { + return fmt.Errorf("table %q is not allowed for DisplayName migration", quotedTableName) + } + glog.Info("Adding DisplayName column to " + quotedTableName) - switch driverName { - case "mysql": - scope.Raw( - "ALTER TABLE " + quotedTableName + " ADD COLUMN DisplayName VARCHAR(255) NULL;", - ).Exec() - scope.Raw("UPDATE " + quotedTableName + " SET DisplayName = Name").Exec() - scope.Raw("ALTER TABLE " + quotedTableName + " MODIFY COLUMN DisplayName VARCHAR(255) NOT NULL").Exec() - case "pgx": - scope.Raw( - "ALTER TABLE " + quotedTableName + " ADD COLUMN DisplayName VARCHAR(255);", - ).Exec() - scope.Raw("UPDATE " + quotedTableName + " SET DisplayName = Name").Exec() - scope.Raw("ALTER TABLE " + quotedTableName + " ALTER COLUMN DisplayName SET NOT NULL").Exec() - } + err := db.Transaction(func(tx *gorm.DB) error { + var stmts []string + switch driverName { + case "mysql": + stmts = []string{ + "ALTER TABLE " + quotedTableName + " ADD COLUMN DisplayName VARCHAR(255);", + "UPDATE " + quotedTableName + " SET DisplayName = Name;", + "ALTER TABLE " + quotedTableName + " MODIFY COLUMN DisplayName VARCHAR(255) NOT NULL;", + } + case "pgx": + stmts = []string{ + "ALTER TABLE " + quotedTableName + " ADD COLUMN DisplayName VARCHAR(255);", + "UPDATE " + quotedTableName + " SET DisplayName = Name;", + "ALTER TABLE " + quotedTableName + " ALTER COLUMN DisplayName SET NOT NULL;", + } + default: + return fmt.Errorf("unsupported driver: %s", driverName) + } - scope.CommitOrRollback() + for _, stmt := range stmts { + if err := tx.Exec(stmt).Error; err != nil { + return err + } + } + return nil + }) + if err != nil { + return err + } + return nil +} - return db.GetErrors() +// EnsureUniqueCompositeIndex enforces a composite unique index on a given model. +// It checks both the existence of a database-level constraint and the corresponding index. +// This function serves as a safety fallback in case AutoMigrate fails to create the index +// due to GORM's conservative merge strategy or legacy residual indexes from earlier versions. +// +// This is particularly critical when upgrading from GORM v1 to v2, as v1's tag-based index +// declarations were often silently ignored or misapplied, especially when partial indexes +// (e.g., on "Name" only) were created before a composite index (e.g., on "Name"+"Namespace"). +// +// Arguments: +// - db: the active GORM database connection +// - model: the target model (passed by reference) +// - indexName: the logical name of the composite unique index as defined in the model's GORM tag +func EnsureUniqueCompositeIndex(db *gorm.DB, model interface{}, indexName string) { + if !db.Migrator().HasConstraint(model, indexName) { + if err := db.Migrator().CreateConstraint(model, indexName); err != nil { + glog.Fatalf("Failed to create constraint %s. Error: %v", indexName, err) + } + } + if !db.Migrator().HasIndex(model, indexName) { + if err := db.Migrator().CreateIndex(model, indexName); err != nil { + glog.Fatalf("Failed to create index %s. Error: %v", indexName, err) + } + } } func InitDBClient(initConnectionTimeout time.Duration) *storage.DB { @@ -352,9 +404,19 @@ func InitDBClient(initConnectionTimeout time.Duration) *storage.DB { driverName := common.GetStringConfig("DBDriverName") arg := initDBDriver(driverName, initConnectionTimeout) + var dialector gorm.Dialector + switch driverName { + case "mysql": + dialector = mysql.Open(arg) + case "pgx": + dialector = postgres.Open(arg) + default: + glog.Fatalf("Unsupported driver %v", driverName) + } + // db is safe for concurrent use by multiple goroutines // and maintains its own pool of idle connections. - db, err := gorm.Open(driverName, arg) + db, err := gorm.Open(dialector, &gorm.Config{}) util.TerminateIfError(err) // If pipeline_versions table is introduced into DB for the first time, @@ -369,28 +431,38 @@ func InitDBClient(initConnectionTimeout time.Duration) *storage.DB { } } - if db.HasTable(&model.Pipeline{}) { - scope := db.NewScope(&model.Pipeline{}) - if !scope.Dialect().HasColumn(scope.TableName(), "DisplayName") { - errs := addDisplayNameColumn(db, scope, scope.QuotedTableName(), driverName) - if len(errs) > 0 { - glog.Fatalf("Failed to add DisplayName column to the %s table. Error(s): %v", scope.TableName(), errs) + // Use GORM Statement to retrieve the fully quoted table name, + // respecting the model's TableName() override and DB dialect quoting rules. + if db.Migrator().HasTable(&model.Pipeline{}) { + stmt := &gorm.Statement{DB: db} + if err := stmt.Parse(&model.Pipeline{}); err != nil { + glog.Fatalf("Failed to parse Pipeline model for table quoting: %v", err) + } + quotedTableName := stmt.Quote(stmt.Table) + hasColumn := db.Migrator().HasColumn(&model.Pipeline{}, "DisplayName") + if !hasColumn { + if err := addDisplayNameColumn(db, quotedTableName, driverName); err != nil { + glog.Fatalf("Failed to add DisplayName column to the %s table. Error: %v", quotedTableName, err) } } } - if db.HasTable(&model.PipelineVersion{}) { - scope := db.NewScope(&model.PipelineVersion{}) - if !scope.Dialect().HasColumn(scope.TableName(), "DisplayName") { - errs := addDisplayNameColumn(db, scope, scope.QuotedTableName(), driverName) - if len(errs) > 0 { - glog.Fatalf("Failed to add DisplayName column to the %s table. Error(s): %v", scope.TableName(), errs) + if db.Migrator().HasTable(&model.PipelineVersion{}) { + stmt := &gorm.Statement{DB: db} + if err := stmt.Parse(&model.PipelineVersion{}); err != nil { + glog.Fatalf("Failed to parse PipelineVersion model for table quoting: %v", err) + } + quotedTableName := stmt.Quote(stmt.Table) + hasColumn := db.Migrator().HasColumn(&model.PipelineVersion{}, "DisplayName") + if !hasColumn { + if err := addDisplayNameColumn(db, quotedTableName, driverName); err != nil { + glog.Fatalf("Failed to add DisplayName column to the %s table. Error: %v", quotedTableName, err) } } } // Create table - response := db.AutoMigrate( + err = db.AutoMigrate( &model.DBStatus{}, &model.DefaultExperiment{}, &model.Experiment{}, @@ -403,102 +475,45 @@ func InitDBClient(initConnectionTimeout time.Duration) *storage.DB { &model.ResourceReference{}, ) - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to initialize the databases. Error: %s", response.Error) - } - - var textFormat string - switch driverName { - case "mysql": - textFormat = client.MYSQL_TEXT_FORMAT - case "pgx": - textFormat = client.PGX_TEXT_FORMAT - default: - glog.Fatalf("Unsupported database driver %s, please use `mysql` for MySQL, or `pgx` for PostgreSQL.", driverName) - } - - response = db.Model(&model.Experiment{}).RemoveIndex("Name") - if response.Error != nil { - glog.Fatalf("Failed to drop unique key on experiment name. Error: %s", response.Error) - } - - response = db.Model(&model.Pipeline{}).RemoveIndex("Name") - if response.Error != nil { - glog.Fatalf("Failed to drop unique key on pipeline name. Error: %s", response.Error) - } - - response = db.Model(&model.ResourceReference{}).ModifyColumn("Payload", textFormat) - if response.Error != nil { - glog.Fatalf("Failed to update the resource reference payload type. Error: %s", response.Error) - } - - response = db.Model(&model.Run{}).AddIndex("experimentuuid_createatinsec", "ExperimentUUID", "CreatedAtInSec") - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create index experimentuuid_createatinsec on run_details. Error: %s", response.Error) + if ignoreAlreadyExistError(driverName, err) != nil { + glog.Fatalf("Failed to initialize the databases. Error: %s", err.Error) } - response = db.Model(&model.Run{}).AddIndex("experimentuuid_conditions_finishedatinsec", "ExperimentUUID", "Conditions", "FinishedAtInSec") - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create index experimentuuid_conditions_finishedatinsec on run_details. Error: %s", response.Error) - } - - response = db.Model(&model.Run{}).AddIndex("namespace_createatinsec", "Namespace", "CreatedAtInSec") - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create index namespace_createatinsec on run_details. Error: %s", response.Error) - } - - response = db.Model(&model.Run{}).AddIndex("namespace_conditions_finishedatinsec", "Namespace", "Conditions", "FinishedAtInSec") - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create index namespace_conditions_finishedatinsec on run_details. Error: %s", response.Error) + err = db.Migrator().DropIndex(&model.Experiment{}, "Name") + if err != nil { + glog.Fatalf("Failed to drop unique key on experiment name. Error: %s", err) } - response = db.Model(&model.Pipeline{}).AddUniqueIndex("name_namespace_index", "Name", "Namespace") - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create index name_namespace_index on run_details. Error: %s", response.Error) + err = db.Migrator().DropIndex(&model.Pipeline{}, "Name") + if err != nil { + glog.Fatalf("Failed to drop unique key on pipeline name. Error: %s", err) } - switch driverName { - case "pgx": - response = db.Model(&model.RunMetric{}). - AddForeignKey("\"RunUUID\"", "run_details(\"UUID\")", "CASCADE" /* onDelete */, "CASCADE" /* onUpdate */) - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create a foreign key for RunUUID in run_metrics table. Error: %s", response.Error) - } - response = db.Model(&model.PipelineVersion{}). - AddForeignKey("\"PipelineId\"", "pipelines(\"UUID\")", "CASCADE" /* onDelete */, "CASCADE" /* onUpdate */) - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create a foreign key for PipelineId in pipeline_versions table. Error: %s", response.Error) - } - response = db.Model(&model.Task{}). - AddForeignKey("\"RunUUID\"", "run_details(\"UUID\")", "CASCADE" /* onDelete */, "CASCADE" /* onUpdate */) - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create a foreign key for RunUUID in task table. Error: %s", response.Error) - } - case "mysql": - response = db.Model(&model.RunMetric{}). - AddForeignKey("RunUUID", "run_details(UUID)", "CASCADE" /* onDelete */, "CASCADE" /* onUpdate */) - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create a foreign key for RunUUID in run_metrics table. Error: %s", response.Error) - } - response = db.Model(&model.PipelineVersion{}). - AddForeignKey("PipelineId", "pipelines(UUID)", "CASCADE" /* onDelete */, "CASCADE" /* onUpdate */) - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create a foreign key for PipelineId in pipeline_versions table. Error: %s", response.Error) - } - response = db.Model(&model.Task{}). - AddForeignKey("RunUUID", "run_details(UUID)", "CASCADE" /* onDelete */, "CASCADE" /* onUpdate */) - if ignoreAlreadyExistError(driverName, response.Error) != nil { - glog.Fatalf("Failed to create a foreign key for RunUUID in task table. Error: %s", response.Error) - } - - // This is a workaround because AutoMigration does not detect that the column went from not null to nullable. - response = db.Model(&model.Job{}).ModifyColumn("WorkflowSpecManifest", client.MYSQL_TEXT_FORMAT_NULL) - if response.Error != nil { - glog.Fatalf("Failed to make the WorkflowSpecManifest column nullable on jobs. Error: %s", response.Error) - } - default: - glog.Fatalf("Driver %v is not supported, use \"mysql\" for MySQL, or \"pgx\" for PostgreSQL", driverName) - } + err = db.Migrator().AlterColumn(&model.ResourceReference{}, "Payload") + if err != nil { + glog.Fatalf("Failed to update the resource reference payload type. Error: %s", err) + } + + // Manual AddForeignKey() and AddIndex() calls have been removed. + // Both Methods are GORM v1 legacy which no longer exist in GORM v2. + // Foreign key constraints are now defined and managed by GORM via struct tags 'constraint' and 'index'. + // This ensures a single source of truth for schema definitions and avoids duplicate or out-of-sync DDL. + + // In both GORM v1 and v2, AutoMigrate is a non-destructive operation and + // will *not* overwrite pre-existing index structures. For example, if a + // single-column index (e.g., on "Name") already exists, the creation of a + // composite unique index may silently fail. + // + // Historically, GORM v1 addressed this by explicitly calling AddUniqueIndex() + // to enforce the intended constraint. Since AddUniqueIndex() is deprecated in + // GORM v2, we now use EnsureUniqueCompositeIndex(), which leverages the v2 + // Migrator API to ensure proper constraint creation and prevent future data + // integrity issues. + // + // See: https://gorm.io/docs/migration.html#Auto-Migration + EnsureUniqueCompositeIndex(db, &model.Pipeline{}, "namespace_name") + EnsureUniqueCompositeIndex(db, &model.Experiment{}, "idx_name_namespace") + EnsureUniqueCompositeIndex(db, &model.PipelineVersion{}, "idx_pipelineid_name") // Data backfill for pipeline_versions if this is the first time for // pipeline_versions to enter mlpipeline DB. @@ -510,9 +525,8 @@ func InitDBClient(initConnectionTimeout time.Duration) *storage.DB { glog.Fatalf("Failed to backfill experiment UUID in run_details table: %s", err) } - response = db.Model(&model.Pipeline{}).ModifyColumn("Description", textFormat) - if response.Error != nil { - glog.Fatalf("Failed to update pipeline description type. Error: %s", response.Error) + if err := db.Migrator().AlterColumn(&model.Pipeline{}, "Description"); err != nil { + glog.Fatalf("Failed to update pipeline description type. Error: %s", err) } // Because PostgreSQL was supported later, there's no need to delete the relic index @@ -530,8 +544,11 @@ func InitDBClient(initConnectionTimeout time.Duration) *storage.DB { } defer rows.Close() } - - return storage.NewDB(db.DB(), storage.NewMySQLDialect()) + newdb, err := db.DB() + if err != nil { + glog.Fatalf("Failed to retrieve *sql.DB from gorm.DB. Error: %v", err) + } + return storage.NewDB(newdb, storage.NewMySQLDialect()) } // Initializes Database driver. Use `driverName` to indicate which type of DB to use: @@ -539,7 +556,7 @@ func InitDBClient(initConnectionTimeout time.Duration) *storage.DB { // 2) "pgx" for PostgreSQL func initDBDriver(driverName string, initConnectionTimeout time.Duration) string { var sqlConfig, dbName string - var mysqlConfig *mysql.Config + var mysqlConfig *mysqlStd.Config switch driverName { case "mysql": mysqlConfig = client.CreateMySQLConfig( @@ -568,6 +585,12 @@ func initDBDriver(driverName string, initConnectionTimeout time.Duration) string var db *sql.DB var err error + + // lyk add this for debugging + glog.Infof("Attempting to connect to Postgres at %s, dsn: %s", + common.GetStringConfig(postgresHost), + sqlConfig) + operation := func() error { db, err = sql.Open(driverName, sqlConfig) if err != nil { @@ -620,6 +643,8 @@ func initDBDriver(driverName string, initConnectionTimeout time.Duration) string default: glog.Fatalf("Driver %v is not supported, use \"mysql\" for MySQL, or \"pgx\" for PostgreSQL", driverName) } + // lyk add this for debugging + glog.Infof("Final Postgres DSN: %s", sqlConfig) return sqlConfig } @@ -719,7 +744,11 @@ func initPipelineVersionsFromPipelines(db *gorm.DB) { func backfillExperimentIDToRunTable(db *gorm.DB) error { // check if there is any row in the run table has experiment ID being empty - rows, err := db.CommonDB().Query("SELECT \"ExperimentUUID\" FROM run_details WHERE \"ExperimentUUID\" = '' LIMIT 1") + sqlDB, err := db.DB() + if err != nil { + return err + } + rows, err := sqlDB.Query("SELECT \"ExperimentUUID\" FROM run_details WHERE \"ExperimentUUID\" = '' LIMIT 1") if err != nil { return err } @@ -733,7 +762,7 @@ func backfillExperimentIDToRunTable(db *gorm.DB) error { return nil } - _, err = db.CommonDB().Exec(` + _, err = sqlDB.Exec(` UPDATE run_details, resource_references SET diff --git a/backend/src/apiserver/client_manager/client_manager_test.go b/backend/src/apiserver/client_manager/client_manager_test.go new file mode 100644 index 00000000000..d65eb6751da --- /dev/null +++ b/backend/src/apiserver/client_manager/client_manager_test.go @@ -0,0 +1,12 @@ +package clientmanager + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestIsAllowedTable(t *testing.T) { + assert.True(t, isAllowedTable("pipeline")) + assert.False(t, isAllowedTable("users")) +} diff --git a/backend/src/apiserver/model/db_status.go b/backend/src/apiserver/model/db_status.go index a4db08aeb1a..9fccda3915f 100644 --- a/backend/src/apiserver/model/db_status.go +++ b/backend/src/apiserver/model/db_status.go @@ -15,5 +15,5 @@ package model type DBStatus struct { - HaveSamplesLoaded bool `gorm:"column:HaveSamplesLoaded; not null; primary_key;"` + HaveSamplesLoaded bool `gorm:"column:HaveSamplesLoaded; not null; primaryKey;"` } diff --git a/backend/src/apiserver/model/default_experiment.go b/backend/src/apiserver/model/default_experiment.go index 76de3b9d2c3..1417afbebf5 100644 --- a/backend/src/apiserver/model/default_experiment.go +++ b/backend/src/apiserver/model/default_experiment.go @@ -15,5 +15,5 @@ package model type DefaultExperiment struct { - DefaultExperimentId string `gorm:"column:DefaultExperimentId; not null; primary_key;"` + DefaultExperimentId string `gorm:"column:DefaultExperimentId; not null; primaryKey;"` } diff --git a/backend/src/apiserver/model/experiment.go b/backend/src/apiserver/model/experiment.go index 0adde8e8f42..1d11226839f 100644 --- a/backend/src/apiserver/model/experiment.go +++ b/backend/src/apiserver/model/experiment.go @@ -15,13 +15,15 @@ package model type Experiment struct { - UUID string `gorm:"column:UUID; not null; primary_key;"` - Name string `gorm:"column:Name; not null; unique_index:idx_name_namespace;"` - Description string `gorm:"column:Description; not null;"` - CreatedAtInSec int64 `gorm:"column:CreatedAtInSec; not null;"` - LastRunCreatedAtInSec int64 `gorm:"column:LastRunCreatedAtInSec; not null;"` - Namespace string `gorm:"column:Namespace; not null; unique_index:idx_name_namespace;"` - StorageState StorageState `gorm:"column:StorageState; not null;"` + UUID string `gorm:"column:UUID; not null; primaryKey;"` + // For details on type lengths and index safety, refer to comments in the Pipeline struct. + Name string `gorm:"column:Name; not null; uniqueIndex:idx_name_namespace; type:varchar(128);"` + Description string `gorm:"column:Description; not null;"` + CreatedAtInSec int64 `gorm:"column:CreatedAtInSec; not null;"` + LastRunCreatedAtInSec int64 `gorm:"column:LastRunCreatedAtInSec; not null;"` + // For details on type lengths and index safety, refer to comments in the Pipeline struct. + Namespace string `gorm:"column:Namespace; not null; uniqueIndex:idx_name_namespace; type:varchar(63)"` + StorageState StorageState `gorm:"column:StorageState; not null;"` } // Note: Experiment.StorageState can have values: "STORAGE_STATE_UNSPECIFIED", "AVAILABLE" or "ARCHIVED" diff --git a/backend/src/apiserver/model/job.go b/backend/src/apiserver/model/job.go index 3871e050b7e..83bff73a2d9 100644 --- a/backend/src/apiserver/model/job.go +++ b/backend/src/apiserver/model/job.go @@ -86,7 +86,7 @@ func (s StatusState) ToV2() StatusState { } type Job struct { - UUID string `gorm:"column:UUID; not null; primary_key;"` + UUID string `gorm:"column:UUID; not null; primaryKey;"` DisplayName string `gorm:"column:DisplayName; not null;"` /* The name that user provides. Can contain special characters*/ K8SName string `gorm:"column:Name; not null;"` /* The name of the K8s resource. Follow regex '[a-z0-9]([-a-z0-9]*[a-z0-9])?'*/ Namespace string `gorm:"column:Namespace; not null;"` @@ -98,9 +98,15 @@ type Job struct { UpdatedAtInSec int64 `gorm:"column:UpdatedAtInSec; default:0;"` Enabled bool `gorm:"column:Enabled; not null;"` ExperimentId string `gorm:"column:ExperimentUUID; not null;"` + // ResourceReferences are deprecated. Use Namespace, ExperimentId // PipelineSpec.PipelineId, PipelineSpec.PipelineVersionId - ResourceReferences []*ResourceReference + + // ResourceReferences are logical links to other KFP resources. + // These references are not unidirectional (i.e., no clear ownership), + // so we avoid declaring a foreign key. The field is excluded from GORM mapping + // to prevent auto-inferred relations and schema validation errors. + ResourceReferences []*ResourceReference `gorm:"-"` Trigger PipelineSpec Conditions string `gorm:"column:Conditions; not null;"` diff --git a/backend/src/apiserver/model/pipeline.go b/backend/src/apiserver/model/pipeline.go index 3760e7b6cda..8bfd72e0867 100644 --- a/backend/src/apiserver/model/pipeline.go +++ b/backend/src/apiserver/model/pipeline.go @@ -31,17 +31,27 @@ const ( ) type Pipeline struct { - UUID string `gorm:"column:UUID; not null; primary_key;"` + UUID string `gorm:"column:UUID; not null; primaryKey;"` CreatedAtInSec int64 `gorm:"column:CreatedAtInSec; not null;"` - Name string `gorm:"column:Name; not null; unique_index:namespace_name;"` // Index improves performance of the List ang Get queries - DisplayName string `gorm:"column:DisplayName; not null"` - Description string `gorm:"column:Description; size:65535;"` // Same as below, set size to large number so it will be stored as longtext + // Name is limited to varchar(128) to ensure the composite index (Namespace, Name) + // stays within MySQL’s 767-byte prefix limit for indexable columns. + // MySQL uses utf8mb4 encoding by default, where each character can take up to 4 bytes. + // In the worst case: 63 (namespace) * 4 + 128 (name) * 4 = 764 bytes ≤ 767 bytes. + // https://dev.mysql.com/doc/refman/8.4/en/column-indexes.html + // Even though Namespace rarely uses its full 63-character capacity in practice, + // MySQL calculates index length based on declared size, not actual content. + // Therefore, keeping Name at varchar(128) is a safe upper bound. + Name string `gorm:"column:Name; not null; uniqueIndex:namespace_name; type:varchar(128);"` // Index improves performance of the List and Get queries + DisplayName string `gorm:"column:DisplayName; not null"` + Description string `gorm:"column:Description; type:longtext;"` // this column will be stored as longtext; // TODO(gkcalat): this is deprecated. Consider removing and adding data migration logic at the server startup. - Parameters string `gorm:"column:Parameters; size:65535;"` + Parameters string `gorm:"column:Parameters; type:longtext;"` // this column will be stored as longtext; Status PipelineStatus `gorm:"column:Status; not null;"` // TODO(gkcalat): this is deprecated. Consider removing and adding data migration logic at the server startup. DefaultVersionId string `gorm:"column:DefaultVersionId;"` // deprecated - Namespace string `gorm:"column:Namespace; unique_index:namespace_name; size:63;"` + // Namespace is restricted to varchar(63) due to Kubernetes' naming constraints: + // https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-label-names + Namespace string `gorm:"column:Namespace; uniqueIndex:namespace_name; type:varchar(63);"` } func (p Pipeline) GetValueOfPrimaryKey() string { @@ -83,6 +93,11 @@ func (p *Pipeline) GetModelName() string { return "pipelines" } +// TableName overrides GORM's table name inference. +func (Pipeline) TableName() string { + return "pipelines" +} + func (p *Pipeline) GetField(name string) (string, bool) { if field, ok := pipelineAPIToModelFieldMap[name]; ok { return field, true diff --git a/backend/src/apiserver/model/pipeline_spec.go b/backend/src/apiserver/model/pipeline_spec.go index 2f92c5c68bd..f103c835244 100644 --- a/backend/src/apiserver/model/pipeline_spec.go +++ b/backend/src/apiserver/model/pipeline_spec.go @@ -27,21 +27,21 @@ type PipelineSpec struct { PipelineName string `gorm:"column:PipelineName; not null;"` // Pipeline YAML definition. This is the pipeline interface for creating a pipeline. - // Set size to 65535 so it will be stored as longtext. - // https://dev.mysql.com/doc/refman/8.0/en/column-count-limit.html - // TODO(gkcalat): consider increasing the size limit > 32MB (<4GB for MySQL, and <1GB for PostgreSQL). - PipelineSpecManifest string `gorm:"column:PipelineSpecManifest; size:33554432;"` + // Stored as longtext to support large manifests (up to 4GB in MySQL). + // https://dev.mysql.com/doc/refman/8.0/en/blob.html + // TODO(kaikaila): consider enforcing a soft limit if needed for performance. + PipelineSpecManifest string `gorm:"column:PipelineSpecManifest; type:longtext;"` // Argo workflow YAML definition. This is the Argo Spec converted from Pipeline YAML. // This is deprecated. Use the pipeline ID, pipeline version ID, or pipeline spec manifest. - WorkflowSpecManifest string `gorm:"column:WorkflowSpecManifest; size:33554432;"` + WorkflowSpecManifest string `gorm:"column:WorkflowSpecManifest; type:longtext;"` // Store parameters key-value pairs as serialized string. // This field is only used for V1 API. For V2, use the `Parameters` field in RuntimeConfig. // At most one of the fields `Parameters` and `RuntimeConfig` can be non-empty // This string stores an array of map[string]value. For example: // {"param1": Value1} will be stored as [{"name": "param1", "value":"value1"}]. - Parameters string `gorm:"column:Parameters; size:65535;"` + Parameters string `gorm:"column:Parameters; type:longtext;"` // Runtime config of the pipeline, only used for v2 template in API v1beta1 API. RuntimeConfig diff --git a/backend/src/apiserver/model/pipeline_version.go b/backend/src/apiserver/model/pipeline_version.go index c5ef19cd453..b981defc8f2 100644 --- a/backend/src/apiserver/model/pipeline_version.go +++ b/backend/src/apiserver/model/pipeline_version.go @@ -29,23 +29,28 @@ const ( ) type PipelineVersion struct { - UUID string `gorm:"column:UUID; not null; primary_key;"` + UUID string `gorm:"column:UUID; not null; primaryKey;"` CreatedAtInSec int64 `gorm:"column:CreatedAtInSec; not null; index;"` - Name string `gorm:"column:Name; not null; unique_index:idx_pipelineid_name;"` - DisplayName string `gorm:"column:DisplayName; not null"` + // Explicitly specify varchar(127) + // so that the combined index (PipelineId + Name) does not exceed 767 bytes in utf8mb4, + // For details on type lengths and index safety, refer to comments in the Pipeline struct. + Name string `gorm:"column:Name; not null; type:varchar(127); uniqueIndex:idx_pipelineid_name;"` + DisplayName string `gorm:"column:DisplayName; not null"` // TODO(gkcalat): this is deprecated. Consider removing and adding data migration logic at the server startup. - Parameters string `gorm:"column:Parameters; not null; size:65535;"` // deprecated + Parameters string `gorm:"column:Parameters; not null; type:longtext;"` // deprecated // PipelineVersion belongs to Pipeline. If a pipeline with a specific UUID // is deleted from Pipeline table, all this pipeline's versions will be // deleted from PipelineVersion table. - PipelineId string `gorm:"column:PipelineId; not null; index; unique_index:idx_pipelineid_name;"` - // Pipeline *Pipeline `gorm:"foreignKey:PipelineId; constraint:OnUpdate:CASCADE,OnDelete:SET NULL;"` - Status PipelineVersionStatus `gorm:"column:Status; not null;"` + // Explicitly specify varchar(64). Refer to Name field comments for details. + // nolint:staticcheck // [ST1003] Field name matches upstream legacy naming + PipelineId string `gorm:"column:PipelineId; not null; index; uniqueIndex:idx_pipelineid_name;type:varchar(64)"` + Pipeline Pipeline `gorm:"foreignKey:PipelineId;references:UUID;constraint:OnDelete:CASCADE,OnUpdate:CASCADE"` // This 'belongs to' relation replaces the legacy AddForeignKey constraint previously defined in client_manager.go + Status PipelineVersionStatus `gorm:"column:Status; not null;"` // Code source url links to the pipeline version's definition in repo. CodeSourceUrl string `gorm:"column:CodeSourceUrl;"` - Description string `gorm:"column:Description; size:65535;"` // Set size to large number so it will be stored as longtext - PipelineSpec string `gorm:"column:PipelineSpec; not null; size:33554432;"` // Same as common.MaxFileLength (32MB in server). Argo imposes 700kB limit - PipelineSpecURI string `gorm:"column:PipelineSpecURI; not null; size:65535;"` // Can store references to ObjectStore files + Description string `gorm:"column:Description; type:longtext;"` // the previous comment says the 'size:65535' is set so it will be stored as longtext; + PipelineSpec string `gorm:"column:PipelineSpec; not null; type:longtext;"` // Same as common.MaxFileLength (32MB in server). Argo imposes 700kB limit + PipelineSpecURI string `gorm:"column:PipelineSpecURI; not null; type:longtext;"` // Can store references to ObjectStore files } func (p PipelineVersion) GetValueOfPrimaryKey() string { @@ -82,6 +87,10 @@ func (p *PipelineVersion) GetModelName() string { return "pipeline_versions" } +// TableName overrides GORM's table name inference. +func (PipelineVersion) TableName() string { + return "pipeline_versions" +} func (p *PipelineVersion) GetField(name string) (string, bool) { if field, ok := p.APIToModelFieldMap()[name]; ok { return field, true diff --git a/backend/src/apiserver/model/resource_reference.go b/backend/src/apiserver/model/resource_reference.go index 511a51fdf99..bcbd48da025 100644 --- a/backend/src/apiserver/model/resource_reference.go +++ b/backend/src/apiserver/model/resource_reference.go @@ -147,27 +147,28 @@ type ResourceType string type Relationship string // Resource reference table models the relationship between resources in a loosely coupled way. +// This model has a composite primary key consisting of ResourceUUID, ResourceType, and ReferenceType. type ResourceReference struct { // ID of the resource object - ResourceUUID string `gorm:"column:ResourceUUID; not null; primary_key;"` + ResourceUUID string `gorm:"column:ResourceUUID; not null; primaryKey;"` // The type of the resource object - ResourceType ResourceType `gorm:"column:ResourceType; not null; primary_key; index:referencefilter;"` + ResourceType ResourceType `gorm:"column:ResourceType; not null; primaryKey; index:referencefilter;"` - // The ID of the resource that been referenced to. + // The ID of the referenced resource. ReferenceUUID string `gorm:"column:ReferenceUUID; not null; index:referencefilter;"` - // The name of the resource that been referenced to. + // The name of the referenced resource. ReferenceName string `gorm:"column:ReferenceName; not null;"` - // The type of the resource that been referenced to. - ReferenceType ResourceType `gorm:"column:ReferenceType; not null; primary_key; index:referencefilter;"` + // The type of the referenced resource. + ReferenceType ResourceType `gorm:"column:ReferenceType; not null; primaryKey; index:referencefilter;"` - // The relationship between the resource object and the resource that been referenced to. + // The relationship between the resource object and the referenced resource. Relationship Relationship `gorm:"column:Relationship; not null;"` - // The json formatted blob of the resource reference. - Payload string `gorm:"column:Payload; not null; size:65535;"` + // JSON-encoded metadata blob about the reference + Payload string `gorm:"column:Payload; not null; type: longtext"` } type ReferenceKey struct { diff --git a/backend/src/apiserver/model/run.go b/backend/src/apiserver/model/run.go index 26cc7c36850..516fd47938c 100644 --- a/backend/src/apiserver/model/run.go +++ b/backend/src/apiserver/model/run.go @@ -205,22 +205,29 @@ func (Run) TableName() string { } type Run struct { - UUID string `gorm:"column:UUID; not null; primary_key"` + UUID string `gorm:"column:UUID; not null; primaryKey"` DisplayName string `gorm:"column:DisplayName; not null;"` /* The name that user provides. Can contain special characters*/ K8SName string `gorm:"column:Name; not null;"` /* The name of the K8s resource. Follow regex '[a-z0-9]([-a-z0-9]*[a-z0-9])?'*/ Description string `gorm:"column:Description; not null;"` - - Namespace string `gorm:"column:Namespace; not null;"` - ExperimentId string `gorm:"column:ExperimentUUID; not null;"` + // Namespace is restricted to varchar(63) due to Kubernetes' naming constraints: + // https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-label-names + Namespace string `gorm:"column:Namespace; type:varchar(63); not null;index:namespace_createatinsec,priority:1; index:namespace_conditions_finishedatinsec,priority:1"` + // varchar(64) is carefully chosen to ensure composite index constraints remain + // within MySQL's 767-byte limit + // e.g., ExperimentId(varchar(64)) + Conditions(varchar(125)) + FinishedAtInSec(8 bytes) = 764 bytes < 767 bytes + // For details on type lengths and index safety, refer to comments in the Pipeline struct. + // nolint:staticcheck // [ST1003] Field name matches upstream legacy naming + ExperimentId string `gorm:"column:ExperimentUUID;type:varchar(64); not null; index:experimentuuid_createatinsec,priority:1; index:experimentuuid_conditions_finishedatinsec,priority:1"` RecurringRunId string `gorm:"column:JobUUID; default:null;"` StorageState StorageState `gorm:"column:StorageState; not null;"` ServiceAccount string `gorm:"column:ServiceAccount; not null;"` - Metrics []*RunMetric + Metrics []*RunMetric `gorm:"foreignKey:RunUUID;references:UUID;constraint:OnDelete:CASCADE,OnUpdate:CASCADE"` // This 'has-many' relation replaces the legacy AddForeignKey constraint previously defined in client_manager.go // ResourceReferences are deprecated. Use Namespace, ExperimentId, // RecurringRunId, PipelineSpec.PipelineId, PipelineSpec.PipelineVersionId - ResourceReferences []*ResourceReference + // gorm:"-" tag is added to avoid declaring a foreign key. Refer to Job model for reasons. + ResourceReferences []*ResourceReference `gorm:"-"` PipelineSpec @@ -301,30 +308,33 @@ func (r *Run) ToV2() *Run { // Stores runtime information about a pipeline run. type RunDetails struct { - CreatedAtInSec int64 `gorm:"column:CreatedAtInSec; not null;"` + CreatedAtInSec int64 `gorm:"column:CreatedAtInSec; not null;index:experimentuuid_createatinsec,priority:2; index:namespace_createatinsec,priority:2"` ScheduledAtInSec int64 `gorm:"column:ScheduledAtInSec; default:0;"` - FinishedAtInSec int64 `gorm:"column:FinishedAtInSec; default:0;"` + FinishedAtInSec int64 `gorm:"column:FinishedAtInSec; default:0; index:experimentuuid_conditions_finishedatinsec,priority:3;index:namespace_conditions_finishedatinsec,priority:3"` // Conditions were deprecated. Use State instead. - Conditions string `gorm:"column:Conditions; not null;"` + // varchar(125) is carefully chosen to ensure composite index constraints remain + // within MySQL's 767-byte limit (e.g., when combined with ExperimentId and FinishedAtInSec). + // For details on type lengths and index safety, refer to comments in the Pipeline struct. + Conditions string `gorm:"column:Conditions; type:varchar(125); not null; index:experimentuuid_conditions_finishedatinsec,priority:2;index:namespace_conditions_finishedatinsec,priority:2"` State RuntimeState `gorm:"column:State; default:null;"` - StateHistoryString string `gorm:"column:StateHistory; default:null; size:65535;"` + StateHistoryString string `gorm:"column:StateHistory; default:null; type: longtext;"` StateHistory []*RuntimeStatus `gorm:"-;"` // Serialized runtime details of a run in v2beta1 - PipelineRuntimeManifest string `gorm:"column:PipelineRuntimeManifest; not null; size:33554432;"` + PipelineRuntimeManifest string `gorm:"column:PipelineRuntimeManifest; not null; type:longtext;"` // Serialized Argo CRD in v1beta1 - WorkflowRuntimeManifest string `gorm:"column:WorkflowRuntimeManifest; not null; size:33554432;"` + WorkflowRuntimeManifest string `gorm:"column:WorkflowRuntimeManifest; not null; type:longtext;"` PipelineContextId int64 `gorm:"column:PipelineContextId; default:0;"` PipelineRunContextId int64 `gorm:"column:PipelineRunContextId; default:0;"` TaskDetails []*Task } type RunMetric struct { - RunUUID string `gorm:"column:RunUUID; not null; primary_key;"` - NodeID string `gorm:"column:NodeID; not null; primary_key;"` - Name string `gorm:"column:Name; not null; primary_key;"` + RunUUID string `gorm:"column:RunUUID; not null; primaryKey;"` + NodeID string `gorm:"column:NodeID; not null; primaryKey;"` + Name string `gorm:"column:Name; not null; primaryKey;"` NumberValue float64 `gorm:"column:NumberValue;"` Format string `gorm:"column:Format;"` - Payload string `gorm:"column:Payload; not null; size:65535;"` + Payload string `gorm:"column:Payload; not null; type:longtext;"` } type RuntimeStatus struct { diff --git a/backend/src/apiserver/model/runtime_config.go b/backend/src/apiserver/model/runtime_config.go index f58ef1e8f05..eff5ef4a72d 100644 --- a/backend/src/apiserver/model/runtime_config.go +++ b/backend/src/apiserver/model/runtime_config.go @@ -16,10 +16,10 @@ package model type RuntimeConfig struct { // Store parameters key-value pairs as serialized string. - Parameters string `gorm:"column:RuntimeParameters; size:65535;"` + Parameters string `gorm:"column:RuntimeParameters; type:longtext;"` // A path in a object store bucket which will be treated as the root // output directory of the pipeline. It is used by the system to // generate the paths of output artifacts. Ref:(https://www.kubeflow.org/docs/components/pipelines/pipeline-root/) - PipelineRoot string `gorm:"column:PipelineRoot; size:65535;"` + PipelineRoot string `gorm:"column:PipelineRoot; type:longtext;"` } diff --git a/backend/src/apiserver/model/task.go b/backend/src/apiserver/model/task.go index 5c5d7092e64..8c4fe38d6d6 100644 --- a/backend/src/apiserver/model/task.go +++ b/backend/src/apiserver/model/task.go @@ -19,11 +19,15 @@ import ( ) type Task struct { - UUID string `gorm:"column:UUID; not null; primary_key"` + UUID string `gorm:"column:UUID; not null; primaryKey"` Namespace string `gorm:"column:Namespace; not null;"` // PipelineName was deprecated. Use RunId instead. - PipelineName string `gorm:"column:PipelineName; not null;"` - RunId string `gorm:"column:RunUUID; not null;"` + PipelineName string `gorm:"column:PipelineName; not null;"` + // RunId is limited to varchar(191) to make it indexable as a foreign key. + // For details on type lengths and index safety, refer to comments in the Pipeline struct. + // nolint:staticcheck // [ST1003] Field name matches upstream legacy naming + RunId string `gorm:"column:RunUUID; type:varchar(191); not null;"` // Note: field name (RunId) ≠ column name (RunUUID). The former should be the foreign key instead of the letter. + Run Run `gorm:"foreignKey:RunId;references:UUID;constraint:OnDelete:CASCADE,OnUpdate:CASCADE"` // This 'belongs to' relation replaces the legacy AddForeignKey constraint previously defined in client_manager.go PodName string `gorm:"column:PodName; not null;"` MLMDExecutionID string `gorm:"column:MLMDExecutionID; not null;"` CreatedTimestamp int64 `gorm:"column:CreatedTimestamp; not null;"` @@ -33,13 +37,13 @@ type Task struct { Name string `gorm:"column:Name; default:null"` ParentTaskId string `gorm:"column:ParentTaskUUID; default:null"` State RuntimeState `gorm:"column:State; default:null;"` - StateHistoryString string `gorm:"column:StateHistory; default:null; size:65535;"` - MLMDInputs string `gorm:"column:MLMDInputs; default:null; size:65535;"` - MLMDOutputs string `gorm:"column:MLMDOutputs; default:null; size:65535;"` - ChildrenPodsString string `gorm:"column:ChildrenPods; default:null; size:65535;"` + StateHistoryString string `gorm:"column:StateHistory; default:null; type:longtext;"` + MLMDInputs string `gorm:"column:MLMDInputs; default:null; type:longtext;"` + MLMDOutputs string `gorm:"column:MLMDOutputs; default:null; type:longtext;"` + ChildrenPodsString string `gorm:"column:ChildrenPods; default:null; type:longtext;"` StateHistory []*RuntimeStatus `gorm:"-;"` ChildrenPods []string `gorm:"-;"` - Payload string `gorm:"column:Payload; default:null; size:65535;"` + Payload string `gorm:"column:Payload; default:null; type:longtext;"` } func (t Task) ToString() string { diff --git a/backend/src/apiserver/storage/db_fake.go b/backend/src/apiserver/storage/db_fake.go index 276d3a68898..6596ed5f75d 100644 --- a/backend/src/apiserver/storage/db_fake.go +++ b/backend/src/apiserver/storage/db_fake.go @@ -16,20 +16,20 @@ package storage import ( "github.com/golang/glog" - "github.com/jinzhu/gorm" "github.com/kubeflow/pipelines/backend/src/apiserver/model" "github.com/kubeflow/pipelines/backend/src/common/util" - _ "github.com/mattn/go-sqlite3" + "gorm.io/driver/sqlite" + "gorm.io/gorm" ) func NewFakeDB() (*DB, error) { // Initialize GORM - db, err := gorm.Open("sqlite3", ":memory:") + dbInstance, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) if err != nil { return nil, util.Wrap(err, "Could not create the GORM database") } // Create tables - db.AutoMigrate( + if err := dbInstance.AutoMigrate( &model.Experiment{}, &model.Job{}, &model.Pipeline{}, @@ -40,8 +40,15 @@ func NewFakeDB() (*DB, error) { &model.Task{}, &model.DBStatus{}, &model.DefaultExperiment{}, - ) - return NewDB(db.DB(), NewSQLiteDialect()), nil + ); err != nil { + return nil, util.Wrap(err, "Failed to automigrate models") + } + + sqlDB, err := dbInstance.DB() + if err != nil { + return nil, util.Wrap(err, "Failed to get generic database object from GORM DB") + } + return NewDB(sqlDB, NewSQLiteDialect()), nil } func NewFakeDBOrFatal() *DB { diff --git a/backend/src/apiserver/storage/experiment_store.go b/backend/src/apiserver/storage/experiment_store.go index 3f704d9415d..eea2f134174 100644 --- a/backend/src/apiserver/storage/experiment_store.go +++ b/backend/src/apiserver/storage/experiment_store.go @@ -196,8 +196,8 @@ func (s *ExperimentStore) scanRows(rows *sql.Rows) ([]*model.Experiment, error) var experiments []*model.Experiment for rows.Next() { var uuid, name, description, namespace, storageState string - var createdAtInSec sql.NullInt64 - var lastRunCreatedAtInSec sql.NullInt64 + var createdAtInSec int64 + var lastRunCreatedAtInSec int64 err := rows.Scan(&uuid, &name, &description, &createdAtInSec, &lastRunCreatedAtInSec, &namespace, &storageState) if err != nil { return experiments, err @@ -206,8 +206,8 @@ func (s *ExperimentStore) scanRows(rows *sql.Rows) ([]*model.Experiment, error) UUID: uuid, Name: name, Description: description, - CreatedAtInSec: createdAtInSec.Int64, - LastRunCreatedAtInSec: lastRunCreatedAtInSec.Int64, + CreatedAtInSec: createdAtInSec, + LastRunCreatedAtInSec: lastRunCreatedAtInSec, Namespace: namespace, StorageState: model.StorageState(storageState).ToV2(), } diff --git a/backend/src/cache/client_manager.go b/backend/src/cache/client_manager.go index d7384fce953..d7496f5aef7 100644 --- a/backend/src/cache/client_manager.go +++ b/backend/src/cache/client_manager.go @@ -24,11 +24,12 @@ import ( "github.com/cenkalti/backoff" "github.com/golang/glog" - "github.com/jinzhu/gorm" "github.com/kubeflow/pipelines/backend/src/cache/client" "github.com/kubeflow/pipelines/backend/src/cache/model" "github.com/kubeflow/pipelines/backend/src/cache/storage" "github.com/kubeflow/pipelines/backend/src/common/util" + "gorm.io/driver/mysql" + "gorm.io/gorm" ) const ( @@ -51,7 +52,12 @@ func (c *ClientManager) KubernetesCoreClient() client.KubernetesCoreInterface { } func (c *ClientManager) Close() { - c.db.Close() + sqlDB, err := c.db.DB.DB() + if err != nil { + log.Printf("Failed to retrieve underlying sql.DB: %v", err) + return + } + sqlDB.Close() } func (c *ClientManager) init(params WhSvrDBParameters, clientParams util.ClientParameters) { @@ -77,22 +83,22 @@ func initDBClient(params WhSvrDBParameters, initConnectionTimeout time.Duration) // db is safe for concurrent use by multiple goroutines // and maintains its own pool of idle connections. - db, err := gorm.Open(driverName, arg) + db, err := gorm.Open(mysql.Open(arg), &gorm.Config{}) util.TerminateIfError(err) // Create table - response := db.AutoMigrate(&model.ExecutionCache{}) - if response.Error != nil { + err = db.AutoMigrate(&model.ExecutionCache{}) + if err != nil { glog.Fatalf("Failed to initialize the databases.") } - response = db.Model(&model.ExecutionCache{}).ModifyColumn("ExecutionOutput", "longtext") - if response.Error != nil { - glog.Fatalf("Failed to update the execution output type. Error: %s", response.Error) + err = db.Migrator().AlterColumn(&model.ExecutionCache{}, "ExecutionOutput") + if err != nil { + glog.Fatalf("Failed to update the execution output type. Error: %s", err) } - response = db.Model(&model.ExecutionCache{}).ModifyColumn("ExecutionTemplate", "longtext not null") - if response.Error != nil { - glog.Fatalf("Failed to update the execution template type. Error: %s", response.Error) + err = db.Migrator().AlterColumn(&model.ExecutionCache{}, "ExecutionTemplate") + if err != nil { + glog.Fatalf("Failed to update the execution template type. Error: %s", err) } var tableNames []string diff --git a/backend/src/cache/model/execution_cache.go b/backend/src/cache/model/execution_cache.go index 18ec531db7d..66fec0c91d2 100644 --- a/backend/src/cache/model/execution_cache.go +++ b/backend/src/cache/model/execution_cache.go @@ -15,7 +15,7 @@ package model type ExecutionCache struct { - ID int64 `gorm:"column:ID; not null; primary_key; AUTO_INCREMENT; index:composite_id_idx"` + ID int64 `gorm:"column:ID; not null; primaryKey; AUTO_INCREMENT; index:composite_id_idx"` ExecutionCacheKey string `gorm:"column:ExecutionCacheKey; not null; index:idx_cache_key;"` ExecutionTemplate string `gorm:"column:ExecutionTemplate; not null;"` ExecutionOutput string `gorm:"column:ExecutionOutput; not null;"` diff --git a/backend/src/cache/server/client_manager_fake.go b/backend/src/cache/server/client_manager_fake.go index 7779bcfed6b..b3af65aecf3 100644 --- a/backend/src/cache/server/client_manager_fake.go +++ b/backend/src/cache/server/client_manager_fake.go @@ -67,7 +67,11 @@ func (f *FakeClientManager) DB() *storage.DB { } func (f *FakeClientManager) Close() error { - return f.db.Close() + sqlDB, err := f.db.DB.DB() + if err != nil { + return err + } + return sqlDB.Close() } func (f *FakeClientManager) KubernetesCoreClient() client.KubernetesCoreInterface { diff --git a/backend/src/cache/storage/db.go b/backend/src/cache/storage/db.go index 7b9154477e1..6a1e32af1ce 100644 --- a/backend/src/cache/storage/db.go +++ b/backend/src/cache/storage/db.go @@ -15,7 +15,7 @@ package storage import ( - "github.com/jinzhu/gorm" + "gorm.io/gorm" ) // DB a struct wrapping plain sql library with SQL dialect, to solve any feature diff --git a/backend/src/cache/storage/db_fake.go b/backend/src/cache/storage/db_fake.go index 58cec438fc3..3df72f89f95 100644 --- a/backend/src/cache/storage/db_fake.go +++ b/backend/src/cache/storage/db_fake.go @@ -18,19 +18,22 @@ import ( "fmt" "github.com/golang/glog" - "github.com/jinzhu/gorm" "github.com/kubeflow/pipelines/backend/src/cache/model" - _ "github.com/mattn/go-sqlite3" + "gorm.io/driver/sqlite" + "gorm.io/gorm" ) func NewFakeDB() (*DB, error) { // Initialize GORM - db, err := gorm.Open("sqlite3", ":memory:") + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) if err != nil { return nil, fmt.Errorf("Could not create the GORM database: %v", err) } // Create tables - db.AutoMigrate(&model.ExecutionCache{}) + err = db.AutoMigrate(&model.ExecutionCache{}) + if err != nil { + return nil, fmt.Errorf("AutoMigrate failed: %v", err) + } return NewDB(db), nil } diff --git a/backend/src/cache/storage/execution_cache_store.go b/backend/src/cache/storage/execution_cache_store.go index 7be8c48d41d..c2150a5fab6 100644 --- a/backend/src/cache/storage/execution_cache_store.go +++ b/backend/src/cache/storage/execution_cache_store.go @@ -140,7 +140,7 @@ func (s *ExecutionCacheStore) CreateExecutionCache(executionCache *model.Executi r, err := s.db.Table("execution_caches").Where("ExecutionCacheKey = ?", executionCache.ExecutionCacheKey).Rows() if err != nil { - log.Printf("failed to get execution cache with key: %s, err: %v", executionCache.ExecutionCacheKey, err) + log.Printf("Failed to get execution cache with key: %s, err: %v", executionCache.ExecutionCacheKey, err) return nil, err } @@ -159,22 +159,19 @@ func (s *ExecutionCacheStore) CreateExecutionCache(executionCache *model.Executi newExecutionCache.StartedAtInSec = now newExecutionCache.EndedAtInSec = now - ok := s.db.NewRecord(&newExecutionCache) - if !ok { - return nil, fmt.Errorf("failed to create new execution cache row for key: %s", executionCache.ExecutionCacheKey) - } + // GORM v2 removed NewRecord(); it was unreliable as it only checked whether the primary key was 0 / "" / nil var rowInsert model.ExecutionCache d := s.db.Create(&newExecutionCache).Scan(&rowInsert) if d.Error != nil { - return nil, d.Error + return nil, fmt.Errorf("failed to create new execution cache: %w", d.Error) } log.Printf("cache entry created successfully with key: %s, template: %v, row id: %d", executionCache.ExecutionCacheKey, executionCache.ExecutionTemplate, rowInsert.ID) return &rowInsert, nil } else { - log.Printf("%d row(s) already exist for cache key: %s", rowCount, executionCache.ExecutionCacheKey) - return executionCache, nil + // return an error to prevent returning non-inserted user-provided structs and align with the unit test + return nil, fmt.Errorf("execution cache with key %s already exists, failed to create new execution cache", executionCache.ExecutionCacheKey) } } diff --git a/backend/src/cache/storage/execution_cache_store_test.go b/backend/src/cache/storage/execution_cache_store_test.go index 72cb7c6da11..9fb1e9ba949 100644 --- a/backend/src/cache/storage/execution_cache_store_test.go +++ b/backend/src/cache/storage/execution_cache_store_test.go @@ -24,6 +24,12 @@ import ( "github.com/stretchr/testify/require" ) +func closeDB(t *testing.T, db *DB) { + sqlDB, err := db.DB.DB() + require.NoError(t, err) + sqlDB.Close() +} + func createExecutionCache(cacheKey string, cacheOutput string) *model.ExecutionCache { return &model.ExecutionCache{ ExecutionCacheKey: cacheKey, @@ -37,7 +43,7 @@ func createExecutionCache(cacheKey string, cacheOutput string) *model.ExecutionC func TestCreateExecutionCache(t *testing.T) { db := NewFakeDBOrFatal() - defer db.Close() + defer closeDB(t, db) executionCacheStore := NewExecutionCacheStore(db, util.NewFakeTimeForEpoch()) executionCacheExpected := model.ExecutionCache{ ID: 1, @@ -70,7 +76,7 @@ func TestCreateExecutionCacheWithDuplicateRecord(t *testing.T) { EndedAtInSec: 1, } db := NewFakeDBOrFatal() - defer db.Close() + defer closeDB(t, db) executionCacheStore := NewExecutionCacheStore(db, util.NewFakeTimeForEpoch()) executionCacheStore.CreateExecutionCache(executionCache) cache, err := executionCacheStore.CreateExecutionCache(executionCache) @@ -80,7 +86,7 @@ func TestCreateExecutionCacheWithDuplicateRecord(t *testing.T) { func TestGetExecutionCache(t *testing.T) { db := NewFakeDBOrFatal() - defer db.Close() + defer closeDB(t, db) executionCacheStore := NewExecutionCacheStore(db, util.NewFakeTimeForEpoch()) executionCacheStore.CreateExecutionCache(createExecutionCache("testKey", "testOutput")) @@ -102,7 +108,7 @@ func TestGetExecutionCache(t *testing.T) { func TestGetExecutionCacheWithEmptyCacheEntry(t *testing.T) { db := NewFakeDBOrFatal() - defer db.Close() + defer closeDB(t, db) executionCacheStore := NewExecutionCacheStore(db, util.NewFakeTimeForEpoch()) executionCacheStore.CreateExecutionCache(createExecutionCache("testKey", "testOutput")) @@ -114,7 +120,7 @@ func TestGetExecutionCacheWithEmptyCacheEntry(t *testing.T) { func TestGetExecutionCacheWithLatestCacheEntry(t *testing.T) { db := NewFakeDBOrFatal() - defer db.Close() + defer closeDB(t, db) executionCacheStore := NewExecutionCacheStore(db, util.NewFakeTimeForEpoch()) executionCacheStore.CreateExecutionCache(createExecutionCache("testKey", "testOutput")) @@ -137,7 +143,7 @@ func TestGetExecutionCacheWithLatestCacheEntry(t *testing.T) { func TestGetExecutionCacheWithExpiredDatabaseCacheStaleness(t *testing.T) { db := NewFakeDBOrFatal() - defer db.Close() + defer closeDB(t, db) executionCacheStore := NewExecutionCacheStore(db, util.NewFakeTimeForEpoch()) executionCacheToPersist := &model.ExecutionCache{ ExecutionCacheKey: "testKey", @@ -155,7 +161,7 @@ func TestGetExecutionCacheWithExpiredDatabaseCacheStaleness(t *testing.T) { func TestGetExecutionCacheWithExpiredAnnotationCacheStaleness(t *testing.T) { db := NewFakeDBOrFatal() - defer db.Close() + defer closeDB(t, db) executionCacheStore := NewExecutionCacheStore(db, util.NewFakeTimeForEpoch()) executionCacheToPersist := &model.ExecutionCache{ ExecutionCacheKey: "testKey", @@ -175,7 +181,7 @@ func TestGetExecutionCacheWithExpiredAnnotationCacheStaleness(t *testing.T) { func TestGetExecutionCacheWithExpiredMaximumCacheStaleness(t *testing.T) { db := NewFakeDBOrFatal() - defer db.Close() + defer closeDB(t, db) executionCacheStore := NewExecutionCacheStore(db, util.NewFakeTimeForEpoch()) executionCacheToPersist := &model.ExecutionCache{ ExecutionCacheKey: "testKey", diff --git a/backend/test/integration/db_test.go b/backend/test/integration/db_test.go index 76d34b322c2..0ded6ec1740 100644 --- a/backend/test/integration/db_test.go +++ b/backend/test/integration/db_test.go @@ -24,8 +24,18 @@ import ( _ "github.com/jackc/pgx/v5/stdlib" cm "github.com/kubeflow/pipelines/backend/src/apiserver/client_manager" + "github.com/kubeflow/pipelines/backend/src/apiserver/model" + "gorm.io/driver/mysql" + "gorm.io/driver/postgres" + "gorm.io/gorm" ) +// verifyCompositeIndex asserts that the given gorm.DB has created both the unique constraint and index named idx on model mdl. +func verifyCompositeIndex(t *testing.T, gdb *gorm.DB, mdl interface{}, idx string) { + assert.True(t, gdb.Migrator().HasConstraint(mdl, idx), "constraint %s should exist", idx) + assert.True(t, gdb.Migrator().HasIndex(mdl, idx), "index %s should exist", idx) +} + type DBTestSuite struct { suite.Suite } @@ -52,6 +62,16 @@ func (s *DBTestSuite) TestInitDBClient_MySQL() { duration, _ := time.ParseDuration("1m") db := cm.InitDBClient(duration) assert.NotNil(t, db) + + // Extract underlying *sql.DB and wrap in gorm.DB + sqlDB := db.DB + dialector := mysql.New(mysql.Config{Conn: sqlDB}) + gdb, err := gorm.Open(dialector, &gorm.Config{}) + if err != nil { + t.Fatalf("failed to open gorm.DB for MySQL: %v", err) + } + // Verify composite unique constraint and index on the Pipeline model + verifyCompositeIndex(t, gdb, &model.Pipeline{}, "namespace_name") } // Test PostgreSQL initializes correctly @@ -70,6 +90,16 @@ func (s *DBTestSuite) TestInitDBClient_PostgreSQL() { duration, _ := time.ParseDuration("1m") db := cm.InitDBClient(duration) assert.NotNil(t, db) + + // Extract underlying *sql.DB and wrap in gorm.DB + sqlDB := db.DB + dialector := postgres.New(postgres.Config{Conn: sqlDB}) + gdb, err := gorm.Open(dialector, &gorm.Config{}) + if err != nil { + t.Fatalf("failed to open gorm.DB for PostgreSQL: %v", err) + } + // Verify composite unique constraint and index on the Pipeline model + verifyCompositeIndex(s.T(), gdb, &model.Pipeline{}, "namespace_name") } func TestDB(t *testing.T) { diff --git a/go.mod b/go.mod index a300e223690..d8ec8c20401 100644 --- a/go.mod +++ b/go.mod @@ -27,13 +27,12 @@ require ( github.com/gorilla/mux v1.8.0 github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.2 github.com/grpc-ecosystem/grpc-gateway v1.16.0 - github.com/jackc/pgx/v5 v5.5.4 - github.com/jinzhu/gorm v1.9.1 + github.com/jackc/pgx/v5 v5.5.5 github.com/kubeflow/pipelines/api v0.0.0-20250102152816-873e9dedd766 github.com/kubeflow/pipelines/kubernetes_platform v0.0.0-20240725205754-d911c8b73b49 github.com/kubeflow/pipelines/third_party/ml-metadata v0.0.0-20240416215826-da804407ad31 github.com/lestrrat-go/strftime v1.0.4 - github.com/mattn/go-sqlite3 v1.14.19 + github.com/mattn/go-sqlite3 v1.14.22 github.com/minio/minio-go/v7 v7.0.94 github.com/peterhellberg/duration v0.0.0-20191119133758-ec6baeebcd10 github.com/pkg/errors v0.9.1 @@ -53,6 +52,10 @@ require ( google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0 google.golang.org/protobuf v1.36.6 gopkg.in/yaml.v3 v3.0.1 + gorm.io/driver/mysql v1.5.7 + gorm.io/driver/postgres v1.5.11 + gorm.io/driver/sqlite v1.5.7 + gorm.io/gorm v1.26.1 k8s.io/api v0.30.1 k8s.io/apimachinery v0.30.1 k8s.io/client-go v0.30.1 @@ -99,11 +102,9 @@ require ( github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/colinmarc/hdfs/v2 v2.4.0 // indirect github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect - github.com/denisenkom/go-mssqldb v0.12.3 // indirect github.com/doublerebel/bellows v0.0.0-20160303004610-f177d92a03d3 // indirect github.com/dustin/go-humanize v1.0.1 // indirect github.com/emicklei/go-restful/v3 v3.11.0 // indirect - github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5 // indirect github.com/evanphx/json-patch v5.8.0+incompatible // indirect github.com/evanphx/json-patch/v5 v5.9.0 // indirect github.com/evilmonkeyinc/jsonpath v0.8.1 // indirect diff --git a/go.sum b/go.sum index 05b95026bef..c5c4b57b81b 100644 --- a/go.sum +++ b/go.sum @@ -19,9 +19,6 @@ cloud.google.com/go/storage v1.43.0 h1:CcxnSohZwizt4LCzQHWvBf1/kvtHUn7gk9QERXPyX cloud.google.com/go/storage v1.43.0/go.mod h1:ajvxEa7WmZS1PxvKRq4bq0tFT3vMd502JwstCcYv0Q0= filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA= filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4= -github.com/Azure/azure-sdk-for-go/sdk/azcore v0.19.0/go.mod h1:h6H6c8enJmmocHUbLiiGY6sx7f9i+X3m1CHdd5c6Rdw= -github.com/Azure/azure-sdk-for-go/sdk/azidentity v0.11.0/go.mod h1:HcM1YX14R7CJcghJGOYCgdezslRSVzqwLf/q+4Y2r/0= -github.com/Azure/azure-sdk-for-go/sdk/internal v0.7.0/go.mod h1:yqy467j36fJxcRV2TzfVZ1pCb5vxm4BtZPUdYWe/Xo8= github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI= github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0= github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA= @@ -131,10 +128,7 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/denisenkom/go-mssqldb v0.12.3 h1:pBSGx9Tq67pBOTLmxNuirNTeB8Vjmf886Kx+8Y+8shw= -github.com/denisenkom/go-mssqldb v0.12.3/go.mod h1:k0mtMFOnU+AihqFxPMiF05rtiDrorD1Vrm1KEz5hxDo= github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= -github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ= github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM= @@ -155,8 +149,6 @@ github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.m github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ= github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= -github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5 h1:Yzb9+7DPaBjB8zlTR87/ElzFsnQfuHnVUVqpZZIcV5Y= -github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5/go.mod h1:a2zkGnVExMxdzMo3M0Hi/3sEU+cWnZpSni0O6/Yb/P0= github.com/evanphx/json-patch v4.2.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/evanphx/json-patch v5.8.0+incompatible h1:1Av9pn2FyxPdvrWNQszj1g6D6YthSmvCfcN6SYclTJg= github.com/evanphx/json-patch v5.8.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= @@ -297,6 +289,7 @@ github.com/go-openapi/validate v0.20.1/go.mod h1:b60iJT+xNNLfaQJUqLI7946tYiFEOuE github.com/go-openapi/validate v0.20.3 h1:GZPPhhKSZrE8HjB4eEkoYAZmoWA4+tCemSgINH1/vKw= github.com/go-openapi/validate v0.20.3/go.mod h1:goDdqVGiigM3jChcrYJxD2joalke3ZXeftD16byIjA4= github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg= +github.com/go-sql-driver/mysql v1.7.0/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI= github.com/go-sql-driver/mysql v1.8.1 h1:LedoTUt/eveggdHS9qUFC1EFSa8bU2+1pZjSRpvNJ1Y= github.com/go-sql-driver/mysql v1.8.1/go.mod h1:wEBSXgmK//2ZFJyE+qWnIsVGmvmEKlqwuVSjsCm7DZg= github.com/go-stack/stack v1.8.0 h1:5SgMzNM5HxrEjV0ww2lTmX6E2Izsfxas4+YHWRs3Lsk= @@ -332,10 +325,6 @@ github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PU github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= -github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe h1:lXe2qZdvpiX5WZkZR4hgp4KJVfY3nMkvmwbVkpv1rVY= -github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0= -github.com/golang-sql/sqlexp v0.1.0 h1:ZCD6MBpcuOVfGVqsEmY5/4FtYiKz6tSyUv9LPEDei6A= -github.com/golang-sql/sqlexp v0.1.0/go.mod h1:J4ad9Vo8ZCWQ2GMrC4UCQy1JpCbwU9m3EOqtpKwwwHI= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/glog v1.2.4 h1:CNNw5U8lSiiBk7druxtSHHTsRWcxKoac6kZKm2peBBc= github.com/golang/glog v1.2.4/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w= @@ -448,8 +437,8 @@ github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsI github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a h1:bbPeKD0xmW/Y25WS6cokEszi5g+S0QxI/d45PkRi7Nk= github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= -github.com/jackc/pgx/v5 v5.5.4 h1:Xp2aQS8uXButQdnCMWNmvx6UysWQQC+u1EoizjguY+8= -github.com/jackc/pgx/v5 v5.5.4/go.mod h1:ez9gk+OAat140fv9ErkZDYFWmXLfV+++K0uAOiwgm1A= +github.com/jackc/pgx/v5 v5.5.5 h1:amBjrZVmksIdNjxGW/IiIMzxMKZFelXbUoPNb+8sjQw= +github.com/jackc/pgx/v5 v5.5.5/go.mod h1:ez9gk+OAat140fv9ErkZDYFWmXLfV+++K0uAOiwgm1A= github.com/jackc/puddle/v2 v2.2.1 h1:RhxXJtFG022u4ibrCSMSiu5aOq1i77R3OHKNJj77OAk= github.com/jackc/puddle/v2 v2.2.1/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8= @@ -464,8 +453,6 @@ github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh6 github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs= github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY= github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc= -github.com/jinzhu/gorm v1.9.1 h1:lDSDtsCt5AGGSKTs8AHlSDbbgif4G4+CKJ8ETBDVHTA= -github.com/jinzhu/gorm v1.9.1/go.mod h1:Vla75njaFJ8clLU1W44h34PjIkijhjHIYnZxMqCdxqo= github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E= github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc= github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ= @@ -534,8 +521,8 @@ github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0 github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/markbates/oncer v0.0.0-20181203154359-bf2de49a0be2/go.mod h1:Ld9puTsIW75CHf65OeIOkyKbteujpZVXDpWK6YGZbxE= github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0= -github.com/mattn/go-sqlite3 v1.14.19 h1:fhGleo2h1p8tVChob4I9HpmVFIAkKGpiukdrgQbWfGI= -github.com/mattn/go-sqlite3 v1.14.19/go.mod h1:2eHXhiwb8IkHr+BDWZGa96P6+rkvnG63S2DGjv9HUNg= +github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU= +github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y= github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 h1:jWpvCLoY8Z/e3VKvlsiIGKtc+UG6U5vzxaoagmhXfyg= github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0/go.mod h1:QUyp042oQthUoa9bqDv0ER0wrtXnBruoNd7aNjkbP+k= github.com/minio/crc64nvme v1.0.1 h1:DHQPrYPdqK7jQG/Ls5CTBZWeex/2FMS3G5XGkycuFrY= @@ -568,7 +555,6 @@ github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lN github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= -github.com/modocache/gover v0.0.0-20171022184752-b58185e213c5/go.mod h1:caMODM3PzxT8aQXRPkAt8xlV/e7d7w8GM5g0fa5F0D8= github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc= github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= @@ -600,7 +586,6 @@ github.com/peterhellberg/duration v0.0.0-20191119133758-ec6baeebcd10 h1:Jf08dx6h github.com/peterhellberg/duration v0.0.0-20191119133758-ec6baeebcd10/go.mod h1:x5xjkH61fUOJVgCCDgqNzlJvdLXiYpmMzSuum2FBOaw= github.com/philhofer/fwd v1.1.3-0.20240916144458-20a13a1f6b7c h1:dAMKvw0MlJT1GshSTtih8C2gDs04w8dReiOGXrGLNoY= github.com/philhofer/fwd v1.1.3-0.20240916144458-20a13a1f6b7c/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM= -github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4/go.mod h1:4OwLy04Bl9Ef3GJJCoec+30X3LQs/0/m4HFRt/2LUSA= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= @@ -748,9 +733,7 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.0.0-20201016220609-9e8e0b390897/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= -golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.3.0/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4= golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= @@ -800,8 +783,6 @@ golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210421230115-4e50805a0758/go.mod h1:72T/g9IO56b78aLF+1Kcs5dz7/ng1VjMUvfKvpfy+jM= -golang.org/x/net v0.0.0-20210610132358-84b48f89b13b/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211216030914-fe4d6282115f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= @@ -1007,6 +988,15 @@ gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gorm.io/driver/mysql v1.5.7 h1:MndhOPYOfEp2rHKgkZIhJ16eVUIRf2HmzgoPmh7FCWo= +gorm.io/driver/mysql v1.5.7/go.mod h1:sEtPWMiqiN1N1cMXoXmBbd8C6/l+TESwriotuRRpkDM= +gorm.io/driver/postgres v1.5.11 h1:ubBVAfbKEUld/twyKZ0IYn9rSQh448EdelLYk9Mv314= +gorm.io/driver/postgres v1.5.11/go.mod h1:DX3GReXH+3FPWGrrgffdvCk3DQ1dwDPdmbenSkweRGI= +gorm.io/driver/sqlite v1.5.7 h1:8NvsrhP0ifM7LX9G4zPB97NwovUakUxc+2V2uuf3Z1I= +gorm.io/driver/sqlite v1.5.7/go.mod h1:U+J8craQU6Fzkcvu8oLeAQmi50TkwPEhHDEjQZXDah4= +gorm.io/gorm v1.25.7/go.mod h1:hbnx/Oo0ChWMn1BIhpy1oYozzpM15i4YPuHDmfYtwg8= +gorm.io/gorm v1.26.1 h1:ghB2gUI9FkS46luZtn6DLZ0f6ooBJ5IbVej2ENFDjRw= +gorm.io/gorm v1.26.1/go.mod h1:8Z33v652h4//uMA76KjeDH8mJXPm1QNCYrMeatR0DOE= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/manifests/kustomize/third-party/mysql/base/mysql-deployment.yaml b/manifests/kustomize/third-party/mysql/base/mysql-deployment.yaml index 407961b9964..70ea2310b54 100644 --- a/manifests/kustomize/third-party/mysql/base/mysql-deployment.yaml +++ b/manifests/kustomize/third-party/mysql/base/mysql-deployment.yaml @@ -22,7 +22,7 @@ spec: # Ext4, Btrfs etc. volumes root directories have a lost+found directory that should not be treated as a database. # ignore-db-dir option has been deprecated in mysql v5.7.16. # - # If upgrading MySQL to v8.0 fails, try removing /var/lib/mysql/lost+found folder in + # If upgrading MySQL to v8.0 fails, try removing /var/lib/mysql/lost+found folder in # mysql-pv-claim (mysql-persistent-storage): # # kubectl exec -it -n kubeflow -- bash @@ -34,7 +34,7 @@ spec: - --datadir - /var/lib/mysql # MLMD workloads (metadata-grpc-deployment and metadata-writer) depend on mysql_native_password authentication plugin. - # mysql_native_password plugin implements native authentication; that is, authentication based on the password + # mysql_native_password plugin implements native authentication; that is, authentication based on the password # hashing method in use from before the introduction of pluggable authentication in MySQL 8.0. # # The mysql_native_password authentication plugin is deprecated as of MySQL 8.0.34, disabled by default