[Draft] Add expected_workspace_status in databricks_mws_workspaces to support LPW #5019

tinglin-db · 2025-09-11T22:08:36Z

Changes

To support least privileged workspaces on GCP, adding a new field expected_workspace_status, which will be translated to workspace_state in the API request.

Tests

Added a unit test TestResourceWorkspaceCreateGcpWithExpectedProvisioning and an integration test TestMwsAccGcpWorkspacesWithExpectedProvisioning

make test run locally
using Go SDK
has entry in NEXT_CHANGELOG.md file

Tested E2E Locally

Terraform will perform the following actions:

  # databricks_mws_workspaces.this will be created
  + resource "databricks_mws_workspaces" "this" {
      + account_id                = (sensitive value)
      + cloud                     = (known after apply)
      + creation_time             = (known after apply)
      + effective_compute_mode    = (known after apply)
      + expected_workspace_status = "PROVISIONING"
      + gcp_workspace_sa          = (known after apply)
      + id                        = (known after apply)
      + is_no_public_ip_enabled   = true
      + location                  = "us-central1"
      + pricing_tier              = (known after apply)
      + workspace_id              = (known after apply)
      + workspace_name            = "tlin-classic-test-tf-1"
      + workspace_status          = (known after apply)
      + workspace_status_message  = (known after apply)
      + workspace_url             = (known after apply)

      + cloud_resource_container {
          + gcp {
              + project_id = "databricks-cal-dev-testing"
            }
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

databricks_mws_workspaces.this: Creating...
databricks_mws_workspaces.this: Creation complete after 2s [id=9fcbb245-7c44-4522-9870-e38324104cf8/2181571221551671]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

  # databricks_mws_workspaces.this will be updated in-place
  ~ resource "databricks_mws_workspaces" "this" {
      - expected_workspace_status = "PROVISIONING" -> null
        id                        = "9fcbb245-7c44-4522-9870-e38324104cf8/2181571221551671"
      + network_id                = "a124db3a-0928-4698-97eb-70e751c87934"
        # (14 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

databricks_mws_workspaces.this: Modifying... [id=9fcbb245-7c44-4522-9870-e38324104cf8/2181571221551671]
databricks_mws_workspaces.this: Modifications complete after 6s [id=9fcbb245-7c44-4522-9870-e38324104cf8/2181571221551671]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

… LPW

nkvuong

a few comments

mws/resource_mws_workspaces.go

alexott · 2025-09-12T06:45:28Z

please run make fmt ws

…ete ws in provisioning state

tinglin-db · 2025-09-25T22:18:19Z

jenkins trigger all

tinglin-db · 2025-09-26T22:27:34Z

mws/mws_workspaces_test.go

 	})
 }

+func TestMwsAccGcpWorkspacesWithExpectedProvisioning(t *testing.T) {


Integration test run (link)
On its last run, this integration test failed during the post-test destroy due to not being able to delete a still PROVISIONING workspace. Error message:

Error: -26T17:50:17.498Z [ERROR] sdk.helper_resource: Error running post-test destroy, there may be dangling resources: test_terraform_path=/tmp/plugintest-terraform1770350163/terraform test_step_number=1 | Error: cannot delete mws workspaces: INVALID_STATE: workspace 49063535405170 cannot be deleted because it is in status PROVISIONING but not must be in one of: PROVISIONING

I pushed a new commit to catch INVALID_STATE_TRANSITION errors in DELETE, but this is probably not the best approach - not sure if there is a way to have the integration test instead run through both CREATE and UPDATE so that the workspace is in a RUNNING status? cc @panchenxue-databricks

You probably need to support this case of deleting workspaces in PROVISIONING, since users could attempt to delete a partially created workspace, or they could modify an attribute that requires recreation (like deployment name).

For this test, you could also just add an additional step to put the workspace in a running state by updating expected_workspace_status.

alexott · 2025-09-27T10:11:51Z

@tinglin-db Now it fails with:

Error: cannot create mws workspaces: Workspace limit exceeded, only 20 workspaces are allowed for your account

that's why I think that deletion in the provisioning state should be handled on backend, otherwise there will be dangling resources.

mws/mws_workspaces_test.go

mgyucht · 2025-10-02T12:25:02Z

mws/resource_mws_workspaces.go

 	WorkspaceURL                        string                   `json:"workspace_url,omitempty" tf:"computed"`
 	WorkspaceStatus                     string                   `json:"workspace_status,omitempty" tf:"computed"`
 	WorkspaceStatusMessage              string                   `json:"workspace_status_message,omitempty" tf:"computed"`
+	ExpectedWorkspaceStatus             string                   `json:"expected_workspace_status,omitempty"`


We should also document this in docs/resources/mws_workspaces.md.

Is this true even if the feature is still in private preview?

Yes. Private preview services, methods and fields (and their documentation) are still part of the SDKs. Currently, private preview methods are not shown in the REST API documentation. For example: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/feature_engineering_feature

I would like us to add badges on individual fields if their stability level lags behind the resource level, but we haven't gotten to that yet. In the meantime, we can add a note in the documentation that this feature is in private preview.

Added documentation, let me know if the wording works

mws/resource_mws_workspaces.go

mgyucht · 2025-10-02T13:25:48Z

mws/resource_mws_workspaces.go

+			if expectedStatus == WorkspaceStatusProvisioning {
+				log.Printf("[INFO] Expected status is PROVISIONING, skipping wait in read function")
+				err = nil
+			} else {
+				err = workspacesAPI.WaitForExpectedStatus(workspace, d.Timeout(schema.TimeoutRead))
+			}


We should have one place for logic around waiting for workspace status. I believe this is unnecessary because the implementation of WaitForExpectedStatus already handles this properly.

I modified WaitForExpectedStatus to take in the expected_status. We need to pass it in from d.Get("expected_workspace_status") because the field is not returned by the GET during polling

mws/resource_mws_workspaces.go

NEXT_CHANGELOG.md

… logic, removed deletion skip, removed read skip

mgyucht

Aligned on the pathway forward, let's wrap this up.

mws/resource_mws_workspaces.go

mgyucht · 2025-10-08T16:00:10Z

mws/resource_mws_workspaces.go

 		return err
 	}
-	return a.WaitForRunning(ws, timeout)
+	return a.WaitForExpectedStatus(ws, WorkspaceStatusRunning, timeout)


Let's set the expected workspace status that is provided by the user in this request.

mgyucht · 2025-10-08T16:02:12Z

mws/mws_workspaces_test.go

 	})
 }

+func TestMwsAccGcpWorkspacesWithExpectedProvisioning(t *testing.T) {


Separately, two more test cases:

New workspace in provisioning state -> set expected state to RUNNING

New workspace in provisioning state -> unset expected state, workspace should progress to RUNNING.

Sounds good, added these test cases

mgyucht

Very close, some nits but this is basically looking good to me.

mgyucht · 2025-10-10T10:32:40Z

mws/resource_mws_workspaces.go

 	WorkspaceURL                        string                   `json:"workspace_url,omitempty" tf:"computed"`
 	WorkspaceStatus                     string                   `json:"workspace_status,omitempty" tf:"computed"`
 	WorkspaceStatusMessage              string                   `json:"workspace_status_message,omitempty" tf:"computed"`
+	ExpectedWorkspaceStatus             string                   `json:"expected_workspace_status,omitempty"`


Yes. Private preview services, methods and fields (and their documentation) are still part of the SDKs. Currently, private preview methods are not shown in the REST API documentation. For example: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/feature_engineering_feature

I would like us to add badges on individual fields if their stability level lags behind the resource level, but we haven't gotten to that yet. In the meantime, we can add a note in the documentation that this feature is in private preview.

mgyucht · 2025-10-10T10:33:34Z

mws/resource_mws_workspaces.go

 }

-var workspaceRunningUpdatesAllowed = []string{"credentials_id", "network_id", "storage_customer_managed_key_id", "private_access_settings_id", "managed_services_customer_managed_key_id", "custom_tags"}
+var workspaceRunningUpdatesAllowed = []string{"credentials_id", "network_id", "storage_customer_managed_key_id", "private_access_settings_id", "managed_services_customer_managed_key_id", "custom_tags", "expected_workspace_status"}


I know this isn't your fault, but would you mind formatting this list with one entry per line? Changes to this list are easier to review this way.

mgyucht · 2025-10-10T14:42:01Z

mws/resource_mws_workspaces.go

-func (a WorkspacesAPI) WaitForRunning(ws Workspace, timeout time.Duration) error {
+// If expected_workspace_status is specified, WaitForExpectedStatus will wait until workspace is in the expected status.
+// If not, it will wait until workspace is running, and otherwise will try to explain why it failed.
+func (a WorkspacesAPI) WaitForExpectedStatus(ws Workspace, expected_status string, timeout time.Duration) error {


Suggested change

func (a WorkspacesAPI) WaitForExpectedStatus(ws Workspace, expected_status string, timeout time.Duration) error {

func (a WorkspacesAPI) WaitForExpectedStatus(ws Workspace, expectedStatus string, timeout time.Duration) error {

mgyucht · 2025-10-10T14:49:59Z

mws/resource_mws_workspaces.go

-			err = workspacesAPI.WaitForRunning(workspace, d.Timeout(schema.TimeoutRead))
+			// The expected_workspace_status field is input only.
+			// Therefore, we need to read it from the original Terraform configuration.
+			expectedStatus := d.Get("expected_workspace_status").(string)


Suggested change

expectedStatus := d.Get("expected_workspace_status").(string)

// PROVISIONING workspace import may fail because the "expected_workspace_status" is not included in the state during import, nor is it returned by the API.

// As a result, the provider will (likely) wait for RUNNING state, which will never happen, and timeout.

// TODO: fix thisl.

expectedStatus := d.Get("expected_workspace_status").(string)

Added comment

github-actions · 2025-10-13T00:45:10Z

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/terraform

Inputs:

PR number: 5019
Commit SHA: 775ba1456d22a09558e96ebb828bbd45f0bd1355

Checks will be approved automatically on success.

Add expected_workspace_status in databricks_mws_workspaces to support…

8439599

… LPW

tinglin-db requested review from a team as code owners September 11, 2025 22:08

tinglin-db requested review from tanmay-db and removed request for a team September 11, 2025 22:08

added documentation + test

fbdc35a

tinglin-db changed the title ~~Add expected_workspace_status in databricks_mws_workspaces to support LPW~~ [Draft] Add expected_workspace_status in databricks_mws_workspaces to support LPW Sep 11, 2025

tinglin-db added 3 commits September 11, 2025 17:19

Modified test cases

b65b35a

added integration test

24bc883

remove debug logging

21cca40

nkvuong reviewed Sep 12, 2025

View reviewed changes

mws/resource_mws_workspaces.go Outdated Show resolved Hide resolved

mws/resource_mws_workspaces.go Outdated Show resolved Hide resolved

mws/resource_mws_workspaces.go Outdated Show resolved Hide resolved

tinglin-db added 3 commits September 11, 2025 22:01

address comments + add validation for expected_workspace_status

a85bca6

remove tf: optional

396c183

uncomment test file

c4522cf

tinglin-db temporarily deployed to test-trigger-is September 12, 2025 06:43 — with GitHub Actions Inactive

fmt

df2044b

tinglin-db temporarily deployed to test-trigger-is September 12, 2025 07:33 — with GitHub Actions Inactive

Merge branch 'main' into lpwterraform

bd8c44b

alexott temporarily deployed to test-trigger-is September 17, 2025 09:46 — with GitHub Actions Inactive

tinglin-db and others added 2 commits September 18, 2025 18:02

modified read logic, workspaceRunningUpdatesAllowed

6362f73

Merge branch 'main' into lpwterraform

0bf4141

alexott temporarily deployed to test-trigger-is September 19, 2025 07:26 — with GitHub Actions Inactive

fix integration test to skip destruction due to not being able to del…

a507db9

…ete ws in provisioning state

Merge branch 'main' into lpwterraform

9d710e3

tinglin-db temporarily deployed to test-trigger-is September 26, 2025 08:49 — with GitHub Actions Inactive

catch INVALID_STATE_TRANSITION for deleting PROVISIONING workspaces

c5efa0c

tinglin-db temporarily deployed to test-trigger-is September 26, 2025 22:12 — with GitHub Actions Inactive

tinglin-db commented Sep 26, 2025

View reviewed changes

mgyucht reviewed Oct 2, 2025

View reviewed changes

NEXT_CHANGELOG.md Outdated Show resolved Hide resolved

changed API call to expected_workspace_status, removed redundant wait…

08466fc

… logic, removed deletion skip, removed read skip

tinglin-db force-pushed the lpwterraform branch from 3af696c to 08466fc Compare October 7, 2025 21:52

mgyucht reviewed Oct 8, 2025

View reviewed changes

tinglin-db temporarily deployed to test-trigger-is October 8, 2025 17:23 — with GitHub Actions Inactive

tinglin-db added 2 commits October 8, 2025 10:37

more fixes

0b7b0f7

added tests + more fixes

05bf244

mgyucht reviewed Oct 10, 2025

View reviewed changes

address last nits

c2104e4

tinglin-db force-pushed the lpwterraform branch from a5a7519 to c2104e4 Compare October 13, 2025 00:42

Update NEXT_CHANGELOG.md

775ba14

	func (a WorkspacesAPI) WaitForExpectedStatus(ws Workspace, expected_status string, timeout time.Duration) error {
	func (a WorkspacesAPI) WaitForExpectedStatus(ws Workspace, expectedStatus string, timeout time.Duration) error {

-			expectedStatus := d.Get("expected_workspace_status").(string)
+			// PROVISIONING workspace import may fail because the "expected_workspace_status" is not included in the state during import, nor is it returned by the API.
+			// As a result, the provider will (likely) wait for RUNNING state, which will never happen, and timeout.
+			// TODO: fix thisl.
+			expectedStatus := d.Get("expected_workspace_status").(string)

[Draft] Add expected_workspace_status in databricks_mws_workspaces to support LPW #5019

Are you sure you want to change the base?

[Draft] Add expected_workspace_status in databricks_mws_workspaces to support LPW #5019

Conversation

tinglin-db commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

nkvuong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexott commented Sep 12, 2025

Uh oh!

tinglin-db commented Sep 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexott commented Sep 27, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mgyucht left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tinglin-db Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgyucht left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

tinglin-db commented Sep 11, 2025 •

edited

Loading

tinglin-db Oct 8, 2025 •

edited

Loading