Skip to content

gcp: skip terracurl destroy for migration exec instead of failing on stale token - #280

Open
ankur512512 wants to merge 1 commit into
fleetdm:mainfrom
ankur512512:fix-gcp-terracurl-destroy-skip
Open

gcp: skip terracurl destroy for migration exec instead of failing on stale token#280
ankur512512 wants to merge 1 commit into
fleetdm:mainfrom
ankur512512:fix-gcp-terracurl-destroy-skip

Conversation

@ankur512512

Copy link
Copy Markdown

Summary

  • terracurl_request.exec (in gcp/byo-project/cloud_run.tf, used to trigger the Fleet DB migration job) wires up a real destroy_url/destroy_method/destroy_headers for its destroy step, but the Authorization bearer token in destroy_headers is captured once at create/apply time via data.google_client_config.default.access_token and frozen into resource state.
  • OAuth access tokens are short-lived (~1 hour). In practice exec_migration is usually only flipped back to false well after the initial apply, so by the time this resource is destroyed, the stored token has expired and GCP rejects the destroy request.
  • When that happens, terraform apply/destroy fails with an unhelpful Error: unable to make request: request failed, retries exceeded: %!s(<nil>) — no indication that it was actually a stale-token 401/403. (Root cause traced into the terracurl provider itself: resource_curl.go's destroy retry loop sets lastError = err after getting back a non-matching response code, but err is still nil from the preceding successful client.Do() call a few lines up — the real per-attempt error is never captured.)
  • The destroy call was already documented in-code as a no-op formality ("we don't use the terracurl_request data source, as that will result in repeated job runs on every refresh") — it isn't undoing anything real. So rather than retrying a request that's expected to eventually fail on a stale token, this sets destroy_skip = true and drops the now-unused destroy_url/destroy_method/destroy_response_codes/destroy_headers attributes.

Reproduction (hit this on a real deploy)

  1. terraform apply with fleet_config.exec_migration = true to run initial migrations.
  2. Some time later (past typical token TTL), set exec_migration = false and terraform apply again to remove the trigger resource.
  3. Destroy fails with the opaque retries exceeded: <nil> error above; a manual curl with a freshly minted token against the same destroy URL succeeds immediately with 200, confirming it's a stale-token issue, not a permissions/network one.

Test plan

  • terraform fmt -check -diff on gcp/byo-project — clean
  • terraform init && terraform validate on gcp/byo-project — succeeds
  • Verified in a live GCP deployment: with this change, terraform apply no longer attempts the fragile destroy path when exec_migration is toggled off after the token would otherwise have expired.

Rollback

Revert this commit — restores the previous destroy_url/destroy_method/destroy_response_codes/destroy_headers block exactly as it was.

…stale token

The destroy_url/destroy_method/destroy_headers on terracurl_request.exec
send an Authorization bearer token that was captured at create/apply time
via data.google_client_config.default.access_token. That token is baked
into resource state and will have expired (OAuth access tokens are
short-lived, ~1 hour) by the time this resource is actually destroyed in
most real deployments, since exec_migration is typically only flipped
back to false well after the initial apply.

When that happens, GCP rejects the destroy request, and the terracurl
provider's retry loop fails to record the actual response as the retry
error (a bug in the provider itself: internal/provider/resource_curl.go
sets lastError = err after a non-matching response code, but err is still
nil from the preceding successful client.Do() call), so the surfaced
error is an unhelpful "request failed, retries exceeded: <nil>" with no
indication it was a stale-token 401/403.

The destroy call was already documented as a no-op formality (it never
undoes the migration), so destroy_skip = true avoids the fragile path
entirely instead of retrying a request that is expected to fail.
@ankur512512
ankur512512 requested review from a team and ddribeiro as code owners August 26, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant