Skip to content

Gate managed Agent installation on workload readiness - #3397

Closed
fanny-jiang wants to merge 3 commits into
mainfrom
fanny/TON-833/gate-managed-install-readiness
Closed

Gate managed Agent installation on workload readiness#3397
fanny-jiang wants to merge 3 commits into
mainfrom
fanny/TON-833/gate-managed-install-readiness

Conversation

@fanny-jiang

@fanny-jiang fanny-jiang commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Prevents managed Agent installation from reporting success until the DatadogAgent has reconciled and every enabled workload is ready.

The installation remains RUNNING and the DatadogAgent remains marked partial while readiness is pending. The Fleet daemon retries readiness checks every five seconds and reports a terminal error after ten minutes. The readiness start time is persisted on the DatadogAgent so restarting the operator does not reset the deadline.

Readiness requires:

  • no active DatadogAgent or DatadogAgentInternal reconciliation errors
  • the Linux Agent and Cluster Agent workloads to be ready unless explicitly disabled
  • the Cluster Checks Runner and OTel Agent Gateway workloads to be ready when their features are enabled
  • the managed Windows-profile DatadogAgentInternal to reconcile successfully and its Agent workload to be ready

Fresh, unobserved all-zero workload statuses remain pending. A Windows Agent workload with a legitimate desired count of zero is accepted only after its DaemonSet generation has been observed and its native status also reports zero workloads.

The DatadogAgent status now aggregates OTel Agent Gateway status from DatadogAgentInternal profiles and detects changes to that status. Repeated RUNNING state writes are also deduplicated without discarding the current pending-readiness error.

Motivation

The EKS managed add-on installation could report success immediately after creating the DatadogAgent resource, before its controllers had reconciled or its workloads were available.

https://datadoghq.atlassian.net/browse/TON-833

Additional Notes

Explicitly disabled components are skipped. Optional components are checked only when enabled by the accepted DatadogAgent spec.

Minimum Agent Versions

No new minimum versions.

  • Agent: unchanged
  • Cluster Agent: unchanged

Describe your test plan

  • Run CGO_ENABLED=0 go test ./... -count=1
  • Run make lint
  • Verify pending or unobserved workloads keep the installation RUNNING and partial
  • Verify all enabled Linux and optional workloads becoming ready transitions the installation to DONE
  • Verify Windows readiness for pending, ready, and legitimate desired-zero DaemonSets
  • Verify active DatadogAgent and DatadogAgentInternal reconciliation errors block completion
  • Verify the persisted ten-minute readiness deadline produces a terminal ERROR and survives retries
  • Verify explicitly disabled components are skipped
  • Verify repeated RUNNING state updates preserve the pending error without unnecessary ConfigMap patches

Checklist

  • PR has at least one valid label: bug
  • PR has a milestone or the qa/skip-qa label
  • All commits are signed (see: signing commits)

@fanny-jiang fanny-jiang added the bug Something isn't working label Aug 24, 2026
@fanny-jiang

Copy link
Copy Markdown
Contributor Author

@codex review

@fanny-jiang fanny-jiang added this to the v1.30.0 milestone Aug 24, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 574c8c4ee3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread pkg/fleet/managed_agent_installation.go Outdated
Comment thread pkg/fleet/managed_agent_installation_command.go
@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Aug 24, 2026

Copy link
Copy Markdown

Tests  Code Coverage

🔄 Datadog auto-retried 1 job - 1 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 86.62%
Overall Coverage: 50.55% (+0.24%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a5b674d | Docs | View more details | Give us feedback!

@fanny-jiang
fanny-jiang marked this pull request as ready for review August 24, 2026 20:11
@fanny-jiang
fanny-jiang requested a review from a team August 24, 2026 20:11
@fanny-jiang
fanny-jiang requested a review from a team as a code owner August 24, 2026 20:11

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b2bb7a59f0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

}{
{name: "Cluster Agent", enabled: !managedAgentInstallationComponentDisabled(dda, v2alpha1.ClusterAgentComponentName), status: dda.Status.ClusterAgent},
{name: "Cluster Checks Runner", enabled: managedAgentInstallationOptionalComponentEnabled(dda, v2alpha1.ClusterChecksRunnerComponentName), status: dda.Status.ClusterChecksRunner},
{name: "OTel Agent Gateway", enabled: managedAgentInstallationOptionalComponentEnabled(dda, v2alpha1.OtelAgentGatewayComponentName), status: dda.Status.OtelAgentGateway},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Aggregate OTel gateway status before checking readiness

When a Fleet configuration enables the OTel Agent Gateway, this entry always waits on dda.Status.OtelAgentGateway, but the active DDAI reconciler writes the deployment status to DatadogAgentInternal.Status.OtelAgentGateway while addDDAIStatusToDDAStatus in internal/controller/datadogagent/reconcile_common.go aggregates only Agent, Cluster Agent, and CCR status. Consequently the top-level field remains nil even after the gateway is healthy, so every such managed installation stays RUNNING and then becomes ERROR after ten minutes; aggregate the OTel status or check the DDAI/workload directly.

Useful? React with 👍 / 👎.

@fanny-jiang
fanny-jiang force-pushed the fanny/TON-833/gate-managed-install-readiness branch from b2bb7a5 to a5b674d Compare August 27, 2026 16:53
@fanny-jiang

Copy link
Copy Markdown
Contributor Author

Closing this PR in favor of a backend-mediated approach

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants