[DADP-182] Enable ADP by default for Linux Operator workloads - #3401
[DADP-182] Enable ADP by default for Linux Operator workloads#3401thieman wants to merge 23 commits into
Conversation
Add a CLI and environment option for enabling the data plane by default on Linux DatadogAgentInternals. Propagate the resolved value into feature options while excluding Windows profile DDAIs and preserving ordinary DDAIs with stray Windows annotations.
Use the same resolved Data Plane default for sidecar activation and DogStatsD port ownership so default-enabled Linux workloads delegate DogStatsD consistently.
Extend the feature-default regression case to verify the Agent Data Plane sidecar requirement and its delegation environment variables. This ensures omitted DataPlane CRD configuration exercises the complete default-enabled ADP path.
Configure all Core/ADP interaction flags on the shared unprivileged Agent container when ADP is enabled under the single-container strategy. Keep the DogStatsD UDP host port on that shared container instead of routing it to an ADP sidecar.
Mirror DaemonSet feature-hook dispatch for ExtendedDaemonSets so single-container workloads receive shared Agent Data Plane configuration.
Exercise an Operator-managed Linux DatadogAgent with the data-plane block omitted and the Operator default enabled through Helm environment values. Verify the single container configuration, ADP socket ownership, DogStatsD delivery, and failure diagnostics in the real Kind suite.
Honor the deprecated false annotation before the Linux rollout default and document the effective Data Plane behavior for optimized and single-container workloads.
Synchronize bundle CRD documentation and assert that both legacy annotation values emit the migration warning.
Regenerate the Marketplace CRD template from the updated Operator API documentation.
|
This reverts commit 7e2d99c.
Keep the rollout behavior focused on runtime code and tests. Regenerate distribution schemas as part of the versioned release workflow.
| "When true (requires --untaintControllerEnabled), the Untaint controller removes the startup taint only after both the node Agent and Datadog CSI node-server pods are Ready. Requires Pod watch coverage of CSI namespaces (DD_CSIDRIVER_WATCH_NAMESPACE).") | ||
| flag.BoolVar(&opts.rolloutOnConfigMapChangeEnabled, "rolloutOnConfigMapChangeEnabled", true, | ||
| "Automatically roll out Agent/Cluster Agent/Cluster Check Runner/OTel Agent Gateway workloads when a ConfigMap referenced by their pod template changes content out-of-band") | ||
| flag.BoolVar(&opts.defaultDataPlaneLinuxEnabled, "defaultDataPlaneLinuxEnabled", false, "Enable the data plane by default for Linux DatadogAgentInternals") |
There was a problem hiding this comment.
Change help to "Enable the Agent Data Plane by default on Linux"
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Fixed in fb31281: the flag now reads Enable the Agent Data Plane by default on Linux.
| | Controller revisions | `--createControllerRevisions` | `DD_CREATE_CONTROLLER_REVISIONS` | `false` | | ||
| | Linux Data Plane default | `--defaultDataPlaneLinuxEnabled` | `DD_DEFAULT_DATA_PLANE_LINUX_ENABLED` | `false` | | ||
|
|
||
| The Linux Data Plane default applies only when `spec.features.dataPlane.enabled` is unset. An explicit CRD value takes precedence, followed by the deprecated `agent.datadoghq.com/adp-enabled` annotation. Set `spec.features.dataPlane.enabled: false` to opt an individual workload out. Windows profiles do not inherit this default. With the optimized container strategy, Data Plane runs in a sidecar. With the single container strategy, the Agent image's s6 supervisor runs it as a process in the Agent container. |
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Removed in fb31281. The detailed rollout behavior remains in the chart value documentation and PR description rather than duplicating it here.
| // It should do nothing if the feature doesn't need to configure it. | ||
| func (f *dataPlaneFeature) ManageSingleContainerNodeAgent(managers feature.PodTemplateManagers) error { | ||
| return f.ManageNodeAgent(managers) | ||
| if !f.enabled { |
There was a problem hiding this comment.
It looks like you mostly inlined a bunch of ManageNodeAgent here, can you reduce the duplication? What actually needs to change? Also are there any other uses of ManageNodeAgent or did you get rid of the only one?
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Reduced in fb31281. ManageSingleContainerNodeAgent and ManageNodeAgent now delegate to one helper with the Core and ADP target containers as parameters. ManageNodeAgent remains the optimized DaemonSet path; the EDS loop also uses it for optimized workloads.
| }) | ||
|
|
||
| // --- Subtest: DSD UDP, single-container ADP enabled by the Operator default --- | ||
| s.T().Run("Single-container DSD UDP uses the Operator ADP default", func(t *testing.T) { |
There was a problem hiding this comment.
This is a pretty involved test for the single container use case, do we have an equivalent for multi-container?
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Yes: DSD UDP with ADP immediately above is the multi-container counterpart for the ADP sidecar, UDP host-port ownership, and fake-intake metric delivery. The new test is separate because it must assert the one-container/s6 process contract. I am currently fixing the live single-container E2E failure rather than treating the existing multi-container test as proof of that distinct runtime path.
Assert that both the DatadogAgent and its internal resource retain the requested container strategy before validating the rendered runtime pod.
Exercise runtime default, legacy annotations, and explicit CRD override precedence directly to protect the rollout opt-out contract.
Exercise both enabled and disabled paths of the shared single-container ADP environment configuration.
Use one target-container helper for optimized and single-container Agent workloads, and align the Linux default flag help text with its user-facing behavior.
Service Discovery requires the privileged system-probe container, so its Agent-version default prevents the unprivileged single-container strategy from applying. Disable it explicitly in the focused ADP runtime fixture.
Keep the single-container shape, configuration, listener ownership, and metric-delivery assertions while removing temporary diagnostic collection that duplicated the suite helpers and emitted broad pod and container-log output.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d2fd3b7a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| f.enabled = featureutils.IsDataPlaneEnabled(dda, ddaSpec) | ||
| f.enabled = featureutils.IsDataPlaneEnabled(dda, ddaSpec, f.defaultEnabled) |
There was a problem hiding this comment.
When the Operator/Helm default is enabled, a workload with no explicit Data Plane setting is enabled unconditionally, including workloads that pin a node Agent image from before the agent-data-plane executable was included. The generated sidecar then runs agent-data-plane ... from that pinned image and CrashLoops, making the Agent DaemonSet unavailable after an Operator upgrade. Apply the default only when the selected Agent image meets the minimum ADP version, while still allowing an explicit workload-level enablement to take precedence.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Verified. Unpinned workloads use the current default Agent image, but spec.override.nodeAgent.image supports pinned older versions. Fixed in 22b5aab: the Operator-level default now applies only when the selected Agent image is 7.81.0 or newer. Explicit spec.features.dataPlane.enabled and the legacy annotation retain their existing opt-in behavior for older images. Regression coverage includes a pinned 7.80 image (default remains disabled), a pinned 7.81 image (default enables ADP), and explicit enablement on 7.80.
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Updated the eligibility threshold per release coordination: 88d016d now applies the global default only to selected Agent images at 7.83.0 or newer. Explicit CRD and legacy-annotation opt-ins remain unchanged. Default-path tests use the published 7.83.0-rc.5 image; the complete CGO-disabled Operator suite passes.
Apply the Operator-level default only to Agent 7.81 and newer, including an explicitly selected node Agent image. Explicit CRD configuration and legacy annotation opt-ins retain their existing behavior for older versions.
Raise the default eligibility threshold to Agent 7.83. Explicit Data Plane and legacy annotation opt-ins remain available for older Agent versions. Pin default-path tests to an available 7.83 release candidate.
Resolve upstream changes while retaining the Linux ADP default and single-container DaemonSet handling. ExtendedDaemonSet runtime support was removed from main, so the obsolete EDS-specific path and test are not retained.
Exercise the Linux ADP default and explicit CRD opt-out in one GKE Autopilot suite using compatible Agent 7.83 release-candidate images. The opt-out workload uses loopback intake endpoints so it cannot emit test telemetry to the real intake.
Log only the matching Agent pod states and DatadogAgentInternal conditions when the GKE Autopilot ADP assertions fail, preserving the existing behavioral assertions.
2 participants
Configuration
New controls
--defaultDataPlaneLinuxEnabledfalseDD_DEFAULT_DATA_PLANE_LINUX_ENABLEDfalse--defaultDataPlaneLinuxEnabled; an explicit CLI argument takes precedence.defaultDataPlaneEnabled.linuxtruein thedatadog-operatorHelm chartDD_DEFAULT_DATA_PLANE_LINUX_ENABLED=true. The default applies to Agent 7.83+; non-Helm installations retain the binary default unless configured.Existing and deprecated controls
spec.features.dataPlane.enabledtrueorfalseis the authoritative per-workload setting. In particular,falseopts an individual workload out of the Linux default.agent.datadoghq.com/adp-enabled"true"enables ADP and"false"opts out. Both values log a deprecation warning.Enablement precedence is: explicit
spec.features.dataPlane.enabled, then the legacy annotation, then the Operator Linux default. The global default applies only when the selected Agent image is version 7.83 or later; explicit CRD and legacy-annotation opt-ins remain available for older Agent images. Windows profiles do not inherit the Linux default.Summary
Relates to DADP-182.
Validation
Automated local validation
CGO_ENABLED=0 make ci-testmake lint-e2eGOWORK=off go test -C test/e2e/ ./tests/k8s_suite/... --tags=e2e -run '^$'On this Darwin ARM host, the normal CGO-enabled suite crashes in the third-party
go-m1cpuinitialization path; the complete suite passed withCGO_ENABLED=0.Prior local Kind runtime validation
Built DADP implementation commit
d87eb29bwithmake docker-build-ci IMG=dadp-operator:dadp-182, loaded that image into Kind, and verified the running Operator reports that commit.containerStrategy: single; the Agent pod contained onlyunprivileged-single-agent; all four ADP environment variables were set; the container declared UDP/8125 as a HostPort; and the s6-managedagent-data-planeprocess owned the UDP/8125 listener.features.dataPlaneblock removed, proving the Operator default enabled ADP. The resulting pod contained theagent-data-planesidecar; Core receivedDD_DATA_PLANE_ENABLEDandDD_DATA_PLANE_DOGSTATSD_ENABLED; ADP received the remote-agent/config-stream settings; only ADP declared UDP/8125 as a HostPort; and the ADP sidecar process owned the UDP/8125 listener.This validation used Agent
7.81.1, before the subsequent change that gates the default at Agent 7.83+.GKE Autopilot coverage
The GKE Autopilot suite now uses the published Agent
7.83.0-rc.5image and deploys two workloads in one cluster:features.dataPlane.enabled; the Operator receivesDD_DEFAULT_DATA_PLANE_LINUX_ENABLED=true; assertions require the ADP sidecar and its Core/ADP environment contract.spec.features.dataPlane.enabled: false; assertions require the ADP sidecar to be absent. Its Agent and Cluster Agent endpoints are redirected to loopback, so it cannot emit test telemetry to the real intake.make lint-e2eand the GKE suite compilation pass locally. Executingmake e2e-gke-autopilot-testsremains pending GCP Application Default Credentials, a configured GCP/Pulumi environment, and a registry-accessible branch Operator image. The DDCI Kubernetes-version matrix remains the final cross-version validation.