Skip to content

[Backport v1.30] Fix YAML mapper for helm2dda for Helm's configmap based configurations - #3409

Open
dd-octo-sts[bot] wants to merge 1 commit into
v1.30from
backport-3373-to-v1.30
Open

[Backport v1.30] Fix YAML mapper for helm2dda for Helm's configmap based configurations#3409
dd-octo-sts[bot] wants to merge 1 commit into
v1.30from
backport-3373-to-v1.30

Conversation

@dd-octo-sts

@dd-octo-sts dd-octo-sts Bot commented Aug 28, 2026

Copy link
Copy Markdown

Backport 96de46d from #3373.


What does this PR do?

Fixes the following issues when the helm2dda flag is run for the Datadog Plugin for kubectl.

When using datadog.otelCollector.config in Helm like:

datadog:
  otelCollector:
    config: |
      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317

As a result, the mappings come out with three duplicate entries which the latter two are invalid:

spec:
  features:
    otelCollector:
      conf:
        configData: |
          receivers:
            otlp:
              protocols:
                grpc:
                  endpoint: 0.0.0.0:4317
        configMap:
          items: |
            receivers:
              otlp:
                protocols:
                  grpc:
                    endpoint: 0.0.0.0:4317
          name: |
            receivers:
              otlp:
                protocols:
                  grpc:
                    endpoint: 0.0.0.0:4317

In addition, using clusterAgent.confd like:

clusterAgent:
  confd:
    mysql.yaml: |
      cluster_check: true
      instances:
        - host: 1.2.3.4
          port: 3306

As a result, the mappings for extraConfd are missing the configDataMap like:

  override:
    clusterAgent:
      extraConfd:
        mysql.yaml: |-
          cluster_check: true
          instances:
            - host: 1.2.3.4
              port: 3306
  • Of note datadog.confd is unaffected.

In addition, agents.customAgentConfig and clusterAgent.datadog_cluster_yaml does not render in the generated DatadogAgent at all and errors out with:

2026/08/19 16:08:37 ERROR source value key was not found in mapping key=agents.customAgentConfig.log_level
2026/08/19 16:08:37 ERROR source value key was not found in mapping key=agents.customAgentConfig.tags
2026/08/19 16:08:37 ERROR source value key was not found in mapping key=clusterAgent.datadog_cluster_yaml.log_level

Motivation

CONS-8504

Additional Notes

Some things to flag which was not covered in this PR:
clusterAgent.advancedConfd is also broken as it would render something like the following which isn't valid:

  override:
    clusterAgent:
      extraConfd:
        configData:
          mysql.d:
            1.yaml: |
              cluster_check: true
              instances:
                - host: 1.2.3.4
                  port: 3306
            2.yaml: |-
              cluster_check: true
              instances:
                - host: 5.6.7.8
                  port: 3306

Despite my changes rendering the DatadogAgent CR properly, it still produced these phantom errors (first error might be expected as there is no equivalent in the DatadogAgent):

026/08/19 16:18:00 ERROR DDA destination key not found sourceKey=agents.useConfigMap
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=clusterAgent.confd.mysql.yaml
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=agents.customAgentConfig.log_level
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=agents.customAgentConfig.tags
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=datadog.confd.mysql.yaml
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=clusterAgent.datadog_cluster_yaml.log_level
  • Furthermore, did not want to make further changes to the error handling as I don't want to potentially prevent the plugin from suppressing legitimate errors.

Describe your test plan

Tested changes locally with the following values.yaml:

datadog:
  apiKey: <your api key>
  otelCollector:
    enabled: true
    config: |
      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
  operator:
    enabled: true
    migration:
      preview: true
  confd:
    mysql.yaml: |
      instances:
        - host: 1.2.3.4
          port: 3306
agents:
  useConfigMap: true
  customAgentConfig:
    log_level: debug
    tags: [foo:bar]
clusterAgent:
  confd:
    mysql.yaml: |
      cluster_check: true
      instances:
        - host: 1.2.3.4
          port: 3306
  datadog_cluster_yaml:
    log_level: debug

Then run the helm2dda flag with:

# run this in the root of the repo
make kubectl-datadog
export PATH="<path to datadog operator repo>/datadog-operator/bin:$PATH"

# then ran
kubectl datadog helm2dda \
  --sourcePath=values.yaml \
  --destPath=helm2dda.yaml \
  --namespace=datadog

Desired Output:

2026/08/19 16:18:00 ERROR DDA destination key not found sourceKey=agents.useConfigMap
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=clusterAgent.confd.mysql.yaml
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=agents.customAgentConfig.log_level
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=agents.customAgentConfig.tags
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=datadog.confd.mysql.yaml
2026/08/19 16:18:00 ERROR source value key was not found in mapping key=clusterAgent.datadog_cluster_yaml.log_level

Mapped DatadogAgent custom resource:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  namespace: datadog
spec:
  features:
    otelCollector:
      conf:
        configData: |
          receivers:
            otlp:
              protocols:
                grpc:
                  endpoint: 0.0.0.0:4317
      enabled: true
  global:
    credentials:
      apiKey: <api key>
  override:
    clusterAgent:
      customConfigurations:
        datadog-cluster.yaml:
          configData: |
            log_level: debug
      extraConfd:
        configDataMap:
          mysql.yaml: |
            cluster_check: true
            instances:
              - host: 1.2.3.4
                port: 3306
    nodeAgent:
      customConfigurations:
        datadog.yaml:
          configData: |
            log_level: debug
            tags:
            - foo:bar
      extraConfd:
        configDataMap:
          mysql.yaml: |
            instances:
              - host: 1.2.3.4
                port: 3306

2026/08/19 16:18:00 INFO YAML file successfully written path=helm2dda.yaml
2026/08/19 16:18:00 ERROR mapper run failed error="mapping completed with 6 error(s): the mapped DDA may contain misconfigurations"
  • The errors are expected as this PR does not fix those, see Additional Notes above.

Checklist

  • PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
  • PR has a milestone or the qa/skip-qa label
  • All commits are signed (see: signing commits)

#3373)

Fix YAML mapper for helm2dda for datadog.otelCollector.config and clusterAgent.confd

Add fixes for agents.customAgentConfig and clusterAgent.cluster_yaml

Restrict multi-key table fallback to mapFunc destinations

Allowing any multi-key table through the fallback (added to support
agents.customAgentConfig/clusterAgent.datadog_cluster_yaml) let it also
apply to plain string/list destinations like
agents.podSecurity.seLinuxContext, which have separate leaf mappings for
their sub-fields. That produced invalid duplicate/extra fields in the
output. Now the multi-key fallback only fires for mapFunc-based
destinations (e.g. mapCustomConfigFile), matching Codex review feedback
on PR #3373.

Co-authored-by: Cursor <cursoragent@cursor.com>

add test coverage for mapCustomConfigFile

Co-authored-by: Cursor <cursoragent@cursor.com>

fix multi-key table handling for helm2dda confd/customAgentConfig mappings

extend helm2dda test fixtures to cover multi-key confd and customAgentConfig mapping

Co-authored-by: patrick.liang <patrick.liang@datadoghq.com>
(cherry picked from commit 96de46d)
@dd-octo-sts dd-octo-sts Bot added bug Something isn't working mergequeue-status: done labels Aug 28, 2026
@dd-octo-sts
dd-octo-sts Bot requested a review from a team as a code owner August 28, 2026 14:36
@dd-octo-sts dd-octo-sts Bot added backport label added by backport action bot label added by backport bot team/container-platform labels Aug 28, 2026
@dd-octo-sts dd-octo-sts Bot added this to the v1.30.0 milestone Aug 28, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a221ed9294

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

if mapVal, ok := utils.GetPathMap(sourceValues[sourceKey]); ok && mapVal != nil {
pathVal = mapVal
} else if tableVal, err := sourceValues.Table(sourceKey); err == nil && len(tableVal) == 1 {
} else if tableVal, err := sourceValues.Table(sourceKey); err == nil && len(tableVal) > 0 && !hasMappedDescendant {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve parent mappings when descendants are intentionally empty

When a table has any descendant mapping entry, this condition now prevents its parent mapping from running even if those descendants are intentionally unmapped. For example, agents.updateStrategy: {type: OnDelete} should be copied through the parent mapping at mapping_datadog_helm_to_datadogagent_crd.yaml:150, while its type descendant at line 152 has an empty destination; hasMappedDescendant therefore skips the parent and the child emits an error without adding anything, so the generated DatadogAgent loses the requested update strategy. Distinguish descendants that replace the parent mapping from empty bookkeeping entries.

Useful? React with 👍 / 👎.

@datadog-official

datadog-official Bot commented Aug 28, 2026

Copy link
Copy Markdown

Pipelines  Code Coverage

⚠️ Warnings

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

DataDog/datadog-operator | build_operator_check_image_amd64

View more details · View in GitLab

ℹ️ Info

🎯 Code Coverage (details)
Patch Coverage: 97.22%
Overall Coverage: 50.17% (+0.06%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a221ed9 | Docs | View more details | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport label added by backport action bot label added by backport bot bug Something isn't working mergequeue-status: done team/container-platform

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant