[CONTP-1569] [CONTINT-5553] Add podCollectionMode field on kubeStateMetricsCore for node-side pod collection

[zhuminyi]

What does this PR do?

Adds features.kubeStateMetricsCore.podCollectionMode options are default and node_kubelet on the
v2alpha1 DatadogAgent CRD.

  apiVersion: datadoghq.com/v2alpha1
  kind: DatadogAgent
  metadata:
    name: datadog
    namespace: datadog
  spec:
    features:
      clusterChecks:
        enabled: true
        useClusterChecksRunners: true  
      kubeStateMetricsCore:
        enabled: true
        podCollectionMode: node_kubelet   # THE SINGLE NEW TOGGLE 

When set to node_kubelet, the operator:

• Updates the operator-generated cluster-side KSM ConfigMap (unless the user supplied their own conf) by:
  • Setting pod_collection_mode: cluster_unassigned and cluster_aggregates_enabled: true on the primary instance so it collects unscheduled pods while suppressing cluster-aggregate metrics.
  • Adding a second instance with pod_collection_mode: cluster_aggregates_only to collect those aggregate metrics centrally.
• Generates a separate node-agent ConfigMap containing a pods-only check with pod_collection_mode: node_kubelet, cluster_aggregates_enabled: true, and collectors: [pods].

When a user supplies features.kubeStateMetricsCore.conf, the operator still creates the node-side check but does not modify the user-owned cluster-side YAML. The user must configure both cluster_unassigned and cluster_aggregates_only cluster-side instances themselves.

This is equivalent to the following manual configuration:

spec:
  features:
    kubeStateMetricsCore:
      enabled: true
      conf:
        configData: |-
          cluster_check: true
          init_config:
          instances:
            - skip_leader_election: true
              cluster_aggregates_enabled: true # silence aggregate metrics
              pod_collection_mode: cluster_unassigned # collect only unscheduled pods
            - skip_leader_election: true
              pod_collection_mode: cluster_aggregates_only # collect aggregate metrics centrally

  override:
    nodeAgent:
      extraConfd:
        configDataMap:
          kubernetes_state_core.yaml: |-
            init_config:
            instances:
              - collectors:
                  - pods
                cluster_aggregates_enabled: true # silence aggregate metrics
                pod_collection_mode: node_kubelet # collect scheduled pods from the node's Kubelet

Result: scheduled-pod metrics are emitted by each node agent locally from the Kubelet (no API-server traffic for pods). Unscheduled pods and every other KSM resource continue to be collected by the primary cluster-side instance, while cluster-aggregate metrics are collected by the dedicated cluster_aggregates_only instance. Default behavior is semantically unchanged when the field is unset.

Motivation

In large clusters, pod metrics dominate the cardinality of the monolithic kubernetes_state_core
cluster check, making it the scaling bottleneck. The agent has supported offloading pod collection to
node-agents via pod_collection_mode: node_kubelet since 7.58, but customers had to assemble two
coordinated configs manually (features.kubeStateMetricsCore.conf.configData for the cluster side,
override.nodeAgent.extraConfd for the node side) and carefully avoid double-collection or losing
unscheduled-pod metrics as well as live without .total aggregate metrics. This field collapses that into a single typed toggle.

Describe your test plan

Unit tests added in this PR:

• TestKsmCheckConfigPodCollectionOnNode

  • When enabled, asserts the cluster-side configuration contains:
    • A cluster_unassigned instance with the full collector list and
      cluster_aggregates_enabled: true.
    • A separate cluster_aggregates_only instance without a collector list or
      cluster_aggregates_enabled.
  • When disabled, asserts there is only one instance and no
    pod_collection_mode is emitted.

• Test_ksmFeature_buildKSMCorePodsOnNodeConfigMap

  • Asserts the node-side ConfigMap metadata and complete parsed YAML shape.
  • Verifies the check has no cluster_check or skip_leader_election fields.
  • Verifies the pods-only instance contains:
    • pod_collection_mode: node_kubelet
    • cluster_aggregates_enabled: true
    • collectors: [pods]

Manual end-to-end validation

# Both ConfigMaps are generated by the operator:
kubectl get cm -n datadog | grep state-metrics
# Expect:
#   <dda>-kube-state-metrics-core-config
#   <dda>-kube-state-metrics-core-pods-on-node-config

# The node agent mounts the node-side check:
kubectl get ds -n datadog datadog-agent \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="agent")].volumeMounts[?(@.name=="ksm-core-pods-on-node-config")]}'
# Expect: mountPath /etc/datadog-agent/conf.d/kubernetes_state_core.d, readOnly: true

# The node-side KSM check is loaded:
kubectl exec -n datadog ds/datadog-agent -c agent -- agent configcheck \
  | sed -n '/kubernetes_state_core check/,/===/p'
# Expect:
#   pod_collection_mode: node_kubelet
#   cluster_aggregates_enabled: true
#   collectors: [pods]

# The node-side check is running and emitting metrics:
kubectl exec -n datadog ds/datadog-agent -c agent -- agent status \
  | awk '/kubernetes_state_core/,/Status:/' | grep -E "Total Runs|Metric Samples"
# Expect: Total Runs > 0 and Metric Samples > 0 after approximately 30 seconds.

# The cluster-side configuration contains the split collector and aggregator:
kubectl exec -n datadog deploy/datadog-cluster-agent -- agent clusterchecks \
  | sed -n '/===== Checks on .*cluster-checks-runner/,/===== /p' \
  | grep -E "kubernetes_state_core|pod_collection_mode|cluster_aggregates_enabled"
# Expect:
#   cluster_unassigned with cluster_aggregates_enabled: true
#   a separate cluster_aggregates_only instance

Minimum Agent Versions

• Node Agent: v7.82.0+
• Cluster Agent: v7.82.0+
• Cluster Checks Runner: v7.82.0+ when the KSM check runs there

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 83.53%
• Overall Coverage: 50.75% (+0.58%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3e30362 | Docs | View more details | Give us feedback!}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 81.52866% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.58%. Comparing base (4e59e1d) to head (c2b6cd3).
⚠️ Report is 223 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/testutils/builder.go	0.00%	12 Missing ⚠️
...atadogagent/feature/kubernetesstatecore/feature.go	83.07%	5 Missing and 6 partials ⚠️
...adogagent/feature/kubernetesstatecore/configmap.go	92.50%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3027      +/-   ##
==========================================
+ Coverage   41.50%   41.58%   +0.07%     
==========================================
  Files         335      334       -1     
  Lines       28714    28795      +81     
==========================================
+ Hits        11919    11975      +56     
- Misses      16001    16019      +18     
- Partials      794      801       +7     

Flag	Coverage Δ
unittests	41.58% <81.52%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../datadogagent/feature/kubernetesstatecore/const.go	100.00% <ø> (ø)
...adogagent/feature/kubernetesstatecore/configmap.go	92.07% <92.50%> (-5.15%)	⬇️
...atadogagent/feature/kubernetesstatecore/feature.go	80.00% <83.07%> (+0.74%)	⬆️
pkg/testutils/builder.go	0.00% <0.00%> (ø)

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e59e1d...c2b6cd3. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Mathew-Estafanous]

A couple small nits but everything else looks good.

[linear-code]

CONTP-191