Skip to content

Conversation

upodroid
Copy link

@upodroid upodroid commented Jul 8, 2025

  • add standard kubernetes labels
  • add one metric used for node exporter dashboards

@upodroid upodroid requested a review from a team as a code owner July 8, 2025 13:30
@CLAassistant
Copy link

CLAassistant commented Jul 8, 2025

CLA assistant check
All committers have signed the CLA.

@@ -99,6 +99,11 @@ extraLogProcessingStages: ""
# @section -- Log Processing
labelsToKeep:
- app.kubernetes.io/name
- app.kubernetes.io/instance
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -140,6 +140,7 @@
- node_systemd_service_restart_total
- node_systemd_unit_state
- node_textfile_scrape_error
- node_time_seconds
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@upodroid
Copy link
Author

upodroid commented Jul 8, 2025

#1296 had good pointers for how to combine LGTM stack + Alloy + kube-stack-prometheus dashboard/rules/alerts. It would be great to document that in the examples.

# helm upgrade --install k8s-monitoring grafana/k8s-monitoring -n observability -f values.yaml
# https://github.com/grafana/k8s-monitoring-helm/issues/1296

cluster:
  name: observability

global:
  scrapeInterval: 30s

destinations:
  - name: loki
    type: loki
    url: http://loki-gateway.observability.svc.cluster.local/loki/api/v1/push
  - name: mimir
    type: prometheus
    url: http://mimir-nginx.observability.svc.cluster.local/api/v1/push

clusterEvents:
  enabled: true

clusterMetrics:
  enabled: true
  apiServer:
    enabled: true
    jobLabel: apiserver
  kubelet:
    jobLabel: kubelet
    metricsTuning:
      # No filtering, keep all metrics
      useIntegrationAllowList: false
    extraMetricProcessingRules: |
      rule {
        replacement   = "/metrics"
        target_label  = "metrics_path"
      }
    scrapeInterval: 10s
  kubeletResource:
    jobLabel: kubelet
    metricsTuning:
      # No filtering, keep all metrics
      useIntegrationAllowList: false
    scrapeInterval: 10s
  cadvisor:
    jobLabel: kubelet
    metricsTuning:
      # No filtering, keep all metrics
      useIntegrationAllowList: false
    extraMetricProcessingRules: |
      rule {
        replacement   = "/metrics/cadvisor"
        target_label  = "metrics_path"
      }
    scrapeInterval: 10s
  node-exporter:
    jobLabel: node-exporter
    metricsTuning:
      # No filtering, keep all metrics
      useDefaultAllowList: false
  kube-state-metrics:
    jobLabel: kube-state-metrics
    metricsTuning:
      # No filtering, keep all metrics
      useDefaultAllowList: false
    metricLabelsAllowlist:
      - pods=[*]
      - deployments=[*]
  windows-exporter:
    deploy: false
    enabled: false

prometheusOperatorObjects:
  enabled: true
  crds:
    deploy: true
podLogs:
  enabled: true
nodeLogs:
  enabled: true
alloy-metrics:
  enabled: true
  liveDebugging:
    enabled: true
  # https://github.com/grafana/k8s-monitoring-helm/issues/1073
  extraConfig: |
    mimir.rules.kubernetes "default" {
      address = "http://mimir-ruler.observability.svc:8080"
      tenant_id = "anonymous"
    }
alloy-logs:
  enabled: true
  liveDebugging:
    enabled: true
alloy-singleton:
  enabled: true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants