Skip to content

feat(metis): add Prometheus metrics collection and HTTP endpoint - #1327

Open
YifeiZhuang wants to merge 3 commits into
kubernetes:masterfrom
YifeiZhuang:metis-metrics
Open

feat(metis): add Prometheus metrics collection and HTTP endpoint#1327
YifeiZhuang wants to merge 3 commits into
kubernetes:masterfrom
YifeiZhuang:metis-metrics

Conversation

@YifeiZhuang

Copy link
Copy Markdown
Contributor

This change introduces Prometheus metrics for the Metis daemon and CNI plugin to monitor IP address allocation, capacity utilization, gRPC RPC latencies, and controller sync actions.

Key Features & Changes:

  1. Metrics Package (pkg/metrics):

    • Created a standalone pkg/metrics package using promauto. promauto is better for Metis because all metrics in pkg/metrics are static package-level variables initialized once when the process starts. It eliminates repetitive init() boilerplate while guaranteeing that all metrics are registered before any handler or daemon component uses them.
    • Exported metrics for:
      • Daemon gRPC requests (metis_grpc_server_handled_total, metis_rpc_latency_seconds)
      • CNI plugin calls (metis_cni_plugin_handled_total, metis_cni_rpc_latency_seconds)
      • Dynamic allocation requests (metis_outgoing_dynamic_ip_alloc_request_total, metis_dynamic_ip_alloc_rpc_latency_seconds)
      • Local store IP & CIDR capacity/utilization (metis_local_store_ip_total, metis_local_store_cidr_block_total)
      • Watcher & Monitor operations (metis_watcher_cidr_operation_count, metis_monitor_action_count, metis_pending_dynamic_request)
    • Documented all metrics with detailed Go doc comments.
  2. Configurable Metrics HTTP Server:
    - Added --metrics-port daemon option (default 9996).
    - If --metrics-port > 0, an HTTP server exposes /metrics using promhttp.Handler(),
    and metrics collection is enabled across all daemon sub-components (Server, Engine, Monitor, Watcher).
    - If --metrics-port <= 0 (or 0), the HTTP server is not started and Prometheus metric recording is disabled.

  3. Store IP & CIDR Accounting Refactoring:

    • Refactored NetworkIPUsage into nested IPs (IPUsage) and CIDRs (CIDRUsage) structs
      for clean separation of IP states (allocated, cooldown, draining, deleting, active total, total)
      and CIDR block states (ready, draining, deleting).

cc. @arvindbr8 @gnossen @zhaoqsh

This change introduces comprehensive Prometheus metrics for the Metis daemon
and CNI plugin to monitor IP address allocation, capacity utilization, gRPC RPC
latencies, and controller sync actions.

Key Features & Changes:
1. Metrics Package (`pkg/metrics`):
   - Created a standalone `pkg/metrics` package using `promauto` to prevent circular
     dependencies between `pkg/daemon` and `pkg/cni`.
   - Exported metrics for:
     - Daemon gRPC requests (`metis_grpc_server_handled_total`, `metis_rpc_latency_seconds`)
     - CNI plugin calls (`metis_cni_plugin_handled_total`, `metis_cni_rpc_latency_seconds`)
     - Dynamic allocation requests (`metis_outgoing_dynamic_ip_alloc_request_total`, `metis_dynamic_ip_alloc_rpc_latency_seconds`)
     - Local store IP & CIDR capacity/utilization (`metis_local_store_ip_total`, `metis_local_store_cidr_block_total`)
     - Watcher & Monitor operations (`metis_watcher_cidr_operation_count`, `metis_monitor_action_count`, `metis_pending_dynamic_request`)
   - Documented all metrics with detailed Go doc comments.

2. Configurable Metrics HTTP Server:
   - Added `--metrics-port` daemon option (default 9996).
   - If `--metrics-port > 0`, an HTTP server exposes `/metrics` using `promhttp.Handler()`,
     and metrics collection is enabled across all daemon sub-components (`Server`, `Engine`, `Monitor`, `Watcher`).
   - If `--metrics-port <= 0` (or `0`), the HTTP server is not started and Prometheus metric recording is disabled.

3. Store IP & CIDR Accounting Refactoring:
   - Refactored `NetworkIPUsage` into nested `IPs` (`IPUsage`) and `CIDRs` (`CIDRUsage`) structs
     for clean separation of IP states (allocated, cooldown, draining, deleting, active total, total)
     and CIDR block states (ready, draining, deleting).

4. Testing:
   - Added unit tests for metric initialization and HTTP `/metrics` endpoint scraping in `pkg/metrics`.
   - Updated existing unit tests in `pkg/daemon`, `pkg/cni`, and `pkg/store` to align with new function signatures.
@kubernetes-prow kubernetes-prow Bot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Aug 31, 2026
@kubernetes-prow

Copy link
Copy Markdown

This issue is currently awaiting triage.

If the repository mantainers determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 31, 2026
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: YifeiZhuang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Aug 31, 2026
@kubernetes-prow kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 4, 2026
@kubernetes-prow

Copy link
Copy Markdown

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant