feat(metis): add Prometheus metrics collection and HTTP endpoint - #1327
feat(metis): add Prometheus metrics collection and HTTP endpoint#1327YifeiZhuang wants to merge 3 commits into
Conversation
This change introduces comprehensive Prometheus metrics for the Metis daemon
and CNI plugin to monitor IP address allocation, capacity utilization, gRPC RPC
latencies, and controller sync actions.
Key Features & Changes:
1. Metrics Package (`pkg/metrics`):
- Created a standalone `pkg/metrics` package using `promauto` to prevent circular
dependencies between `pkg/daemon` and `pkg/cni`.
- Exported metrics for:
- Daemon gRPC requests (`metis_grpc_server_handled_total`, `metis_rpc_latency_seconds`)
- CNI plugin calls (`metis_cni_plugin_handled_total`, `metis_cni_rpc_latency_seconds`)
- Dynamic allocation requests (`metis_outgoing_dynamic_ip_alloc_request_total`, `metis_dynamic_ip_alloc_rpc_latency_seconds`)
- Local store IP & CIDR capacity/utilization (`metis_local_store_ip_total`, `metis_local_store_cidr_block_total`)
- Watcher & Monitor operations (`metis_watcher_cidr_operation_count`, `metis_monitor_action_count`, `metis_pending_dynamic_request`)
- Documented all metrics with detailed Go doc comments.
2. Configurable Metrics HTTP Server:
- Added `--metrics-port` daemon option (default 9996).
- If `--metrics-port > 0`, an HTTP server exposes `/metrics` using `promhttp.Handler()`,
and metrics collection is enabled across all daemon sub-components (`Server`, `Engine`, `Monitor`, `Watcher`).
- If `--metrics-port <= 0` (or `0`), the HTTP server is not started and Prometheus metric recording is disabled.
3. Store IP & CIDR Accounting Refactoring:
- Refactored `NetworkIPUsage` into nested `IPs` (`IPUsage`) and `CIDRs` (`CIDRUsage`) structs
for clean separation of IP states (allocated, cooldown, draining, deleting, active total, total)
and CIDR block states (ready, draining, deleting).
4. Testing:
- Added unit tests for metric initialization and HTTP `/metrics` endpoint scraping in `pkg/metrics`.
- Updated existing unit tests in `pkg/daemon`, `pkg/cni`, and `pkg/store` to align with new function signatures.
|
This issue is currently awaiting triage. If the repository mantainers determine this is a relevant issue, they will accept it by applying the The DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: YifeiZhuang The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
b8f48e3 to
0d2edc3
Compare
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
This change introduces Prometheus metrics for the Metis daemon and CNI plugin to monitor IP address allocation, capacity utilization, gRPC RPC latencies, and controller sync actions.
Key Features & Changes:
Metrics Package (
pkg/metrics):pkg/metricspackage usingpromauto.promautois better for Metis because all metrics in pkg/metrics are static package-level variables initialized once when the process starts. It eliminates repetitive init() boilerplate while guaranteeing that all metrics are registered before any handler or daemon component uses them.metis_grpc_server_handled_total,metis_rpc_latency_seconds)metis_cni_plugin_handled_total,metis_cni_rpc_latency_seconds)metis_outgoing_dynamic_ip_alloc_request_total,metis_dynamic_ip_alloc_rpc_latency_seconds)metis_local_store_ip_total,metis_local_store_cidr_block_total)metis_watcher_cidr_operation_count,metis_monitor_action_count,metis_pending_dynamic_request)Configurable Metrics HTTP Server:
- Added
--metrics-portdaemon option (default 9996).- If
--metrics-port > 0, an HTTP server exposes/metricsusingpromhttp.Handler(),and metrics collection is enabled across all daemon sub-components (
Server,Engine,Monitor,Watcher).- If
--metrics-port <= 0(or0), the HTTP server is not started and Prometheus metric recording is disabled.Store IP & CIDR Accounting Refactoring:
NetworkIPUsageinto nestedIPs(IPUsage) andCIDRs(CIDRUsage) structsfor clean separation of IP states (allocated, cooldown, draining, deleting, active total, total)
and CIDR block states (ready, draining, deleting).
cc. @arvindbr8 @gnossen @zhaoqsh