adding pool alert metrics by abhilashshetty04 · Pull Request #833 · openebs/mayastor-extensions

abhilashshetty04 · 2026-04-03T08:19:23Z

This PR adds additional metrics on the diskpool to expose alert releated information.

Abhinandan-Purkait · 2026-04-06T09:36:27Z

Code wise looks good. Although it would be good to see how this will be consumed and mapped to charts?

tiagolobocastro · 2026-04-08T14:38:43Z

+        self.alert_status
+    }
+
+    /// Get the collection of notice alert reasons for the pool.


What does this mean, what is this number, the enum value, the count.. ?

I thought of concatenating all i32 into one f64 value. Pasted example below. attention reason has
IoError and IoStrallIntermittent both.

./io-engine-client pool list -ojson
[
{
"capacity": 1069547520,
"cluster_size": 4194304,
"committed": 0,
"disk_capacity": 1073741824,
"disk_info": [
{
"errors": {
"alerts": {
"attention": [
4,
2
],
"critical": [
1
],
"notice": [],
"status": 3,
"warning": []
},
"io_error_count": 3,
"io_error_threshold": 8,
"io_stall_transition_count": 2,
"io_stall_transition_threshold": 3,
"io_stalled": true
},
"uri": "uring:///dev/mapper/lvmvg-newlv"
}
],
"disks": [
"uring:///dev/mapper/lvmvg-newlv"
],
"encrypted": false,
"errors": {
"alerts": {
"attention": [
4,
2
],
"critical": [
1
],
"notice": [],
"status": 3,
"warning": []
},
"io_error_count": 3,
"io_error_threshold": 8,
"io_stall_transition_count": 2,
"io_stall_transition_threshold": 3,
"io_stalled": true
},
"max_expandable_size": 137271181312,
"md_info": {
"md_page_size": 4096,
"md_pages": 256,
"md_used_pages": 1
},
"name": "pool-uring",
"page_size": 4096,
"pooltype": 0,
"state": 4,
"used": 0,
"uuid": "03de92fb-1f6b-4121-ba3b-ee31934b9abc"
}
]
/bin # %

After discussing with @krishnaGajabi , Im thinking of exposing a variant present or not using labels. Will see how that works.

tiagolobocastro · 2026-04-08T14:43:42Z

+
+    /// Get the alert status for the pool.
+    pub(crate) fn alert_status(&self) -> i32 {
+        self.alert_status


Is this the enum value?
In the node exporter, online is being created as a gauge, would it be better to do the same here, and create gauges for each status? (I don't know how these are typically exported...)

abhilashshetty04 · 2026-04-13T08:16:27Z

./io-engine-client pool list -ojson
[
{
"capacity": 5360320512,
"cluster_size": 4194304,
"committed": 0,
"disk_capacity": 5368709120,
"disk_info": [
{
"errors": {
"alerts": {
"attention": [
4
],
"critical": [],
"notice": [],
"status": 2,
"warning": [
3
]
},
"io_error_count": 3,
"io_error_threshold": 8,
"io_stall_transition_count": 3,
"io_stall_transition_threshold": 3,
"io_stalled": false
},
"uri": "uring:///dev/mapper/lvmvg-new"
}
],
"disks": [
"uring:///dev/mapper/lvmvg-new"
],
"encrypted": false,
"errors": {
"alerts": {
"attention": [
4
],
"critical": [],
"notice": [],
"status": 2,
"warning": [
3
]
},
"io_error_count": 3,
"io_error_threshold": 8,
"io_stall_transition_count": 3,
"io_stall_transition_threshold": 3,
"io_stalled": false
},
"max_expandable_size": 137271181312,
"md_info": {
"md_page_size": 4096,
"md_pages": 1280,
"md_used_pages": 1
},
"name": "pool-uring",
"page_size": 4096,
"pooltype": 0,
"state": 4,
"used": 0,
"uuid": "dba451a1-345d-4659-aa71-b252bf1b94ec"
}
]

This has attention and warning alert reasons. Please see alert with label name pool-uring.

When in stalled state, alert type is Critical:

Copilot

Copilot wasn't able to review any files in this pull request.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: Abhilash Shetty <abhilash.shetty@datacore.com>

Copilot

Copilot wasn't able to review any files in this pull request.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tiagolobocastro

Can you also add some tests as well?

tiagolobocastro · 2026-04-13T14:57:19Z

+    let opts = Opts::new(metric_name, metric_desc)
+        .subsystem("diskpool_alert")
+        .variable_labels(vec![
+            "node".to_string(),


let's place these in a vec and reuse it here and below to avoid discrepancies

node is part of Collector struct now. Is that what you meant?

tiagolobocastro · 2026-04-13T17:25:21Z

+        };
+        let cache_deref = cache.deref_mut();
+        let mut metric_family = Vec::with_capacity(3 * cache_deref.pool_mut().pools.capacity());
+        let node_name = match get_node_name() {


Why not place the name in the collector self?

tiagolobocastro · 2026-04-13T17:26:10Z

+            }
+        };
+        let cache_deref = cache.deref_mut();
+        let mut metric_family = Vec::with_capacity(3 * cache_deref.pool_mut().pools.capacity());


What is this doing? creating a vector with 3 x pool byte capacity??

Not byte capacity, cache_deref.pool_mut().pools holds Vec of PoolInfo. capacity is number of elements (poolInfo) present in the cache. cache is populated before the collect is called.

It should be 10 here as this Collector exports 10 metrics per pool.

tiagolobocastro · 2026-04-13T17:28:48Z

+    fn update_alerts(&mut self, pool_alerts: &Vec<i32>) {
+        for i in pool_alerts {
+            match i {
+                0 => self.unknown = 1,


seems quite brittle to just use these as raw int... should use the actual types

We get Vec from gRPC. Here, For each metrics like io_alert_critical_reason or io_alert_attention_reason we have labels (io_error, io_stall_intermittent_exc etc) with value 0 or 1 (unset or set).

Hence keeping everything in i32 seemed straighforward.

abhilashshetty04 requested a review from a team as a code owner April 3, 2026 08:19

Abhinandan-Purkait approved these changes Apr 6, 2026

View reviewed changes

tiagolobocastro reviewed Apr 8, 2026

View reviewed changes

abhilashshetty04 force-pushed the pool_alert_exporter branch from 8f786f7 to 2ec2498 Compare April 13, 2026 08:07

Abhinandan-Purkait requested a review from Copilot April 13, 2026 10:18

Copilot AI reviewed Apr 13, 2026

View reviewed changes

feat(exporter): adding pool alert metrics

8b62329

Signed-off-by: Abhilash Shetty <abhilash.shetty@datacore.com>

Abhinandan-Purkait force-pushed the pool_alert_exporter branch from 2ec2498 to 8b62329 Compare April 13, 2026 10:39

Abhinandan-Purkait requested a review from Copilot April 13, 2026 10:39

Copilot AI reviewed Apr 13, 2026

View reviewed changes

tiagolobocastro reviewed Apr 13, 2026

View reviewed changes

Conversation

abhilashshetty04 commented Apr 3, 2026

Uh oh!

Abhinandan-Purkait commented Apr 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhilashshetty04 commented Apr 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

tiagolobocastro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhilashshetty04 Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abhilashshetty04 Apr 15, 2026 •

edited

Loading