Use SPDK dynamic scheduler

[judahrand]

Is your feature request related to a problem? Please describe.
When running Mayastor on clusters which have variable workloads CPU usage and power consumption increases significantly.

Describe the solution you'd like
SPDK has a dynamic scheduler which, I believe, should reduce the power consumption without significantly impacting performance. Crucially, when threads are idle they are switched into interrupt mode and stop polling.

Additional context
I've tried deploying Mayastor at home but the increased power consumption was enough for me to abandon it. I don't have much understanding of SPDK but thought I'd put this up as from my reading it seems like this might help. I'd welcome being further educated though!

[judahrand]

It would also be fantastic if Mayastor could run in interrupt mode for cases where low latency is not important! This was added for NVMEoF TCP in the newest version of SPDK, I believe.

https://spdk.io/release/2024/09/27/v24.09_release/

[tiagolobocastro]

Indeed seems SPDK now has interrupt mode for nvme-tcp. We have to evaluate this mode itself and then figure out how we can integrate with it.
We have our own reactors which would need adapting to support this.
I think this would be a very good enhancement for Mayastor.

[skwair]

Just adding some measurements I did to this issue for reference: I'm running a Kubernetes cluster with Talos 1.9.0 on 3 NUCs (Intel Ultra 5 125H, 48GB RAM), idling at around 11-13W (measured at the plug) each.

After deploying Mayastor, each machine increases its power consumption by ~17W, totaling to ~30W per unit, which is pretty significant.

Looking forward to see how switching to the interrupt mode could improve this!

[jr42]

I have a smart home setup with 3 mini PCs (Intel N150) running talos. This is definitely needed.

As a workaround I reduced to a single io-engine and pinned it to a single core. This core I set to the lowest frequency (700MHz in my case). That is still enough to saturate a GbE link and the downsides are slightly higher latency (still better than longhorn v1) and wasting 3/4 of a core. On the upside I save 70€ in electricity a year that way. Enough to upgrade one mini pc every year.

values.yaml:

mayastor:
  io_engine:
    envcontext: "iova-mode=pa"
    cpuCount: 1
    coreList: 
      - 3

worker.yaml:

machine:
  sysfs:
    devices.system.cpu.cpu3.cpufreq.scaling_max_freq: "700000"

When replicating this check, if your CPU supports single core frequency scaling and figure out the frequency needed to saturate your link.

[thomasdba]

any updates or plan ? thx.

[lion7]

I'm also very interested in this feature. Like others I'm running Talos on my home lab server which usually is rather idle. After installing mayastor both the power consumption and noise significantly increased. Normally the fans only start spinning when I actually do some heavy stuff on this server, but now it's constantly at high rpms. It would be nice if this can be avoided in this use case.

[jdblack]

I moved to longhorn. Mayastore was unusable for me

…

On Thu, Aug 14, 2025, 10:50 AM Gerard de Leeuw ***@***.***> wrote: *lion7* left a comment (openebs/mayastor#1745) <#1745 (comment)> I'm also very interested in this feature. Like others I'm running Talos on my home lab server which usually is rather idle. After installing mayastor both the power consumption and noise significantly increased. Normally the fans only start spinning when I actually do some heavy stuff on this server, but now it's constantly at high rpms. It would be nice if this can be avoided in this use case. — Reply to this email directly, view it on GitHub <#1745 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIO4LPXVYKHNUFKSKSF24T3NQBPDAVCNFSM6AAAAACD3KV5YCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCOBWG4YTENBXGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[tiagolobocastro]

any updates or plan ? thx.

Not at the moment, don't think anyone has managed to take a look at this; if someone from the community is willing to help out here, it'd be great.
Btw some way of reducing the power consumption may be to configure the the cpu min/max limits set for the io-engine on the helm-chart. Of course this also adversely affects the peak performance, but if perf is not a case for you then could be worth testing with this.

[gabrielgbs97]

The main problem with SPDK on virtualized environments is that polling mode inducts a high CPU usage (it would be manageable when using interrupt mode). This has some issues:

• Pure polling wastes CPU cycles on 'nothing useful' to maintain low latency. Also requires setting a high number of CPU reservations (hi io-engines !). Overprovisioning CPU resources is not common in Kubernetes, this makes the Kubernetes cluster have beefy nodes and 80% of reserved resources are infrastructure. I experienced stability issues (I was ready to trade for low performance at least) if downgrading io-engine CPU reservations and limits aggressively.
• High CPU usage on vCPUs makes overprovisioning in an on-prem virtualized infrastructure a risky strategy. Kubernetes VMs are noisy neighbors...

For our use case this is stopping us from enabling mayastor on production cases. HA storage will run under good old NFS backend.

[tiagolobocastro]

• I experienced stability issues (I was ready to trade for low performance at least) if downgrading io-engine CPU reservations and limits aggressively

Would you be able to elaborate on what issues you faced; this is not something we test so if there's something wrong with this approach it would be worth documenting this.