Add cpu.max Support for the SCX Scheduler

[multics69]

This patch set adds cpu.max support [1] to the SCX scheduler.
It introduces a new library for CPU bandwidth control and integrates it with LAVD.
The series consists of four major parts:

1. cpu.max library implementation
2. ATQ/rbtree modifications required by the library
3. LAVD integration for cpu.max
4. Utility scripts for debugging and testing

---

1. Overview of the cpu.max Feature

The cpu.max interface controls CPU bandwidth for non-root cgroups using three parameters:

• $QUOTA
• $PERIOD
• $BURST

A cgroup may consume up to $QUOTA runtime in each $PERIOD.
Using max for $QUOTA removes the limit.
At the end of each period, unused time is discarded unless $BURST is specified—allowing carryover up to $BURST.

---

2. Design Overview

Our implementation follows the cgroup v2 cpu.max semantics but differs from the in-kernel CFS version for efficiency and simplicity.

(1) Interpreting Quota and Period as CPU Utilization

We treat quota / period as CPU utilization rather than time-based replenishment.

For example, if a cgroup has:

quota = 20000 µs
period = 200000 µs

it may use up to 10% of a single CPU.
All cgroups are replenished by a single BPF timer, avoiding one timer per cgroup.

(2) Eventual Quota Enforcement via Task Admission Control

To reduce overhead in the critical path, we enforce quota eventually rather than immediately.

• In CFS, quota is checked when selecting a task.
• In SCX, we check it on enqueue (task admission control).

Tasks from throttled cgroups are deferred to a backlog queue instead of
DSQ. When quota is replenished by the BPF timer, deferred tasks are moved
back to their DSQ, ensuring zero additional overhead in ops.dispatch().

---

3. API for BPF Schedulers

BPF schedulers interact with the library using the following API.

Initialization

• scx_cgroup_bw_lib_init() → called in ops.init()
• scx_cgroup_bw_init() / scx_cgroup_bw_exit() → called in ops.cgroup_init() / ops.cgroup_exit()
• scx_cgroup_bw_set() → called when cpu.max parameters change

Runtime Checks

Before enqueueing (ops.enqueue()) or selecting (ops.select_cpu()), call
scx_cgroup_bw_throttled() to check if the cgroup is throttled.

If throttled:

• Defer task using scx_cgroup_bw_put_aside()
• When replenished, call scx_cgroup_bw_reenqueue() (from ops.dispatch())

Re-enqueued tasks use scheduler-defined policies (e.g., time slice, vtime, DSQ).
A callback must be registered with:

REGISTER_SCX_CGROUP_BW_ENQUEUE_CB(eqcb)

where:

int eqcb(u64 pid);

If a throttled task is dequeued before being unthrottled (e.g., due to sched_setaffinity(2)),
the scheduler should call scx_cgroup_bw_cancel() to maintain proper state.

Reporting the Used Time

A task must report its consumed execution time periodically (ops .tick()) and when it stops (ops.stopping()) using scx_cgroup_bw_consume().

API Summary

See scheds/include/lib/cgroup.h for detailed documentation.

int scx_cgroup_bw_lib_init(struct scx_cgroup_bw_config *config);
int scx_cgroup_bw_init(struct cgroup *cgrp, struct scx_cgroup_init_args *args);
int scx_cgroup_bw_exit(struct cgroup *cgrp);
int scx_cgroup_bw_set(struct cgroup *cgrp, u64 period_us, u64 quota_us, u64 burst_us);
int scx_cgroup_bw_throttled(struct cgroup *cgrp);
int scx_cgroup_bw_consume(struct cgroup *cgrp, u64 consumed_ns);
int scx_cgroup_bw_put_aside(struct task_struct *p, u64 taskc, u64 vtime, struct cgroup *cgrp);
int scx_cgroup_bw_reenqueue(void);
int scx_cgroup_bw_cancel(u64 taskc);
#define REGISTER_SCX_CGROUP_BW_ENQUEUE_CB(eqcb)

---

4. Prerequisites for BPF Schedulers

Schedulers must allocate their per-task context (e.g., struct task_ctx in LAVD) in the BPF arena,
as the library internally uses ATQ (Arena-based Task Queue) for managing throttled tasks.

Each task context must begin with struct scx_task_common (defined in scheds/include/lib/atq.h)
or reserve equivalent space (struct atq_ctx in LAVD).
This structure embeds a red-black tree node and related state required by ATQ.

---

5. Patch Structure

(1) ATQ / Rbtree Changes

• lib/atq: factor out task insertion into scx_atq_insert_node
• lib/rbtree: add noalloc/nofree variants of the API
• lib/rbtree: turn rbtree_insert_mode from a per-insert into a per-tree attribute
• lib/rbtree: adjust inlining to pass verification
• lib/atq: only use embedded rbnodes on scx_atq_insert_*()
• lib/rbtree: remove RB_ALLOC check out of rb_insert codepath and update selftests
• lib/selftests: add selftests for embedded rbtree node
• lib/rbtree: initialize node color to red for embedded nodes
• include/lib: expose rb_integrity_check as public API
• lib/rbtree: wipe the values of ->left and ->right in embedded rbnode_t instances before use
• selftests/atq: expand and fix tests
• lib/selftests: exclude cgroup_bw.bpf.c from selftests
• lib/selftests: hardcode Clang version to be the system one
• include: fix userspace-side spinlock definitions
• lib/atq: change signature to require scx_task_common
• lib/atq: add atq_remove_node call
• include/atq: add ATQ lock/unlock calls
• lib/atq: add unlocked insert ATQ calls

(2) cpu.max Library Implementation

• lib: cgroup_bw: Add skeleton for CPU bandwidth control.
• lib: cgroup_bw: Implement scx_cgroup_bw_lib_init().
• lib: cgroup_bw: Implement scx_cgroup_bw_init().
• lib: cgroup_bw: Implement scx_cgroup_bw_set().
• lib: cgroup_bw: Implement scx_cgroup_bw_exit().
• lib: cgroup_bw: Implement scx_cgroup_bw_throttled().
• lib: cgroup_bw: Implement scx_cgroup_bw_consume().
• lib: cgroup_bw: Implement scx_cgroup_bw_put_aside().
• lib: cgroup_bw: Implement replenish timer and cbw_reenqueue_cgroup().
• lib: cgroup_bw: Implement scx_cgroup_bw_cancel().

(3) LAVD Integration

• scx_lavd: Initial integration with CPU bandwidth control.
• scx_lavd: Move task_ctx to the BPF arena.
• scx_lavd: Support cpu.max at enqueue-like paths.
• scx_lavd: Implement ops.deque() for throttling cancelling.

(4) Scripts

• scripts: Add a script to get a cgroup path from it id
• scripts: Add a script to set up cgroups for basic testing of cpu.max.

---

6. Reference

[1] Cgroup v2 Documentation — CPU controller

---

Signed-off-by: Changwoo Min [email protected]
Signed-off-by: Emil Tsalapatis [email protected]

[etsal]

Short note: This feature will temporarily break schedulers that use the old ATQ API, notably p2dq. There will be a subsequent patch that fixes this issue.

CC @hodgesds

[htejun]

Let's land this one after the november release is cut.

[JakeHillion]

Short note: This feature will temporarily break schedulers that use the old ATQ API, notably p2dq. There will be a subsequent patch that fixes this issue.

This is kind of a pain, CI is failing on main because of this. Can we please make sure to land the fixes today?

[sirlucjan]

This is fix: #3038 @JakeHillion