Add sched-ext/scx by ananthb · Pull Request #203 · flatcar/sysext-bakery

ananthb · 2026-01-22T20:23:06Z

Run userspace schedulers with sched-ext/scx

Add the scx repository to run sched-ext/scx userspace schedulers.
sched-ext is a Kernel subsystem that allows running task schedulers
in userspace. This repository adds a host of userspace schedulers and
tooling to load, run, and unload them.

How to use

TODO

Testing done

[Describe the testing you have done before submitting this PR. Please include both the commands you issued as well as the output you got.]

Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

scx.sysext/create.sh

t-lo · 2026-02-03T16:59:16Z

@ananthb Thank you for the PR! One comment, otherwise LGTM.

Out of curiosity: Why are we using Fedora instead of Alpine? Alpine is much more lightweight and builds significantly faster, particularly when cross-compiling (i.e. when the whole build container runs in software emulation). Fedora is not a blocker, I'm just curious.

ananthb · 2026-02-03T18:17:46Z

@t-lo literally just what I was running on the laptop when I wrote it. I'll take a crack at getting this compiling on Alpine. Shouldn't be hard.

ananthb · 2026-02-03T20:03:43Z

Using an alpine base image now.

t-lo · 2026-02-27T14:42:20Z

Hmmm, that doesn't seem to work. I've built a test sysext and ran it:

./bakery.sh create scx v1.0.20
./bakery.sh boot scx.raw

Then, in the VM:

core@localhost ~ $ /usr/bin/scx_bpfland 
-bash: /usr/bin/scx_bpfland: cannot execute: required file not found

It seems to be dynamically linked:

core@localhost ~ $ ldd /usr/bin/scx_bpfland 
        linux-vdso.so.1 (0x00007fe856a5a000)
        libelf.so.1 => /usr/lib64/libelf.so.1 (0x00007fe8563af000)
        libz.so.1 => /usr/lib64/libz.so.1 (0x00007fe856394000)
        libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fe856366000)
        libc.musl-x86_64.so.1 => not found
        libc.so.6 => /usr/lib64/libc.so.6 (0x00007fe856171000)
        /lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007fe856a5c000)

Can we produce statically linked Rust binaries? I tried a bit myself using

RUSTFLAGS="-C target-feature=+crt-static" cargo build --release

but that ran into linker issues. I'm not too knowledgeable with Rust, sorry.

Add the scx repository to enable running sched-ext/scx userspace schedulers. Signed-off-by: Ananth Bhaskararaman <antsub@gmail.com>

ananthb · 2026-03-01T14:39:00Z

Can you give it a shot now?

t-lo

Not quite there but getting in shape.

t-lo · 2026-03-02T17:36:36Z

scx.sysext/build.sh

+  zlib-dev \
+  zstd-dev \


Suggested change

zlib-dev \

zstd-dev \

zlib-dev \

zlib-static \

zstd-dev \

zstd-static \

We can add the static libraries here so we don't need to check later.

t-lo · 2026-03-02T17:40:34Z

scx.sysext/build.sh

+export RUSTFLAGS="-C target-feature=+crt-static"
+export PKG_CONFIG_ALL_STATIC=1
+export LIBBPF_SYS_STATIC=1
+export RUSTFLAGS="${RUSTFLAGS} -C link-arg=-lzstd"


Suggested change

export RUSTFLAGS="-C target-feature=+crt-static"

export PKG_CONFIG_ALL_STATIC=1

export LIBBPF_SYS_STATIC=1

export RUSTFLAGS="${RUSTFLAGS} -C link-arg=-lzstd"

TARGET="$(echo "${target}" | tr a-z- A-Z_)"

export CARGO_TARGET_${TARGET}_RUSTFLAGS="-C target-feature=+crt-static -lz -lzstd -lpthread -L /usr/lib -C link-arg=-Wl,-static"

export CARGO_TARGET_${TARGET}_PKG_CONFIG_ALL_STATIC=1

export CARGO_TARGET_${TARGET}_LIBBPF_SYS_STATIC=1

Since we're cross-compiling (via CARGO_BUILD_TARGET=... above) we need to use CARGO_TARGET_... env variable names (as per https://github.com/cross-rs/cross/wiki/Configuration#cargo-configuration). Also, we can link host libraries directly (via -L /usr/lib) since we're using the host target (host: line in rustc info).

t-lo · 2026-03-02T17:44:33Z

scx.sysext/build.sh

+# Verify binaries are fully static (no NEEDED entries).
+for bin in target/release/scx_*; do
+  if [ -x "${bin}" ]; then
+    needed="$(scanelf --needed --nobanner -F '%N' "${bin}")"
+    if [ -n "${needed}" ]; then
+      echo "ERROR: ${bin} is not fully static (NEEDED: ${needed})"
+      exit 1
+    fi
+  fi
+done


Suggested change

# Verify binaries are fully static (no NEEDED entries).

for bin in target/release/scx_*; do

if [ -x "${bin}" ]; then

needed="$(scanelf --needed --nobanner -F '%N' "${bin}")"

if [ -n "${needed}" ]; then

echo "ERROR: ${bin} is not fully static (NEEDED: ${needed})"

exit 1

fi

fi

done

I think we can drop this; we've verified in the PR review that cargo / rustc are actually building static binaries.

t-lo · 2026-03-02T17:45:21Z

scx.sysext/build.sh

+# Ensure libz static is available for fully static linking.
+if [ ! -f /usr/lib/libz.a ] && [ ! -f /lib/libz.a ] && [ ! -f /usr/local/lib/libz.a ]; then
+  zlib_version="1.3.1"
+  workdir="/tmp/zlib-${zlib_version}"
+  zlib_url="https://zlib.net/zlib-${zlib_version}.tar.gz"
+  if ! curl -fsSL "${zlib_url}" -o /tmp/zlib.tar.gz; then
+    curl -fsSL "https://zlib.net/fossils/zlib-${zlib_version}.tar.gz" -o /tmp/zlib.tar.gz
+  fi
+  tar -C /tmp -xf /tmp/zlib.tar.gz
+  cd "${workdir}"
+  ./configure --static
+  make -j"$(nproc)"
+  make install
+  cd /opt/scx
+  export LIBRARY_PATH="/usr/local/lib:${LIBRARY_PATH:-}"
+  export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:${PKG_CONFIG_PATH:-}"
+fi
+
+# Ensure libzstd static is available for libelf static linking.
+if [ ! -f /usr/lib/libzstd.a ] && [ ! -f /lib/libzstd.a ] && [ ! -f /usr/local/lib/libzstd.a ]; then
+  zstd_version="1.5.7"
+  workdir="/tmp/zstd-${zstd_version}"
+  curl -fsSL "https://github.com/facebook/zstd/releases/download/v${zstd_version}/zstd-${zstd_version}.tar.gz" -o /tmp/zstd.tar.gz
+  tar -C /tmp -xf /tmp/zstd.tar.gz
+  cd "${workdir}"
+  make -j"$(nproc)"
+  make install PREFIX=/usr/local
+  cd /opt/scx
+  export LIBRARY_PATH="/usr/local/lib:${LIBRARY_PATH:-}"
+  export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:${PKG_CONFIG_PATH:-}"
+fi


Suggested change

# Ensure libz static is available for fully static linking.

if [ ! -f /usr/lib/libz.a ] && [ ! -f /lib/libz.a ] && [ ! -f /usr/local/lib/libz.a ]; then

zlib_version="1.3.1"

workdir="/tmp/zlib-${zlib_version}"

zlib_url="https://zlib.net/zlib-${zlib_version}.tar.gz"

if ! curl -fsSL "${zlib_url}" -o /tmp/zlib.tar.gz; then

curl -fsSL "https://zlib.net/fossils/zlib-${zlib_version}.tar.gz" -o /tmp/zlib.tar.gz

fi

tar -C /tmp -xf /tmp/zlib.tar.gz

cd "${workdir}"

./configure --static

make -j"$(nproc)"

make install

cd /opt/scx

export LIBRARY_PATH="/usr/local/lib:${LIBRARY_PATH:-}"

export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:${PKG_CONFIG_PATH:-}"

fi

# Ensure libzstd static is available for libelf static linking.

if [ ! -f /usr/lib/libzstd.a ] && [ ! -f /lib/libzstd.a ] && [ ! -f /usr/local/lib/libzstd.a ]; then

zstd_version="1.5.7"

workdir="/tmp/zstd-${zstd_version}"

curl -fsSL "https://github.com/facebook/zstd/releases/download/v${zstd_version}/zstd-${zstd_version}.tar.gz" -o /tmp/zstd.tar.gz

tar -C /tmp -xf /tmp/zstd.tar.gz

cd "${workdir}"

make -j"$(nproc)"

make install PREFIX=/usr/local

cd /opt/scx

export LIBRARY_PATH="/usr/local/lib:${LIBRARY_PATH:-}"

export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:${PKG_CONFIG_PATH:-}"

fi

Not needed anymore as we're directly installing the static versions of our lib dependencies via apk.

t-lo · 2026-03-02T17:52:25Z

scx.sysext/build.sh

+
+# Install binaries (only executables, not .d dependency files)
+mkdir -p /install_root/usr/bin
+find target/release -maxdepth 1 -name 'scx_*' -type f -executable -exec cp {} /install_root/usr/bin/ \;


Suggested change

find target/release -maxdepth 1 -name 'scx_*' -type f -executable -exec cp {} /install_root/usr/bin/ \;

find "target/${target}/release" -maxdepth 1 -name 'scx*' -type f -executable -exec cp \{\} /install_root/usr/bin/ \;

We're cross-compiling, so build output would to go a sub-directory, e.g. target/x86_64-alpine-linux-musl/release for x86.
Also, if we drop the _ in our filter (scx_* => scx*) we'd also get scxtop and scxcash (tool for cache usage) which I think could be desirable?

t-lo · 2026-03-02T17:57:44Z

docs/scx.md

+- `scx_rusty` - multi-domain BPF + user space hybrid scheduler
+- `scx_rustland` - BPF + user space rust scheduler
+- `scx_simple` - simple scheduler for demonstration purposes
+


Suggested change

- scx_beerland

- scx_flash

- scx_layered

- scx_rlfifo

- scx_wd40

- scx_chaos

- scx_cosmos

- scx_mitosis

- scx_p2dq

- scx_tickless

Tools include:

- `scxtop` - top-like utility for scheduler telemetry

- `scxcash` - Cache usage analyser

Full list of schedulers (as of 1.0.10). Feel free to add short descriptions 😉

t-lo · 2026-03-02T17:59:33Z

scx.sysext/files/usr/share/scx/scx

+#   scx_rusty    - multi-domain BPF + user space hybrid scheduler
+#   scx_rustland - BPF + user space rust scheduler
+#   scx_simple   - simple scheduler for demonstration purposes
+#


Suggested change

#

# scx_beerland

# scx_flash

# scx_layered

# scx_rlfifo

# scx_wd40

# scx_chaos

# scx_cosmos

# scx_mitosis

# scx_p2dq

# scx_tickless

#

t-lo · 2026-03-02T18:03:03Z

Your new approach of cross-compiling looks great! It wasn't using the correct RUSTFLAGS but it provided inspiration on how to tackle this. Added a few suggestions to make things work and clean stuff up a bit, also to add scxtop and scxcash to the sysext. Please have a look and ensure everything works. ./bakery.sh boot scx.raw is a great way to test build results on your local machine.

Great work, keep pushing!

ananthb · 2026-03-02T19:28:33Z

Yep I got a basic compile going with podman but I stopped short of actually running it. I'm on NixOS so getting the shell right was more work than I had time for. I can give it another shot this weekend!

ananthb force-pushed the add-sched-ext branch 2 times, most recently from 05fde51 to 17191cb Compare January 27, 2026 14:02

ananthb marked this pull request as ready for review January 27, 2026 14:03

ananthb requested a review from a team as a code owner January 27, 2026 14:03

ananthb force-pushed the add-sched-ext branch from 17191cb to f8e4b86 Compare January 28, 2026 09:26

t-lo reviewed Feb 3, 2026

View reviewed changes

scx.sysext/create.sh Outdated Show resolved Hide resolved

ananthb force-pushed the add-sched-ext branch from f8e4b86 to e84cf62 Compare February 3, 2026 20:01

github-actions bot mentioned this pull request Feb 22, 2026

Monthly contributions report 2026-01-22 - 2026-02-21 flatcar/Flatcar#2024

Open

Add sched-ext/scx

0f06c76

Add the scx repository to enable running sched-ext/scx userspace schedulers. Signed-off-by: Ananth Bhaskararaman <antsub@gmail.com>

ananthb force-pushed the add-sched-ext branch from e84cf62 to 0f06c76 Compare March 1, 2026 14:37

t-lo reviewed Mar 2, 2026

View reviewed changes

	find target/release -maxdepth 1 -name 'scx_*' -type f -executable -exec cp {} /install_root/usr/bin/ \;
	find "target/${target}/release" -maxdepth 1 -name 'scx*' -type f -executable -exec cp \{\} /install_root/usr/bin/ \;

+- scx_beerland
+- scx_flash
+- scx_layered
+- scx_rlfifo
+- scx_wd40
+- scx_chaos
+- scx_cosmos
+- scx_mitosis
+- scx_p2dq
+- scx_tickless
+Tools include:
+- `scxtop` - top-like utility for scheduler telemetry
+- `scxcash` - Cache usage analyser

-#
+# scx_beerland
+# scx_flash
+# scx_layered
+# scx_rlfifo
+# scx_wd40
+# scx_chaos
+# scx_cosmos
+# scx_mitosis
+# scx_p2dq
+# scx_tickless
+#

Conversation

ananthb commented Jan 22, 2026

Run userspace schedulers with sched-ext/scx

How to use

Testing done

Uh oh!

Uh oh!

t-lo commented Feb 3, 2026

Uh oh!

ananthb commented Feb 3, 2026

Uh oh!

ananthb commented Feb 3, 2026

Uh oh!

t-lo commented Feb 27, 2026

Uh oh!

ananthb commented Mar 1, 2026

Uh oh!

t-lo left a comment

Choose a reason for hiding this comment

Uh oh!

t-lo Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

t-lo Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

t-lo Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

t-lo Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

t-lo Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

t-lo Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

t-lo Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

t-lo commented Mar 2, 2026

Uh oh!

ananthb commented Mar 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants