Skip to content

Commit c8215f8

Browse files
chrishenziechrishenzie
committed
Add writable-cgroups experimental plugin
Adds a new `writable-cgroups` plugin, designed to enable safe delegation of cgroup management to containers. This plugin allows containers to mount `/sys/fs/cgroup` as read-write, enabling workloads (like AI/ML frameworks) to manage their own sub-cgroups. This plugin serves as a reference implementation and test-bed for validating the `nsdelegate` security model proposed in KEP-5474 as an alternative to introducing new Kubernetes API fields. Signed-off-by: Chris Henzie <chrishenzie@gmail.com>
1 parent ed596cb commit c8215f8

6 files changed

Lines changed: 787 additions & 1 deletion

File tree

‎Makefile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,8 @@ PLUGINS := \
5454
$(BIN_PATH)/wasm \
5555
$(BIN_PATH)/network-device-injector \
5656
$(BIN_PATH)/network-logger \
57-
$(BIN_PATH)/rdt
57+
$(BIN_PATH)/rdt \
58+
$(BIN_PATH)/writable-cgroups
5859

5960
ifneq ($(V),1)
6061
Q := @

‎plugins/writable-cgroups/README.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Writable Cgroups NRI Plugin (Experimental)
2+
3+
This is an experimental NRI plugin designed to safely enable writable cgroups
4+
(`/sys/fs/cgroup`) inside containers.
5+
6+
## Purpose & Context
7+
8+
This plugin serves as a test-bed for validating the behavior and security model
9+
of "delegated cgroup management" in Kubernetes environments. It was developed in
10+
response to [KEP-5474](https://github.com/kubernetes/enhancements/issues/5474),
11+
which originally proposed adding explicit API support for writable cgroups.
12+
13+
However, the evolving consensus is to move away from API additions and instead
14+
leverage the Linux kernel's
15+
[`nsdelegate` mount option](https://man7.org/linux/man-pages/man7/cgroups.7.html#:~:text=Cgroups%20v2%20delegation%3A%20nsdelegate%20and%20cgroup%20namespaces)
16+
directly within container runtimes. This approach allows runtimes to
17+
automatically provide safe, writable cgroup access when the host is correctly
18+
configured, removing the need for user-facing API changes.
19+
20+
For a detailed design rationale and decision log, please see the
21+
[Delegated Cgroup Management Design Document](https://docs.google.com/document/d/1MJZADe-_fO95wwolUvrxGcm6rhWZRgti__7mjlu-dV8/edit?usp=sharing).
22+
23+
## How It Works
24+
25+
This plugin intercepts container creation requests and checks for the presence
26+
of the `nsdelegate` mount option on the host's root cgroup hierarchy.
27+
28+
1. **Safety Check:** On startup, the plugin inspects the host's mount table (via
29+
`/host/proc/1/mountinfo` by default). It verifies that the `cgroup2`
30+
filesystem is mounted with the `nsdelegate` option.
31+
2. **Conditional Activation:**
32+
* If `nsdelegate` is **absent** on the host: The plugin logs a warning and
33+
takes **no action**, ensuring safety. Containers retain the default
34+
Read-Only cgroup mount.
35+
* If `nsdelegate` is **present** on the host: The plugin proceeds to check
36+
for enabling annotations.
37+
3. **Enabling:** If the safety check passes AND a container is annotated, the
38+
plugin modifies the container spec to mount `/sys/fs/cgroup` as **Read-Write
39+
(`rw`)** instead of Read-Only (`ro`).
40+
41+
### Why is this safe?
42+
43+
When `nsdelegate` is enabled on the host, the kernel enforces strict boundaries
44+
at the cgroup namespace level. Even with a Read-Write mount, a container:
45+
46+
* **Cannot** modify its own resource limits (e.g., `memory.max`) set by the
47+
runtime (writes are denied with `EPERM`).
48+
* **Can** create sub-cgroups and manage resources for its own child processes.
49+
50+
## Usage
51+
52+
### Prerequisites
53+
54+
* A container runtime with NRI support enabled.
55+
* The host system must have cgroup v2 enabled and mounted with the `nsdelegate`
56+
option.
57+
58+
### Deployment
59+
60+
Deploy the plugin binary to your node and ensure it is registered with the NRI
61+
service.
62+
63+
**Command Line Arguments:**
64+
65+
* `-idx`: Plugin index.
66+
* `-socket-path`: Path to the NRI socket.
67+
* `-host-mount-file`: Path to the host's mountinfo file (default:
68+
`/host/proc/1/mountinfo`). Ensure this file is accessible to the plugin (e.g.,
69+
via a bind mount in a DaemonSet).
70+
71+
### Annotations
72+
73+
To enable writable cgroups for a workload, add the following annotation to your
74+
Pod:
75+
76+
**Pod-Level (Applies to all containers):**
77+
78+
```yaml
79+
annotations:
80+
cgroups.noderesource.dev/writable: "true"
81+
```
82+
83+
**Container-Level (Applies to a specific container):**
84+
85+
```yaml
86+
annotations:
87+
cgroups.noderesource.dev/writable.container.<container_name>: "true"
88+
```
89+
90+
## Status
91+
92+
This plugin is **experimental** and intended for testing and validation purposes
93+
only. It is not recommended for production use until the behavior is
94+
standardized in upstream container runtimes.

‎plugins/writable-cgroups/go.mod‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
module github.com/containerd/nri/plugins/writable-cgroups
2+
3+
go 1.24.0
4+
5+
require (
6+
github.com/containerd/nri v0.6.1
7+
github.com/moby/sys/mountinfo v0.7.2
8+
github.com/sirupsen/logrus v1.9.3
9+
github.com/stretchr/testify v1.8.4
10+
)
11+
12+
require (
13+
github.com/containerd/log v0.1.0 // indirect
14+
github.com/containerd/ttrpc v1.2.7 // indirect
15+
github.com/davecgh/go-spew v1.1.1 // indirect
16+
github.com/golang/protobuf v1.5.3 // indirect
17+
github.com/knqyf263/go-plugin v0.9.0 // indirect
18+
github.com/kr/text v0.2.0 // indirect
19+
github.com/opencontainers/runtime-spec v1.3.0 // indirect
20+
github.com/pmezard/go-difflib v1.0.0 // indirect
21+
github.com/tetratelabs/wazero v1.11.0 // indirect
22+
golang.org/x/sys v0.38.0 // indirect
23+
google.golang.org/genproto/googleapis/rpc v0.0.0-20230731190214-cbb8c96f2d6d // indirect
24+
google.golang.org/grpc v1.57.1 // indirect
25+
google.golang.org/protobuf v1.34.1 // indirect
26+
gopkg.in/yaml.v3 v3.0.1 // indirect
27+
)
28+
29+
replace github.com/containerd/nri => ../..

‎plugins/writable-cgroups/go.sum‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
github.com/brianvoe/gofakeit/v7 v7.12.1 h1:df1tiI4SL1dR5Ix4D/r6a3a+nXBJ/OBGU5jEKRBmmqg=
2+
github.com/brianvoe/gofakeit/v7 v7.12.1/go.mod h1:QXuPeBw164PJCzCUZVmgpgHJ3Llj49jSLVkKPMtxtxA=
3+
github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
4+
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
5+
github.com/containerd/ttrpc v1.2.7 h1:qIrroQvuOL9HQ1X6KHe2ohc7p+HP/0VE6XPU7elJRqQ=
6+
github.com/containerd/ttrpc v1.2.7/go.mod h1:YCXHsb32f+Sq5/72xHubdiJRQY9inL4a4ZQrAbN1q9o=
7+
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
8+
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
9+
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
10+
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
11+
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
12+
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
13+
github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
14+
github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
15+
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
16+
github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
17+
github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
18+
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
19+
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
20+
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
21+
github.com/google/pprof v0.0.0-20240424215950-a892ee059fd6 h1:k7nVchz72niMH6YLQNvHSdIE7iqsQxK1P41mySCvssg=
22+
github.com/google/pprof v0.0.0-20240424215950-a892ee059fd6/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
23+
github.com/knqyf263/go-plugin v0.9.0 h1:CQs2+lOPIlkZVtcb835ZYDEoyyWJWLbSTWeCs0EwTwI=
24+
github.com/knqyf263/go-plugin v0.9.0/go.mod h1:2z5lCO1/pez6qGo8CvCxSlBFSEat4MEp1DrnA+f7w8Q=
25+
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
26+
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
27+
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
28+
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
29+
github.com/moby/sys/mountinfo v0.7.2 h1:1shs6aH5s4o5H2zQLn796ADW1wMrIwHsyJ2v9KouLrg=
30+
github.com/moby/sys/mountinfo v0.7.2/go.mod h1:1YOa8w8Ih7uW0wALDUgT1dTTSBrZ+HiBLGws92L2RU4=
31+
github.com/onsi/ginkgo/v2 v2.19.1 h1:QXgq3Z8Crl5EL1WBAC98A5sEBHARrAJNzAmMxzLcRF0=
32+
github.com/onsi/ginkgo/v2 v2.19.1/go.mod h1:O3DtEWQkPa/F7fBMgmZQKKsluAy8pd3rEQdrjkPb9zA=
33+
github.com/onsi/gomega v1.34.0 h1:eSSPsPNp6ZpsG8X1OVmOTxig+CblTc4AxpPBykhe2Os=
34+
github.com/onsi/gomega v1.34.0/go.mod h1:MIKI8c+f+QLWk+hxbePD4i0LMJSExPaZOVfkoex4cAo=
35+
github.com/opencontainers/runtime-spec v1.3.0 h1:YZupQUdctfhpZy3TM39nN9Ika5CBWT5diQ8ibYCRkxg=
36+
github.com/opencontainers/runtime-spec v1.3.0/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0=
37+
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
38+
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
39+
github.com/prometheus/procfs v0.6.0 h1:mxy4L2jP6qMonqmq+aTtOx1ifVWUgG/TAmntgbh3xv4=
40+
github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
41+
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
42+
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
43+
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
44+
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
45+
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
46+
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
47+
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
48+
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
49+
github.com/tetratelabs/wazero v1.11.0 h1:+gKemEuKCTevU4d7ZTzlsvgd1uaToIDtlQlmNbwqYhA=
50+
github.com/tetratelabs/wazero v1.11.0/go.mod h1:eV28rsN8Q+xwjogd7f4/Pp4xFxO7uOGbLcD/LzB1wiU=
51+
golang.org/x/net v0.38.0 h1:vRMAPTMaeGqVhG5QyLJHqNDwecKTomGeqbnfZyKlBI8=
52+
golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
53+
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
54+
golang.org/x/sys v0.38.0 h1:3yZWxaJjBmCWXqhN1qh02AkOnCQ1poK6oF+a7xWL6Gc=
55+
golang.org/x/sys v0.38.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
56+
golang.org/x/text v0.23.0 h1:D71I7dUrlY+VX0gQShAThNGHFxZ13dGLBHQLVl1mJlY=
57+
golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4=
58+
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg=
59+
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
60+
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
61+
google.golang.org/genproto/googleapis/rpc v0.0.0-20230731190214-cbb8c96f2d6d h1:pgIUhmqwKOUlnKna4r6amKdUngdL8DrkpFeV8+VBElY=
62+
google.golang.org/genproto/googleapis/rpc v0.0.0-20230731190214-cbb8c96f2d6d/go.mod h1:TUfxEVdsvPg18p6AslUXFoLdpED4oBnGwyqk3dV1XzM=
63+
google.golang.org/grpc v1.57.1 h1:upNTNqv0ES+2ZOOqACwVtS3Il8M12/+Hz41RCPzAjQg=
64+
google.golang.org/grpc v1.57.1/go.mod h1:Sd+9RMTACXwmub0zcNY2c4arhtrbBYD1AUHI/dt16Mo=
65+
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
66+
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
67+
google.golang.org/protobuf v1.34.1 h1:9ddQBjfCyZPOHPUiPxpYESBLc+T8P3E+Vo4IbKZgFWg=
68+
google.golang.org/protobuf v1.34.1/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
69+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
70+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
71+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
72+
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
73+
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
74+
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

0 commit comments

Comments
 (0)