Proper SELinux support

[stgraber]

Per https://discuss.linuxcontainers.org/t/unable-to-add-lxc-rootfs-mount-options-context/20943/14 we should try and at least set the correct context mount options whenever we mount a container or VM data directory.

Ideally we'd have something more robust where we can handle per-instance labeling but that would require someone with more SELinux knowledge than I have to get this done properly, put testing in place and keep an eye on it so it doesn't regress :)

So for now, we'll try to go for low hanging fruits and just make things better where feasible.

[mschiff]

Hi Stéphane,

in the meantime I have developed a (mostly) working SELinux-Module for incus! Getting containers and VMs running on a SELinux system is not yet straight forward, but it is doable and working with one major issue which I am not able to solve without some basic incus support here.

It would we great to have some very basic support for this in incus as it would make running qemu VMs much better:

The missing piece is the SELinux-labels of all related files for a specific VM.

A possibility to pass a context to incus which it will use to label VM specific dirs before creating supporting files and starting a VM would eliminate this last big roadblock.

Currently its like that:

after creating/starting a VM I get:

(sysadm_r)@host ~ # ls -lZ /var/lib/incus/devices/vm1/
dr-x------. 6 nobody root system_u:object_r:unlabeled_t:s0 14  6. Dez 17:59 config.mount

(sysadm_r)@host ~ # ls -lZ /var/lib/incus/devices/vm1/config.mount/
-r--------. 1 nobody root system_u:object_r:unlabeled_t:s0        2  6. Dez 17:59 agent-mounts.json
-rw-r--r--. 1 nobody root system_u:object_r:unlabeled_t:s0      819  6. Dez 17:59 agent.conf
-r--------. 1 nobody root system_u:object_r:unlabeled_t:s0      725  6. Dez 17:59 agent.crt
-r--------. 1 nobody root system_u:object_r:unlabeled_t:s0      288  6. Dez 17:59 agent.key
dr-x------. 2 nobody root system_u:object_r:unlabeled_t:s0        3  6. Dez 17:59 files
-r-x------. 1 nobody root system_u:object_r:unlabeled_t:s0 21721508 19. Nov 14:24 incus-agent
-rwx------. 1 nobody root system_u:object_r:unlabeled_t:s0     1302  6. Dez 17:59 install.sh
lrwxrwxrwx. 1 root   root system_u:object_r:unlabeled_t:s0       11  6. Dez 17:59 lxd-agent
dr-x------. 2 nobody root system_u:object_r:unlabeled_t:s0        2  6. Dez 17:59 nics
-r--------. 1 nobody root system_u:object_r:unlabeled_t:s0      688  6. Dez 17:59 server.crt
dr-x------. 2 nobody root system_u:object_r:unlabeled_t:s0        4  6. Dez 17:59 systemd
dr-x------. 2 nobody root system_u:object_r:unlabeled_t:s0        3  6. Dez 17:59 udev

(sysadm_r)@host ~ # ls -lZ /var/lib/incus/devices/vm1/config.mount/
-r--------. 1 nobody root system_u:object_r:unlabeled_t:s0        2  6. Dez 17:59 agent-mounts.json
-rw-r--r--. 1 nobody root system_u:object_r:unlabeled_t:s0      819  6. Dez 17:59 agent.conf
-r--------. 1 nobody root system_u:object_r:unlabeled_t:s0      725  6. Dez 17:59 agent.crt
-r--------. 1 nobody root system_u:object_r:unlabeled_t:s0      288  6. Dez 17:59 agent.key
dr-x------. 2 nobody root system_u:object_r:unlabeled_t:s0        3  6. Dez 17:59 files
-r-x------. 1 nobody root system_u:object_r:unlabeled_t:s0 21721508 19. Nov 14:24 incus-agent
-rwx------. 1 nobody root system_u:object_r:unlabeled_t:s0     1302  6. Dez 17:59 install.sh
lrwxrwxrwx. 1 root   root system_u:object_r:unlabeled_t:s0       11  6. Dez 17:59 lxd-agent
dr-x------. 2 nobody root system_u:object_r:unlabeled_t:s0        2  6. Dez 17:59 nics
-r--------. 1 nobody root system_u:object_r:unlabeled_t:s0      688  6. Dez 17:59 server.crt
dr-x------. 2 nobody root system_u:object_r:unlabeled_t:s0        4  6. Dez 17:59 systemd
dr-x------. 2 nobody root system_u:object_r:unlabeled_t:s0        3  6. Dez 17:59 udev

The unlabeled_t is the problem here and could be solved by labeleing the root node of the filesystem before creating files and dirs inside.

A solution would be if incusd labeled the root dir of the mount /var/lib/incus/devices/vm1/config.mount to system_u:object_r:container_var_lib_t:s0 after mounting and before creating the files inside.

Or is there any hook I could use for that or maybe one can be added so I could specify the command to use myself?

Many thanks in advance for considering.

-Marc

[stgraber]

We got some funding coming in now to tackle the SELinux issue and try to label things correctly for both containers and VMs. Ideally we'd want Incus to automatically:

• Detect that it's on a SELinux system
• Detect the correct set of labels on that system for:
  • container rootfs
  • container config
  • VM disks
  • VM config
• Apply correct labels on instance creation

I believe you gave us all the right pointers for the labels to use in the forum thread, so it's mostly going to be a matter of getting a test system up and running and then iterate on the logic until we get it behaving.

If you could provide instructions on how to set up a suitable SELinux test environment in a VM, including your new SELinux module, that'd be pretty useful.

PS: The funding I mentioned is aimed for the Incus project at large and can be used to sub-contract some part of this work, so if you have time, interest and the ability to take on some contract work, we may be able to compensate some of your work on this.

[mschiff]

We got some funding coming in now to tackle the SELinux issue and try to label things correctly for both containers and VMs.

This is GREAT news ;-)

Ideally we'd want Incus to automatically:

* Detect that it's on a SELinux system

* Detect the correct set of labels on that system for:
  
  * container rootfs
  * container config
  * VM disks
  * VM config

* Apply correct labels on instance creation

For best security the SELinux MLS/MCS Policy is being used most of the time. Then every container is running in the same domain (with same process label) but with a different category. That way the containers are segregated from each other.

One way to achive that is if the user sets a different category for each container. If containers need to share data, they need to share at least one category (You can specifiy single multiple categories per label). This is what I do now by adding something like

  raw.lxc: |
    lxc.selinux.context = system_u:system_r:container_t:s0:c100

to the container config (c100 being the category)

The other (more automatic) way would be, that a user can enable "automatic container segregation" and then incusd would use random categories for each container. Propably in the for :cX,cY and X being 512-1024 and cY being 0-1024, so there still is room fixed cats.

I believe you gave us all the right pointers for the labels to use in the forum thread, so it's mostly going to be a matter of getting a test system up and running and then iterate on the logic until we get it behaving.

Sounds good!

If you could provide instructions on how to set up a suitable SELinux test environment in a VM, including your new SELinux module, that'd be pretty useful.

My module is currently in "PR pending state". It is usable already, but still needs some tweaks. For example currently all containers are runnning in the "spc_t" type which is a "super privileged container". With this, MCS categories will not be enforced. The correct type will be "container_t".

What you basically need:

• Gentoo system with SELinux enabled hardened profile, using systemd. -> default/linux/amd64/23.0/hardened/selinux/systemd ( https://www.gentoo.org/downloads/#amd64-advanced -> "H/SELinux stage 3" (this uses openrc though, which is great and maybe not really relevant for developing but you can switch to systemd after installing this)
• a git clone of the hardened-refpolicy git repo -> https://github.com/gentoo/hardened-refpolicy
• my module files (incus.fc, incus.if, incus.te) -> Add new incus module, add required rules to other modules gentoo/hardened-refpolicy#6

I might also work with a Red Hat based system, which will be easier to setup, but I did not test the module on redhat yet, although, the types being used etc. should be the same. That being said, there are interfaces in the policy that might differ and so in a future version there might be some conditionals in the module based on target distro.

I am happy to help setting up such an envirenment.

PS: The funding I mentioned is aimed for the Incus project at large and can be used to sub-contract some part of this work, so if you have time, interest and the ability to take on some contract work, we may be able to compensate some of your work on this.

Thank you for this! I woud be very interested, but unfortunately I am fully "booked" and cannot take more projects at the moment :-/

[stgraber]

@mschiff I'm looking at landing some basic support in this week's release of Incus as this lines up with a delivery milestone on our end too.

I've been re-reading the notes from here and the forum and I've got a functional environment to test changes to Incus with (latest stable Fedora with SELinux in enforce mode and Incus running fine with both containers and VMs on it).

My goal is to get a baseline support in this week, something we can then refine/extend as folks start using it. I'll start by adding an environment variable similar to what we have for AppArmor and then add some basic detection of SELinux, that will then let us put in SELinux-specific logic in place in both the container and VM codepaths.

For container's I'll have that generate an SELinux context along the line of what you suggested, using the database id of the container as the category. Normally containers don't really share data, though we'll have to see exactly how that goes with custom storage volumes.

Those don't really belong to a given container but can be attached to one or more containers so getting that to behave correctly may get interesting :)

I think Incus 6.17 will ship with the new logic requiring the env variable (INCUS_SECURITY_SELINUX=true), then once we're happy with the behavior and there's no obvious issues with it, we can make it default to true and keep the variable as an opt-out (similar to AppArmor).

Btw, I see the Gentoo PR got closed and there's a mention of it having been merged upstream but I'm not seeing a link to that upstream inclusion, do you have more details?

[mschiff]

This is really great news @stgraber !

I will be more than happy to help and test it then.

Btw, I see the Gentoo PR got closed and there's a mention of it having been merged upstream but I'm not seeing a link to that upstream inclusion, do you have more details?

This is the PR: SELinuxProject/refpolicy#951

I will write some more details later about what I am currently configuring in incus for SELinux enabled containers and VMs

[mschiff]

until now, to make use of SELinux for incus controlled containers and VMs I do the following:

• Use the new incus module contained in the reference policy introduced new container domain for containers running an init system -> container_init_t and virtual machines will be started as qemu_t

LXC containers

• change storage to block_mode in profile:

config:
  security.idmap.isolated: "yes"
description: Default Incus profile for project selinux-test
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    initial.block.mount_options: defcontext=system_u:object_r:container_file_t:s0
    initial.zfs.block_mode: "true"
    path: /
    pool: default
    type: disk
name: default

• create instance (e.g. incus create images:ubuntu/24.04 inst01)
• set file labels and category via mount options:
  incus storage volume set default container/inst01 block.mount_options=defcontext=system_u:object_r:container_file_t:s0:c900
• set container domain via small script:

#!/usr/bin/env bash
#
# set selinux category for incus LXC instance
#

if [[ $# -ne 3 ]]; then
        echo "Syntax: $(basename "$0") <instance> <category> <domain>"
        echo "Example: $(basename "$0") mylxc c100 spc_t"
        echo "Example: $(basename "$0") mylxc c100 container_init_t"
        echo "Example: $(basename "$0") mylxc c100.c200 container_init_t"
        exit 1
fi

set -e

INSTANCE="$1"
CAT="$2"
DOM="$3"

T=$(mktemp)
trap 'rm -f $T' EXIT

incus config show "$INSTANCE" > "$T"

if grep -q lxc.selinux.context "$T"; then
        # cat exists, change it
        echo "Updating category to $CAT"
        sed -i "/lxc.selinux.context/ s/system_u:system_r:.*:s0:.*/system_u:system_r:${DOM}:s0:$CAT/" "$T"
else
        # add cat
        echo "Setting category to $CAT"
        sed -i "/^config:$/a \ \ raw.lxc: |\n    lxc.selinux.context = system_u:system_r:${DOM}:s0:$CAT" "$T"
fi
incus config edit "$INSTANCE" < "$T"

• start instance

qemu virtual machines

• virtual machines will run as qemu_t but I could not find any way to set a category for the process context
• same for storage: I could not find a way to have files for a VM labeled before starting the instance and because of that the current SELinux module allows incusd to manage "unlabeled_t" files:

        # incusd needs support to label VM files before launching
        # until then, VM files will unfortunately be unlabeled
        kernel_manage_unlabeled_dirs(incusd_t)
        kernel_manage_unlabeled_files(incusd_t)
        kernel_manage_unlabeled_symlinks(incusd_t)
        kernel_mounton_unlabeled_dirs(incusd_t)

requirements for proper SELinux support

In order to have proper selinux support in incus, I think the following will be required. This is just a brain dump of what comes to my mind right now. Maybe it would be useful to create some document somewhere to gather all requirements to have proper support which can be reviewed by others as well?

process context (domain)

• It should be possible to configure the process context for a container and virtual machine via profile and in instance configs
• Default for lxc containers should be container_init_t type
• Default for qemu instances should be qemu_t type
• The category is to segregate instances running under the same domain. I think it should be possible to have auto-generated categories and also to have them configurable manually for example to have the same cat for all instances within a project or something like that or if you have your own scheme for building cat numbers.
• Generating a random cat may use more than one number (cats) to have a greater pool of numbers. Values may be from c0 to c1023 (although the maximum value for the Reference Policy is a compile time option). The cat can be constructed with lists of categories (e.g. :c162,c645) and/or ranges (e.g. :c0.1023 for all categories). And for access a process needs all cats, that an object (file) is labeled with.

storage

• It should be possible to configure file contexts (labels) via mount options for block storage
• If should be possible to use custom labels for filesystem level storage and here incusd must label all files after creating and before starting an instance
• Labels should be the same as the process contexts by default but may also be changed on a profile or instance base for other use cases

[stgraber]

Just updating this issue with some details of where we're at now:

• Introduce control environment variables for SELinux (so folks can opt-in to the initial support)
• Detect most common SELinux setups (Fedora and refpolicy)
• Set correct runtime context for containers
• Generate and apply category for containers (using the DB ID)
• Handle block mode storage
• Set correct runtime context for VMs
• Generate and apply category for VMs (using the DB ID)
• Add support for filesystem label on the VM config drive
• Add support for filesystem label on the container volume (possibly a different label for rootfs)
• Add support for filesystem labels on custom volumes
• Validate handling of custom volumes added to multiple containers
• Flip the switch so SELinux support is enabled by default

[stgraber]

I have very limited experience with SELinux and none of my development or production systems use it, so it would be extremely helpful for folks who actually use SELinux to contribute the missing bit of logic here!