fast-fstests is a tool that wraps fstests with pytest, opening up possibilities to better integrate with CI/CD pipelines. It leverages multiple machines allowing for parallel testing. This can be configured optionally with mkosi to automatically handle the creation and destruction of vms.
Runtime data in seconds collected on my machine.
| group | fstests (s) | fast-fstests 5vms (s) | fast-fstests 10vms (s) | fast-fstests 15vms (s) | fast-fstests 20vms (s) | fast-fstests 25vms (s) | fast-fstests 30vms (s) |
|---|---|---|---|---|---|---|---|
| auto* | 5780 | 1620 | 1090 | 920 | 870 | 840 | 950 |
| quick | 1500 | 680 | 540 | 490 | 435 | 425 | 450 |
| btrfs/auto* | 1470 | 430 | 310 | 255 | 310 | 310 | 285 |
| btrfs/quick | 390 | 170 | 120 | 125 | 105 | 110 | 110 |
*I excluded btrfs/187 and generic/562 as they are outliers that take around 30min on my machine.
There is between a 3-7x speed improvement, bringing down the time to run auto from 96 minutes to 14 minutes.
- fstests
- pytest
- pytest-xdist - for parallelizing pytest
- filelock - locking mechanism for parallel tests
- mkosi - for managing virtual machines
- mkosi-kernel - for configuring mkosi
- fast-fstests can be configured via a pytest.ini file or via cli arguments.
- Example included at pytest.ini.example.
- fast-fstests options set in pytest.ini are overridden by command line flags, unless it's a list argument, in which case command line flags will append to pytest.ini options.
| pytest.ini option | command line flag | description |
|---|---|---|
| targetpaths | --targetpath | Specify targetpaths to run fstests on. HOSTNAME:PATH-TO-FSTESTS eg. vm1:/home/fstests |
| mkosi | --mkosi | Specify the number of mkosi hosts to create. |
| mkosi_config_dir | --mkosi-config-dir | Path to mkosi configuration directory. |
| mkosi_options | --mkosi-options | Options to pass to mkosi when launching qemu. |
| mkosi_fstests_dir | --mkosi-fstests-dir | Path to fstests on mkosi. |
| mkosi_setup_timeout | --mkosi-setup-timeout | How long to wait in seconds for mkosi setup before aborting. (default 60s) |
| host_fstests_dir | --host-fstests-dir | Path to fstests on host. |
| tests | --tests | List of tests to run e.g. btrfs/001 or generic/100. (can't be used with group) |
| group | --group | Name of group to run e.g. btrfs/quick or auto. (can't be used with tests) |
| excludes | --excludes | List of tests to exclude. |
| exclude_file | --exclude-file | Path to an exclude file with a test per line to exclude from test run. Cannot be used with --excludes. |
| random | --random | Whether to randomize the order that tests are run. |
| no_cleanup_on_failure | --no-cleanup-on-failure | Preserve machine if a there was a test failure. Requires manual cleanup before next fast-fstest run. mkosi --machine {MACHINE} ssh poweroff |
| results_path | --results-path | Path to results directory. All test outputs stored here. |
cd .../fast-fstests
pip install pytest
pip install pytest-xdist
pip install filelock
pytest src/fast-fstests.py --targetpath host1:/fstests --targetpath host2:/home/fstests --group btrfs/auto
Though mkosi is optional I find the convenience of not managing vms very appealing. Here are directions for getting everything setup with mkosi. Caveat this guide is not intended as a comprehensive mkosi setup guide, it is only detailing some additional configuration needed to get mkosi working with fast-fstests. It is highly recommended that before starting this guide you already have a working mkosi and mkosi-kernel setup that can run fstests.
- mkosi-kernel profile
I like to create a separate profile to hold all of the mkosi configurations for running fast-fstests. I've included an example in fast-fstests.conf.example. Some important configurations are:
- Ssh=yes
(necessary to remotely execute fstests on vm) - Ephemeral=yes
(necessary to launch multiple vms from the same image) - Include=modules/fstests
(include the fstests module) - BuildSources=[PATH TO FSTESTS]:fstests
(tell mkosi where fstests is on your machine) - ExtraTrees=[PATH TO MKOSI-KERNEL]/mkosi.builddir/centos~9~x86-64/fstests:/fstests
(copies the built fstests to /fstests on the vm)
- Build the image
mkosi --profile fast-fstests -f build
- Check if fstests runs
mkosi --profile fast-fstests qemu
cd .../fstests
./check
./check should start running tests without needing to compile anything.
- Check if ephemeral machines are working
mkosi --profile fast-fstests --machine 1 qemu
In a separate shell:
mkosi --profile fast-fstests --machine 2 qemu
Both should launch successfully.
- Check if ssh is working
mkosi --profile fast-fstests qemu
Once qemu vm is up and running, run this in a separate shell:
mkosi ssh
Should successfully ssh into qemu vm.
- If all steps above are working you should be good to go!
pytest src/fast-fstests.py --mkosi 5 --group btrfs/auto
Test results can be found at results_path/{TEST}/{LATEST_TIMESTAMP}/ and include return code, stdout and stderr.