Skip to content

Status/2025Q3/sta-wpc-ci-automation.adoc: Add report #549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Status/2025Q3/sta-wpc-ci-automation.adoc: Add report #549

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions website/content/en/status/report-2025-07-2025-09/sta-wpc-ci-automation.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
=== STA Work Package C: CI/CD Automation

Contact: Siva Mahadevan <[email protected]>

In this quarter, as part of the infrastructure modernization work commissioned by the Sovereign Tech Agency (STA), I have been working on the in-tree CI Makefile targets.
I also worked on bringing our CI test reports to a clean state on our tier-1 architectures (amd64 and aarch64).
This report is a supplement to the overall STA status report and will describe the work done in more detail.

==== tests/ci improvements

The [.filename]#tests/ci# subdirectory in the src tree was introduced in link:https://cgit.freebsd.org/src/commit/?id=cb9d4bb1fbb9ac0eb9f211656e91f9d5254c166c[commit: Add preliminary in-tree CI infrastructure for developers] by mailto:[email protected][Moin Rahman] and aims to provide an easy way for developers to replicate the CI testing run by link:https://ci.freebsd.org[our Jenkins cluster].
In this quarter, the following improvements were made by the team:

New functionality:

* link:https://cgit.freebsd.org/src/commit/?id=f6e5bcd4716c2974c7ceb1fbbfc0ef8db8e13d8b[tests/ci: run ci-full kyua tests in parallel]
* link:https://cgit.freebsd.org/src/commit/?id=92a7f2d577630d670643f855a1d123a2260102af[tests/ci: Add KYUA_TEST_FILTERS to allow user to select specific tests]
* link:https://cgit.freebsd.org/src/commit/?id=932cf5fa44b7ff3b7ea21ccb32bbbefff11c160c[tests/ci: Add CIENV variable]

Bug fixes:

* link:https://cgit.freebsd.org/src/commit/?id=ed4c64bc792fd2509d8fd6805f8e174c2db62338[tests/ci: Use suitable variable for qemu-user-static existence check]
* link:https://cgit.freebsd.org/src/commit/?id=2f50ea54c6e3b15f655e800eb680ddcb80e30b4f[tests/ci: fix race condition in bhyveload boot]
* link:https://cgit.freebsd.org/src/commit/?id=57f725a6a6c19b12015854589764babe58252155[tests/ci: fix missing /usr/local/{sbin,bin} in freebsdci rc PATH]
* link:https://cgit.freebsd.org/src/commit/?id=9f3e4b28f9dec27b73eeebd161d119d1e9864081[tests/ci: Fix missing qemu devices]
* link:https://cgit.freebsd.org/src/commit/?id=10fb5e7117e522540167fa02692e1e7b8e38eec0[tests/ci: Fix race condition with ci-extractmeta]
* link:https://cgit.freebsd.org/src/commit/?id=66fe805070d121d4b93f63b7b02a70d76069edfc[tests/ci: Fix wrong chflags target path in 'beforeclean' target]
* link:https://cgit.freebsd.org/src/commit/?id=9ec37e8ff49c6b555e8d4f522adc9378157df170[tests/ci: Use QEMU blockdev declaration for all platforms]
* link:https://cgit.freebsd.org/src/commit/?id=e990e460d50c628dfa681c5693bd8ece87b001ad[tests/ci: Fix unescaped kld_list var in rc.conf]

With these changes, a developer can run CI with these example commands as root:

[source,shell]
----
# Fully parallel CI:
make ci

# Single-threaded CI
make PARALLEL_JOBS=1 ci

# Single-threaded CI, running a subset of the tests as described in kyua-test(1)
make PARALLEL_JOBS=1 KYUA_TEST_FILTERS='/path/to/testcase /path/to/another:testname1' ci

# Run smoke (boot) tests
make CITYPE=smoke ci
----

==== Test case management

link:https://ci.freebsd.org/tinderbox[Tinderbox] has been reporting that our supported platforms are failing in CI since a link:https://ci.freebsd.org/job/FreeBSD-main-amd64-test/26756/[run from the last quarter].
As the backlog grows larger, it becomes harder for users and developers to notice a new failure and pin it to a particular commit.

To complement the [.filename]#tests/ci# CI/CD automation improvements, along with link:https://github.com/markjdb/bricoler[Bricoler] to help with more granular investigations, I worked on cleaning up the failing test backlog on tier-1 architectures.
The following patches and bug reports were submitted as a result of this (still ongoing) work:

New bug reports filed to track failing or flaky tests:

* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288991[PR 288991: sys/netinet/output:output_raw_flowid_mpath_success]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289096[PR 289096: lib/libexecinfo/sigtramp_test:test_backtrace_sigtramp]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289165[PR 289165: usr.bin/limits/limits_test:cputime_soft_flag]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289240[PR 289240: sys/netlink/netlink_socket:overflow]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289239[PR 289239: sys/netpfil/pf/sctp:pfsync]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289236[PR 289236: sys/kern/exterr_test:gettext_extended]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289382[PR 289382: sys/netinet6/lpm6:lpm6_test1_success]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289628[PR 289628: sys/netpfil/pf/nat:endpoint_independent_pass]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289630[PR 289630: libexec/rc/rc_subr_test:wait_for_pids_progress]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289237[PR 289237: sys/fs/fusefs/last_local_modify:main]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289084[PR 289084: lib/libc/string/memcmp_test:{diff,neq}]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289477[PR 289477: sys/netpfil/pf/route_to:prefer_ipv6_nexthop_mixed_af_random_table_ipv4]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289299[PR 289299: sys/netpfil/pf/rules_counter:keepcounters]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289684[PR 289684: sys/netlink/test_snl:snl_parse_errmsg_capped]
* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289146[PR 289146: sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action]

Unskip tests that are wrongly skipped in CI:

* link:https://cgit.freebsd.org/src/commit/?id=6ff78a63d8cd0dd64ae79cbda5cb03572c1e17f5[tests/sys/netpfil: unskip tests that no longer need to be skipped]
* link:https://cgit.freebsd.org/src/commit/?id=178b9c2364740ead21f584dc30c3aa45bd0bb8bf[tests/ci: Add missing kmods and pkgs to unskip tests]

Test case metadata fixes:

* link:https://cgit.freebsd.org/src/commit/?id=83b08f40264177ecf03fce808474815816f9846f[fix parallel execution of swapon tests]
* link:https://cgit.freebsd.org/src/commit/?id=1e570722dced7a3afc0a35b10af047b835e51567[cap_dns/tests/dns_test: mark tests as needing network access]
* link:https://cgit.freebsd.org/src/commit/?id=68fe0d9cc03bd80f63a5317a633d2426ae286316[pfctl tests: use require.kmods instead of manual check for pf]
* link:https://cgit.freebsd.org/src/commit/?id=df88c711a26c04192761e96dc08a6f3f9e42afed[cap_net/net_test: require 'allow_network_access']
* link:https://cgit.freebsd.org/src/commit/?id=5dadfac51c3876b0ba6fde831cb4417594167fed[tests/mac_portacl: enable is_exclusive for now]
* link:https://cgit.freebsd.org/src/commit/?id=cacfd42b0c57b3455ca4a23e21de071fa0cc9ed0[tests/sys/mqueue: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=215a0654186926b2e4497d83f3eb45b5eefcb23f[tests/sys/netlink: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=308c9c995794f1ae89bba1d43aab9c51f7e90a09[tests/sys/opencrypto: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=26d3dfa8733892adf107b045a8c98d480b4bb21d[tests/sys/aio: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=c444bfca0063d0820f3fc72cb429809f5f52a970[tests/pf/ioctl: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=559b0f6b82867b60a2d5cb1b2744114603cb17a7[tests/sys/netmap: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=74898ce961e1f4818eba500dfa3a06bc30c1d9b3[tests/sndstat: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=7a1a6d1e2a836d189447ba725b0e256014613ab8[tests/socket_accf: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=4079513328d5656e31cb853a799176ff2500f79b[tests/sys/net: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=427be10491e37160a82eef31622676c655a29f1e[tests/sys/netinet: use require.kmods property instead of ad-hoc checks]
* link:https://cgit.freebsd.org/src/commit/?id=f53228c4e398468c71d3ed8a372cc29ccae6a0c1[tests/vmm_cred_jail: use require.kmods property instead of ad-hoc checks]

mark tests as "expected fail" (xfail), currently WIP:

* link:https://cgit.freebsd.org/src/commit/?id=df997faca8411bdf8008c25de5db3ed719019d21[atf_pytest: fix xfail detection from pytest report]

==== Tooling (WIP)

To catch errors more quickly, instead of relying on link:https://ci.freebsd.org[Jenkins] to update the test report, I ran local CI multiple times daily.
To help with this, I worked on some tooling to speed up the testing/debugging cycles.
I am maintaining the following (currently uncommitted) tools:

* link:https://codeberg.org/svmhdvn/mypoudriere/src/branch/main/myfreebsd[parallel CI runner built on top of tests/ci]
* link:https://codeberg.org/svmhdvn/depistage[(VERY WIP) automated CI bug report/triage system]

Sponsor: The FreeBSD Foundation