diff --git a/website/content/en/status/report-2025-07-2025-09/sta-wpc-ci-automation.adoc b/website/content/en/status/report-2025-07-2025-09/sta-wpc-ci-automation.adoc new file mode 100644 index 00000000000..4f0088b2b53 --- /dev/null +++ b/website/content/en/status/report-2025-07-2025-09/sta-wpc-ci-automation.adoc @@ -0,0 +1,111 @@ +=== STA Work Package C: CI/CD Automation + +Contact: Siva Mahadevan + +In this quarter, as part of the infrastructure modernization work commissioned by the Sovereign Tech Agency (STA), I have been working on the in-tree CI Makefile targets. +I also worked on bringing our CI test reports to a clean state on our tier-1 architectures (amd64 and aarch64). +This report is a supplement to the overall STA status report and will describe the work done in more detail. + +==== tests/ci improvements + +The [.filename]#tests/ci# subdirectory in the src tree was introduced in link:https://cgit.freebsd.org/src/commit/?id=cb9d4bb1fbb9ac0eb9f211656e91f9d5254c166c[commit: Add preliminary in-tree CI infrastructure for developers] by mailto:bofh@FreeBSD.org[Moin Rahman] and aims to provide an easy way for developers to replicate the CI testing run by link:https://ci.freebsd.org[our Jenkins cluster]. +In this quarter, the following improvements were made by the team: + +New functionality: + +* link:https://cgit.freebsd.org/src/commit/?id=f6e5bcd4716c2974c7ceb1fbbfc0ef8db8e13d8b[tests/ci: run ci-full kyua tests in parallel] +* link:https://cgit.freebsd.org/src/commit/?id=92a7f2d577630d670643f855a1d123a2260102af[tests/ci: Add KYUA_TEST_FILTERS to allow user to select specific tests] +* link:https://cgit.freebsd.org/src/commit/?id=932cf5fa44b7ff3b7ea21ccb32bbbefff11c160c[tests/ci: Add CIENV variable] + +Bug fixes: + +* link:https://cgit.freebsd.org/src/commit/?id=ed4c64bc792fd2509d8fd6805f8e174c2db62338[tests/ci: Use suitable variable for qemu-user-static existence check] +* link:https://cgit.freebsd.org/src/commit/?id=2f50ea54c6e3b15f655e800eb680ddcb80e30b4f[tests/ci: fix race condition in bhyveload boot] +* link:https://cgit.freebsd.org/src/commit/?id=57f725a6a6c19b12015854589764babe58252155[tests/ci: fix missing /usr/local/{sbin,bin} in freebsdci rc PATH] +* link:https://cgit.freebsd.org/src/commit/?id=9f3e4b28f9dec27b73eeebd161d119d1e9864081[tests/ci: Fix missing qemu devices] +* link:https://cgit.freebsd.org/src/commit/?id=10fb5e7117e522540167fa02692e1e7b8e38eec0[tests/ci: Fix race condition with ci-extractmeta] +* link:https://cgit.freebsd.org/src/commit/?id=66fe805070d121d4b93f63b7b02a70d76069edfc[tests/ci: Fix wrong chflags target path in 'beforeclean' target] +* link:https://cgit.freebsd.org/src/commit/?id=9ec37e8ff49c6b555e8d4f522adc9378157df170[tests/ci: Use QEMU blockdev declaration for all platforms] +* link:https://cgit.freebsd.org/src/commit/?id=e990e460d50c628dfa681c5693bd8ece87b001ad[tests/ci: Fix unescaped kld_list var in rc.conf] + +With these changes, a developer can run CI with these example commands as root: + +[source,shell] +---- +# Fully parallel CI: +make ci + +# Single-threaded CI +make PARALLEL_JOBS=1 ci + +# Single-threaded CI, running a subset of the tests as described in kyua-test(1) +make PARALLEL_JOBS=1 KYUA_TEST_FILTERS='/path/to/testcase /path/to/another:testname1' ci + +# Run smoke (boot) tests +make CITYPE=smoke ci +---- + +==== Test case management + +link:https://ci.freebsd.org/tinderbox[Tinderbox] has been reporting that our supported platforms are failing in CI since a link:https://ci.freebsd.org/job/FreeBSD-main-amd64-test/26756/[run from the last quarter]. +As the backlog grows larger, it becomes harder for users and developers to notice a new failure and pin it to a particular commit. + +To complement the [.filename]#tests/ci# CI/CD automation improvements, along with link:https://github.com/markjdb/bricoler[Bricoler] to help with more granular investigations, I worked on cleaning up the failing test backlog on tier-1 architectures. +The following patches and bug reports were submitted as a result of this (still ongoing) work: + +New bug reports filed to track failing or flaky tests: + +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288991[PR 288991: sys/netinet/output:output_raw_flowid_mpath_success] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289096[PR 289096: lib/libexecinfo/sigtramp_test:test_backtrace_sigtramp] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289165[PR 289165: usr.bin/limits/limits_test:cputime_soft_flag] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289240[PR 289240: sys/netlink/netlink_socket:overflow] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289239[PR 289239: sys/netpfil/pf/sctp:pfsync] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289236[PR 289236: sys/kern/exterr_test:gettext_extended] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289382[PR 289382: sys/netinet6/lpm6:lpm6_test1_success] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289628[PR 289628: sys/netpfil/pf/nat:endpoint_independent_pass] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289630[PR 289630: libexec/rc/rc_subr_test:wait_for_pids_progress] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289237[PR 289237: sys/fs/fusefs/last_local_modify:main] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289084[PR 289084: lib/libc/string/memcmp_test:{diff,neq}] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289477[PR 289477: sys/netpfil/pf/route_to:prefer_ipv6_nexthop_mixed_af_random_table_ipv4] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289299[PR 289299: sys/netpfil/pf/rules_counter:keepcounters] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289684[PR 289684: sys/netlink/test_snl:snl_parse_errmsg_capped] +* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289146[PR 289146: sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action] + +Unskip tests that are wrongly skipped in CI: + +* link:https://cgit.freebsd.org/src/commit/?id=6ff78a63d8cd0dd64ae79cbda5cb03572c1e17f5[tests/sys/netpfil: unskip tests that no longer need to be skipped] +* link:https://cgit.freebsd.org/src/commit/?id=178b9c2364740ead21f584dc30c3aa45bd0bb8bf[tests/ci: Add missing kmods and pkgs to unskip tests] + +Test case metadata fixes: + +* link:https://cgit.freebsd.org/src/commit/?id=83b08f40264177ecf03fce808474815816f9846f[fix parallel execution of swapon tests] +* link:https://cgit.freebsd.org/src/commit/?id=1e570722dced7a3afc0a35b10af047b835e51567[cap_dns/tests/dns_test: mark tests as needing network access] +* link:https://cgit.freebsd.org/src/commit/?id=68fe0d9cc03bd80f63a5317a633d2426ae286316[pfctl tests: use require.kmods instead of manual check for pf] +* link:https://cgit.freebsd.org/src/commit/?id=df88c711a26c04192761e96dc08a6f3f9e42afed[cap_net/net_test: require 'allow_network_access'] +* link:https://cgit.freebsd.org/src/commit/?id=5dadfac51c3876b0ba6fde831cb4417594167fed[tests/mac_portacl: enable is_exclusive for now] +* link:https://cgit.freebsd.org/src/commit/?id=cacfd42b0c57b3455ca4a23e21de071fa0cc9ed0[tests/sys/mqueue: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=215a0654186926b2e4497d83f3eb45b5eefcb23f[tests/sys/netlink: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=308c9c995794f1ae89bba1d43aab9c51f7e90a09[tests/sys/opencrypto: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=26d3dfa8733892adf107b045a8c98d480b4bb21d[tests/sys/aio: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=c444bfca0063d0820f3fc72cb429809f5f52a970[tests/pf/ioctl: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=559b0f6b82867b60a2d5cb1b2744114603cb17a7[tests/sys/netmap: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=74898ce961e1f4818eba500dfa3a06bc30c1d9b3[tests/sndstat: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=7a1a6d1e2a836d189447ba725b0e256014613ab8[tests/socket_accf: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=4079513328d5656e31cb853a799176ff2500f79b[tests/sys/net: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=427be10491e37160a82eef31622676c655a29f1e[tests/sys/netinet: use require.kmods property instead of ad-hoc checks] +* link:https://cgit.freebsd.org/src/commit/?id=f53228c4e398468c71d3ed8a372cc29ccae6a0c1[tests/vmm_cred_jail: use require.kmods property instead of ad-hoc checks] + +mark tests as "expected fail" (xfail), currently WIP: + +* link:https://cgit.freebsd.org/src/commit/?id=df997faca8411bdf8008c25de5db3ed719019d21[atf_pytest: fix xfail detection from pytest report] + +==== Tooling (WIP) + +To catch errors more quickly, instead of relying on link:https://ci.freebsd.org[Jenkins] to update the test report, I ran local CI multiple times daily. +To help with this, I worked on some tooling to speed up the testing/debugging cycles. +I am maintaining the following (currently uncommitted) tools: + +* link:https://codeberg.org/svmhdvn/mypoudriere/src/branch/main/myfreebsd[parallel CI runner built on top of tests/ci] +* link:https://codeberg.org/svmhdvn/depistage[(VERY WIP) automated CI bug report/triage system] + +Sponsor: The FreeBSD Foundation