Status/2025Q3/sta-wpc-ci-automation.adoc: Add report

Author: svmhdvn

website/content/en/status/report-2025-07-2025-09/sta-wpc-ci-automation.adoc

@@ -0,0 +1,104 @@
+=== STA Work Package C: CI/CD Automation
+
+Contact: Siva Mahadevan <smahadevan@freebsdfoundation.org>
+
+In this quarter, as part of the infrastructure modernization work commissioned by the Sovereign Tech Agency (STA), I have been working on the in-tree CI Makefile targets. I also worked on bringing our CI test reports to a clean state on our tier-1 architectures (amd64 and aarch64). This report is a supplement to the overall STA status report and will describe the work done in more detail.
+
+==== /tests/ci improvements
+
+The /tests/ci subdirectory of the src tree was introduced in link:https://cgit.freebsd.org/src/commit/?id=cb9d4bb1fbb9ac0eb9f211656e91f9d5254c166c[commit: Add preliminary in-tree CI infrastructure for developers] by bofh@ and aims to provide an easy way for developers to replicate the CI testing run by link:https://ci.freebsd.org[our Jenkins cluster]. In this quarter, the following improvements were made by the team:
+
+New functionality:
+
+* link:https://cgit.freebsd.org/src/commit/?id=f6e5bcd4716c2974c7ceb1fbbfc0ef8db8e13d8b[tests/ci: run ci-full kyua tests in parallel]
+* link:https://cgit.freebsd.org/src/commit/?id=92a7f2d577630d670643f855a1d123a2260102af[tests/ci: Add KYUA_TEST_FILTERS to allow user to select specific tests]
+* link:https://cgit.freebsd.org/src/commit/?id=932cf5fa44b7ff3b7ea21ccb32bbbefff11c160c[tests/ci: Add CIENV variable]
+
+Bug fixes:
+
+* link:https://cgit.freebsd.org/src/commit/?id=ed4c64bc792fd2509d8fd6805f8e174c2db62338[tests/ci: Use suitable variable for qemu-user-static existence check]
+* link:https://cgit.freebsd.org/src/commit/?id=2f50ea54c6e3b15f655e800eb680ddcb80e30b4f[tests/ci: fix race condition in bhyveload boot]
+* link:https://cgit.freebsd.org/src/commit/?id=57f725a6a6c19b12015854589764babe58252155[tests/ci: fix missing /usr/local/{sbin,bin} in freebsdci rc PATH]
+* link:https://cgit.freebsd.org/src/commit/?id=9f3e4b28f9dec27b73eeebd161d119d1e9864081[tests/ci: Fix missing qemu devices]
+* link:https://cgit.freebsd.org/src/commit/?id=10fb5e7117e522540167fa02692e1e7b8e38eec0[tests/ci: Fix race condition with ci-extractmeta]
+* link:https://cgit.freebsd.org/src/commit/?id=66fe805070d121d4b93f63b7b02a70d76069edfc[tests/ci: Fix wrong chflags target path in 'beforeclean' target]
+* link:https://cgit.freebsd.org/src/commit/?id=9ec37e8ff49c6b555e8d4f522adc9378157df170[tests/ci: Use QEMU blockdev declaration for all platforms]
+* link:https://cgit.freebsd.org/src/commit/?id=e990e460d50c628dfa681c5693bd8ece87b001ad[tests/ci: Fix unescaped kld_list var in rc.conf]
+
+With these changes, a developer can run CI with these example commands as root:
+
+[source,shell]
+----
+# Fully parallel CI:
+make ci
+
+# Single-threaded CI
+make PARALLEL_JOBS=1 ci
+
+# Single-threaded CI, running a subset of the tests as described in kyua-test(1)
+make PARALLEL_JOBS=1 KYUA_TEST_FILTERS='/path/to/testcase /path/to/another:testname1' ci
+
+# Run smoke (boot) tests
+make CITYPE=smoke ci
+----
+
+==== Test case management
+
+Since the last quarter, link:https://ci.freebsd.org/tinderbox[Tinderbox] has been reporting that our supported platforms are failing in CI since a link:https://ci.freebsd.org/job/FreeBSD-main-amd64-test/26756/[run from the last quarter]. As the backlog grows larger, it becomes harder for users and developers to notice a new failure and pin a failure to a particular commit.
+
+To complement the /tests/ci CI/CD automation improvements, along with link:https://github.com/markjdb/bricoler[Bricoler] to help with more granular investigations, I worked on cleaning up the failing test backlog on tier-1 architectures. The following patches and bug reports were submitted as a result of this (still ongoing) work:
+
+New bug reports filed to track failing or flaky tests:
+
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288991[PR 288991: sys/netinet/output:output_raw_flowid_mpath_success]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289096[PR 289096: lib/libexecinfo/sigtramp_test:test_backtrace_sigtramp]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289165[PR 289165: usr.bin/limits/limits_test:cputime_soft_flag]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289240[PR 289240: sys/netlink/netlink_socket:overflow]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289239[PR 289239: sys/netpfil/pf/sctp:pfsync]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289236[PR 289236: sys/kern/exterr_test:gettext_extended]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289382[PR 289382: sys/netinet6/lpm6:lpm6_test1_success]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289628[PR 289628: sys/netpfil/pf/nat:endpoint_independent_pass]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289630[PR 289630: libexec/rc/rc_subr_test:wait_for_pids_progress]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289237[PR 289237: sys/fs/fusefs/last_local_modify:main]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289084[PR 289084: lib/libc/string/memcmp_test:{diff,neq}]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289477[PR 289477: sys/netpfil/pf/route_to:prefer_ipv6_nexthop_mixed_af_random_table_ipv4]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289299[PR 289299: sys/netpfil/pf/rules_counter:keepcounters]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289684[PR 289684: sys/netlink/test_snl:snl_parse_errmsg_capped]
+* link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289146[PR 289146: sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action]
+
+Unskip tests that are wrongly skipped in CI:
+
+* link:https://cgit.freebsd.org/src/commit/?id=6ff78a63d8cd0dd64ae79cbda5cb03572c1e17f5[tests/sys/netpfil: unskip tests that no longer need to be skipped]
+* link:https://cgit.freebsd.org/src/commit/?id=178b9c2364740ead21f584dc30c3aa45bd0bb8bf[tests/ci: Add missing kmods and pkgs to unskip tests]
+
+Test case metadata fixes:
+
+* link:https://cgit.freebsd.org/src/commit/?id=83b08f40264177ecf03fce808474815816f9846f[fix parallel execution of swapon tests]
+* link:https://cgit.freebsd.org/src/commit/?id=1e570722dced7a3afc0a35b10af047b835e51567[cap_dns/tests/dns_test: mark tests as needing network access]
+* link:https://cgit.freebsd.org/src/commit/?id=68fe0d9cc03bd80f63a5317a633d2426ae286316[pfctl tests: use require.kmods instead of manual check for pf]
+* link:https://cgit.freebsd.org/src/commit/?id=df88c711a26c04192761e96dc08a6f3f9e42afed[cap_net/net_test: require 'allow_network_access']
+* link:https://cgit.freebsd.org/src/commit/?id=5dadfac51c3876b0ba6fde831cb4417594167fed[tests/mac_portacl: enable is_exclusive for now]
+* link:https://cgit.freebsd.org/src/commit/?id=cacfd42b0c57b3455ca4a23e21de071fa0cc9ed0[tests/sys/mqueue: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=215a0654186926b2e4497d83f3eb45b5eefcb23f[tests/sys/netlink: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=308c9c995794f1ae89bba1d43aab9c51f7e90a09[tests/sys/opencrypto: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=26d3dfa8733892adf107b045a8c98d480b4bb21d[tests/sys/aio: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=c444bfca0063d0820f3fc72cb429809f5f52a970[tests/pf/ioctl: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=559b0f6b82867b60a2d5cb1b2744114603cb17a7[tests/sys/netmap: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=74898ce961e1f4818eba500dfa3a06bc30c1d9b3[tests/sndstat: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=7a1a6d1e2a836d189447ba725b0e256014613ab8[tests/socket_accf: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=4079513328d5656e31cb853a799176ff2500f79b[tests/sys/net: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=427be10491e37160a82eef31622676c655a29f1e[tests/sys/netinet: use require.kmods property instead of ad-hoc checks]
+* link:https://cgit.freebsd.org/src/commit/?id=f53228c4e398468c71d3ed8a372cc29ccae6a0c1[tests/vmm_cred_jail: use require.kmods property instead of ad-hoc checks]
+
+mark tests as "expected fail" (xfail), currently WIP:
+
+* link:https://cgit.freebsd.org/src/commit/?id=df997faca8411bdf8008c25de5db3ed719019d21[atf_pytest: fix xfail detection from pytest report]
+
+==== Tooling (WIP)
+
+To catch errors more quickly, instead of relying on link:https://ci.freebsd.org[Jenkins] to update the test report, I ran local CI multiple times daily. To help with this, I worked on some tooling to speed up the testing/debugging cycles. I am maintaining the following (currently uncommitted) tools:
+
+* link:https://codeberg.org/svmhdvn/mypoudriere/src/branch/main/myfreebsd[parallel CI runner built on top of /tests/ci]
+* link:https://codeberg.org/svmhdvn/depistage[(VERY WIP) automated CI bug report/triage system]
+
+Sponsor: The FreeBSD Foundation