feat: add concept of "profiles" and initial performance/debug profiles - #607
feat: add concept of "profiles" and initial performance/debug profiles#607Christopher Obbard (obbardc) wants to merge 12 commits into
Conversation
Christopher Obbard (obbardc)
changed the title
|
Loïc asked claude for a review, please find it below Overall this is a clean, well-decomposed series: the mechanism ( Things verified rather than assumed, since they're the load-bearing bits:
Substantive1. The performance profile gets no hardware boot validation. Related caveat: forcing 2. A whole extra rootfs for a cmdline-only difference. For 3. On a tree with an existing default 4. The image recipe never validates the profile name, only agreement with the rootfs. So Robustness / duplication5. The valid-profile list lives in three places — the 6. Dead branch in the image recipe: 7. The rootfs validation runs unconditionally, unlike the "Validate snapshot timestamp" precedent it's modelled on, which is wrapped in 8. Style drift on reading buildinfo. The recipe uses 9. Docs and commit messages10. 11. The new PREFIX comment cites 12. 13. Trivia: None of the above is a blocker except arguably #3, which will bite the first person who follows the README example. #1 is the one most worth the author's view on before merge, since it determines whether this profile is ever actually booted on hardware. |
Loïc Minier (lool)
left a comment
Loïc Minier (lool)
left a comment
There was a problem hiding this comment.
Hey, so I went through all the commits, really nice split and logical incremental changes – everything made sense and is inline with exactly what I would expect for these features. TL;DR: I think this is close to what we should merge for an initial version of this.
I didn't review line by line, I mostly connected with the approach and found this sound.
In review comments, I'm only flagging a minor cosmetic thing in a comment with naming of variables.
Otherwise it also bugs me a bit that we "variants" and "profiles" and that people might find this confusing. Saying "default" might also be confusing to refer to one or the other default, perhaps we should pick unique names for each of the defaults.
The Makefile duplication is also not something new and unrelated to this PR, but it's starting to get worse.
I also asked claude to review the PR, and it was insightful (will attach), albeit most bits were harmless. From the Claude notes, point 1), 2), 10), 11), 12) are worth considering IMO.
One thing made me pause. We're on a path to grow our CI significantly, yet while the changes are easy to review and maintain, they are the opposite of what we usually thrive for with Debian: an universal OS that adapts to different situations. So I was wondering if we should handle some of these more dynamically at build time or even at runtime. For instance, we could have a command to select between profiles and it would set the kernel cmdline and kernel flavor. Leaving this thought here as
| echo "VARIANT_ID={{$variantid}}" >>/etc/buildinfo | ||
| {{- end }} | ||
| {{- if ne $profile "default" }} | ||
| echo "BUILD_PROFILE_ID={{$profile}}" >>/etc/buildinfo |
There was a problem hiding this comment.
Unrelated, but looking back at it, I find VARIANT_ID slightly weird instead of VARIANT
Could BUILD_PROFILE_ID be simply PROFILE? Do we fear it will clash with something?
There was a problem hiding this comment.
nope, I am happy to change it to VARIANT and PROFILE.
VARIANT_IDcame from https://www.freedesktop.org/software/systemd/man/latest/os-release.html- and really
BUILD_PROFILE_IDwas because I really wanted to callPROFILEBUILD_PROFILE.
Happy to change the names to whatever you think makes sense. I'd lean on VARIANT_ID and PROFILE_ID everywhere, what do you think?
Also, I think having default as the default for both and including default for both in /etc/buildinfo makes the most sense, but opinions welcome...
There was a problem hiding this comment.
Fine with me; I had forgotten about the freedesktop spec for this, it doesn't seem to have a concept like profile (a few things come close, but are different), so +1 on PROFILE_ID – it does state unknown fields should be ignored.
I was also considering amending BUILD_ID to include profile, but since we're building out of the same workflow runs, I guess that's alright (technically, there could be package updates between the two flavors being built).
There was a problem hiding this comment.
OK; changed the variables to PROFILE=x and VARIANT=x. Further changes welcome ;-).
I have taken an action to open a separate issue about adding metadata into /etc/os-release based on https://www.freedesktop.org/software/systemd/man/latest/os-release.html as it doesn't belong in this PR yet I don't think :-)
435f81c to
ff46d7c
Compare
ff46d7c to
95c67d1
Compare
I agree, I think for
Yeah, I am not sure the best way to handle this. Maybe we need to consider cleaning that up very shortly.
I'll look at those one-by-one. Very detailed. I've split the "debug" into a separate branch locally, since that depends on another kernel. That can be proposed/merged after this one which adds "performance" (no kernel changes needed).
Yeah, I think that this needs to be considered as part of #507 - my vision is that as we get bigger, we should create a shared/generic minimal "rootfs", then extend it more as the recipes increase (may be other options too). I think that cleanup should come as soon as this is in though. |
Christopher Obbard (obbardc)
changed the title
Some callers omit the variant input. Give it an explicit "default" value so an omitted input selects the default image variant directly rather than relying on the empty-string fallback. This is behaviour-neutral: both an empty variant and "default" select the default image and omit the variant from artifact names. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Label each debos build job with its suite and variant so matrix jobs are easier to identify in the Actions UI. Show "default" when no variant is specified. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
PREFIX is a very generic name for what is really the filename prefix shared by every artifact a build publishes. Rename it to ARTIFACT_PREFIX, which says what it prefixes and spell the expression building it over several lines so that further components can be added to it without the line growing unreadable. While here document the property the next commit depends on: every matrix leg of a build uploads to the same S3 directory, so this prefix is the only thing keeping one suite's or variant's artifacts apart from another and anything consuming an artifact has to reconstruct it. No functional change: the prefix built for a given suite and variant is the same as before. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
95c67d1 to
0d0d8a8
Compare
The LAVA templates hardcode "{{SUITE}}-flash-{ufs,emmc}.tar.gz" which
only ever names the default variant's artifacts. A build publishes every
image it makes into one shared directory and tells them apart by the
filename prefix, so the suite alone can only ever address the default
image of that suite. Anything else that gets built is unreachable from
LAVA.
Substitute {{ARTIFACT_PREFIX}} instead and add the input feeding it. The
prefix is passed in by the caller rather than rebuilt here from suite,
variant and profile: how those combine into a filename is debos.yml's
business, and reproducing the rule in this workflow would mean two
places to update whenever the naming grows a component. The caller wrote
the build matrix, so it already knows exactly which images exist and can
name one. It cannot be read back from the build job either: that is a
matrix, and a reusable workflow's outputs are last-writer-wins across
its legs, so an output would be some arbitrary leg's prefix. That is
also why "url" survives being read from the matrix at all - every leg
uploads to one shared S3 directory and so reports the same URL, the same
property behind the duplicated build_url artifact in #397.
The input defaults to the suite, which is how the default image is
named, so this commit boots exactly what it booted before: callers that
say nothing keep testing each suite's default image. A later commit uses
it to reach the images that were unreachable.
The artifacts holding the results are scoped by the prefix too. They
were scoped by suite alone, so two calls testing different images of one
suite in the same run would delete each other's results (the action
removes the previous same-named artifact) and each publish job would
collect the other's. The results artifact grows the prefix as a suffix
rather than a prefix so the download pattern can anchor on it: "trixie"
is itself a prefix of "trixie-weston-multimedia", so a leading glob
would match both.
test-on-pr.yml publishes results from its own job rather than through
lava-test.yml's, so it hardcodes those artifact names and has to follow
the same change. The job-details pattern is unaffected, as those keep
the prefix at the front.
Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Since multi-variant builds landed, build.yml has been building the weston-multimedia variant of both suites without ever booting either one: the LAVA templates could only address a suite's default image, so the sweep silently flashed that instead. The variant has had no hardware coverage at all, and a failure to boot it would have gone unnoticed. Now that the caller names the image, add both weston-multimedia images to the sweep. The test matrix becomes an explicit list of images rather than a bare suite axis, since it no longer maps one-to-one onto the suites and each entry costs a full LAVA sweep across every board; that is also why it is a list and not a cross product of the build matrix, which would grow a sweep for every combination anyone ever builds. Only the entries that need one carry an artifact_prefix: a suite's default image is named after the suite alone, which is what lava-test.yml falls back to when the input is unset. The job gains an explicit name because the generated one stops being useful once the matrix has a second column. Naming it after the image also labels the nested job names, which is what makes several concurrent "Publish Tests Results" jobs tellable apart. This doubles the number of LAVA tests; from two to four. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
"Build on PR" builds both variants of both suites for every pull request, but the tests that follow only ever booted trixie's default image: the LAVA templates could not address anything else. So the same gap the daily build had applies to pull requests, and on the workflow where a regression is cheapest to catch, before it reaches main. Boot all four images the build produces. Each entry is a full LAVA sweep across every board, taking pull request testing from one sweep to four, but that is simply what validating four images costs: an image that is built and never booted buys nothing, and the variants differ in the graphical session that has to come up before a board reaches a login prompt, which is the part a pull request is most likely to break. This workflow publishes results from its own job rather than through lava-test.yml's, because it needs the pull request from the triggering run's payload. That job now collects the results of every image the test job booted instead of naming one prefix, so the list of images lives in the test matrix alone with nothing here to keep in sync: results are replaced in place by name and so are always the latest per board and image, and job details are still scoped to the run attempt because they keep the LAVA job id and are never replaced. More than one image means "Job 4711 on qcs6490-rb3gen2-vision-kit" no longer identifies anything on its own, so the pull request comment now says which image each job booted. lava-action names a job-details file after the test_job_file_name_prefix it is given, which lava-test.yml builds from the artifact prefix, so the image is recovered from the filename rather than tracked separately. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Add a "profile" recipe variable selecting the intended runtime configuration of the image. Only "default" is accepted for now; later commits will add "debug" and "performance" profiles. Both recipes check that the profile is valid and the rootfs recipe records the chosen profile in /etc/buildinfo as "PROFILE=<profile>". The image recipe checks that rootfs.tar was built with the profile it is invoked with; as the two recipes are separate debos invocations they can easily disagree. A rootfs.tar with no /etc/buildinfo or no PROFILE= entry is rejected as well, as its profile cannot be established. In CI, the profile is validated before the build, passed to every debos invocation and (unless it is the default profile) appended to ARTIFACT_PREFIX so that profiles of the same suite and variant don't overwrite each other's artifacts. Nothing downstream needs to change for this: lava-test.yml is handed the prefix it should boot rather than deriving it, so the naming rule stays in this one workflow. The profile is passed to both recipes even though it is expected to only affect the image recipe in the long run. Threading it through the rootfs as well is intentional while the profile set is still settling: the rootfs records PROFILE= so that the image recipe can verify the two independent debos invocations agree on it, as otherwise an image built from a stale or default rootfs.tar would be indistinguishable from a real profile build. Once the profiles have settled and the rootfs is profile-independent, the record and the check can be dropped and one rootfs build can shared between all profiles. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Move the kernel command line passed to filesystem-deploy into a $kernelcmdline variable alongside the other recipe variables. This makes it easier for later commits to extend the command line without changing the filesystem-deploy action directly. No functional change. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
0d0d8a8 to
4020da3
Compare
|
Loïc Minier (@lool) in response to your very in-depth claude review: Review 1 - The performance profile gets no hardware boot validationThis is a genuine issue to fix now (and also was an issue with multimedia images!). I have fixed that in this PR, but I will move that to its own PR once validated, it doesn't belong here. Review 2 - A whole extra rootfs for a cmdline-only differenceYeah, this was deliberate. The long-term intent is actually the opposite: profile is expected to affect only the image recipe (e.g. kernel cmdline) not rootfs contents. It's currently passed to the rootfs anyway while the profile set is still settling. Once the profiles have settled and the rootfs is demonstrably profile-independent, we can remove the profile from the rootfs (e.g. the default rootfs can be shared) and only the image and flash stages need repeating per profile. I'd rather do that as a deliberate cleanup after all profiles are implemented. Review 3 - stale
|
Christopher Obbard (obbardc)
changed the title
Add a "performance" profile. It is similar to the performance flavour meta-qcom builds from its ci/performance.yml. It builds with the default kernel configuration: no kernel options are changed for this profile as none are needed for now. It appends "quiet systemd.tty.term.console=linux" to the kernel command line so that the kernel doesn't print the boot log to the (slow) console and so that systemd still formats its own console output once that boot log is hidden. This reduces verbose kernel output during boot, suppresses driver and module initialisation messages, filesystem checks and other non-critical logs to improve overall boot time. Fixes: #604 Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Build a separate image in the daily build with the performance profile, only for trixie and the weston-multimedia variant to keep the number of CI jobs minimal and add it to the list of images LAVA tests. This deliberately runs a full extra build - rootfs, both disk images and the flash bundle - even though the performance profile currently differs from the default only by the kernel command line. "Minimal" here means the number of jobs rather than the work inside them; sharing the rootfs across profiles depends on the profile variable becoming image-only and is left as a follow-up improvement. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Add a "debug" profile. It is similar to the debug flavour meta-qcom builds from its ci/debug.yml. It enables ftrace from the very start of the boot by appending "ftrace=tracing_on trace_buf_size=5M trace_event=<events>" to the kernel command line, where <events> covers the timer, irq, workqueue, sched, power, regulator, thermal and rpmh tracepoints meta-qcom traces. The events leading to an early hang or a slow boot are therefore already recorded by the time userspace comes up, rather than only from the point a tracing session is started by hand. The profile is not ready for use yet: it still installs the default kernel, which is built without the debug options the tracepoints above are only half the story without. That needs the custom qcom-next-debug kernel package and installing it will be added in a follow-up pull request. This commit puts the profile and its kernel command line in place so that image can be built and published under its own name. Related: #568 Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Build a separate image in the daily build with the debug profile, only for trixie and the weston-multimedia variant to keep the number of CI jobs minimal, like the performance profile, and add it to the list of images LAVA tests. The profile is not ready for use yet - it still installs the default kernel rather than the qcom-next-debug package tracked by #568 - but building and booting it from now on means the follow-up that installs that kernel lands on a job which is already known to produce a bootable image. Booting it is worth a LAVA sweep of its own: tracing every timer, irq, workqueue and scheduler event from the start of the boot is exactly the kind of change that can slow a board down or keep it from reaching a login prompt, and that only hardware can tell apart from the default image. As with the performance profile this runs a full extra build - rootfs, both disk images and the flash bundle - even though the debug profile currently differs from the default only by the kernel command line. Sharing the rootfs across profiles remains a follow-up improvement. Related: #568 Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
4020da3 to
c05878b
Compare
2 participants
Add a
profiledebos variable to select the intended runtimeconfiguration of an image, as well as initial
performanceanddebugprofiles.Notes:
The debug profile is created but isn't yet ready for use, since it depends on the custom
qcom-next-debugkernel package (see Builddebugkernel variant withqcom_debug.configfragment pkg-linux-qcom#69 and Create a new "debug" debian build for Glymur #568). Installing the debug kernel will be added in a follow-up PR.This PR depends on some CI fixes, the first 6 comments. I have included them in this PR to show everything wired up, but these commits should be in their own PR.
Closes #604