Support bundles should include GZ `/var/fm` contents

[rmustacc]

Currently support bundles don't include any fault information that a system can generate. This should be included in support bundles for diagnosability and to help us correlate other issues that are going on. Specifically we should probably add /var/fm to the list of GZ-specific things to include. We should also ensure that some of the rotated FM logs are pruned periodically to ensure that we can't run a ramdisk out of space this way. While things are already in the logrotate commands, it wasn't immediately obviously to me that we ever would clean up or archive those files and not potentially fill up the ramdisk.

[davepacheco]

Just wanted to mention that if any of this work involves the sled agent DebugCollector, I've got a pretty big change there that should make this a bunch easier. (I'm not sure if that would involved in this or if the support bundle stuff will grab the files right out of /var/fm.)

[smklein]

@hawkw, @papertigers we should coordinate on this - couple things:

1. Do we want to expose an endpoint out of sled agent to "just grab all the files from /var/vm"? We have some existing endpoints for support bundles -- e.g., support_logs_download -- but the support bundle collector can nab whatever it needs. @davepacheco , if you're adding something for the DebugCollector, we could use that.
2. Mechanically adding this to the bundle (on the Nexus side) should be straightforward, because we've already gone through a similar process for zone logs (see: save_zone_log_zip_or_error in nexus/src/app/background/tasks/support_bundle/steps/host_info.rs). In that example, we used: "assemble + expose the contents of /var/fm as a zipfile which we export from the sled agent -> transfer that to Nexus -> Unzip it, re-assemble it into the support bundle".

[papertigers]

@smklein sorry for the late reply, but I think if you want each sled-agent to grab a copy of it's /var/fm logs then we could easily add another method like support_logs_download. I am not familar with the DebugCollector that @davepacheco mentioned so it may make sense for us to chat about this a bit more before we decide on a path forward.

[davepacheco]

The "pretty big change" I mentioned above landed a few days ago. The gist is that the set of files that sled agent archives from the live system into the debug dataset is now closer to configuration than handwritten code. Here's the example that configures SMF log files:

omicron/sled-agent/config-reconciler/src/debug_collector/file_archiver/rules.rs

Lines 129 to 130 in 3b27679

	directory: VAR_SVC_LOG.parse().unwrap(),
	regex: "^.*\\.log$".parse().unwrap(),

If you wanted to add, say, /var/fm/fmd/fltlog (so that this file gets archived regularly into the debug dataset), it may be as simple as adding a new entry in that list for this file. Note that I'm just talking about sled agent's periodic archival of live state into debug datasets. I'm not sure how this translates to support bundles.

It's very possible though that this is the wrong approach. The archiver is oriented around things like rotated log files and core files. These are immutable and no longer needed in their original locations and the archiver deletes them. I'm not familiar with most of the data in /var/fm. If it's more like occasionally-changing files that you don't want to delete when they get archived, we might need something different. (You can actually configure an archive rule to not delete the data. In that case, I think you'll wind up getting a full new copy in the debug dataset every 5 minutes or so.)

I'm happy to chat more about it.

[papertigers]

Yea I would be curious to chat more about this at some point. Does the new archive system follow the naming conventions of how archived log files were named before? Just thinking about what may have broken in support bundles if things changed at all since we rely on oxlog for finding log files on a sled. I can't really spend time on this currently given other priorities but maybe we could discuss this in the next few weeks.

[davepacheco]

The change I landed did not change any runtime behavior at all. Core files, crash dumps, log files, etc. are all archived to the same spot in the debug dataset that they were before.

The only thing I'm aware of that oxlog currently misses are live log files: #9271.