Define a methodology to fast forward the software stack on a specific system

[pearce8]

• Construct a list of currently available spack externals on the system
• Compare with the list of spack externals in system.py
• Provide a list of spack externals in system.py which are no longer available (versions or at all)
• Provide a list of currently available spack externals which are not in system.py
• (ultimately) propose updates to system.py, removing deprecated and adding new software versions

@matthewc2003 please use your work on Frontier as a way to build+test this methodology. Please ping me as you get answers for the questions above or need to discuss.

@slabasan @michaelmckinsey1 FYI.

[matthewc2003]

Most of the functionality requested is present in the "benchpark system external" command. As such I extended it to provide information on missing packages as well as a bit of reformatting to make it more human readable. I am currently looking through the output on Frontier to validate that packages I can find by hand are detected by this utility as well, basically sanity checking.

TODOs:

1. Use the information gathered to propose the changes to system.py. Starting with the just version updates, until I can validate packages the search fail to discover actually do not exist.

Foreseeable limitations:

1. Modules need to be loaded with lmod to be discovered by spack external find (makes sense)
2. Scan only hits packages loaded by default and I'm not sure about how we want to manage handling multiple versions of the same package.

Questions/Notes:

1. How do we handle upstream spack installations, specifically those managed by Lmod? It seems like we can add the managed spack installation as upstream to the local spack repo, but that might have packages that aren't officially supported on the system and may not be generalizable across systems. I think it would be nice if all/most Lmod based modules are discovered as spack externals where relevant. Currently looking into the feasibility of this.
2. The current plan is to automatically generate the spack packages needed by the benchmarks in Benchpark and then explicitly look for them in an automated fashion, updating the outdated system spec right?

@pearce8