Compare with most recent build

[simonbyrne]

For some CI infra, it would be helpful if we could compare with the most recent (or perhaps n most recent?) builds on the main branch. This could be helpful for regression testing (i.e. have our results changed?), and performance (how has our scaling changed).

Buildkite provides two mechanisms for storing data:

• artifacts: these are scoped to a particular step, and are stored for 6 months. It can be queried by build id and step key: https://buildkite.com/docs/agent/v3/cli-artifact#downloading-artifacts
• meta-data: these are scoped to a build (i.e. are shared across steps), and are limited in size, but are retained indefinitely. They (somewhat annoyingly) need to be queried by job id of the build: https://buildkite.com/docs/agent/v3/cli-meta-data#getting-data

Initially we will use artifacts, as they are more useful for storing data. The main problem is then "what is the step id of the previous job"? The best way to do this is probably something like:

1. Find the most recent commit that is also a commit on the main branch. Once the repository is checked out, we can do:

   git merge-base HEAD main
   

2. Query the buildkite API for this comnit:

   curl -H "Authorization: Bearer $TOKEN" 'https://api.buildkite.com/v2/organizations/clima/pipelines/climaatmos-scaling/builds?commit=$COMMIT&per_page=1' | jq -r 'first.id'
   

3. If this doesn't exist (e.g. in ClimaAtmos-scaling, we only run it nightly, not on every commit), just get the last one on the main branch

   curl -H "Authorization: Bearer $TOKEN" 'https://api.buildkite.com/v2/organizations/clima/pipelines/climaatmos-scaling/builds?branch=main&per_page=1' | jq -r 'first.id'
   

4. store this in the build meta-data (similar to 729741a), so that we don't need to store it for subsequent jobs in this build (prev_build_id?)
5. set an appropriate environment variable (PREV_BUILDKITE_BUILD_ID?)

In that way, the comparison code can then download the artifact by

buildkite-agent artifact download --build $PREV_BUILDKITE_BUILD_ID --step $BUILDKITE_STEP_KEY artifact_name destination

[trontrytel]

In the EDMF part of the ClimaAtmos CI we have plots that show the state of the main branch and the PR branch. Checkout any recent profile plot (two lines for the main and PR) and contour plots (two panels for the main and the PR).

[sriharshakandala]

@charleskawczynski : Is this issue still active?