ADR Suggestion: Freezing Dependency Versions on Release

[rozyczko]

Strict version pinning at release time (Hard Freeze)

To reduce the risk of unexpected breakages from dependency updates, improve reproducibility, and provide a more stable experience for users, we propose to adopt strict version pinning at release time ("Hard Freeze"). This ensures that users always install the tested and validated environment corresponding to each release.

Summary of the proposal

• All dependencies shall be pinned to exact versions in requirements.txt or later in pyproject.toml at the time of release, in the develop branch, before merging with main.
  e.g.

  [project]
  dependencies = [
      "numpy==1.24.2",
      "scipy==1.10.0",
      "matplotlib==3.7.1"
  ]

• Running python setup.py build (or later pip install .) will install the exact versions of dependencies that were tested and validated for the release.

To keep development workflows clean and flexible:

• Immediately after release, the develop branch shall unpin all dependencies in requirements.txt or pyproject.toml so that it uses the latest available versions for development and CI testing.

[jamescrake-merani]

We don't currently have a develop branch. Are you proposing that we create one? Or are you referring to the release branches that we create?

Regardless, I think the idea of this proposal seems sensible. This is especially important because we need to use pins on the Flatpak so it will be very useful to have a set of pins that have already been tested.

Also I have some questions (or perhaps points that we need to discuss in more detail) regarding the timings. When exactly should we lock in the pins given our current release schedule? I was thinking perhaps after the feature code freeze so that when we enter the bug fixing phase, we don't get new bugs introduced by dependency changes. But I am open to other suggestions.

[rozyczko]

I am merely proposing that we try to address the issue we've had for ages. Yes, we don't have develop but the proposed workflow doesn't have to use it.
Instead of develop, we can have the release branch pinned after code freeze, to iron out all the versioning issues.
Then, after the release, we merge back to main and either

• unpin dependencies in the release branch and update main
• update main with the release branch and then unpin dependencies

I would prefer the second solution since it leaves the release branch "untouched" and corresponds 1:1 with the released code.

[krzywon]

I like this suggestion a lot, but a process for getting the dependencies for each of our supported OSes should be included in the ADR, and who should make the changes. My only concern is how long the list might grow to, especially if different OS use different dependency versions.

At the time of a code freeze, the release manager must review the build logs for every supported OS release and capture the version for every named dependency within the requirements list.

I also prefer the 2nd option in the unpinning process. After a release, even though tags exist, branches don't result in headless git mode on checkout, so the release branch should not be modified.

[krzywon]

Discussion from technical meeting:

Who does it? - Release manager ensures it happens, but anyone can do it.
All OSes use same version of dependencies... If one OS doesn't work with a pinned package, rollback to allow stable release, keeping the version the same for all OSes.

[krzywon]

First release to try this - v6.1.0. If it doesn't work, we can roll back and use our existing release process.

[krzywon]

Adding comments from the general meeting today:

• A trial run of this process for v6.1.0 is generally considered a good idea.
• @lucas-wilkins commented that for distributed apps, pinning dependencies makes sense, but for wheels (https://github.com/orgs/SasView/discussions/3161), this could cause conflicts, especially when users pip install into existing environments
• @wpotrzebowski asked, when the release branch is merged back into main, do the pinned versions stay, or should dependencies remain unpinned in main? The answer was dependencies should remain unpinned in main for development, unless there is a compelling reason for pinning a dependency. This should be specified in any final ADR.

[krzywon]

Process for 6.1.0:

• Add pip list to ci.yml to generate list of dependency list on github
• Compile list of dependencies from all OSes
• Compare OS lists and generate single list based on all
• Do we pin dependencies of dependecies or just ones listed in requirements.txt? Likely the former in case a subdependency adds breaking changes.
• MacOS architectures... TBD

Many issues to overcome

[jamescrake-merani]

So pip freeze actually does the same thing as pip list, but the former command outputs in the requirements file format meaning we can directly run pip install -r on its output.

For 6.1.0, I propose the following procedure:

• On the day of the code freeze, we create a fresh environment for each platform, and install the latest versions of packages. Its important that we use the same version of Python as the CI.
• We then run pip freeze, and pipe its result into a requirements file in the build tools directory. We can use a name such as {platform_name}-requirements.txt
• Importantly, we treat macOS Intel, and ARM machines as separate platforms. This is because some wheels are dependent on a specific processor architecture.
• These requirement files will then be used by the CI, and should also be used in development environments.

I notice as well that the CI builds for both Python 3.10, and 3.11. This may cause problems for packages that specify the Python versions they support. For example, some packages' latest versions may support 3.11 but not 3.10, and thus pip won't install them on 3.10. For this reason, I propose only building for one version of Python for the CI. I'd suggest that we go with Python 3.11 for this release. This only has to apply to our 6.1.0 release, and we can keep both versions of Python being built for on main. I just think this would simplify our testing if we only have to worry about one version of Python.

I will bring these proposals to the fortnightly discussion but feel free to discuss this here before then.

[butlerpd]

agree that one python version for the release branch should be sufficient and we can use all the simplification help we can get IMO

[krzywon]

Moving forward, we could have separate requirements files for releases. The release requirements files would be of the form requirements-{platform_name}.txt as suggested by @jamescrake-merani, and have a separate action step specific for branches that match the pattern release*.

Example yml step to add (untested):

      - name: Install Python dependencies (Release Branches Only)
        if: contains(github.ref, "release")
        run: |
          python -m pip install --upgrade pip
          python -m pip install wheel setuptools
          python -m pip install -r build_tools/requirements-${{ matrix.os }}.txt

      - name: Install Python dependencies
        if: !contains(github.ref, "release")
        run: |
          python -m pip install --upgrade pip
          python -m pip install wheel setuptools
          python -m pip install -r build_tools/requirements.txt

[wpotrzebowski]

I am a bit confused here. If I understand correctly, the current proposal for GH action it will install requirements from the previous release to next release (whenever branch is created). E.g. if we now save requirements-6.1.0.txt and then create let say release_6.1.1 branch it will install requirements-6.1.0.txt, which may be quite old and not essentially what we want?

[krzywon]

The suggestion is not to have separate requirements files for each release, but for each operating system (matrix.os). When the release manager is preparing the release branch, the OS-specific files should be updated to use the latest, stable, versions of each dependency.

[krzywon]

I had to do one odd thing in #3358 that weren't discussed here. I had to create python version specific requirements files: If our tests were against newer versions of python, rather than our current CI (release off 3.11, test against 3.10 and 3.11), I could have pinned all dependencies to the release version. This should change in the future.

[krzywon]

A few notes from the PR (thanks @llimeht for the suggestions):

• I have turned off py 3.10 tests for the release branch. This should likely be merged as-is into main once the release is finalized.
• Include the hash for each dependency to prevent malicious code inclusion. There are existing tools to help manage this, but this likely won't happen for v6.1.0
• Any resulting ADR from this proposal should include a guide on the process for pinning including the how and when it should be performed, including setting the branch name in ci.yml, any scripts to run as part of the CI to generate the list, and at what point in the process should the release requirements files be updated.

[krzywon]

Comments from technical meeting

• Always use a single python version (latest) for releases and testing (suggestion)

[krzywon]

One unintended consequence of our release-specific versioning - dependabot changes. We currently have dependabot running to suggest dependency updates listed in our build_tools directory once a week. Our main requirements file is mostly using unpinned dependencies, so we get very few suggestions from this, but now that our 6.1.0 release branch is merged into main, the release specific dependency lists, which are 100% pinned, are also being checked by the bot. These pinned versions will only change per release so we have a clear record of the dependency versions used in each forthcoming release.

Moving forward, we have a few options to handle this:

1. Once the release branch is in main, having versions stepped up doesn't hurt us, so long as the main branch isn't broken, so accept all dependabot pull requests.
   a. Coupled with the previous suggestion, we can bundle dependencies in our dependabot config so they are all incremented in a single PR by the bot.
2. Move the release yamls into their own directory (one level deeper in build_tools?) so they aren't caught by dependabot.
3. Kill dependabot and continue to use unpinned dependency lists in main, and fully pinned dependency lists for releases.

[jamescrake-merani]

I have written a revised version of the ADR based on suggestions that have been made since the original draft:

Strict version pinning at release time (Hard Freeze)

Preamble

To reduce the risk of unexpected breakages from dependency updates, improve reproducibility, and provide a more stable experience for users, we propose to adopt strict version pinning at release time ("Hard Freeze"). This ensures that users always install the tested and validated environment corresponding to each release.

Proposal

• During a release cycle, all dependencies shall be pinned to exact versions after the first code freeze.
• Each platform shall have a different set of dependency pinnings.
• The versions that each dependency is pinned on shall be determined by creating a fresh virtual environment, and installing the latest versions of all dependencies. Then, the pip freeze command should be used to determine the requirements.txt text for that platform. This command is run as part of the CI, and the output from that can be used.
• For the purpose of this ADR, both macOS Intel, and macOS ARM builds shall be considered separate platforms, as this has the potential to affect the dependencies that are available to both platforms.
• Dependency pinnings shall not apply to the PyPi distribution of SasView.

Changing pinnings

• Dependency pins should only be altered post code-freeze in very limited circumstances. Such as:
  • When a bug is spotted which is found to be related to a dependency, and where downgrading/upgrading the pin is found to resolve the issue. If possible, we should consider making changes to our code first if this can fix the issue before changing the dependency pinning.
  • Where a package is found to have some instability which can be rectified by downgrading to an older more stable version, or upgrading to a version which includes fixes to bugs which are specific to that package.
• Dependencies pinnings, where altered, should be altered for all platforms unless there is a strong reason to only alter them for a specific platform. If this is necessary, the release manager should consider the impact this may have on testing, as differing versions of dependencies could potentially lead to platform-specific bugs.

CI

• Our CI should be configured so that, on the main branch, builds do not use pinned dependencies.
• Any release branch, on the other hand, should always use pinned dependencies for builds.

[wtheller]

Is there a concern that users with computers that are subject to regular security scans will find that Sasview breaks when a fixed version of a dependency package is found to have an issue and must be updated? The security folks sometimes deliver messages about security udates as "update or else".

The matter is more or less addressed in the "changing pinnings" updated proposal provided by James, but the text does not mention security issues.

[krzywon]

@wtheller, this is a valid concern, but any installed version will break if the security scans are auto-removing dependencies. The only time we are pinning is during the final release stages and this is only for the bundled releases. If a release is found to have a security issue that needs resolving, we should be able to release a new version, using the appropriate dependencies, fairly quickly. Also, during the release process, as you note, if a security issue is found, the procedure allows for an increment or roll back of dependencies.

For developers, updating dependencies on their local system should happen on a semi-regular basis anyway, regardless of what they are testing.

[butlerpd]

As mentioned in the email to developers, this ADR was brought up for final approval at the October 7th meeting. The single remaining comment after the final draft was deemed answered satisfactorily and no further comments were made. The ADR was thus approved as finalized. Final steps to create the final ADR can now proceed.

[krzywon]

Finalized in #3831 - Closing this now.