Lockfile introduction / better dependency management

[joecrowley-synergy]

I've encountered once (possibly twice) issues with package updates resulting in failed tests, highlighting compatibility issues (e.g. #220). I would like to have included more restriction on dependency versions and / or lock files introduced to the repo. Options:

• Add a requirements.txt to the repo and utilise something like pip install -r requirements.txt.
• Go through dependencies and better define each version constraint manually for each named, especially max versions.
• Introduce a dependency management tool (e.g. Poetry) and have it define dependency version constraints automatically (e.g. poetry add pydantic would look at currently installed versions of packages and add the necessary version constraint to the project.dependencies of pyproject.toml file)
• If the above is adopted, introduce lockfile (e.g. poetry.lock) automatically generated by the tool (as opposed to the requirements.txt approach).

Pros:

• New comers can have some confidence that the versions of packages they are installing for the project are what is being used as reference when going through Getting Started tutorials
• Bug reviews will (likely) become less shrouded in mystery, eliminating conflicting package issues

Cons:

• Less flexible
• Managing updates becomes a new manual, sometimes tedious and soul crushing, task (could be a positive tho).

[joshvote]

Hi Joe,

This is an echo to a conversation we had internally at BSGIP a few years back. You're on the money with the PRO's / CON's and they mirror the discussions we had internally.

I'd like to say our current approach has been working OK but we have definitely gotten stung twice in the last few days with alembic/pytest_asyncio both updating and breaking things:

#224
#221

The big reason we leaned into "unpinned" dependencies is that with our small team, we figured we'd never end up properly rolling forward our dependencies and that with good test coverage, the dependency issues became kind of moot.

All that being said, we could look at doing a lockfile to correspond with a version tag. That way, people at least have a known "good" configuration to reference. We can create a simple action that will just run tests using that lockfile instead of our current dependencies and that should be good.

I'll leave this issue open until we pop it in place. The key requirement from my side is ensuring that it's integrated/tested with our current actions (either on release tagging or as part of a PR)

[joecrowley-synergy]

Thanks Josh, that sounds like a good way forward to me. This will at least help me better eliminate project package issues vs my local installation environment.