Backlog: Python requirements auto-upgrade

[pcrespov]

A backlog of features to support automatic dependency upgrade:

• follow up on workaround with pytest-sugar: added py in requirements
• prune pylint from tests and move them to tools?
• tools in containers or separate virtualenv like pre-commit ???
• how to sync/ guarantee same tool versions with pre-commit?
• Do we really need to pin tools? why not leaving them open to latest always? Could add constraints for delicate cases (e.g. pylint)

setup

• review setup.py (see catalog/setup.py as reference)
  • create setup template. describe rationale
  • how to include intra-dependencies in a relative fashion?
• shall we use setuptools with pep517?

Constraints

• catching up and removing constraints

repo-wide common packages

• same tooling deps and configs (e.g. mypy)
• limit testing lbiraries

Test

• compare pip freeze vs compiled reqs (i.e. *.txt)

Vulnerabilities

• repo-wide constraints
• repo-wide upgrades and auto-commit

Keep some dependencies in sync repo-wide

• upgrade only testing and tooling dependencies

Reduce the number of dependencies

how to choose the right libraries?, how to avoid duplication? how to determine the "cost" of adding a given dependency? how to track whether a dependency is still needed?

• remove unused dependencies
• dependencies coverage (how much a library is actually used, e.g. avoid adding a dependency for only one function)

Tools

Some of the tooling used for development is installed in the .venv so vscode has access to it. These libraries can be independently added for every package in requirements/_tools.in. The main drawback is that they might add constraints to upstream dependencies (e.g. cannot upgrade a library in _base.in because downstream there is a tool that uses an old version of it).
For that very reason, tools like pre-commit use an independent venv for all the tools it needs in the hooks
An alternative is installing the tool in a container and using the CLI to trigger it (e.g. instead of calling pylint entrypoint created in the venv, we call docker run tools/pylint .... Consider the latter! See

• Maintenance: review and cleanup python formatting & static analysis tooling #2731
• Check https://docs.renovatebot.com/

Upgrades Strategies

• testing and tooling upgrades
• service upgrades
• target libraries upgrades
  • key libraries
  • vulnerabilities
  • removing constraints
• @sanderegg proposed to add constraints in packages dependencies instead of freezing them. This means that the tests runs will always take the latest versions of requirements fitting constraints on the main libraries (e.g. pydanctic<2, etc). This means we do not need to freeze anymore packages and in exchange we can add extras. The latter will avoid installing libraries in services that are not needed. This will not only reduce the size of the service but also free from incompatibilites with other libraries the service installs.

Monitoring

• When to upgrade?
• What was upgraded? first level or deeper level requirement?
• How many upgrades a library had ? Which failed? Determine how robust is it.
• Uniformity? repo wide how many versions of a given library are?
• recurrent tasks
  • how to automate or remind us recurrent tasks? e.g. pip-tools was constraint because of an incompatibility with pip upgrades ... how to remind us when to remove this constraint? upgrade list of key libraries (fastapi, aiohttp, ...)

[pcrespov]

TODO: review notes in python-dependencies.md or packages-notes.md

[pcrespov]

@colinRawlings please do not hesitate to add inputs to this list. I am looking forward to your suggestions to make maintenance both more reliable and automatic