Additional features for pip dependencies: uniform logging, private repos and comes_from info

[nestormh]

Hello, in our project we have been using Medikit for a while, but we were missing some features which were useful for us. With this aim, we created a fork. These changes are now included in this PR for discussion, improvement, and ideally for potential inclusion in your main project if you find them useful.

Added features

These are the newly added features:

Uniform dependencies

Due to the nature of our project, we wanted to have different dependencies for different extras, but in some cases, we needed to have the same versions for all the extras. With this aim, we have added the parameter python.use_uniform_requirements. When True, for a certain requirement, finds a valid version for all the extras using it (including inherited requirements) and sets all of them for all the requirements*.txt files implied. We are not considering setup.py for this feature, to keep some flexibility after the process.

Addition of private repositories

It is now possible to use private repositories using git, for instance including in the python add_requirements something like: requirement @ git+ssh://[email protected]/user_name/requirement_2.git@{checkpoint_or_tag}, which will be then translated into equivalent requirements in the setup.py and requirements*.txt files. More information

Show the requirement which inherits a certain requirement

Just a visualization feature. When python.show_comes_from_info is True, presents a comment with the generated requirements, indicating the requirement which caused the installation of a dependency, to understand better the changes.

Extended information

New parameters provided:

• python.use_uniform_requirements: If True, requirements are set to be the same for all the extras when there are coincidences, directly or on the inherited dependencies.
• python.show_comes_from_info: If True, information about the origin of a dependency is shown.

Valid requirement examples:

boto3 ~=1.10,
requirement_1 @ git+https://{token}@github.com/user_name/requirement_1.git@{checkpoint_or_tag},
requirement_2 @ git+ssh://[email protected]/user_name/requirement_2.git@{checkpoint_or_tag},

Projectfile usage example:

# Test (see github.com/python-medikit)

from medikit import require

NAME = 'Test'

with require('python') as python:
    python.setup(
        name=NAME,
        description='This is just a test',
        license='Apache License, Version 2.0',
        url='http://www.an-example.com/',
        download_url='http://www.an-example.com/',
        author='An author',
        author_email='[email protected]',
    )

    # New parameters are configured as follows
    python.use_uniform_requirements = True
    python.show_comes_from_info = True

   # You can try commenting/uncommenting the following to see the behavior
    python.add_requirements(
        'boto3 ~=1.10',
        # f'httpie @ git+https://github.com/httpie/[email protected]',
        dev=[
            'boto3 ~= 1.11',
            # 'httpie >= 2.2.0',
            # f'httpie @ git+https://github.com/httpie/[email protected]',
        ],
        extra1=[
            'boto3 == 1.12',
            # f'httpie @ git+https://{token}@github.com/httpie/httpie.git@5945845420c019db81e8c09f0ce174a7eeebcdd8',
            f'httpie @ git+https://github.com/httpie/[email protected]',
            # 'httpie < 2.4.0'
        ],
        extra2=[
            'boto3 ~=1.11',
            # f'httpie @ git+https://github.com/httpie/[email protected]',
        ],
        extra3=[
            # 'boto3 == 1.12',
            # 'httpie == 2.2.0',
        ]
    )

with require('git') as git:
    pass

with require('make') as make:
    pass

# vim: ft=python:

• In a git repository, if there are submodules, it will explore them along all of them
• If use_uniform_requirements== True, requirement versions will be unified among all the extras.
  NOTE: In case a private repository version is mixed with a different version from another repository, or even with
  a packaged version (included inherited versions), an IncompatibleRequirements exception will be triggered.

For instance, the example above will become:

cat requirements.txt 

-e .
boto3==1.12.0
botocore==1.15.49                       # From: boto3==1.17.45
jmespath==0.10.0                        # From: boto3==1.17.45
python-dateutil==2.8.1                  # From: botocore==1.20.45
s3transfer==0.3.6                       # From: boto3==1.17.45
six==1.15.0                     # From: python-dateutil==2.8.1
urllib3==1.25.11                        # From: botocore==1.20.45

cat requirements-dev.txt

-e .[dev]
-r requirements.txt
boto3==1.12.0
botocore==1.15.49                       # From: boto3==1.17.45
jmespath==0.10.0                        # From: boto3==1.17.45
python-dateutil==2.8.1                  # From: botocore==1.20.45
s3transfer==0.3.6                       # From: boto3==1.17.45
six==1.15.0                     # From: python-dateutil==2.8.1
urllib3==1.25.11                        # From: botocore==1.20.45

cat requirements-extra1.txt

-e .[extra1]
-r requirements.txt
boto3==1.12.0
botocore==1.15.49                       # From: boto3==1.12
certifi==2020.12.5                      # From: requests==2.25.1
chardet==4.0.0                  # From: requests==2.25.1
docutils==0.15.2                        # From: botocore==1.15.49
git+https://github.com/httpie/[email protected]
idna==2.10                      # From: requests==2.25.1
jmespath==0.10.0                        # From: boto3==1.12
pygments==2.8.1                 # From: httpie@ git+https://github.com/httpie/[email protected]
python-dateutil==2.8.1                  # From: botocore==1.15.49
requests==2.25.1                        # From: httpie@ git+https://github.com/httpie/[email protected]
s3transfer==0.3.6                       # From: boto3==1.12
six==1.15.0                     # From: python-dateutil==2.8.1
urllib3==1.25.11                        # From: botocore==1.15.49

cat requirements-extra2.txt
-e .[extra2]
-r requirements.txt
boto3==1.12.0
botocore==1.15.49                       # From: boto3==1.17.45
jmespath==0.10.0                        # From: boto3==1.17.45
python-dateutil==2.8.1                  # From: botocore==1.20.45
s3transfer==0.3.6                       # From: boto3==1.17.45
six==1.15.0                     # From: python-dateutil==2.8.1
urllib3==1.25.11                        # From: botocore==1.20.45

cat setup.py 
# Generated by Medikit 0.8.0 on 2021-04-06.
# All changes will be overriden.
# Edit Projectfile and run “make update” (or “medikit update”) to regenerate.

from setuptools import setup, find_packages
from codecs import open
from os import path

here = path.abspath(path.dirname(__file__))

# Py3 compatibility hacks, borrowed from IPython.
try:
    execfile
except NameError:

    def execfile(fname, globs, locs=None):
        locs = locs or globs
        exec(compile(open(fname).read(), fname, "exec"), globs, locs)


# Get the long description from the README file
try:
    with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
        long_description = f.read()
except:
    long_description = ''

# Get the classifiers from the classifiers file
tolines = lambda c: list(filter(None, map(lambda s: s.strip(), c.split('\n'))))
try:
    with open(path.join(here, 'classifiers.txt'), encoding='utf-8') as f:
        classifiers = tolines(f.read())
except:
    classifiers = []

version_ns = {}
try:
    execfile(path.join(here, 'Test/_version.py'), version_ns)
except EnvironmentError:
    version = 'dev'
else:
    version = version_ns.get('__version__', 'dev')

setup(
    author='An author',
    author_email='[email protected]',
    description='This is just a test',
    license='Apache License, Version 2.0',
    name='Test',
    version=version,
    long_description=long_description,
    classifiers=classifiers,
    packages=find_packages(exclude=['ez_setup', 'example', 'test']),
    include_package_data=True,
    install_requires=['boto3 ~= 1.10'],
    extras_require={
        'dev': ['boto3 ~= 1.11'],
        'extra1': [
            'boto3 == 1.12',
            'httpie @ git+https://github.com/httpie/[email protected]'
        ],
        'extra2': ['boto3 ~= 1.11']
    },
    url='http://www.an-example.com/',
    download_url='http://www.an-example.com/'.format(version=version),
)

We hope you find these features interesting. Waiting for your comments.

[coveralls]

Pull Request Test Coverage Report for Build 529

• 35 of 66 (53.03%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.5%) to 47.116%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
medikit/feature/python.py	35	66	53.03%

Totals
Change from base Build 514:	0.5%
Covered Lines:	1015
Relevant Lines:	1936

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 523

• 12 of 68 (17.65%) changed or added relevant lines in 1 file are covered.
• 7 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-2.0%) to 44.619%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
medikit/feature/python.py	12	68	17.65%

Files with Coverage Reduction	New Missed Lines	%
medikit/feature/python.py	7	45.7%

Totals
Change from base Build 514:	-2.0%
Covered Lines:	976
Relevant Lines:	1938

---

💛 - Coveralls