Development

Development

See CONTRIBUTING.rst for details: https://github.com/nexB/scancode-toolkit/blob/master/CONTRIBUTING.rst

Code layout and conventions

Source code is in src/. Tests are in tests/.

There is one Python package for each major feature under src/ and a corresponding directory with the same name under tests (but this is not a package by design).

Each test script is named test_XXXX and while we love to use py.test as a test runner, these tests have no dependencies on py.test, only on the unittest module.

When source or tests need data files, we store these in a data subdirectory.

We use PEP8 conventions with a relaxed line length that can be up to 90'ish characters long when needed to keep the code clear and readable.

We store pre-built vendored native binaries in bin/ sub-directories of each src/ packages. These binaries are organized by OS and architecture. This ensure that ScanCode works out of the box either using a checkout or a download, without needing a compiler and toolchain to be installed. The corresponding source code for the pre-built binaries are store in a separate repository at https://github.com/nexB/scancode-thirdparty-src

We store vendored thirdparty components and libraries in the thirdparty directory. Python libraries are stored as wheels, eventually pre-built if the corresponding wheel is not available in the Pypi repository. Some of these components may be advanced builds with bug fixes or advanced patches.

We write tests, a lot of tests, thousands of tests. Several tests are data-driven and use data files as test input and sometimes data files as test expectation (in this case using either JSON or YAML files). The tests should pass on Linux 64 bits, Windows 32 and 64 bits and on MacOSX 10.6.8 and up. We maintain two CI loops with Travis (Linux) at https://travis-ci.org/nexB/scancode-toolkit and Appveyor (Windows) at https://ci.appveyor.com/project/nexB/scancode-toolkit

When finding bugs or adding new features, we add tests. See existing test code for examples.

###pip requirements and the configure script

ScanCode use the configure/configure.bat and etc/configure.py scripts to run install a virtualenv, install pip requirements and more in a self-contained way that require no network connectivity.

Earlier unreleased versions of ScanCode where using buildout to install and configure eventually complex dependencies. We had some improvements that were merged in the upstream buildout to support bootstrapping and installing without a network connection and When we migrated to use pip and wheels as new, improved and faster way to install and configure dependencies we missed some of the features of buildout like the recipes, being able to invoke arbitrary Python or shell scripts after installing packages and have scripts or requirements that are operating system-specific.

The etc/configure.py scriptdoes all this (even though not all of its capabilities are used here yet).

###ScanCode requirements and third-party Python libraries### In a somewhat unconventional way, all the required libraries are vendored aka. copied in the repo itself in the thirdparty/ directory. If ScanCode were only a library it would not make sense. But its is first an application and having a well defined frozen set of dependencies is important for apps. The benefit of this approach (combined with the configure script) means that a mere checkout of the repository contains everything needed to run ScanCode except for a Python interpreter.

###Using ScanCode as a Python library

ScanCode can be used alright as a Python library .. BUT even though scancode-toolkit can be built as a Python wheel, all its dependencies are not yet available as in Pypi. You will have to wait for this (or help with the packaging of missing Pypi dependencies).

These missing Pypi dependencies either where never published on Pypi by their respective upstream authors or have some advanced patches and bug fixes not yet available as a release on Pypi).

If you are interested to use ScanCode as a library, please say so in a comment on https://github.com/nexB/scancode-toolkit/issues/24

###To cut a release:

• run bumpversion with major, ``minororpatch` to bump the version in:

• src/scancode/__init__.py

• setup.py

• Update the CHANGELOG.rst

• commit changes and push changes to develop:

• git commit -m "commit message"

• git push --set-upstream origin develop

• merge develop branch in master and tag the release.

• git checkout master

• git merge develop

• git tag -a v1.6.1 -m "version 1.6.1"

• git push --set-upstream origin master

• git push --set-upstream origin v1.6.1

• draft a new release in GitHub, using the previous release blurb as a base. Highlight new and noteworthy changes from the CHANGELOG.rst.

• run etc/release/release.sh locally.

• upload the release archives created in the dist/ directory to the GitHub release page.

• save the release as a draft. Use the previous release notes to create notes in the same style. Ensure that the link to thirdparty source code is present.

• test the downloads.

• publish the release.