Skip to content

experimental scancode integration

Jump to bottom
devonfw-core edited this page Jul 16, 2026 · 1 revision

Experimental Scancode Integration

Starting from version 1.4.0 Solicitor can be integrated with the tool ScanCode to include detailed information gathered from the "deep license scan" performed by ScanCode. This includes detected Licenses, Copyrights and Notice-Files.

Warning
 The current integration with ScanCode is experimental: The used ScanCode parameters, interfacing and curations logic and all parts of the data persistence are experimental and thus might result in insufficient quality of results. The current workflow and implementation is subject to change in future versions without further notice.

General workflow

The general workflow when integrating with ScanCode consists of the following 3 steps:

  1. Execute Solicitor in a "classic" way i.e. just based on the data provided via the Readers as described in [Reading License Information with Readers]. Besides the normal reports/documents generated this will also create scripts for downloading the needed OSS source codes and run Scancode.

  2. Download source codes and run ScanCode by executing the generated scripts. The downloaded sources and ScanCode results will be saved to a directory tree in the local filesystem.

  3. Execute Solicitor a second time. For all ApplicationComponents where ScanCode information is available (stored in the local directory tree) the license data as obtained from the Readers is replaced by this information. The data model is enriched with the found copyright and notice file information. Reports (see [Reporting and Creating output documents]) are now based on the ScanCode data (where available).

Prerequisites

Bash

The scripts generated by Solicitor to download sources and run ScanCode are in Bash syntax. So either run it on a system using natively Bash (linux) or install an appropriate environment (e.g. Git Bash) if you are using a windows environment.

ScanCode

Download and install ScanCode (Solicitor is assuming version 32, tested with 32.2.1) from https://github.com/nexB/scancode-toolkit/releases. Make sure that the executable is included in the search PATH for executables.

Activate feature

As the ScanCode integration is still experimental it is currently deactivated by default. To enable it set system property solicitor.feature-flag.scancode=true. (See [Built in Default Properties] for information how to do so.) If this feature flag is not activated then Solicitor will not try to attempt to read ScanCode information from the local file system.

Detailed workflow

Solicitor 1st run

Execute Solicitor in a classic way. As part of the report creation step this will generate two scripts:

  • output/scancode_PROJECTNAME.sh (for downloading the sources, also calls scancodeScan.sh)

  • output/scancodeScan.sh (for running ScanCode on the downloaded sources)

Scripts will include all ApplicationComponents with exception of those where normalizedLicenseType was set to COMMERCIAL.

Download Sources and run Scancode

Change to directory output and execute sh scancode_PROJECTNAME.sh. This will download all sources and process them via ScanCode. This might take several hours to complete. Results are stored in subdirectory Source of the directory output and is organized in a tree structure given by the PackageURL of the ApplicationComponents.

Origin file (package.yaml)

The Scancode integration scripts try to download ApplicationComponent sources from default URLs derived from the PackageUrl (e.g. Maven Central). In cases where the sources are not available at these locations, the download will fail (and the subsequent source scan will be skipped). In this case it is possible to manually download the sources from some other location and store it in the directory structure. Restarting the Scancode integration script might then perform the source scan.

To be able to document the (non default) origin of the ApplicationComponent sources a file package.yaml is created in the components directory in the file system. If the failed source download has been performed manually it is possible to edit this file and correct the data given in this file.

# This file contains metadata about the orgin of the package and the sources.
# This file was automatically created but might manually be edited if the contained data is not correct
sourceDownloadUrl: https://url/pointing/to/the/source/archive.jar  (1)
packageDownloadUrl: https://url/pointing/to/the/binary/archive.jar (2)
packageUrl: pkg:the/packageUrl/of/the/package (3)
# note: to add comments: write them here and remove the hash at the beginning of the line (not yet processed by Solicitor)

  1. URL for downloading the sources - will be available as property ApplicationComponent.sourceDownloadUrl in the Solicitor data model.

  2. URL for downloading the binaries - will be available as property ApplicationComponent.packageDownloadUrl in the Solicitor data model.

  3. PackageUrl of the package. This is for reference as the PackageUrl of the package is not stored anywhere else by default and might be difficult to reconstruct.

The content of the file package.yaml currently just affects the above given two properties, it does not affect the downloading of sources by the scripts.

Solicitor 2nd run

Execute Solicitor a second time. After reading the component/license information from the Readers (but before starting the rule engine) Solicitor will try to look up ScanCode information from the directory tree in output/Sources for all processed ApplicationComponents. If information is found for an ApplicationComponent the following is done:

  • License information (including URL of license text) as obtained from the Readers is replaced by the license info found by ScanCode

  • Copyrights are taken from ScanCode results

  • Info on NOTICE file is taken from the ScanCode results

  • If the ScanCode results contain information about project URLs this is stored as sourceRepoUrl and/or ossHomepage

  • sourceDownloadUrl and packageDownloadUrl are set to the values given in file package.yaml

Output

Main target of the additional information obtained from ScanCode is currently the new report Attributions_PROJECTNAME.html which lists

  • all ApplicationComponents (excluding those which are not OSS licensed)

  • with all found copyrights

  • and all licenses

  • including all different license texts

  • and contents of all found NOTICE files

dataStatus values of the Scancode integration

When using the Scancode integration the following values are used for field ApplicationComponent.dataStatus:

Value Description

ND:DISABLED

No data available. Scancode integration disabled. License info from reader was preserved.

ND:NOT_AVAILABLE

No data available. No scan results existing and no indication that attempting download/scanning has failed. License info from reader was preserved.

ND:PROCESSING_FAILED

No data available. No scan results existing. Processing (downloading or scanning) had failed. License info from reader was preserved.

NL:WITH_ISSUES

Data available but did not contain any license information. Issues were detected in the data which probably need to be curated. License info from reader was preserved.

NL:NO_ISSUES

Data available but did not contain any license information. No curations applied. No issues were detected (despite the fact that no license info was found). License info from reader was preserved.

NL:CURATED

Data available but did not contain any license information. Curations were applied. No issues were detected (despite the fact that no license info was found). License info from reader was preserved.

DA:WITH_ISSUES

Data available (including licenses). Issues were detected in the data which probably need to be curated.

DA:NO_ISSUES

Data available (including licenses). No curations applied. No issues were detected.

DA:CURATED

Data available (including licenses). Curations were applied. No issues were detected.

Automatic mapping of RawLicense data obtained from Scancode to NormalizedLicense

Within the normal workflow NormalizedLicense objects are created from RawLicense objects via the rules given in the different LicenseAssignment and LicenseNameMapping decision tables, see [Phase 1: Determining assigned Licenses]. The "raw" license data obtained from Scancode represents licenses either by SPDX-IDs or (if licenses are detected which do not have a corresponding SPDX-IDs) via LicenseRef-scancode-XXXXX qualifiers. This is an improved data quality as compared to RawLicenses obtained from normal Readers. (See [Reading License Information with Readers].) Solicitor makes use of this improved data quality and by default performs an automatic mapping of RawLicense data to NormalizedLicense s in this case:

  • If the raw license matches a SPDX-ID then a NormalizedLicense is created with normalizedLicenseType set to OSS-SPDX.

  • If the raw license starts with LicenseRef-scancode- then a NormalizedLicense is created with normalizedLicenseType set to SCANCODE.

  • If the raw license matches a given "ignorelist" (see below), then a NormalizedLicense is created with normalizedLicenseType set to IGNORE and normalizeLicense set to Ignore.

  • If the raw license does not match any of the above criteria or matches a "blacklist" (see below) then no automatic mapping is done.

Ignorelist and Blacklist

The ignorelist allows to automatically map licenses so that they are ignored in the further evaluation. The blacklist allows suppressing the automatic mapping of specific licenses. Both lists are configured via properties and are represented by a comma separated list of regular expressions.

The default is:

solicitor.scancode.automapping.blacklistpatterns=.*unknown.*,.*proprietary.*
solicitor.scancode.automapping.ignorelistpatterns=

This prohibits automatic mapping of licenses ids which are ambiguous. No ignore mapping is done by default.

Feature flag

The automatic mapping might be disabled by setting the corresponding feature flag to false:

solicitor.feature-flag.scancode.automapping=false

Correcting data

The data obtained from ScanCode might be affected by false positives (wrongly detected a license or copyright) or false negatives (missed to detect a license or copyright). To compensate such defects there are two mechanisms: Applying Curation information from a "curations" file or changing the license information via the decision table rules.

Curating data via a curations file

Structure of curations file

To define curations you might create a file output/curations.yaml containing the following structure:

artifacts:
  - name: pkg/npm/@somescope/somepackage/1.2.3                  (1)
    url: https://github.com/foo/bar                             (2)
    licenseCurations:                                           (3)
    - operation: REMOVE
      path: "sources/package/readme.md"
      ruleIdentifier: "proprietary-license_unknown_13.RULE"
      matchedText: ".* to be paid .*"
      comment: "just a generic remark, not a license"
    - operation: ADD
      newLicense: "Apache-2.0"
      comment: "License as given on website"
    copyrightCurations:                                         (4)
    - operation: REMOVE
      path: "sources/package/lib/test.js"
      oldCopyright: "(c) R.apv"
      comment: "some minified code fragment, not a copyright"
    excludedPaths:                                              (5)
    - "sources/src"                                             (6)
  - name: pkg/npm/@anotherscope/anotherpackage/4.5.6            (7)
.
.
.

  1. Path of the package information as used in the file tree. Derived from the PackageURL.

  2. URL of the project, will be stored as sourceRepoUrl. (Optional: no change if not existing.)

  3. Rules for curating license findings, see below.

  4. Rules for curation copyright findings, see below.

  5. Excluded paths to be set. Optional. If defined then all scanned files, whose path prefix contain any given string here, are excluded from the ScanCode information.

  6. A single path prefix. All scanned files starting with this path prefix are excluded from the Scancode information.

  7. Further packages to follow.

Rules for curating licenses

Curating licenses is done by REMOVING (i.e. ignoring) specific license findings from ScanCode, by REPLACING the detected license with another one or by ADDING license findings either to specific files or on top level (not related to specific file of the package sources). In addition to the conditions/data which is specific for any of the below described operations it is always possible to define a comment which is intended to be included in any audit trail log for documentation purposes (not yet used/implemented).

Licenses: REMOVE

Removing found licenses is done by defining rules which result in ignoring the license finding(s) of scancode rules in files within the scanned codebase. The following "conditions" are used for defining the rule

  • path of the file within the sources (defined as a regular expression; matches to files[].path in the scancode json file)

  • ruleIdentifier of the rule (defined as a regular expression; matches to files[].licenses[].matched_rule.identifier in the scancode json file)

  • matchedText of the finding (defined as a regular expression; matches to files[].licenses[].matched_text in the scancode json file)

  • oldLicense of the finding (defined as regular expression; matches to files[].licenses[].spdx_license_key

The first three conditions can uniquely identify any license finding listed in the scancode json file. The oldLicense condition can be used to select findings to be ignored based on the found license instead of the ruleIdentifier. All conditions are optional but at least one needs to be defined. By using RegEx syntax the curations can be written very flexible. By using solely oldLicense as a condition it is e.g. possible to remove all findings of a specific license.

Licenses: REPLACE

Instead of removing licenses (ignoring the finding) they might be replaced with a different license key and/or URL pointing to the license text. The conditions are the same as for REMOVE, the replacement is defined as follows

Data:

  • newLicense is the key / id of the license to use instead (replacing files[].licenses[].spdx_license_key)

  • url is the url pointing to the license text

At least one of the two parameters has to be set.

Licenses: ADD

Adding new licenses is done by defining rules which add new license info (to the licenses found in a source file) - or "on top level".

Conditions:

  • path of the file within the sources to which the license should be added (defined as a regular expression; matches to files[].path in the scancode json file). Note that this will only work if there are files[].path in the scancode json for which this conditions matches.

It is not possible to associate licenses to files which are not listed in scancode json. The path condition might be omitted which results in the given license to be added to the result without any relation to a specific path.

Data:

  • newLicense: the key/SPDX-ID of the license to add

  • url: URL to the license text

Rules for curating copyrights

Curating copyrights is based on the same principles as curation of licenses, providing REMOVE, REPLACE and ADD operations.

Copyrights: REMOVE

REMOVING found copyrights is done by defining rules which result in ignoring the copyright finding(s) in files within the scanned codebase. The following "conditions" are used for defining the rule

  • path of the file within the sources (defined as a regular expression; matches to files[].path in the scancode json file)

  • oldCopyright the found copyright text to ignore (defined as a regular expression; matches to files[].copyrights[].copyright in the scancode json file)

At least one of the conditions has to be defined.

Copyright: REPLACE

This follows the above principles. It uses the same conditions as REMOVE and uses a parameter to define the copyright to use instead:

Data: * newCopyright: The copyright entry to use instead of the originally found copyright

Copyright: ADD

Adding new copyrights is done by defining rules which add new copyright info (to the copyrights found in a source file) - or "on top level".

Conditions:

  • path of the file within the sources (defined as a regular expression; if omitted the copyright will be applied on "top level"). Note that it is again only possible to add copyrigts to paths which are listed in the scancode json

Data:

  • newCopyright: the copyright string to add

Redefining all licenses / copyrights of a component

Instead of curating license / copyrights on a "per finding" level as given above it is alternatively possible to completely replace the list of found licenses and/or copyrights with a new list.

Important
 Up to version 1.23.0 this was the only way of doing license / copyright curations. Use of this way of curating data is still possible but discouraged and might be deprecated/removed soon.

The file output/curations.yaml looks as follows when doing curations this way:

artifacts:
  - name: pkg/npm/@somescope/somepackage/1.2.3                  (1)
    url: https://github.com/foo/bar                             (2)
    licenses:                                                   (3)
      - license: MIT                                            (4)
        url: https://raw.githubusercontent.com/foo/bar/LICENSE  (5)
    copyrights:                                                 (6)
      - (c) 2021 Donald Duck                                    (7)
      - "(c) 2019 Mickey Mouse <http://mickey.mouse>"           (8)
    excludedPaths:                                              (9)
    - "sources/src"                                             (10)
  - name: pkg/npm/@anotherscope/anotherpackage/4.5.6            (11)
.
.
.

  1. Path of the package information as used in the file tree. Derived from the PackageURL.

  2. URL of the project, will be stored as sourceRepoUrl. (Optional: no change if not existing.)

  3. Licenses to set. Optional. If defined then all found licenses will be replaced by the list of licenses given here.

  4. SPDX identifier of license.

  5. URL pointing to license text.

  6. Copyrights to set. Optional. If defined then all found copyrights will be replaced by the list of copyrights given here.

  7. A single copyright.

  8. Another copyright. Note that due to YAML syntax any string containing : needs to be enclosed with parentheses

  9. Excluded paths to be set. Optional. If defined then all scanned files, whose path prefix contain any given string here, are excluded from the ScanCode information.

  10. A single path prefix. All scanned files starting with this path prefix are excluded from the Scancode information.

  11. Further packages to follow.

Hierarchical definition of rules

Different version of a package/component or even different packages/components within the same namespace often require mostly the same curations to be applied. To avoid being forced to redefine curations for every single version it is possible to define curations by just specifying a prefix part in the name attribute.

Example of available levels/prefixes for pkg:/maven/ch.qos.logback/logback-classic@1.2.3

  • pkg

  • pkg/maven

  • pkg/maven/ch

  • pkg/maven/ch/qos

  • pkg/maven/ch/qos/logback

  • pkg/maven/ch/qos/logback/logback-classic

  • pkg/maven/ch/qos/logback/logback-classic/1.2.3

The complete tree will be checked for curations. Any found curations will be merged

  • Attribute name: latest encountered in the hierarchy will be taken

  • Attribute note will be joined using delimiter " / "

  • Attribute url: latest encountered in the hierarchy will be taken

  • Attribute copyrights (old style of curations): Lists will be merged

  • Attribute licenses (old style of curations): Lists will be merged

  • Attribute excludedPaths: Lists will be merged

  • Attribute licenseCurations: License curation rule lists (REMOVE/REPLACE/ADD) will be merged; order is more specific ones first; when evaluating for a specific license finding in the scancode json only the first matching curation rule will be taken.

  • Attribute copyrightCurations: Copyright curation rule lists (REMOVE/REPLACE/ADD) will be merged; order is more specific ones first; when evaluating for a specific copyright finding in the scancode json only the first matching curation rule will be taken.

The resulting curation will then be applied to the scancode data of the component.

Decision table rules

As for license information obtained from the Readers the license information from ScanCode can also be altered using decision table rules. A new attribute origin was introduced in the RawLicense entity as well as condition field in decision table LicenseAssignmentV2*.xlsx/xls/csv. The origin attribute in Rawlicense either contains the string scancode if the license information came from ScanCode or it contains the (lowercase) class name of the used Reader.

Using the [Extended comparison syntax] it is possible to qualify whether a rule should apply for licenses found by ScanCode or not:

Value of condition Origin rule applies for …​

scancode

…​ licenses obtained from ScanCode information

NOT:scancode

…​ licenses obtained from normal Readers

(empty)

…​ in both cases

Due the automatic mapping of scancode based RawLicenses to NormalizedLicenses (see Automatic mapping of RawLicense data obtained from Scancode to NormalizedLicense) such explicit mapping rules are only required for licenses not handled by the automatism.

Clone this wiki locally