False negatives for OSPS-LE-02 and OSPS-LE-03

[epipav]

Description

The OSPS-LE-02 (assessment: OSPS-LE-02.01) and OSPS-LE-03 (assessment: OSPS-LE-03.01) evaluations fail to recognize license files when they are located inside a LICENSE/ directory.

According to the returned assessment description, this location is explicitly considered valid:

"While active, the license for the source code MUST be maintained in the corresponding repository's LICENSE file, COPYING file, or LICENSE/ directory."

However, repositories that follow this structure are currently flagged as failing the evaluation.

Example

A concrete example is the matplotlib repository, which stores its license in a LICENSE/ folder. Despite being a documented and valid location, the evaluations report this as an issue.

Expected Behavior

License files should be properly detected when stored in a LICENSE/ directory.
Repositories following this convention should not be incorrectly flagged as failing OSPS-LE-02 and OSPS-LE-03.

[eddie-knight]

Hey, thanks for catching this! I'm not sure what the fix will be yet, but likely we'll need to put some conditional inspection that crawls the license directory if found.

[trumant]

I did some triage and found the following. When I run the main branch against matplotlib/matplotlib I see:

[ERROR] OSPS-LE-02.01: License was not found in a well known location via the GitHub API
[ERROR] OSPS-LE-03.01: License was not found in a well known location via the GitHub API

OSPS-LE-02.01

https://github.com/matplotlib/matplotlib/blob/main/LICENSE/LICENSE looks like a project-specific license that we would not recognize today as an OSI or FSF approved/recognized even if we do examine the contents of LICENSE/LICENSE. GitHub is also unable to identify the project's license.

Our current implementation relies on our ability to determine a SPDX expression or SPDX identifier from the data sources available to the tool.

OSPS-LE-03.01

We do not examine the repository contents at all in our license determination today. Rather we rely on what GitHub tells us in its Repository API response:

curl -sL \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/repos/matplotlib/matplotlib | jq .license

which returns nothing, because this field is nil in the response.

For Discussion

• Should we enhance our license discovery logic to deeply examine LICENSE, COPYING and LICENSE/ contents in the hopes of being able to determine a SPDX identifier with which we can evaluate whether or not it meets the OSI/FSF requirement? If we do, we will at least stand a chance of correctly reporting the status of OSPS-LE-03.01, but we will still report OSPS-LE-02.01 as a failure
• If we do not examine the repository contents more deeply, should we return Needs review instead of Error in this specific case where GitHub's API tells us that it has no idea what license corresponds to the source code repository