✨ Add multi-repo scanning: --repos, --org, and optional --combined output #4793

gabrielsoltz · 2025-09-16T20:51:07Z

What kind of change does this PR introduce?

This PR introduces support for scanning multiple repositories in a single invocation, while preserving existing behavior by default. The only user-visible change for single-repo usage is a small improvement to the “Starting/Finished” banners (they now include the repo label).

New flags

--repos: comma-separated list of repositories to scan (e.g., --repos=owner1/repo1,github.com/owner2/repo2).
--org: GitHub organization handle (e.g., --org=github.com/ossf or ossf).
--combined: after scanning N repos, print a single combined table with all checks together with two new extra columns, REPO and AGGREGATED SCORE.

Precedence when multiple inputs are provided: --repos ➜ --org ➜ --local ➜ --repo (or repo resolved from package managers).

UX / output changes

Before (single repo):

Starting [License]
Starting [Code-Review]
...
Finished [License]
Finished [Code-Review]
...

Now (single or multi-repo):

Starting (owner/repo) [License]
Starting (owner/repo) [Code-Review]
...
Finished (owner/repo) [License]
Finished (owner/repo) [Code-Review]
...

This makes it obvious which repo each check belongs to—especially important when scanning many repos.
No other output changes occur unless --combined is used.

When --combined is set, a final COMBINED RESULTS section is emitted after all per-repo results, e.g.:

COMBINED RESULTS
----------------
| REPO                                             | AGGREGATED SCORE | SCORE   | NAME                 | REASON                      | DOCUMENTATION/REMEDIATION                                |
|--------------------------------------------------|------------------|---------|----------------------|-----------------------------|----------------------------------------------------------|
| github.com/ossf-tests/scorecard-check-...        | 3.3 / 10         | 10 / 10 | Binary-Artifacts     | no binaries found in repo   | https://github.com/ossf/scorecard/blob/main/docs/...     |
...

Implementation details

Introduce buildRepoURLs(ctx, *options.Options) ([]string, error):
Normalizes the input universe and always returns a slice of repo URIs.
Honors precedence --repos ➜ --org ➜ --local ➜ single --repo/pkg-manager.
Refactor rootCmd to iterate over the returned list and run the same scan pipeline for each repo.
ListOrgRepos for listing repositories from the organization, reusing the githubrepo code and logic.

Examples

Scan a list

scorecard --repos=ossf/scorecard,ossf-tests/scorecard-check-branch-protection-e2e

Scan all non-archived repos in an org

scorecard --org=github.com/ossf

Aggregate across many repos

scorecard --repos=ossf/scorecard,ossf-tests/scorecard-check-branch-protection-e2e --combined

PR title follows the guidelines defined in our pull request documentation
Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

Fixes #4792

Special notes for your reviewer

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

codecov · 2025-09-17T05:35:24Z

Codecov Report

❌ Patch coverage is 31.53153% with 228 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.48%. Comparing base (353ed60) to head (feb0b0e).
⚠️ Report is 259 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4793      +/-   ##
==========================================
+ Coverage   66.80%   67.48%   +0.67%     
==========================================
  Files         230      250      +20     
  Lines       16602    19395    +2793     
==========================================
+ Hits        11091    13088    +1997     
- Misses       4808     5428     +620     
- Partials      703      879     +176

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

spencerschrock

Did a partial review of a few things before I ran out of time, I need to dive more into how you build the repo list, scan repos, and present the results. Feel free to address that feedback now or wait until I have more time for full review

clients/githubrepo/roundtripper/roundtripper.go

go.mod

clients/githubrepo/org.go

cmd/internal/org/org.go

cmd/root.go

spencerschrock · 2025-09-18T15:48:29Z

Also, this repo requires DCO.

scorecard/CONTRIBUTING.md

Lines 10 to 14 in 2864c76

    
           > Additionally the Linux Foundation (LF) requires all contributions include per-commit sign-offs. 
        
           > Ensure you use the `-s` or `--signoff` flag for every commit. 
        
           > 
        
           > For more details, see the [LF DCO wiki](https://wiki.linuxfoundation.org/dco) 
        
           > or [this Pi-hole signoff guide](https://docs.pi-hole.net/guides/github/how-to-signoff/).

For instructions on how to fix it:
https://github.com/ossf/scorecard/pull/4793/checks?check_run_id=50569515731

spencerschrock · 2025-09-29T22:12:18Z

Just wanted to update you that this is still on my radar! We are trying to cut a 5.3.0 release this week, so my attention has been on that. But once that's cut this week, I will make time to finish. my review, and happy to cut a 5.4.0 or 5.3.1 shortly after

spencerschrock

I like this approach overall, I think building a slice of repos and looping over them is the correct approach. Most of my issue is with the --combined output format, is there anyway we can split that into its own PR? I think that would help get --repos and --org merged first.

cmd/internal/org/org.go

spencerschrock · 2025-10-03T21:02:28Z

options/flags.go

+
+	FlagCombinedOutput = "combined"


is this intended to be only change the behavior if --format=default? Currently when I run something like this, I get the "combined" output instead of "json"

go run main.go --org=ossf-tests --format json --combined

Or should this be it's own format value? e.g.

go run main.go --org=ossf-tests --format combined

This was not intended, but after reviewing, I definitely prefer your approach to treat combined as a new format option.

This is now changed, now we use --format combined... I intentionally left the start/finish banner for the format default and combined. Maybe these banners should be optional, using another flag (--show-progress or something like that).

options/flags.go

pkg/scorecard/scorecard_result.go

cmd/root.go

spencerschrock · 2025-10-03T21:23:41Z

cmd/root.go

+		if err != nil {
+			// processRepo already logged details; skip this URI.
+			fmt.Fprintf(os.Stderr, "Skipping %s: %v\n", uri, err)
+			continue
+		}


this changes the behavior of the scorecard binary. Previously if any check had a runtime error, rootCmd would return an error, which would lead to a non-zero exit code. But now these runtime errors don't lead to an exit code.

I think continuing to scan all repositories is good, but we should "remember" if we saw an error along the way so we can return it at the end

Makes sense. I added a small implementation for this using sawRuntimeErr. We can improve it, but I think it already aligns with your point.

Thanks, I may change the implementation in a follow-up as it would just be more effective for me to handle my suggested changes.

spencerschrock · 2025-10-03T21:35:12Z

cmd/root.go

+	label := repoLabelFromURI(uri)
+
+	// Start banners with repo label (always show banners even in combined-only mode)
+	if o.Format == options.FormatDefault {
+		if len(enabledProbes) > 0 {
+			printProbeStart(label, enabledProbes)
+		} else {
+			printCheckStart(label, enabledChecks)
+		}


let's get rid of repoLabelFromURI and just use uri instead of label

Otherwise it would be confusing when you analyze repos across forges:

--repos=github.com/some/repo,gitlab.com/other/repo

Which one is github? Which one is gitlab? What if they both have the exact same repo name?

Starting (some/repo) [Maintained] ... Starting (other/repo) [Maintained]

Good point. And much easier. Changed.

The only cosmetic thing is that running with --org, we will get as uri the full URL, like https://github.com/ossf-tests/scorecard-check-pinned-dependencies-e2e...

cmd/root.go

github-actions · 2025-10-14T02:03:57Z

This pull request has been marked stale because it has been open for 10 days with no activity

gabrielsoltz · 2025-10-16T19:58:06Z

Hey @spencerschrock, thank you so much for reviewing my PR — and sorry it took me a while to address your comments; I couldn’t get to it earlier. I left two small cosmetic notes (one about the banners and one about the URI) for us to discuss. Whenever you have a moment, this is ready for another review. ✌️

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Signed-off-by: Jessica Wagantall <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Signed-off-by: Stephen Augustus <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Fix typo in message about pinning dependencies. Signed-off-by: Martin Costello <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Remove redundant code from when `pull_request_target` was used that was removed by ossf#2672. Signed-off-by: Martin Costello <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Signed-off-by: Mike Dolan <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

…4814) Bumps the distroless group with 1 update in the /clients/githubrepo/roundtripper/tokens/server directory: distroless/base. Bumps the distroless group with 1 update in the /cron/internal/bq directory: distroless/base. Bumps the distroless group with 1 update in the /cron/internal/cii directory: distroless/base. Bumps the distroless group with 1 update in the /cron/internal/controller directory: distroless/base. Bumps the distroless group with 1 update in the /cron/internal/webhook directory: distroless/base. Bumps the distroless group with 1 update in the /cron/internal/worker directory: distroless/base. Updates `distroless/base` from `06c713c` to `10136f3` Updates `distroless/base` from `06c713c` to `10136f3` Updates `distroless/base` from `06c713c` to `10136f3` Updates `distroless/base` from `06c713c` to `10136f3` Updates `distroless/base` from `06c713c` to `10136f3` Updates `distroless/base` from `06c713c` to `10136f3` --- updated-dependencies: - dependency-name: distroless/base dependency-version: nonroot dependency-type: direct:production dependency-group: distroless - dependency-name: distroless/base dependency-version: nonroot dependency-type: direct:production dependency-group: distroless - dependency-name: distroless/base dependency-version: nonroot dependency-type: direct:production dependency-group: distroless - dependency-name: distroless/base dependency-version: nonroot dependency-type: direct:production dependency-group: distroless - dependency-name: distroless/base dependency-version: nonroot dependency-type: direct:production dependency-group: distroless - dependency-name: distroless/base dependency-version: nonroot dependency-type: direct:production dependency-group: distroless ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

…4819) * repair GitHub project list with excess path components Approximately 300 repos in the list had 1-3 extra path components, which caused 32 shards to continuously have runtime errors: Could not resolve to a Repository with the name '/' They were fixed by dropping all excess path components using a VSCode find-and-replace regex, then `make add-projects` to remove duplicates: find: github\.com/([^/]+)/([^/]+)/[^/]+ replace: github.com/$1/$2, Signed-off-by: Spencer Schrock <[email protected]> * remove 404 and add renamed repos This was done with a quick one-off script to hit the "get a repository" endpoint to test the new repos: https://api.github.com/repos/OWNER/REPO Signed-off-by: Spencer Schrock <[email protected]> --------- Signed-off-by: Spencer Schrock <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

spencerschrock · 2025-10-21T21:46:54Z

Whenever you have a moment, this is ready for another review

I will get to this tomorrow, thanks for your patience.

spencerschrock · 2025-10-22T21:29:46Z

cmd/root.go

+	// End banners BEFORE RESULTS (always show banners even in combined-only mode)
+	if o.Format == options.FormatDefault {
+		if len(enabledProbes) > 0 {
+			printProbeResults(uri, enabledProbes)
+		} else {
+			printCheckResults(uri, enabledChecks)


as written, this only prints the results for the default format, but the comment makes me think it should also apply to the combined format:

if o.Format == options.FormatDefault || o.Format == options.FormatCombined {

spencerschrock · 2025-10-22T21:52:11Z

cmd/root.go

+		if err != nil {
+			// processRepo already logged details; skip this URI.
+			fmt.Fprintf(os.Stderr, "Skipping %s: %v\n", uri, err)
+			continue
+		}


Thanks, I may change the implementation in a follow-up as it would just be more effective for me to handle my suggested changes.

spencerschrock · 2025-10-22T21:54:42Z

options/options.go

 	CommitDepth     int
 	ShowDetails     bool
 	ShowAnnotations bool
+	CombinedOutput  bool


with the switch to using the existing format flag, this CombinedOutput field is unused

spencerschrock · 2025-10-22T21:55:47Z

options/options.go

+	// FormatCombined specifies that results should be output as a single combined table for multiple repos.
+	FormatCombined = "combined"


Can we imply this is just for the table output since the other formats naturally combine their output?

FormatCombinedTable = "combined-table"

spencerschrock · 2025-10-22T22:03:05Z

pkg/scorecard/scorecard_result.go

+// FormatCombinedResults prints a combined table with extra REPO and AGGREGATED SCORE columns.
+// This expects `results` to contain checks for a single repo; callers scanning
+// multiple repos should aggregate calls or invoke this helper appropriately.
+func FormatCombinedResults(
+	writer io.Writer,
+	results *Result,
+	checkDocs docChecks.Doc,
+	opt *AsStringResultOption,
+	opts *options.Options,
+) error {


one of the reasons I suggested splitting this into a separate PR was to get the other code merged, since I knew I would have trickier feedback on this format that would take another cycle or two.

All the other format options are methods on results. Is there a reason you didn't go with that approach instead of FormatCombinedResults:

func (r *Result) AsCombinedTable(writer io.Writer, checkDocs docChecks.Doc, opt *AsStringResultOption) error

There's a lot of code overlap with AsString, the main difference is the two additional columns. Instead of duplicating the logic, can you make them use a shared helper function with a flag or something?

e.g.

func (r *Result) AsString(writer io.Writer, checkDocs docChecks.Doc, opt *AsStringResultOption) error { combined := false return common(writer, checkDocs, opt, combined) } func (r *Result) AsCombined(writer io.Writer, checkDocs docChecks.Doc, opt *AsStringResultOption) error { combined := true return common(writer, checkDocs, opt, combined) } and then in `common` you can look at that bool to see if you should add the column or not

github-project-automation bot added this to OpenSSF Scorecard Sep 16, 2025

gabrielsoltz changed the title ~~✨ Implement --org and --repos for scanning organizations or multiple repositories~~ ✨ Add multi-repo scanning: --repos, --org, and optional --combined output Sep 16, 2025

gabrielsoltz marked this pull request as ready for review September 16, 2025 21:20

gabrielsoltz requested a review from a team as a code owner September 16, 2025 21:20

gabrielsoltz requested review from raghavkaul and spencerschrock and removed request for a team September 16, 2025 21:20

gabrielsoltz temporarily deployed to gitlab September 17, 2025 05:30 — with GitHub Actions Inactive

gabrielsoltz temporarily deployed to integration-test September 17, 2025 05:30 — with GitHub Actions Inactive

gabrielsoltz temporarily deployed to gitlab September 18, 2025 04:43 — with GitHub Actions Inactive

gabrielsoltz temporarily deployed to integration-test September 18, 2025 04:43 — with GitHub Actions Inactive

spencerschrock reviewed Sep 18, 2025

View reviewed changes

gabrielsoltz force-pushed the feat-support-org-and-repos branch from 3ab90d2 to 75c4984 Compare September 19, 2025 08:57

gabrielsoltz temporarily deployed to gitlab September 20, 2025 03:08 — with GitHub Actions Inactive

gabrielsoltz temporarily deployed to integration-test September 20, 2025 03:09 — with GitHub Actions Inactive

justaugustus mentioned this pull request Sep 20, 2025

✨ multi-scorecard experiment #4502

Closed

2 tasks

justaugustus moved this to In Progress in OpenSSF Scorecard Sep 20, 2025

justaugustus moved this from In Progress to Review in progress in OpenSSF Scorecard Sep 20, 2025

spencerschrock reviewed Oct 3, 2025

View reviewed changes

github-actions bot added the Stale label Oct 14, 2025

dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Oct 16, 2025

gabrielsoltz added 4 commits October 16, 2025 22:06

First Iteration

38009b2

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

switch-to-client

373f1cb

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

implement-outputs

5142e77

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

minor-improvements

385e947

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

jwagantall and others added 20 commits October 16, 2025 22:06

chore: Add Hiero's hiero-cli (ossf#4806)

46ddbbd

Signed-off-by: Jessica Wagantall <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

MAINTAINERS: Add Adam Korczynski (AdamKorcz), ADA Logics (ossf#4808)

aca199d

Signed-off-by: Stephen Augustus <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

fix: fix dependencies typo (ossf#4809)

9783816

Fix typo in message about pinning dependencies. Signed-off-by: Martin Costello <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

🌱 Bump the gomod group across 2 directories with 8 updates (ossf#4815)

590d77e

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

chore: remove redundant code (ossf#4810)

a56c1c7

Remove redundant code from when `pull_request_target` was used that was removed by ossf#2672. Signed-off-by: Martin Costello <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Added key Qwen, Meta-Llama, and OSS GPT repositories (ossf#4811)

52193f8

Signed-off-by: Mike Dolan <[email protected]> Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

🌱 Bump the golang group across 8 directories with 1 update (ossf#4818)

4cc0984

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

🌱 Bump the github-actions group with 3 updates (ossf#4812)

b864b9b

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Delete unused scheme case

cdc4d61

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

migrate tablewriter dependency to v1 new API

776379c

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

better-flags-description

e51c83d

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

better-flags-description

b60fd9d

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

better-comment

b601277

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Rely on ListOrgRepos

5318471

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Add sawRuntimeErr for returning it

93f0f03

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Delete repoLabelFromURI and rely on uri

0df2c8d

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Delete unused checksList

c1448d6

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

Delete combined flag, instead use --format combined

07e8adc

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

gabrielsoltz force-pushed the feat-support-org-and-repos branch from 91600a2 to 07e8adc Compare October 16, 2025 20:06

fix conflict

538b3ee

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

justaugustus removed the Stale label Oct 16, 2025

gabrielsoltz temporarily deployed to gitlab October 16, 2025 20:13 — with GitHub Actions Inactive

gabrielsoltz temporarily deployed to integration-test October 16, 2025 20:13 — with GitHub Actions Inactive

fix linter errors

feb0b0e

Signed-off-by: Gabriel Alejandro Soltz <[email protected]>

gabrielsoltz deployed to integration-test October 16, 2025 21:04 — with GitHub Actions Active

gabrielsoltz temporarily deployed to gitlab October 16, 2025 21:04 — with GitHub Actions Inactive

spencerschrock reviewed Oct 22, 2025

View reviewed changes

		// FormatCombined specifies that results should be output as a single combined table for multiple repos.
		FormatCombined = "combined"

✨ Add multi-repo scanning: --repos, --org, and optional --combined output #4793

Are you sure you want to change the base?

✨ Add multi-repo scanning: --repos, --org, and optional --combined output #4793

Uh oh!

Conversation

gabrielsoltz commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What kind of change does this PR introduce?

New flags

UX / output changes

Before (single repo):

Now (single or multi-repo):

Implementation details

Examples

Scan a list

Scan all non-archived repos in an org

Aggregate across many repos

Which issue(s) this PR fixes

Special notes for your reviewer

Does this PR introduce a user-facing change?

Uh oh!

codecov bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

spencerschrock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spencerschrock commented Sep 18, 2025

Uh oh!

spencerschrock commented Sep 29, 2025

Uh oh!

spencerschrock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Oct 14, 2025

Uh oh!

gabrielsoltz commented Oct 16, 2025

Uh oh!

spencerschrock commented Oct 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spencerschrock Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

gabrielsoltz commented Sep 16, 2025 •

edited

Loading

codecov bot commented Sep 17, 2025 •

edited

Loading

spencerschrock Oct 22, 2025 •

edited

Loading