Skip to content

feat: support multiple control catalog versions#269

Open
vinayada1 wants to merge 1 commit intoossf:mainfrom
vinayada1:feat/multi-catalog-support
Open

feat: support multiple control catalog versions#269
vinayada1 wants to merge 1 commit intoossf:mainfrom
vinayada1:feat/multi-catalog-support

Conversation

@vinayada1
Copy link
Contributor

Summary

Implements the proposal from #257 to handle multiple control catalogs and versions of each.

Changes

  • Rename OSPS evaluation plan variable to OSPS_2025_10 with version-specific catalog ID (osps-baseline-2025-10)
  • Add OSPS_2026_02 evaluation plan (initially mirrors 2025_10, to be updated as the new baseline evolves)
  • Register both evaluation suites in main.go
  • Add OSPS_Baseline_2026_02.yaml catalog with version-specific metadata (id: osps-baseline-2026-02, version: 2026.02)
  • Update CI scripts and example config to default to the newest catalog version

Design Decisions

  1. Each catalog version gets its own Go variable (OSPS_2025_10, OSPS_2026_02) and is registered as a separate evaluation suite with a versioned ID
  2. The new 2026_02 plan starts as a copy of 2025_10 - new or changed assessment steps can be added independently without breaking the previous version
  3. Reusable step functions are shared across both plans; version-specific step implementations can be added to individual plans as needed
  4. Catalog YAML id fields now include the version suffix to match their evaluation suite registration names

Closes #257

@vinayada1 vinayada1 requested a review from a team as a code owner March 18, 2026 18:39
@vinayada1 vinayada1 force-pushed the feat/multi-catalog-support branch from c9bc59a to 4701f2f Compare March 18, 2026 18:46
@jmeridth
Copy link
Member

@vinayada1 This is awesome. My only thought is about the data structure for the different versions:

TLDR I like option B

Current PR approach: Single shared map, registered twice

// One map with ALL step implementations
OSPS = map[string][]gemara.AssessmentStep{...}

// Registered for each version, catalog YAML filters active IDs
for _, catalogID := range []string{"osps-baseline-2025-10", "osps-baseline-2026-02"} {
    orchestrator.AddEvaluationSuite(catalogID, nil, evaluation_plans.OSPS)
}

This is clean right now because 2026-02 is a copy of 2025-10. The catalog YAML does the filtering. But the moment a step implementation needs to differ between versions (e.g., OSPS-AC-01.01 gets a stricter check in 2026-02), this falls apart — we can't have two different implementations for the same key in one map.

Option A: Map of maps (version as key)

var OSPS = map[string]map[string][]gemara.AssessmentStep{
    "osps-baseline-2025-10": { ... },
    "osps-baseline-2026-02": { ... },
}

Pros: Version is a first-class concept. Supports divergent implementations per version. Registration in main.go becomes a clean range loop.

Cons: If most steps are shared, we duplicating the entire map per version. That's a lot of repetition in this file — the current map is already large. Maintenance burden increases: adding a shared step means editing N places.

Option B: Struct with version attribute

type EvaluationPlan struct {
    CatalogID string
    Steps     map[string][]gemara.AssessmentStep
}

var Plans = []EvaluationPlan{
    {CatalogID: "osps-baseline-2025-10", Steps: baseSteps},
    {CatalogID: "osps-baseline-2026-02", Steps: mergedSteps(baseSteps, overrides2026)},
}

Pros: Clean encapsulation. Could support a composition pattern (base + overrides) that avoids full duplication while still allowing divergence. The catalog ID lives next to its steps.

Cons: More upfront structure. The SDK's AddEvaluationSuite takes map[string][]gemara.AssessmentStep directly, so the struct is internal bookkeeping only — it doesn't flow into the SDK API.

My take

The PR's current approach works fine if the design intent is that step implementations are always shared and only the catalog YAML controls which IDs are active per version. That's a reasonable constraint and the simplest thing that works.

However, if we anticipate step implementations diverging between versions (which seems likely as the baseline evolves), I'd lean toward Option B with a composition pattern — keep a shared base map, then per-version overrides that get merged. It avoids the duplication problem of Option A while making version a real concept rather than just a loop of catalog ID strings.

The key question is: does the SDK guarantee that passing extra step implementations (keys not in the catalog YAML) is harmless? The PR comments say yes ("the SDK only runs the relevant subset"). If that's reliable, the current shared-map approach is defensible and we can defer restructuring until implementations actually diverge. If there's any risk of the SDK complaining about unrecognized assessment IDs, we'd want the per-version separation now.

@jmeridth
Copy link
Member

@vinayada1 please sign your commits with git commit -s 😄. That will solve the DCO check.

Implements the proposal from ossf#257 to handle multiple control catalogs
and versions of each.

Changes:
- Auto-discover catalogs from embedded YAML files in data/catalogs/
  so adding a new catalog requires no changes to main.go
- Add AllSteps() function that merges all step maps (OSPS + future
  catalogs like CRA) into one combined map for SDK registration
- Rename existing catalog ID to osps-baseline-2025-10 with version
- Add OSPS_Baseline_2026_02.yaml catalog for the new baseline version
- Add test to verify every catalog assessment ID has a matching step
  implementation, catching missing steps at build time
- Update CI scripts and example config

To add a new catalog (e.g., CRA manufacturers):
1. Drop the catalog YAML into data/catalogs/
2. Create step implementations under evaluation_plans/
3. Add the step map to AllSteps()
4. The test will catch any missing step implementations

Closes ossf#257
@vinayada1 vinayada1 force-pushed the feat/multi-catalog-support branch from 4701f2f to 29982d6 Compare March 18, 2026 20:58
@eddie-knight
Copy link
Collaborator

proposed solution: vinayada1#1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Proposal] Handle multiple control catalogs (and versions of each)

3 participants