feat: support multiple control catalog versions#269
feat: support multiple control catalog versions#269
Conversation
c9bc59a to
4701f2f
Compare
|
@vinayada1 This is awesome. My only thought is about the data structure for the different versions: TLDR I like option BCurrent PR approach: Single shared map, registered twice// One map with ALL step implementations
OSPS = map[string][]gemara.AssessmentStep{...}
// Registered for each version, catalog YAML filters active IDs
for _, catalogID := range []string{"osps-baseline-2025-10", "osps-baseline-2026-02"} {
orchestrator.AddEvaluationSuite(catalogID, nil, evaluation_plans.OSPS)
}This is clean right now because 2026-02 is a copy of 2025-10. The catalog YAML does the filtering. But the moment a step implementation needs to differ between versions (e.g., OSPS-AC-01.01 gets a stricter check in 2026-02), this falls apart — we can't have two different implementations for the same key in one map. Option A: Map of maps (version as key)var OSPS = map[string]map[string][]gemara.AssessmentStep{
"osps-baseline-2025-10": { ... },
"osps-baseline-2026-02": { ... },
}Pros: Version is a first-class concept. Supports divergent implementations per version. Registration in main.go becomes a clean range loop. Cons: If most steps are shared, we duplicating the entire map per version. That's a lot of repetition in this file — the current map is already large. Maintenance burden increases: adding a shared step means editing N places. Option B: Struct with version attributetype EvaluationPlan struct {
CatalogID string
Steps map[string][]gemara.AssessmentStep
}
var Plans = []EvaluationPlan{
{CatalogID: "osps-baseline-2025-10", Steps: baseSteps},
{CatalogID: "osps-baseline-2026-02", Steps: mergedSteps(baseSteps, overrides2026)},
}Pros: Clean encapsulation. Could support a composition pattern (base + overrides) that avoids full duplication while still allowing divergence. The catalog ID lives next to its steps. Cons: More upfront structure. The SDK's AddEvaluationSuite takes map[string][]gemara.AssessmentStep directly, so the struct is internal bookkeeping only — it doesn't flow into the SDK API. My takeThe PR's current approach works fine if the design intent is that step implementations are always shared and only the catalog YAML controls which IDs are active per version. That's a reasonable constraint and the simplest thing that works. However, if we anticipate step implementations diverging between versions (which seems likely as the baseline evolves), I'd lean toward Option B with a composition pattern — keep a shared base map, then per-version overrides that get merged. It avoids the duplication problem of Option A while making version a real concept rather than just a loop of catalog ID strings. The key question is: does the SDK guarantee that passing extra step implementations (keys not in the catalog YAML) is harmless? The PR comments say yes ("the SDK only runs the relevant subset"). If that's reliable, the current shared-map approach is defensible and we can defer restructuring until implementations actually diverge. If there's any risk of the SDK complaining about unrecognized assessment IDs, we'd want the per-version separation now. |
|
@vinayada1 please sign your commits with |
Implements the proposal from ossf#257 to handle multiple control catalogs and versions of each. Changes: - Auto-discover catalogs from embedded YAML files in data/catalogs/ so adding a new catalog requires no changes to main.go - Add AllSteps() function that merges all step maps (OSPS + future catalogs like CRA) into one combined map for SDK registration - Rename existing catalog ID to osps-baseline-2025-10 with version - Add OSPS_Baseline_2026_02.yaml catalog for the new baseline version - Add test to verify every catalog assessment ID has a matching step implementation, catching missing steps at build time - Update CI scripts and example config To add a new catalog (e.g., CRA manufacturers): 1. Drop the catalog YAML into data/catalogs/ 2. Create step implementations under evaluation_plans/ 3. Add the step map to AllSteps() 4. The test will catch any missing step implementations Closes ossf#257
4701f2f to
29982d6
Compare
|
proposed solution: vinayada1#1 |
Summary
Implements the proposal from #257 to handle multiple control catalogs and versions of each.
Changes
OSPSevaluation plan variable toOSPS_2025_10with version-specific catalog ID (osps-baseline-2025-10)OSPS_2026_02evaluation plan (initially mirrors 2025_10, to be updated as the new baseline evolves)main.goOSPS_Baseline_2026_02.yamlcatalog with version-specific metadata (id: osps-baseline-2026-02,version: 2026.02)Design Decisions
OSPS_2025_10,OSPS_2026_02) and is registered as a separate evaluation suite with a versioned IDidfields now include the version suffix to match their evaluation suite registration namesCloses #257