CPU vs. GPU for LST in HLT and updates to the offline #215

VourMa · 2025-12-16T20:11:29Z

The goal of this PR is to introduce two HLT workflows to monitor the agreement between LST on CPU and LST on GPU:

Workflow 0.7541 monitors the LST output tracks when LST is used for track building (most direct comparison of LST), i.e. for alpakaValidationLST,singleIterPatatrack,trackingLST.
Workflow 0.7573 monitors the built tracks in the upcoming new tracking baseline, where LST is used as an extended seeding algorithm (comparison of LST output in a "production" configuration), i.e. for singleIterPatatrack,phase2CAExtension,trackingLST,seedingLST,trackingMkFitCommon,hltTrackingMkFitInitialStep.

The additional CPU reconstruction (SerialSync) and comparison plots are implemented with a new procModifier, alpakaValidationLST. This procModifier needs to be run only in the procModifier combinations mentioned above to take effect, otherwise it produces neither the additional products nor the comparison plots. It is also included in the alpakaValidation modifier chain.

The analyzer that produces the comparison plots has been improved with a new parameter option to skip luminosity and PU plots, as these are not always available in the current workflows for Phase 2 HLT.

With the introduction of the alpakaValidationLST modifier, the offline workflow testing LST on CPU vs. LST on GPU can be made explicit. The code is changed so that workflow 0.704 runs the offline reconstruction without any additional CPU reconstruction, while a new workflow, 0.7041, runs the comparison.

Some screenshots of the content of the DQM file:

slava77

I don't see alpakaValidationLST already added to the alpakaValidation chain.
Or is it logically not practical? (I didn't think all variants well enough).

It sounds like it can be done with no (?) cost. If it's chained by default:

If the trackingLST is not in the producer side of the configuration, a clone of tracks would be compared. This is OK, but wasteful.
Alternatively alpakaValidationLST & trackingLST could be used on the producer sequence and product addition side. This way if the trackingLST was not specified we'd be leaving/allowing the validation module silently fail with empty/missing inputs. This will be lighter on the resources.

slava77 · 2025-12-19T23:19:36Z

HLTrigger/Configuration/python/HLT_75e33/modules/hltInitialStepMkFitSeeds_cfi.py

@@ -12,3 +12,7 @@
 from Configuration.ProcessModifiers.seedingLST_cff import seedingLST
 from Configuration.ProcessModifiers.hltTrackingMkFitInitialStep_cff import hltTrackingMkFitInitialStep
 (trackingLST & seedingLST & hltTrackingMkFitInitialStep).toModify(hltInitialStepMkFitSeeds, seeds = "hltInitialStepTrajectorySeedsLST")
+
+hltInitialStepMkFitSeedsSerialSync = hltInitialStepMkFitSeeds.clone(


I thought the idea was to not cover the seedingLST part: just use the candidate directly from LST and only pass them through a fit

My idea was to have both: the trackingLST only would be the more "direct" comparison, while there would also be the option to compare the soon-to-be default tracking sequence, to test what actually will be running at HLT. The latter includes seedingLST, hence the above modification.

The value I see in these different sequences, and also having them separately from offline, is to probe changes at the configuration level which can matter, like the propagation of triplet pixel seeds, their duplicate cleaning or not within LST, etc..

HLTrigger/Configuration/python/HLT_75e33/modules/hltInitialStepTrackCandidates_cfi.py

HLTrigger/Configuration/python/HLT_75e33/modules/hltInitialStepTracks_cfi.py

HLTrigger/Configuration/python/HLT_75e33/modules/hltInitialStepTrajectorySeedsLST_cfi.py

HLTrigger/Configuration/python/HLT_75e33/sequences/HLTInitialStepSequence_cfi.py

VourMa · 2026-01-05T15:43:19Z

I don't see alpakaValidationLST already added to the alpakaValidation chain. Or is it logically not practical? (I didn't think all variants well enough).

Indeed, when I get everything fully working, I want to add it there. I will do it already in the next commit, so that do not forget.

It sounds like it can be done with no (?) cost. If it's chained by default:

If the trackingLST is not in the producer side of the configuration, a clone of tracks would be compared. This is OK, but wasteful.

Alternatively alpakaValidationLST & trackingLST could be used on the producer sequence and product addition side. This way if the trackingLST was not specified we'd be leaving/allowing the validation module silently fail with empty/missing inputs. This will be lighter on the resources.

I wrote the tracking sequence modifications in such a way so that the alpakaValidationLST only takes effect when other, appropriate LST procModifiers are enabled:

For the trackingLST only variant, the LST sequence is modified:

cmssw/HLTrigger/Configuration/python/HLT_75e33/sequences/HLTInitialStepSequence_cfi.py

Line 42 in 62a129c

alpakaValidationLST.toReplaceWith(_HLTInitialStepSequenceLST, cms.Sequence(

which is only run when trackingLST is enabled:

cmssw/HLTrigger/Configuration/python/HLT_75e33/sequences/HLTInitialStepSequence_cfi.py

Line 50 in 62a129c

    
           (singleIterPatatrack & trackingLST & ~seedingLST).toReplaceWith(HLTInitialStepSequence, _HLTInitialStepSequenceLST.copyAndExclude([HLTHighPtTripletStepSeedingSequence,hltHighPtTripletStepSeedTracksLST]))

For the soon-to-be default tracking, the full set of procModifiers is specified:

cmssw/HLTrigger/Configuration/python/HLT_75e33/sequences/HLTInitialStepSequence_cfi.py

Line 118 in 62a129c

    
           #(alpakaValidation & singleIterPatatrack & trackingLST & seedingLST & hltTrackingMkFitInitialStep).toReplaceWith(HLTInitialStepSequence, HLTInitialStepSequence.copyAndAdd([hltInputLSTSerialSync,hltLSTSerialSync,hltInitialStepTrajectorySeedsLSTSerialSync,hltInitialStepTrackCandidatesSerialSync,hltInitialStepTracksSerialSync]))

Based on the above, I think that there is no duplication of tracks but rather a (hopefully) silent failure of the comparison due to missing inputs in all cases of improper procModifier combination. Even that can be avoided by adding trackingLST in this line?

cmssw/Validation/RecoTrack/python/HLTmultiTrackValidator_cff.py

Line 54 in 62a129c

alpakaValidationLST.toReplaceWith(hltMultiTrackValidation, cms.Sequence(

VourMa · 2026-01-07T18:41:13Z

@slava77 please take a look and let me know of any comments you may have. I will fix them together with the resolution of the conflict and some commit squashing.

slava77

approving, but leaving a few points to either avoid copy-paste or extremely long (no-space) names/strings

slava77 · 2026-01-09T22:37:20Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

+class UpgradeWorkflow_lstOnGPUIters01TrackingOnlyAlpakaValidationLST(UpgradeWorkflowTracking):
+    def setup__(self, step, stepName, stepDict, k, properties):
+        if 'Reco' in step: stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
+        elif 'HARVEST' in step: stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@trackingOnlyValidation+@trackingOnlyDQM', '--procModifiers': 'alpakaValidationLST,trackingIters01,trackingLST'}, stepDict[step][k]])


I'm always puzzled by why we need to repeat so much compared to an existing workflow.
Can this inherit from UpgradeWorkflow_lstOnGPUIters01TrackingOnly and just merge '--procModifiers': 'alpakaValidationLST' (and update only the suffix and offset)?

slava77 · 2026-01-09T22:40:09Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

    '-s':'HARVESTING:@hltValidation'
 }

+upgradeWFs['HLTTiming75e33SingleIterLSTAlpakaValidationLST'] = deepcopy(upgradeWFs['HLTTiming75e33'])


Suggested change

upgradeWFs['HLTTiming75e33SingleIterLSTAlpakaValidationLST'] = deepcopy(upgradeWFs['HLTTiming75e33'])

upgradeWFs['HLTTiming75e33SingleIterLSTAlpakaValidationLST'] = deepcopy(upgradeWFs['HLTTiming75e33AlpakaSingleIterLST'])

similar to the other comment about copy-paste: can this allow to reduce the necessary details?

I wouldn't like to change just the structure of HLT ones for the time being. But it is a good point, and I will keep it in mind for the clean up when we get the new tracking baseline in.

slava77 · 2026-01-09T22:44:03Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

+upgradeWFs['HLTTiming75e33SingleIterCAExtLSTSeedingMkFitBuildingAlpakaValidationLST'].step2 = {
+    # This workflow is meant to and only works for the tracking validation
+    '-s':'DIGI:pdigi_valid,L1TrackTrigger,L1,L1P2GT,DIGI2RAW,HLT:75e33_timing,VALIDATION:hltMultiTrackValidation',
+    '--procModifiers': 'alpakaValidationLST,singleIterPatatrack,phase2CAExtension,trackingLST,seedingLST,trackingMkFitCommon,hltTrackingMkFitInitialStep',


seeing how long the modifier list is, I'd propose to define a modifier chain named e.g. hltSingleIterTrackingBaseline and have something more readable

The (my) hope is to move the new tracking baseline to default not too far in the future, so I wouldn't like to add a ModifierChain (and hence a new file) for that if it is to be removed in a couple of weeks.

slava77 · 2026-01-09T22:48:31Z

HLTrigger/Configuration/python/HLT_75e33/modules/hltInitialStepTrackCandidates_cfi.py

+    lstInput = "hltInputLSTSerialSync",
+    lstPixelSeeds = "hltInputLSTSerialSync"
+)
+(singleIterPatatrack & seedingLST & trackingLST & hltTrackingMkFitInitialStep).toModify(hltInitialStepTrackCandidatesSerialSync,


singleIterPatatrack & seedingLST & trackingLST & hltTrackingMkFitInitialStep is repeated twice and by itself is pretty large, consider to define a shorthand _pataLSTMKF = singleIterPatatrack & seedingLST & trackingLST & hltTrackingMkFitInitialStep and use it in both places

slava77 · 2026-01-09T22:50:32Z

HLTrigger/Configuration/python/HLT_75e33/sequences/HLTInitialStepSequence_cfi.py

@@ -106,6 +115,18 @@

 (singleIterPatatrack & trackingLST & seedingLST & hltTrackingMkFitInitialStep).toReplaceWith(HLTInitialStepSequence, _HLTInitialStepSequenceSingleIterPatatrackLSTSeedingMkFitTracking)

+(alpakaValidationLST & singleIterPatatrack & trackingLST & seedingLST & hltTrackingMkFitInitialStep).toReplaceWith(HLTInitialStepSequence, cms.Sequence(


singleIterPatatrack & trackingLST & seedingLST & hltTrackingMkFitInitialStep is used 3 times in the file, consider a shorthand (by the time it wraps over one line I went over a threshold to make a comment )

VourMa · 2026-01-14T16:00:49Z

@slava77 I should have implemented most of your comments and replied to the rest. Let me know if all looks good now, and I can proceed to the cmssw PR.

slava77

looks good to go, considering followup comments

…to the offline corresponding workflow

VourMa requested a review from slava77 December 16, 2025 20:11

slava77 reviewed Dec 20, 2025

View reviewed changes

VourMa changed the title ~~CPU vs. GPU for LST (and potentially more) in HLT~~ CPU vs. GPU for LST in HLT Jan 7, 2026

VourMa marked this pull request as ready for review January 7, 2026 18:39

slava77 approved these changes Jan 9, 2026

View reviewed changes

VourMa force-pushed the CMSSW_16_0_0_pre3_serialSync branch from d08a0a7 to b4abc99 Compare January 14, 2026 15:46

VourMa changed the base branch from master to CMSSW_16_0_0_pre3_removeUnusedFlag January 14, 2026 15:47

VourMa changed the base branch from CMSSW_16_0_0_pre3_removeUnusedFlag to master January 14, 2026 15:47

slava77 approved these changes Jan 14, 2026

View reviewed changes

VourMa force-pushed the CMSSW_16_0_0_pre3_serialSync branch from b4abc99 to 6fd4c49 Compare January 14, 2026 16:51

VourMa changed the title ~~CPU vs. GPU for LST in HLT~~ CPU vs. GPU for LST in HLT and updates to the offline Jan 14, 2026

VourMa force-pushed the CMSSW_16_0_0_pre3_serialSync branch 2 times, most recently from 2cf3f6b to 0dd7b24 Compare January 15, 2026 10:44

Addition of HLT workflows for testing LST on CPU vs. GPU and updates …

b807f98

…to the offline corresponding workflow

VourMa force-pushed the CMSSW_16_0_0_pre3_serialSync branch from 0dd7b24 to b807f98 Compare January 16, 2026 18:26

	upgradeWFs['HLTTiming75e33SingleIterLSTAlpakaValidationLST'] = deepcopy(upgradeWFs['HLTTiming75e33'])
	upgradeWFs['HLTTiming75e33SingleIterLSTAlpakaValidationLST'] = deepcopy(upgradeWFs['HLTTiming75e33AlpakaSingleIterLST'])

		@@ -106,6 +115,18 @@

		(singleIterPatatrack & trackingLST & seedingLST & hltTrackingMkFitInitialStep).toReplaceWith(HLTInitialStepSequence, _HLTInitialStepSequenceSingleIterPatatrackLSTSeedingMkFitTracking)

		(alpakaValidationLST & singleIterPatatrack & trackingLST & seedingLST & hltTrackingMkFitInitialStep).toReplaceWith(HLTInitialStepSequence, cms.Sequence(

CPU vs. GPU for LST in HLT and updates to the offline #215

Are you sure you want to change the base?

CPU vs. GPU for LST in HLT and updates to the offline #215

Conversation

VourMa commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slava77 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

VourMa commented Jan 5, 2026

Uh oh!

VourMa commented Jan 7, 2026

Uh oh!

slava77 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VourMa commented Jan 14, 2026

Uh oh!

slava77 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

VourMa commented Dec 16, 2025 •

edited

Loading