Add Validation and DQM to Phase2 HLT WF

[AdrianoDee]

This is to remind us that the wf running the Phase2 HLT (39434.75) is not running any validation or DQM for hlt objects.

This will be necessary in the mid-long term and would be useful in the short term for developments to spot eventual changes in the menu behavior (see e.g. #39323).

[cmsbuild]

A new Issue was created by @AdrianoDee .

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[AdrianoDee]

assign upgrade

[AdrianoDee]

assign hlt

[cmsbuild]

New categories assigned: upgrade,hlt

@AdrianoDee,@missirol,@srimanob,@Martin-Grunewald you have been requested to review this Pull request/Issue and eventually sign? Thanks

[srimanob]

I think the way is to run HLT at the ususal step (DIGI+L1+HLT). Then the validation will run as usual. No need to push for extra development.

[srimanob]

I make this #39450 for the first attempt to converge on running HLT together with DIGI step.

[slava77]

What is the current status?
is this topic being tracked in another issue or outside of github?

@beaucero @SohamBhattacharya

[srimanob]

Hi @slava77
I have not finished it yet, sorry. I can make it after May 22.

[slava77]

Hi @slava77 I have not finished it yet, sorry. I can make it after May 22.

I wanted to check if there was some progress
Thanks.

[srimanob]

Hi @slava77
I've made a new PR to enable one-by-one HLT validation module in #41898

[srimanob]

I think this PR can be closed as we have Phase-2 DQM in the release.

[srimanob]

+Upgrade

[mmusich]

#42783 introduces some modification to run the Tracking DQM monitoring in Phase-2

[SohamBhattacharya]

Hi, I have naive question about this.
I see two different approaches to adding HLT validation and DQM.

1. The one started by @srimanob here
2. One developed by @Sam-Harper and @finnlabe here (in cmssw)

If they're both trying to do the same (?) thing (i.e. validate HLT paths), is there a way to consolidate the two approaches to make it simpler?

[mmusich]

If they're both trying to do the same (?) thing (i.e. validate HLT paths), is there a way to consolidate the two approaches to make it simpler?

as far as I can tell , that's not the same thing. The issue here is more targeted to the offline validation of the objects used by the HLT (electrons, muons, tracks, jets, etc. etc.). Basically everything in DQMOffline/Trigger that is currently available for Run 3 (but not for phase-2).

[SohamBhattacharya]

If they're both trying to do the same (?) thing (i.e. validate HLT paths), is there a way to consolidate the two approaches to make it simpler?

as far as I can tell , that's not the same thing. The issue here is more targeted to the offline validation of the objects used by the HLT (electrons, muons, tracks, jets, etc. etc.). Basically everything in DQMOffline/Trigger that is currently available for Run 3 (but not for phase-2).

It seems to me that the updates to HLTriggerOffline in #41898 are to do with validating the TriggerResults (validation for HLT_AK4PFPuppiJet520), which is similar to what's being done in 2 above.
(please correct me if I'm wrong)
While we absolutely need the validation of Phase-2 HLT objects in DQMOffline/Trigger, imo (from the pov of HLT-upgrade) we should first have the Phase-2 HLT path and filter validation. That would show us changes in not only the HLT objects (thus also effectively doing the job of DQMOffline/Trigger as far as the impact on the Phase-2 menu is considered), but also in the upstream quantities that the HLT filters depend on, like the L1 objects.
For example, the egamma issue (fixed in #42791) was missed because the filters were not monitored and changes at L1 were not propagated down to the filters used by the Phase-2 menu.

Hence I was wondering if there's a way to have a validation like this as part of the Phase-2 HLT WF.

[tagging @rovere as I'm not sure if he's in the thread]

[mmusich]

we should first have the Phase-2 HLT path and filter validation.

@SohamBhattacharya I guess you are free to twist the initial aim of this ticket, but in the initial message I read:

is not running any validation or DQM for hlt objects.

I think your proposal should go into a different issue!

[SohamBhattacharya]

we should first have the Phase-2 HLT path and filter validation.

@SohamBhattacharya I guess you are free to twist the initial aim of this ticket, but in the initial message I read:

is not running any validation or DQM for hlt objects.

I think your proposal should go into a different issue!

No problem at all with having this in a different issue.
Only initiating the discussion here as the question of validation was rightfully raised in #42791 -- so wanted to make it clear what kind of validation hlt-upgrade would like to focus on first. :)

[srimanob]

Hi @SohamBhattacharya

My PR (#41898) is for Validation step (i.e. compare with MC) while @mmusich PR (#42783) takes care for DQM (i.e. no need of GEN, MC. Something that you run on Tier-0 during data taking, and also regular relvals, both Data and MC).

The way that both PR work is the same, we create a sequence based on Run-3 and enable only modules which work for Phase-2. See:

• Validation: https://github.com/cms-sw/cmssw/blob/master/HLTriggerOffline/Common/python/HLTValidation_cff.py#L30-L91 (I chose to not create new sequence, but use the original one and exclude what do not work.)

• DQM:

  cmssw/DQMOffline/Trigger/python/DQMOffline_Trigger_cff.py

  Lines 193 to 215 in d9adfea

  	offlineHLTSource4HLTMonitorPD = cms.Sequence(
  	dqmInfoHLTMon *
  	lumiMonitorHLTsequence * # lumi
  	sistripMonitorHLTsequence * # strip
  	sipixelMonitorHLTsequence * # pixel
  	BTVHLTOfflineSource * # BTV
  	bTagHLTTrackMonitoringSequence * # BTV relative track efficiencies
  	trackingMonitorHLT * # tracking
  	BTagAndProbeHLT * # BTag and Probe
  	trackingMonitorHLTDisplacedJet* # EXO : DisplacedJet Tracking
  	egmTrackingMonitorHLT * # EGM tracking
  	hltToOfflineTrackValidatorSequence * # Relative Online to Offline performace
  	vertexingMonitorHLT * # vertexing
  	particleNetMonitoringHLT # HIG: monitoring of HLT PNET taggers (incl. comparisons to Offline PNET)
  	)
  	_offlineHLTSource4HLTMonitorPDPh2 = cms.Sequence(
  	dqmInfoHLTMon *
  	HLTtrackerphase2DQMSource * # phase-2 IT and OT clusters
  	trackingMonitorHLT * # tracking
  	hltToOfflineTrackValidatorSequence * # Relative Online to Offline performace
  	vertexingMonitorHLT # vertexing
  	)

  (Marco creates the specific Phase-2 sequence, and include only working module. Then use replace to replace the sequence.

If I understand @mmusich correctly, the suggestion is to try to enable this full sequence [1]. While on my part, POG, PAG should try to enable their trigger paths and re-enable [2]

[1]

offlineHLTSource4HLTMonitorPD = cms.Sequence(
    dqmInfoHLTMon *
    lumiMonitorHLTsequence *          # lumi
    sistripMonitorHLTsequence *       # strip
    sipixelMonitorHLTsequence *       # pixel
    BTVHLTOfflineSource *             # BTV
    bTagHLTTrackMonitoringSequence *  # BTV relative track efficiencies
    trackingMonitorHLT *              # tracking
    BTagAndProbeHLT *                 # BTag and Probe
    trackingMonitorHLTDisplacedJet*   # EXO : DisplacedJet Tracking 
    egmTrackingMonitorHLT *           # EGM tracking
    hltToOfflineTrackValidatorSequence *  # Relative Online to Offline performace
    vertexingMonitorHLT *             # vertexing
    particleNetMonitoringHLT          # HIG: monitoring of HLT PNET taggers (incl. comparisons to Offline PNET)
)

[2]

hltassociation = cms.Sequence(
    hltMultiTrackValidation
    +hltMultiPVValidation
    +egammaSelectors
    +ExoticaValidationProdSeq
    +hltMultiTrackValidationGsfTracks
    +hltMultiTrackValidationMuonTracks
    )

hltvalidationWithMC = cms.Sequence(
    HLTMuonVal
    +HLTTauVal
    +egammaValidationSequence
    +heavyFlavorValidationSequence
    +HLTJetMETValSeq
    +HLTSusyExoValSeq
    +HiggsValidationSequence
    +ExoticaValidationSequence
    +b2gHLTriggerValidation
    +SMPValidationSequence
    +hltbtagValidationSequence #too noisy for now
    +hltHCALdigisAnalyzer+hltHCALRecoAnalyzer+hltHCALNoiseRates # HCAL
)

[mmusich]

for the record #43094 introduced tracking and vertexing HLT validation (w.r.t tracking particles) for the phase 2 setup.

[rovere]

@mmusich is it possible that some harvesting step/sequence is missing from #43094?
I was not able to find the usual eff/fake/dup plots.

[mmusich]

I was not able to find the usual eff/fake/dup plots.

which workflow are you running?

[mmusich]

is it possible that some harvesting step/sequence is missing from #43094?

yah, seems I forgot these:

cmssw/HLTriggerOffline/Common/python/HLTValidationHarvest_cff.py

Lines 45 to 46 in 7122236

	phase2_common.toReplaceWith(hltpostvalidation, hltpostvalidation.copyAndExclude([postProcessorHLTtrackingSequence,
	postProcessorHLTvertexing,

let me fix.

[mmusich]

let me fix.

see #43330

[mmusich]

For HGCal (DQM):

• Add HGCAL Hit calibration to Phase2 HLT. #44684 (14_0_X)
• Add HGCAL Hit calibration to Phase2 HLT. #44558 (master)

[cmsbuild]

cms-bot internal usage

[mmusich]

For Muon:

• Implement Phase 2 Muon HLT validation #46860

[mmusich]

For intermediate tracking collections:

• add DQM / Validation monitoring for hltInitialStep and hltHighPtTriplet tracks in the Phase2 HLT menu #46823