Add possibility to read triggerbits for Secondary Datasets from the GT #45092

francescobrivio · 2024-05-29T09:55:32Z

PR description:

Upon request of @cms-sw/ppd-l2, in order to migrate the list of trigger paths that define the Secondary Datasets away from the release and into the GT, in this PR I'm updating the HLTrigger/HLTfilters/plugins/HLTHighLevel.cc module to also accept a "tag label" through which we can read the SecondaryDatased-dedicated triggerbits tag.

This issue is discussed in: https://its.cern.ch/jira/browse/CMSBPH-2

Changes in this PR:

Update of HLTHighLevel in 6bfb34d
Update of the ReserveDMu SD configuration in c796cdc
Update of the Prompt GT in autoCond with an updated candidate that contains the new triggerbits tag with label SecondaryDatasetTrigger
- ~~Note to @cms-sw/alca-l2 we will need a properly versioned Prompt GT to update autoCond~~
- ~~GT diff: 140X_dataRun3_Prompt_v2 vs 140X_dataRun3_Prompt_Candidate_2024_05_29_08_51_00~~
- See below for the final GT diffs

Full GT differences:

run2_data
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/140X_dataRun2_v1/140X_dataRun2_v2
run3_data_prompt
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/140X_dataRun3_Prompt_frozen_v1/140X_dataRun3_Prompt_frozen_v3
- Jump from version v1 to v3 since the v2 GT was never propagated to the release (but it was correctly deployed in production, see comment in Add possibility to read triggerbits for Secondary Datasets from the GT #45092 (comment)), so it additionaly includes the change of the DropBoxMetadataRcd tag (from DropBoxMetadata_v8.4_express to DropBoxMetadata_v8.5_express)
run3_data
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/140X_dataRun3_v3/140X_dataRun3_v4
run3_data_PromptAnalysis
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/140X_dataRun3_PromptAnalysis_v1/140X_dataRun3_PromptAnalysis_v2

Triggerbit tag
The new triggerbit tag is AlCaRecoTriggerBits_SecondaryDataset_v1, with 2 IOVs:

From start to year 2023 (included) Payload Inspector plot:
From 2024 onwards Payload Inspector plot:
Currently the ReserveDmu key is the only one implemented, but this can easily be extended to other SecondaryDatasets if needed

PR validation:

Code compiles + scram b runtests runs fine.
Additionally, I have run the following cmsDriver

cmsDriver.py RECO --conditions 140X_dataRun3_Prompt_v2 \
  --datatier RAW-RECO --era Run3 --eventcontent RAW \
  --filein "file:/eos/home-f/fbrivio/AlCa/data/ParkingDoubleMuonLowMass1_Run380470/136ee2f3-e230-40a7-952b-d3a7d12c27ce.root" \
  --fileout "file:skim_ReserveDMu.root" \
  --nThreads 2 --number 200 --scenario pp \
  --step SKIM:@ParkingDoubleMuonLowMass0 \
  --data --processName PAT

in different configurations and checked the number of events saved in ReserveDMu.root

Version	Trigger paths	Evts selected
`master`	all ReserveDMu paths	43
`master`	only 2 paths	5
---	---	---
This PR	all ReserveDMu paths	43
This PR	only 2 ReserveDMu paths	5

Backport:

Not a backport, but eventually a backport to 14_0_X will be opened in order to deploy this in Tier0 and produce directly the ReserveDMu SecondaryDataset

cmsbuild · 2024-05-29T09:55:58Z

cms-bot internal usage

mmusich · 2024-05-29T10:01:33Z

Additionally, I have run the following cmsDriver

is there a workflow to test skims (SKIM:@ParkingDoubleMuonLowMass0 in particular) @cms-sw/pdmv-l2 ?

mmusich · 2024-05-29T10:04:24Z

is there a workflow to test skims

apparently 141.114:

141.114 RunParkingDoubleMuonLowMass2023C+HLTDR3_2023+SKIMPARKINGDOUBLEMUONLOWMASS0RUN3_reHLT_2023+HARVESTRUN3_2023 [1]: input from: /ParkingDoubleMuonLowMass0/Run2023C-v1/RAW with run [] 
                                           [2]: cmsDriver.py step2  --process reHLT -s L1REPACK:Full,HLT:@relval2024 --conditions auto:run3_hlt_relval --data  --eventcontent FEVTDEBUGHLT --datatier FEVTDEBUGHLT --era Run3_2023 -n 100 
                                           [3]: cmsDriver.py step3  --conditions auto:run3_data_prompt_relval -s RAW2DIGI,L1Reco,RECO,SKIM:ReserveDMu+LogError+LogErrorMonitor,PAT,NANO,DQM:@standardDQM+@miniAODDQM+@nanoAODDQM --datatier RECO,MINIAOD,NANOAOD,DQMIO --eventcontent RECO,MINIAOD,NANOEDMAOD,DQM --data  --process reRECO --scenario pp --era Run3_2023 --customise Configuration/DataProcessing/RecoTLR.customisePostEra_Run3 --hltProcess reHLT -n 100 
                                           [4]: cmsDriver.py step4  -s HARVESTING:@standardDQM+@miniAODDQM+@nanoAODDQM --conditions auto:run3_data --data  --filetype DQM --scenario pp --era Run3_2023 -n 100 

1 workflows with 4 steps

mmusich · 2024-05-29T10:04:40Z

test parameters:

workflow = 141.114

cmsbuild · 2024-05-29T10:05:43Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45092/40412

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/AlCa/python/autoCond.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368

cmsbuild · 2024-05-29T10:06:20Z

A new Pull Request was created by @francescobrivio for master.

It involves the following packages:

Configuration/AlCa (alca)
Configuration/Skimming (pdmv)
HLTrigger/HLTfilters (hlt)

@AdrianoDee, @mmusich, @cmsbuild, @Martin-Grunewald, @saumyaphor4252, @miquork, @consuegs, @sunilUIET, @perrotta can you please review it and eventually sign? Thanks.
@mmusich, @youyingli, @Martin-Grunewald, @fabiocos, @silviodonato, @yuanchao, @tocheng, @missirol, @rsreds this is something you requested to watch as well.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

mmusich · 2024-05-29T10:29:57Z

@cmsbuild, please test

cmsbuild · 2024-05-29T12:56:23Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1ec3d5/39604/summary.html
COMMIT: c5dd3d6
CMSSW: CMSSW_14_1_X_2024-05-29-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45092/39604/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

1000.0

----- Begin Fatal Exception 29-May-2024 13:17:45 CEST-----------------------
An exception of category 'NoProductResolverException' occurred while
   [0] Processing  Event run: 165121 lumi: 62 event: 23609118 stream: 0
   [1] Running path 'ReserveDMuPath'
   [2] Calling method for module HLTHighLevel/'ReserveDMu'
Exception Message:
No data of type "AlCaRecoTriggerBits" with label "SecondaryDatasetTrigger" in record "AlCaRecoTriggerBitsRcd"
 Please add an ESSource or ESProducer to your job which can deliver this data.
----- End Fatal Exception -------------------------------------------------

Configuration/AlCa/python/autoCond.py

cmsbuild · 2024-05-30T09:40:25Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45092/40429

This PR adds an extra 28KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/AlCa/python/autoCond.py modified in PR(s): [Dummy] Try creating big-enough cache to cause splitting #44368

mmusich · 2024-06-01T10:37:12Z

+hlt

for cms-AlCaDB@3d0b8e9
PR according description and follow-up review:
- enables reading list of trigger bits from the EventSetup in HLTHighLevel from a label different from the default (null).
- motivation given at Add possibility to read triggerbits for Secondary Datasets from the GT #45092 (comment)

perrotta · 2024-06-01T13:32:56Z

Does anybody understand the origin of the differences in the DQM comparisons for the SiStrip BadComponents and for the Ecal Endcap timing projections in the workflows 141.X?

mmusich · 2024-06-02T00:05:52Z

Does anybody understand the origin of the differences in the DQM comparisons for the SiStrip BadComponents and for the Ecal Endcap timing projections in the workflows 141.X?

Did the snapshot update make the framework see newer payloads on the IoV in question?

perrotta · 2024-06-02T06:59:59Z

Does anybody understand the origin of the differences in the DQM comparisons for the SiStrip BadComponents and for the Ecal Endcap timing projections in the workflows 141.X?

Did the snapshot update make the framework see newer payloads on the IoV in question?

It does not seem so. These are 2023C/D workflows for runs 367131 or 369978.
Both old and new GT had payloads updated for the whole 2023, see e.g. for Strips:

(unless there is something more subtle that I am missing...)

francescobrivio · 2024-06-05T08:35:29Z

Thanks @perrotta I also took a look at all the IOV diffs in the GTs, but indeed I can't see any obvious issue...

mmusich · 2024-06-05T21:24:24Z

I also took a look at all the IOV diffs in the GTs, but indeed I can't see any obvious issue...

well... @francescobrivio @perrotta I let you figure out of that is possible, but by doing this simple test of running:

runTheMatrix.py -l 141.046 --command='--customise_commands="process.GlobalTag.DumpStat = cms.untracked.bool(True)"' -j 8 -t 4 --ibeos

with and without this PR and then comparing the logs of step4 (harvesting, which is the one in which the Strip plots that differ https://tinyurl.com/2y2ffbvd are populated by this class) and I definitely see different payloads being consumed:

@@ -125,11 +125,6 @@ AlCaRecoTriggerBitsRcd / MuonDQMTrigger: frontier://FrontierProd/CMS_CONDITIONS,
   Time look up, payloadIds:
 
 
-AlCaRecoTriggerBitsRcd / SecondaryDatasetTrigger: frontier://FrontierProd/CMS_CONDITIONS, AlCaRecoTriggerBits_SecondaryDataset_v1
-  
-  Time look up, payloadIds:
-
-
 AlCaRecoTriggerBitsRcd / SiStripDQMTrigger: frontier://FrontierProd/CMS_CONDITIONS, AlCaRecoTriggerBits_TrackerDQM_v6_offline
   
   Time look up, payloadIds:
@@ -383,19 +378,19 @@ DropBoxMetadataRcd / : frontier://FrontierProd/CMS_CONDITIONS, DropBoxMetadata_v
 EBAlignmentRcd / : frontier://FrontierProd/CMS_CONDITIONS, EBAlignment_Run1_Run2_Run3_v01_offline
   
   Time look up, payloadIds:
-   366727 - 18446744073709551615 : 4d60842f7f5064c6e443f8e6df608d5ac62702d6
+   359022 - 18446744073709551615 : 7e69558ff6dba47cd9a3c4a251aad68b76c147f8
 
 
 EEAlignmentRcd / : frontier://FrontierProd/CMS_CONDITIONS, EEAlignment_Run1_Run2_Run3_v02_offline
   
   Time look up, payloadIds:
-   366727 - 18446744073709551615 : 93d56ec34dd153dc9e58f35ff220cc86f29e5dc4
+   359022 - 18446744073709551615 : d67eb9abc6423e53709355d5e40bb79c41a40030
 
 
 ESAlignmentRcd / : frontier://FrontierProd/CMS_CONDITIONS, ESAlignment_Run1_Run2_Run3_v01_offline
   
   Time look up, payloadIds:
-   366727 - 18446744073709551615 : d956f1c17b41ab34d3beacb62c79c08c8c00f7d0
+   352319 - 18446744073709551615 : 5cb9cad76f9a51cb4fcb3ecad8e6491402306c52
 
 
 ESAngleCorrectionFactorsRcd / : frontier://FrontierProd/CMS_CONDITIONS, ESAngleCorrectionFactors_V01_hlt
@@ -1697,7 +1692,7 @@ HOAlignmentRcd / : frontier://FrontierProd/CMS_CONDITIONS, HOAlignment_zero_v01_
 HcalChannelQualityRcd / : frontier://FrontierProd/CMS_CONDITIONS, HcalChannelQuality_v9.0_offline
   
   Time look up, payloadIds:
-   1562139735097345 - 1605347106095104 : 74fcb388c3a365a6aec767ba5562f842e82b9c68
+   1535394973745153 - 18446744073709551615 : 00baf5e7f800fa3fa88312f990f95a3176e853ee
 
 
 HcalElectronicsMapRcd / : frontier://FrontierProd/CMS_CONDITIONS, HcalElectronicsMap_v9.1_offline
@@ -1788,7 +1783,7 @@ HcalPedestalWidthsRcd / : frontier://FrontierProd/CMS_CONDITIONS, HcalPedestalWi
 HcalPedestalWidthsRcd / effective: frontier://FrontierProd/CMS_CONDITIONS, HcalPedestalWidths_v4.0_offline_effective
   
   Time look up, payloadIds:
-   369978 - 370174 : 211d46db012f305635fbb7c00061c0d5fe232434
+   362693 - 18446744073709551615 : 3d9ce138453a94c7142ac3e0728e49a81818f9b0
 
 
 HcalPedestalsRcd / : frontier://FrontierProd/CMS_CONDITIONS, HcalPedestals_ADC_v10.1_offline
@@ -1800,7 +1795,7 @@ HcalPedestalsRcd / : frontier://FrontierProd/CMS_CONDITIONS, HcalPedestals_ADC_v
 HcalPedestalsRcd / effective: frontier://FrontierProd/CMS_CONDITIONS, HcalPedestals_v11.0_offline_effective
   
   Time look up, payloadIds:
-   369978 - 370174 : a891aa2ec274f6e258b55021027bfd42244a5d36
+   362693 - 18446744073709551615 : 89ec4929ecb131fe83de7cb83640d4eb64c57ebb
 
 
 HcalQIEDataRcd / : frontier://FrontierProd/CMS_CONDITIONS, HcalQIEData_NormalMode_v9.2_offline
@@ -1824,7 +1819,7 @@ HcalRecoParamsRcd / : frontier://FrontierProd/CMS_CONDITIONS, HcalRecoParams_v11
 HcalRespCorrsRcd / : frontier://FrontierProd/CMS_CONDITIONS, HcalRespCorrs_v9.0_offline
   
   Time look up, payloadIds:
-   362102 - 377782 : 1d35b183fda55c37278ca171edd3592e0ecb1ca7
+   362102 - 18446744073709551615 : 1d35b183fda55c37278ca171edd3592e0ecb1ca7
 
 
 HcalSiPMCharacteristicsRcd / : frontier://FrontierProd/CMS_CONDITIONS, HcalSiPMCharacteristics_v4.0_offline
@@ -3158,7 +3153,7 @@ SiStripBadChannelRcd / : frontier://FrontierProd/CMS_CONDITIONS, SiStripBadChann
 SiStripBadFiberRcd / : frontier://FrontierProd/CMS_CONDITIONS, SiStripBadChannel_FromOfflineCalibration_GR10_v9_offline
   
   Time look up, payloadIds:
-   369978 - 369993 : 051cddb31e53f8b02ffa85860c466524ab09a842
+   362643 - 18446744073709551615 : bead1d3672d1bde3f985b81b2cc0aa9bce4463b5
 
 
 SiStripBadModuleRcd / : frontier://FrontierProd/CMS_CONDITIONS, SiStripBadChannel_FromEfficiencyAnalysis_GR10_v3_offline
@@ -3214,7 +3209,7 @@ SiStripDeDxProton_3D_Rcd / : frontier://FrontierProd/CMS_CONDITIONS, SiStripDeDx
 SiStripDetVOffRcd / : frontier://FrontierProd/CMS_CONDITIONS, SiStripDetVOff_v3_offline
   
   Time look up, payloadIds:
-   7251257414682496576 - 7251335685781498879 : 8183951c4a248562800e0239cb010d413de8bfd3
+   7169398241720628672 - 18446744073709551615 : a7399f72a1889800ba87121097847e9a35c88ca7
 
 
 SiStripFedCablingRcd / : frontier://FrontierProd/CMS_CONDITIONS, SiStripFedCabling_v1_prompt
@@ -3311,4 +3306,4 @@ VeryForwardIdealGeometryRecord / : frontier://FrontierProd/CMS_CONDITIONS, PPSRE
   Time look up, payloadIds:

perrotta · 2024-06-06T12:54:16Z

Thank you @mmusich this makes much more sense indeed! (Sorry, I could not follow too much these days, as this week I am in vacation).

In fact, for the step 4 the auto:run3_data GT is used, which corresponds to:

140X_dataRun3_v3 for the baseline
140X_dataRun3_v4 for the PR

(I was looking at the prompt conditions used in step3, instead).

Looking, for example, at the EBAlignment_Run1_Run2_Run3_v01_offline tag, the GT in _v4 has an additional payload with IOV 366727, which is antecedent the run number processed in that workflow, 369978.

Thus everything makes finally sense!

This PR can be safely signed, then

perrotta · 2024-06-06T12:54:23Z

+alca

AdrianoDee · 2024-06-06T14:39:40Z

+pdmv

cmsbuild · 2024-06-06T14:40:05Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

francescobrivio · 2024-06-10T07:05:59Z

@cms-sw/orp-l2 a kind ping to please merge this PR whenever you can.
AFAIU the backport would be very welcome from @cms-sw/ppd-l2 to produce the SD in a "growing-campaign" after the Technical Stop.

perrotta · 2024-06-10T15:21:18Z

urgent
(there is a backport waiting for it...)

antoniovilela · 2024-06-10T23:21:01Z

+1

francescobrivio added 2 commits May 29, 2024 11:26

Add possibility to read a labelled triggerbits tag in HLTHighLevel

6bfb34d

Read triggerbits from GT for ReserveDMu secondary dataset

c796cdc

cmsbuild added this to the CMSSW_14_1_X milestone May 29, 2024

cmsbuild added hlt-pending alca-pending pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending labels May 29, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels May 29, 2024

cmsbuild added tests-started and removed tests-pending labels May 29, 2024

cmsbuild added tests-rejected and removed tests-started labels May 29, 2024

mmusich reviewed May 29, 2024

View reviewed changes

Configuration/AlCa/python/autoCond.py Outdated Show resolved Hide resolved

cmsbuild added tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels May 30, 2024

cmsbuild added the tests-approved label Jun 1, 2024

cmsbuild added hlt-approved and removed hlt-pending labels Jun 1, 2024

cmsbuild added alca-approved and removed alca-pending labels Jun 6, 2024

cmsbuild added fully-signed pdmv-approved and removed pending-signatures pdmv-pending labels Jun 6, 2024

cmsbuild added the urgent label Jun 10, 2024

cmsbuild added orp-approved and removed orp-pending labels Jun 10, 2024

cmsbuild merged commit 6753b0e into cms-sw:master Jun 10, 2024
12 checks passed

This was referenced Jun 11, 2024

gcc: add --as-needed link option cms-sw/cmsdist#9232

Closed

Replace enums with static constexpr int in EcalChannelStatus_PayloadInspector.cc #45193

Merged

perrotta mentioned this pull request Jun 13, 2024

Move online (frozen) GTs to 141X and add the dedxCalibration record for HI #45220

Merged

francescobrivio deleted the alca-SD_triggerBits_14_1_X branch June 26, 2024 08:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add possibility to read triggerbits for Secondary Datasets from the GT #45092

Add possibility to read triggerbits for Secondary Datasets from the GT #45092

francescobrivio commented May 29, 2024 •

edited

Loading

cmsbuild commented May 29, 2024 •

edited

Loading

mmusich commented May 29, 2024

mmusich commented May 29, 2024

mmusich commented May 29, 2024

cmsbuild commented May 29, 2024

cmsbuild commented May 29, 2024

mmusich commented May 29, 2024

cmsbuild commented May 29, 2024

cmsbuild commented May 30, 2024

mmusich commented Jun 1, 2024

perrotta commented Jun 1, 2024

mmusich commented Jun 2, 2024

perrotta commented Jun 2, 2024

francescobrivio commented Jun 5, 2024

mmusich commented Jun 5, 2024

perrotta commented Jun 6, 2024

perrotta commented Jun 6, 2024

AdrianoDee commented Jun 6, 2024

cmsbuild commented Jun 6, 2024

francescobrivio commented Jun 10, 2024

perrotta commented Jun 10, 2024 •

edited

Loading

antoniovilela commented Jun 10, 2024

Add possibility to read triggerbits for Secondary Datasets from the GT #45092

Add possibility to read triggerbits for Secondary Datasets from the GT #45092

Conversation

francescobrivio commented May 29, 2024 • edited Loading

PR description:

PR validation:

Backport:

cmsbuild commented May 29, 2024 • edited Loading

mmusich commented May 29, 2024

mmusich commented May 29, 2024

mmusich commented May 29, 2024

cmsbuild commented May 29, 2024

cmsbuild commented May 29, 2024

mmusich commented May 29, 2024

cmsbuild commented May 29, 2024

RelVals

cmsbuild commented May 30, 2024

mmusich commented Jun 1, 2024

perrotta commented Jun 1, 2024

mmusich commented Jun 2, 2024

perrotta commented Jun 2, 2024

francescobrivio commented Jun 5, 2024

mmusich commented Jun 5, 2024

perrotta commented Jun 6, 2024

perrotta commented Jun 6, 2024

AdrianoDee commented Jun 6, 2024

cmsbuild commented Jun 6, 2024

francescobrivio commented Jun 10, 2024

perrotta commented Jun 10, 2024 • edited Loading

antoniovilela commented Jun 10, 2024

francescobrivio commented May 29, 2024 •

edited

Loading

cmsbuild commented May 29, 2024 •

edited

Loading

perrotta commented Jun 10, 2024 •

edited

Loading