Add PPS DAQ mapping to 13_3_X real data Run2 and Run3 GTs #43226

perrotta · 2023-11-08T17:11:13Z

PR description:

It adds the PPS DAQ mapping to 13_3_X real data GTs.
At the moment, no MC GTs are updated for it, because MC workflows do not consume the added payload yet.

They were announced in this CMSTalk post

=== Data Tags (all added with "hlt" synchronization):

PPSDAQMapping_TimingDiamond_v1
PPSDAQMapping_TrackingStrip_v1
PPSDAQMapping_TotemTiming_v1
PPSDAQMapping_TotemT2_v1
PPSAnalysisMask_v1

=== GT diffs

perrotta · 2023-11-08T17:12:58Z

please test with #43226

cmsbuild · 2023-11-08T17:17:45Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43226/37573

This PR adds an extra 16KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/AlCa/python/autoCond.py modified in PR(s): Add SimBeamSpotObject record to Run-1/2/3 MC GTs #43197

cmsbuild · 2023-11-08T17:18:08Z

A new Pull Request was created by @perrotta (Andrea Perrotta) for master.

It involves the following packages:

Configuration/AlCa (alca)

@perrotta, @saumyaphor4252, @consuegs can you please review it and eventually sign? Thanks.
@yuanchao, @Martin-Grunewald, @missirol, @rsreds, @tocheng, @mmusich, @fabiocos this is something you requested to watch as well.
@rappoccio, @sextonkennedy, @antoniovilela you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2023-11-08T20:43:53Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43226/37575

This PR adds an extra 16KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/AlCa/python/autoCond.py modified in PR(s): Add SimBeamSpotObject record to Run-1/2/3 MC GTs #43197

cmsbuild · 2023-11-08T20:44:15Z

Pull request #43226 was updated. @consuegs, @cmsbuild, @perrotta, @saumyaphor4252 can you please check and sign again.

perrotta · 2023-11-09T14:35:05Z

Waiting for #43197 to get merged first. Then a rebase will be probably needed

francescobrivio · 2023-11-09T21:35:05Z

Then a rebase will be probably needed

Maybe not, given I'm updating MC GTs only, while you are touching data GTs. GH should be (hopefully) smart enough!

perrotta · 2023-11-13T09:19:46Z

Then a rebase will be probably needed

Maybe not, given I'm updating MC GTs only, while you are touching data GTs. GH should be (hopefully) smart enough!

Indeed, github was smart enough to recognize that changes were applied on different lines, even if nearby

perrotta · 2023-11-13T09:20:53Z

please test
(just to test it without the companion #42711)

cmsbuild · 2023-11-13T13:02:34Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5249f5/35760/summary.html
COMMIT: ca55839
CMSSW: CMSSW_14_0_X_2023-11-12-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43226/35760/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 17 errors in the following unit tests:

---> test TestDQMOnlineClient-beamhlt_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-beamhlt_dqm_sourceclient-legacy had ERRORS
---> test TestDQMOnlineClient-ctpps_dqm_sourceclient had ERRORS
and more ...

RelVals

4.22

----- Begin Fatal Exception 13-Nov-2023 10:57:47 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="TotemDAQMapping" label="TimingDiamond"
 from record TotemReadoutRcd. The two providers are 
1) type="PoolDBESSource" label="GlobalTag"
2) type="TotemDAQMappingESSourceXML" label="totemDAQMappingESSourceXML_TimingDiamond"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

138.4

----- Begin Fatal Exception 13-Nov-2023 10:58:31 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="TotemDAQMapping" label="TimingDiamond"
 from record TotemReadoutRcd. The two providers are 
1) type="PoolDBESSource" label="GlobalTag"
2) type="TotemDAQMappingESSourceXML" label="totemDAQMappingESSourceXML_TimingDiamond"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

138.5

----- Begin Fatal Exception 13-Nov-2023 10:58:36 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="TotemDAQMapping" label="TimingDiamond"
 from record TotemReadoutRcd. The two providers are 
1) type="PoolDBESSource" label="GlobalTag"
2) type="TotemDAQMappingESSourceXML" label="totemDAQMappingESSourceXML_TimingDiamond"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

4.224.22_RunCosmics2011A/step2_RunCosmics2011A.log
4.64.6_MinimumBias2010A/step2_MinimumBias2010A.log
134.813134.813_RunCosmics2015C/step2_RunCosmics2015C.log

Expand to see more relval errors ...

AddOn Tests

hlt_data_Fake

----- Begin Fatal Exception 13-Nov-2023 10:59:25 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="TotemDAQMapping" label="TimingDiamond"
 from record TotemReadoutRcd. The two providers are 
1) type="PoolDBESSource" label="GlobalTag"
2) type="TotemDAQMappingESSourceXML" label="totemDAQMappingESSourceXML_TimingDiamond"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

hlt_data_Fake1

----- Begin Fatal Exception 13-Nov-2023 10:59:30 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="TotemDAQMapping" label="TimingDiamond"
 from record TotemReadoutRcd. The two providers are 
1) type="PoolDBESSource" label="GlobalTag"
2) type="TotemDAQMappingESSourceXML" label="totemDAQMappingESSourceXML_TimingDiamond"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

hlt_data_Fake2

----- Begin Fatal Exception 13-Nov-2023 10:59:34 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="TotemDAQMapping" label="TimingDiamond"
 from record TotemReadoutRcd. The two providers are 
1) type="PoolDBESSource" label="GlobalTag"
2) type="TotemDAQMappingESSourceXML" label="totemDAQMappingESSourceXML_TimingDiamond"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

perrotta · 2023-11-13T13:05:41Z

please test with #42711

cmsbuild · 2023-11-13T15:51:46Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5249f5/35763/summary.html
COMMIT: ca55839
CMSSW: CMSSW_14_0_X_2023-11-13-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43226/35763/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 69 lines from the logs
Reco comparison results: 174 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3363028
DQMHistoTests: Total failures: 10
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3362996
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

perrotta · 2023-11-13T16:12:55Z

+alca

It is needed by and must be merged together with PPS - RawToDigi update to handle mappings from CondDB #42711 (once all necessary signatures will get collected in that PR)
Please don't merge this PR alone!

cmsbuild · 2023-11-13T16:13:19Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2023-11-17T14:34:38Z

+1

perrotta · 2023-11-18T08:27:06Z

Configuration/AlCa/python/autoCond.py

    # GlobalTag for Run2 data 2018B relvals only: HEM-15-16 fail
-    'run2_data_HEfail'             :    '133X_dataRun2_HEfail_v1',
+    'run2_data_HEfail'             :    '133X_dataRun2_HEfail_v2',


There was an error of mine while naming the updated DB: I wrote 133X_dataRun2_Hefail_v2 instead of 133X_dataRun2_HEfail_v2. This originates the error in the IBs for (e.g.) wf 136.8642.

I'm going to fix it in the DB, and there should be no need to fix the PR after that!

Now 133X_dataRun2_HEfail_v2 exists in the IB, identical to 133X_dataRun2_Hefail_v2: https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/133X_dataRun2_Hefail_v2/133X_dataRun2_HEfail_v2

I expect that the IB error in 136.8642 will disappear while testing next CMSSW_14_0_X_2023-11-18-1100

Add PPS DAQ mapping to 13_3_X real data GTs

2616b68

perrotta marked this pull request as draft November 8, 2023 17:11

cmsbuild added this to the CMSSW_14_0_X milestone Nov 8, 2023

cmsbuild added alca-pending pending-signatures tests-pending orp-pending code-checks-pending labels Nov 8, 2023

cmsbuild added tests-started requires-external and removed tests-pending labels Nov 8, 2023

perrotta changed the title ~~Add PPS DAQ mapping to 13_3_X real data GTs~~ [DO NOT MERGE] Add PPS DAQ mapping to 13_3_X real data GTs Nov 8, 2023

perrotta marked this pull request as ready for review November 8, 2023 17:13

perrotta mentioned this pull request Nov 8, 2023

PPS - RawToDigi update to handle mappings from CondDB #42711

Merged

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 8, 2023

update also frozen GT

ca55839

cmsbuild added tests-pending code-checks-pending and removed tests-started requires-external code-checks-approved labels Nov 8, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 8, 2023

perrotta mentioned this pull request Nov 9, 2023

Add SimBeamSpotObject record to Run-1/2/3 MC GTs #43197

Merged

perrotta changed the title ~~[DO NOT MERGE] Add PPS DAQ mapping to 13_3_X real data GTs~~ Add PPS DAQ mapping to 13_3_X real data Run2 and Run3 GTs Nov 9, 2023

cmsbuild added tests-started and removed tests-pending labels Nov 13, 2023

cmsbuild added tests-rejected and removed tests-started labels Nov 13, 2023

cmsbuild added requires-external tests-started and removed tests-rejected labels Nov 13, 2023

cmsbuild added tests-approved and removed tests-started labels Nov 13, 2023

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Nov 13, 2023

cmsbuild added orp-approved and removed orp-pending labels Nov 17, 2023

cmsbuild merged commit eacae5b into cms-sw:master Nov 17, 2023

cmsbuild mentioned this pull request Nov 17, 2023

add verbose option to BeamSpotRcdPrinter and use it in the unit tests #43291

Merged

perrotta deleted the gtsForPpsDaqMapping branch November 18, 2023 06:56

perrotta commented Nov 18, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PPS DAQ mapping to 13_3_X real data Run2 and Run3 GTs #43226

Add PPS DAQ mapping to 13_3_X real data Run2 and Run3 GTs #43226

perrotta commented Nov 8, 2023 •

edited

Loading

perrotta commented Nov 8, 2023

cmsbuild commented Nov 8, 2023

cmsbuild commented Nov 8, 2023

cmsbuild commented Nov 8, 2023

cmsbuild commented Nov 8, 2023

perrotta commented Nov 9, 2023

francescobrivio commented Nov 9, 2023

perrotta commented Nov 13, 2023

perrotta commented Nov 13, 2023

cmsbuild commented Nov 13, 2023

perrotta commented Nov 13, 2023

cmsbuild commented Nov 13, 2023

perrotta commented Nov 13, 2023 •

edited

Loading

cmsbuild commented Nov 13, 2023

rappoccio commented Nov 17, 2023

perrotta Nov 18, 2023

perrotta Nov 18, 2023

Add PPS DAQ mapping to 13_3_X real data Run2 and Run3 GTs #43226

Add PPS DAQ mapping to 13_3_X real data Run2 and Run3 GTs #43226

Conversation

perrotta commented Nov 8, 2023 • edited Loading

PR description:

perrotta commented Nov 8, 2023

cmsbuild commented Nov 8, 2023

cmsbuild commented Nov 8, 2023

cmsbuild commented Nov 8, 2023

cmsbuild commented Nov 8, 2023

perrotta commented Nov 9, 2023

francescobrivio commented Nov 9, 2023

perrotta commented Nov 13, 2023

perrotta commented Nov 13, 2023

cmsbuild commented Nov 13, 2023

Unit Tests

RelVals

RelVals-INPUT

AddOn Tests

perrotta commented Nov 13, 2023

cmsbuild commented Nov 13, 2023

Comparison Summary

perrotta commented Nov 13, 2023 • edited Loading

cmsbuild commented Nov 13, 2023

rappoccio commented Nov 17, 2023

perrotta Nov 18, 2023

Choose a reason for hiding this comment

perrotta Nov 18, 2023

Choose a reason for hiding this comment

perrotta commented Nov 8, 2023 •

edited

Loading

perrotta commented Nov 13, 2023 •

edited

Loading