[12_5_X] Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tracking #39975

francescobrivio · 2022-11-03T13:09:24Z

PR description:

Backport of #39988

Following the addition of a new Era (Run3_pp_on_PbPb_approxSiStripClusters) in #39863, this PR updates the Era in the online DQM clients that use tracking in order to properly cope with the approxSiStripClusters.
@mmusich please advise if I forgot any, or if other modifications are needed

@cms-sw/dqm-l2 I believe this is the only way to properly handle the HI run, but if you have a better plan please let me know and I can update the PR

- Note1 -
Unfortunately I don't know if a cms.Process() can be customized with a different Era (e.g. following a check on the runKey) after it has been initialized with another one (I guess not, but maybe @makortel @Dr15Jones can advise), so the hardcoded solution in this PR is to completely change the Era to the approxSiStripClusters one.
Hence this PR should not be merged, but only applied online in DQM machines in P5 for the HI data taking period (currently scheduled for 17-18^th November).
EDIT: I'm now using the sys.argv to determine the run key and set the proper Era when the cms.Process() is declared,
so this PR should be good for both HI and pp collisions and thus can be merged.

- Note2 -
Visualization clients make use of scenarios, they will be modified in a different PR which includes a new scenario (working on it right now).

PR validation:

Tested locally with:
cmsRun DQM/Integration/python/clients/beam_dqm_sourceclient-live_cfg.py unitTest=True runkey=hi_run
with the HI input streamer files provided in this CMSTalk thread.

Backport:

Backport of #39988

cmsbuild · 2022-11-03T13:09:57Z

A new Pull Request was created by @francescobrivio for CMSSW_12_5_X.

It involves the following packages:

DQM/Integration (dqm)

@emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@threus, @batinkov, @battibass this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Backported from Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tracking #39988

francescobrivio · 2022-11-03T13:15:37Z

FYI @cms-sw/alca-l2 @cms-sw/tracking-pog-l2 @mandrenguyen

rvenditti · 2022-11-03T13:35:28Z

please test

Dr15Jones · 2022-11-03T13:42:44Z

Unfortunately I don't know if a cms.Process() can be customized with a different Era (e.g. following a check on the runKey) after it has been initialized with another one

It can not be. The Modifiers (which is what Era's use) are applied as we load python modules so one can not change what Modifiers are to be used once the Process has started to be built.

francescobrivio · 2022-11-03T14:15:58Z

Unfortunately I don't know if a cms.Process() can be customized with a different Era (e.g. following a check on the runKey) after it has been initialized with another one

It can not be. The Modifiers (which is what Era's use) are applied as we load python modules so one can not change what Modifiers are to be used once the Process has started to be built.

Ok got it! thanks Chris!
@cms-sw/dqm-l2 then I believe this PR is the only way we have to apply the proper era for HI online.

cmsbuild · 2022-11-03T17:59:05Z

-1

Failed Tests: UnitTests RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f0daa8/28773/summary.html
COMMIT: ca3bd49
CMSSW: CMSSW_12_5_X_2022-11-03-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39975/28773/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOnlineClient-beam_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-sistrip_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-pixel_dqm_sourceclient had ERRORS

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 17 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3734344
DQMHistoTests: Total failures: 1010
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3733312
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 216 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

francescobrivio · 2022-11-03T18:05:14Z

-1

Failed Tests: UnitTests RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f0daa8/28773/summary.html COMMIT: ca3bd49 CMSSW: CMSSW_12_5_X_2022-11-03-1100/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39975/28773/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOnlineClient-beam_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-sistrip_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-pixel_dqm_sourceclient had ERRORS

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

Summary:
* No significant changes to the logs found

* Reco comparison results: 17 differences found in the comparisons

* DQMHistoTests: Total files compared: 51

* DQMHistoTests: Total histograms compared: 3734344

* DQMHistoTests: Total failures: 1010

* DQMHistoTests: Total nulls: 0

* DQMHistoTests: Total successes: 3733312

* DQMHistoTests: Total skipped: 22

* DQMHistoTests: Total Missing objects: 0

* DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)

* Checked 216 log files, 49 edm output root files, 51 DQM output files

* [TriggerResults](https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_5_X_2022-11-03-1100+f0daa8/53887/triggerResults): no differences found

Ok as expected this does not run on old data, but @makortel suggested privately a way around this.
I will test it and push it as soon as I manage.

…cking

cmsbuild · 2022-11-04T15:06:01Z

Pull request #39975 was updated. @emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

francescobrivio · 2022-11-04T15:13:05Z

In the last push I'm using the sys.argvto get the running conditions (hi_run for the HI collisions) and set the correct Era
to be used for HI or for pp.

With this last commit tests should pass and this PR can even be integrated in the release, provided @cms-sw/dqm-l2 agrees. I will also open the master PR.

francescobrivio · 2022-11-04T15:14:33Z

@cmsbuild please test

mmusich · 2022-11-04T15:15:28Z

Hence this PR should not be merged, but only applied online in DQM machines in P5 for the HI data taking period (currently scheduled for 17-18th November).

I guess this doesn't apply anymore.

francescobrivio · 2022-11-04T15:16:59Z

Hence this PR should not be merged, but only applied online in DQM machines in P5 for the HI data taking period (currently scheduled for 17-18th November).

I guess this doesn't apply anymore.

I just updated the description :D

DQM/Integration/python/clients/sistriplas_dqm_sourceclient-live_cfg.py

francescobrivio · 2022-11-04T15:40:21Z

backport of #39988

cmsbuild · 2022-11-04T17:53:50Z

Pull request #39975 was updated. @emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

francescobrivio · 2022-11-07T09:54:44Z

@cmsbuild please test

francescobrivio · 2022-11-07T12:18:21Z

urgent

needed in the next 12_5_X release for HI data-taking

cmsbuild · 2022-11-07T14:00:40Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f0daa8/28819/summary.html
COMMIT: 33c8378
CMSSW: CMSSW_12_5_X_2022-11-06-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39975/28819/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3734658
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3734628
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 50 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 216 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

emanueleusai · 2022-11-07T23:40:00Z

PR added on top of 12_5_2 to online machines and tested successfully during the HI run

emanueleusai · 2022-11-07T23:40:06Z

+1

cmsbuild · 2022-11-07T23:40:23Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_5_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_6_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-11-09T02:09:12Z

+1

cmsbuild added this to the CMSSW_12_5_X milestone Nov 3, 2022

cmsbuild added dqm-pending orp-pending pending-signatures tests-pending labels Nov 3, 2022

cmsbuild added tests-started and removed tests-pending labels Nov 3, 2022

cmsbuild added tests-rejected and removed tests-started labels Nov 3, 2022

Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tra…

2b94df4

…cking

francescobrivio force-pushed the alca-dqm_clients_hi_12_5_X branch from ca3bd49 to 2b94df4 Compare November 4, 2022 15:05

cmsbuild added tests-pending and removed tests-rejected labels Nov 4, 2022

francescobrivio changed the title ~~[Do not merge] Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM online clients~~ Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tracking Nov 4, 2022

cmsbuild added tests-started and removed tests-pending labels Nov 4, 2022

mmusich reviewed Nov 4, 2022

View reviewed changes

DQM/Integration/python/clients/sistriplas_dqm_sourceclient-live_cfg.py Show resolved Hide resolved

francescobrivio changed the title ~~Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tracking~~ [12_5_X] Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tracking Nov 4, 2022

cmsbuild added the backport label Nov 4, 2022

add onlineMetaDataInputTag in beamhlt DQM client for HI running

33c8378

cmsbuild added tests-pending and removed tests-started labels Nov 4, 2022

cmsbuild added tests-started and removed tests-pending labels Nov 7, 2022

cmsbuild added the urgent label Nov 7, 2022

cmsbuild added tests-approved and removed tests-started labels Nov 7, 2022

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Nov 7, 2022

cmsbuild added backport-ok and removed backport labels Nov 8, 2022

cmsbuild added orp-approved and removed orp-pending labels Nov 9, 2022

cmsbuild merged commit 3aaac74 into cms-sw:CMSSW_12_5_X Nov 9, 2022

francescobrivio deleted the alca-dqm_clients_hi_12_5_X branch November 14, 2022 17:22

missirol mentioned this pull request Nov 20, 2022

Crash of visualization-live-secondInstance DQM client in run-362243 #40120

Open

francescobrivio mentioned this pull request Jan 3, 2023

Restore DQM live clients process names #40421

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[12_5_X] Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tracking #39975

[12_5_X] Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tracking #39975

francescobrivio commented Nov 3, 2022 •

edited

Loading

cmsbuild commented Nov 3, 2022 •

edited

Loading

francescobrivio commented Nov 3, 2022

rvenditti commented Nov 3, 2022

Dr15Jones commented Nov 3, 2022

francescobrivio commented Nov 3, 2022

cmsbuild commented Nov 3, 2022

francescobrivio commented Nov 3, 2022

Unit Tests

RelVals-INPUT

Comparison Summary

cmsbuild commented Nov 4, 2022

francescobrivio commented Nov 4, 2022

francescobrivio commented Nov 4, 2022

mmusich commented Nov 4, 2022

francescobrivio commented Nov 4, 2022

francescobrivio commented Nov 4, 2022

cmsbuild commented Nov 4, 2022

francescobrivio commented Nov 7, 2022

francescobrivio commented Nov 7, 2022

cmsbuild commented Nov 7, 2022

emanueleusai commented Nov 7, 2022

emanueleusai commented Nov 7, 2022

cmsbuild commented Nov 7, 2022

perrotta commented Nov 9, 2022

[12_5_X] Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tracking #39975

[12_5_X] Use Era Run3_pp_on_PbPb_approxSiStripClusters in DQM clients with tracking #39975

Conversation

francescobrivio commented Nov 3, 2022 • edited Loading

PR description:

PR validation:

Backport:

cmsbuild commented Nov 3, 2022 • edited Loading

francescobrivio commented Nov 3, 2022

rvenditti commented Nov 3, 2022

Dr15Jones commented Nov 3, 2022

francescobrivio commented Nov 3, 2022

cmsbuild commented Nov 3, 2022

Unit Tests

RelVals-INPUT

Comparison Summary

francescobrivio commented Nov 3, 2022

Unit Tests

RelVals-INPUT

Comparison Summary

cmsbuild commented Nov 4, 2022

francescobrivio commented Nov 4, 2022

francescobrivio commented Nov 4, 2022

mmusich commented Nov 4, 2022

francescobrivio commented Nov 4, 2022

francescobrivio commented Nov 4, 2022

cmsbuild commented Nov 4, 2022

francescobrivio commented Nov 7, 2022

francescobrivio commented Nov 7, 2022

cmsbuild commented Nov 7, 2022

Comparison Summary

emanueleusai commented Nov 7, 2022

emanueleusai commented Nov 7, 2022

cmsbuild commented Nov 7, 2022

perrotta commented Nov 9, 2022

francescobrivio commented Nov 3, 2022 •

edited

Loading

cmsbuild commented Nov 3, 2022 •

edited

Loading