Add DMQ FakeBeamMonitor plugin and clients #30690

francescobrivio · 2020-07-14T20:06:01Z

PR description:

Added one new plugin under DQM/BeamMonitor: DQM/BeamMonitor/plugins/FakeBeamMonitor.cc
- It's almost a copy of the BeamMonitor.cc plugin, except that the BeamSpot is not fitted from tracks and PVs, but is randomly generated using a TRandom3.
- No change in the undelying logic with respect to the BeamMonitor is added
- This plugin should be used when tracks/PVs are not available in the event, but a random BeamSpot is still needed (e.g. during MWGR)
Added two clients (basically a copy of the already existing beamspot clients) that run the new FakeBeamMonitor plugin:
- DQM/Integration/python/clients/beamfake_dqm_sourceclient-live_cfg.py
- DQM/Integration/python/clients/beamhltfake_dqm_sourceclient-live_cfg.py

PR validation:

The new plugin and clients have been tested "offline" by me, @gennai and @ggovi using and "EmpySource" and thanks to @smorovic who was running a DAQ with fake runs.
Two customizations are needed in the clients:

the parameter authenticationPath (lines 358 of beamfake_dqm_sourceclient-live_cfg.py and 160 of beamhltfake_dqm_sourceclient-live_cfg.py ) should contain the path to the db.key file, needed for the upload to CondDB. A copy of this key can be found at P5 under ~govi/bs_key/.cms_cond (@ggovi can confirm).
the parameter lastLumiUrl (lines 367 of beamfake_dqm_sourceclient-live_cfg.py and 169 of beamhltfake_dqm_sourceclient-live_cfg.py ) was working fine at the moment of the test, but should be double checked.

Backporting

A backport to CMSSW_11_1_X might be needed.

cmsbuild · 2020-07-14T20:06:20Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-07-14T20:13:17Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30690/16982

This PR adds an extra 40KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/BeamMonitor/plugins/BuildFile.xml modified in PR(s): Clean BuildFiles in and after Reco area #30679

cmsbuild · 2020-07-14T20:13:39Z

A new Pull Request was created by @francescobrivio for master.

It involves the following packages:

DQM/BeamMonitor
DQM/Integration

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@threus, @batinkov, @battibass this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

jfernan2 · 2020-07-14T20:41:59Z

please test

cmsbuild · 2020-07-14T20:42:19Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-07-14-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/7941/console Started: 2020/07/14 22:44

cmsbuild · 2020-07-14T22:01:47Z

+1
Tested at: dfaab53
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03e003/7941/summary.html
CMSSW: CMSSW_11_2_X_2020-07-14-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-07-14T22:01:50Z

Comparison job queued.

cmsbuild · 2020-07-15T02:32:59Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03e003/7941/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2787429
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2787378
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 154 log files, 17 edm output root files, 37 DQM output files

andrius-k · 2020-07-15T08:27:51Z

Hello,
Could you please provide a backport of this PR to 11_1_X for us (DQM) to test it in the playback system?
Regards,
Andrius

francescobrivio · 2020-07-15T08:29:01Z

Sorry this was not supposed to be closed, I pressed the button by mistake. Should be re-opened now.

francescobrivio · 2020-07-15T08:30:24Z

Hello,
Could you please provide a backport of this PR to 11_1_X for us (DQM) to test it in the playback system?
Regards,
Andrius

I will open the backport.

jfernan2 · 2020-07-17T09:33:51Z

@francescobrivio @gennai there were changes to this PR made by @andrius-k yesterday in the test done at the MWGR, as you suggested. We should include them in this PR, right? If we are expecting to make similar test in the future or actually use the code in online you should push these changes to the PR. Andrius can provide the diff if needed. Thanks

cmsbuild · 2020-07-17T10:32:09Z

+1
Tested at: 1992d5b
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03e003/8045/summary.html
CMSSW: CMSSW_11_2_X_2020-07-16-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-07-17T10:32:12Z

Comparison job queued.

francescobrivio · 2020-07-17T10:35:11Z

@francescobrivio @gennai there were changes to this PR made by @andrius-k yesterday in the test done at the MWGR, as you suggested. We should include them in this PR, right? If we are expecting to make similar test in the future or actually use the code in online you should push these changes to the PR. Andrius can provide the diff if needed. Thanks

I don't think this code is meant to be used in online (@gennai please correct me if I'm wrong), but we might want to use it again in similar tests. So if you can provide the diff I will push those changes as well.

andrius-k · 2020-07-17T11:05:34Z

Hi @francescobrivio,
I can see that you pushed another commit yesterday after we deployed the PR to P5. The diff between this PR (from the second commit) and what's currently running in P5 is the following:

diff --git a/CondCore/DBOutputService/src/OnlineDBOutputService.cc b/CondCore/DBOutputService/src/OnlineDBOutputService.cc
index 9f8065c5f44..c1ec99522e9 100644
--- a/CondCore/DBOutputService/src/OnlineDBOutputService.cc
+++ b/CondCore/DBOutputService/src/OnlineDBOutputService.cc
@@ -11,8 +11,9 @@ cond::service::OnlineDBOutputService::OnlineDBOutputService(const edm::Parameter
       m_preLoadConnectionString(iConfig.getUntrackedParameter<std::string>("preLoadConnectionString", "")),
       m_debug(iConfig.getUntrackedParameter<bool>("debugLogging", false)) {
   if (!m_lastLumiUrl.empty()) {
-    startTransaction();
-    m_runNumber = PoolDBOutputService::session().getCurrentRun().run;
+    //startTransaction();
+    //m_runNumber = PoolDBOutputService::session().getCurrentRun().run;
+    m_runNumber = iConfig.getUntrackedParameter<unsigned long long>("runNumber",100000);
   } else {
     m_lastLumiFile = iConfig.getUntrackedParameter<std::string>("lastLumiFile", "");
     //if( m_lastLumiFile.size() == 0 ){
diff --git a/DQM/Integration/python/clients/beamfake_dqm_sourceclient-live_cfg.py b/DQM/Integration/python/clients/beamfake_dqm_sourceclient-live_cfg.py
index 5cc6468f4a2..704ddcad8f6 100644
--- a/DQM/Integration/python/clients/beamfake_dqm_sourceclient-live_cfg.py
+++ b/DQM/Integration/python/clients/beamfake_dqm_sourceclient-live_cfg.py
@@ -34,6 +34,8 @@ noDB = True
 if 'noDB=False' in sys.argv:
     noDB=False

+noDB = False
+
 #---------------
 # Input sources
 if unitTest:
@@ -363,8 +365,8 @@ process.OnlineDBOutputService = cms.Service("OnlineDBOutputService",
     preLoadConnectionString = cms.untracked.string('frontier://FrontierProd/CMS_CONDITIONS'),

     runNumber = cms.untracked.uint64(options.runNumber),
-    lastLumiFile = cms.untracked.string('last_lumi.txt'),
-    #lastLumiUrl = cms.untracked.string('http://ru-c2e14-11-01.cms:11100/urn:xdaq-application:lid=52/getLatestLumiSection'),
+    #lastLumiFile = cms.untracked.string('last_lumi.txt'),
+    lastLumiUrl = cms.untracked.string('http://ru-c2e14-11-01.cms:11100/urn:xdaq-application:lid=52/getLatestLumiSection'),
     writeTransactionDelay = cms.untracked.uint32(options.transDelay),
     latency = cms.untracked.uint32(2),
     autoCommit = cms.untracked.bool(True),

cmsbuild · 2020-07-17T12:32:02Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03e003/8045/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2612943
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2612888
DQMHistoTests: Total skipped: 48
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 146 log files, 17 edm output root files, 35 DQM output files

francescobrivio · 2020-07-17T12:44:49Z

Hi @francescobrivio,
I can see that you pushed another commit yesterday after we deployed the PR to P5. The diff between this PR (from the second commit) and what's currently running in P5 is the following:

diff --git a/CondCore/DBOutputService/src/OnlineDBOutputService.cc b/CondCore/DBOutputService/src/OnlineDBOutputService.cc
index 9f8065c5f44..c1ec99522e9 100644
--- a/CondCore/DBOutputService/src/OnlineDBOutputService.cc
+++ b/CondCore/DBOutputService/src/OnlineDBOutputService.cc
@@ -11,8 +11,9 @@ cond::service::OnlineDBOutputService::OnlineDBOutputService(const edm::Parameter
       m_preLoadConnectionString(iConfig.getUntrackedParameter<std::string>("preLoadConnectionString", "")),
       m_debug(iConfig.getUntrackedParameter<bool>("debugLogging", false)) {
   if (!m_lastLumiUrl.empty()) {
-    startTransaction();
-    m_runNumber = PoolDBOutputService::session().getCurrentRun().run;
+    //startTransaction();
+    //m_runNumber = PoolDBOutputService::session().getCurrentRun().run;
+    m_runNumber = iConfig.getUntrackedParameter<unsigned long long>("runNumber",100000);
   } else {
     m_lastLumiFile = iConfig.getUntrackedParameter<std::string>("lastLumiFile", "");
     //if( m_lastLumiFile.size() == 0 ){
diff --git a/DQM/Integration/python/clients/beamfake_dqm_sourceclient-live_cfg.py b/DQM/Integration/python/clients/beamfake_dqm_sourceclient-live_cfg.py
index 5cc6468f4a2..704ddcad8f6 100644
--- a/DQM/Integration/python/clients/beamfake_dqm_sourceclient-live_cfg.py
+++ b/DQM/Integration/python/clients/beamfake_dqm_sourceclient-live_cfg.py
@@ -34,6 +34,8 @@ noDB = True
 if 'noDB=False' in sys.argv:
     noDB=False

+noDB = False
+
 #---------------
 # Input sources
 if unitTest:
@@ -363,8 +365,8 @@ process.OnlineDBOutputService = cms.Service("OnlineDBOutputService",
     preLoadConnectionString = cms.untracked.string('frontier://FrontierProd/CMS_CONDITIONS'),

     runNumber = cms.untracked.uint64(options.runNumber),
-    lastLumiFile = cms.untracked.string('last_lumi.txt'),
-    #lastLumiUrl = cms.untracked.string('http://ru-c2e14-11-01.cms:11100/urn:xdaq-application:lid=52/getLatestLumiSection'),
+    #lastLumiFile = cms.untracked.string('last_lumi.txt'),
+    lastLumiUrl = cms.untracked.string('http://ru-c2e14-11-01.cms:11100/urn:xdaq-application:lid=52/getLatestLumiSection'),
     writeTransactionDelay = cms.untracked.uint32(options.transDelay),
     latency = cms.untracked.uint32(2),
     autoCommit = cms.untracked.bool(True),

thanks @andrius-k !
I prefer not to touch the CondCore package, since @ggovi wrote that particular module and he might want to work more on that.
About the fake beamspot clients:

the lastLumiFile/Url change has alrady been implemented in: 1992d5b
the parameter noDB = False instead should be read from command line, was there a problem with that?

andrius-k · 2020-07-21T09:37:54Z

Hi @francescobrivio
No problem!
Regarding the noDB argument, unfortunately we can't selectively pass in different arguments to different Online clients. Is it possible for you to set the default value to whatever you would like it to be when running in Online production?

jfernan2 · 2020-07-23T11:19:40Z

@francescobrivio @ggovi could you clarify a but about the noDB parameter, in particular:

we cannot run selectively as @andrius-k said, except for limited tests for which we understand the FakeBeamMonitor is intended for, at least in some circustances
for MWGRs, we understand that noDB should be then equal to False, i.e. the DB will be accesible, a run and a LS are available from DAQ, is this correct?
noDB = True should be used for "offline" tests with no live mode or UnitTest mode, or any other playback mode to not overwrite the DB?
Thanks

silviodonato · 2020-07-27T15:02:28Z

kind reminder @francescobrivio

gennai · 2020-07-27T15:10:20Z

Hi, I can answer on the above points. noDB means that a sql file is created instead of uploading the new payload to the Prod condition DB. You are basically right.
noDB= True should be used for unit test and other local tests (including playback). For MWGR noDB must be = False as we do want to upload the payload to the database.

francescobrivio · 2020-07-27T15:10:37Z

Sorry for the delay. Please see the inline answers.

@francescobrivio @ggovi could you clarify a but about the noDB parameter, in particular:
* we cannot run selectively as @andrius-k said, except for limited tests for which we understand the FakeBeamMonitor is intended for, at least in some circustances

Not sure I understand the run selectively part. Can't this option be bassed from command line?

* for MWGRs, we understand that noDB should be then equal to False, i.e. the DB will be accesible, a run and a LS are available from DAQ, is this correct?

This is correct

* noDB = True should be used for "offline" tests with no live mode or UnitTest mode, or any other playback mode to not overwrite the DB?

Correct.

The noDB = False option (access the DB and upload conditions) should only be used for specific tests (e.g. during MWGR) or in normal data taking periods, as you can see in the "normal" BeamSpot client here

jfernan2 · 2020-07-27T15:48:54Z

Thanks @francescobrivio @gennai
So, the default config we want is noDB = True to avoid real uploading (as it is the PR right now).

noDB = False will only be used on very dedicated ocasions, since DQM core will have to pass it manually by command line on a very limited number of runs. We have to manually switch it off on the client running at P5, which is error prone and not desired, that's what I meant by run selectively

Correct?

gennai · 2020-07-27T15:52:02Z

Thanks @francescobrivio @gennai
So, the default config we want is noDB = True to avoid real uploading (as it is the PR right now).

noDB = False will only be used on very dedicated ocasions, since DQM core will have to pass it manually by command line on a very limited number of runs. We have to manually switch it off on the client running at P5, which is error prone and not desired, that's what I meant by run selectively

Correct?

yes.

jfernan2 · 2020-07-27T15:56:14Z

+1

cmsbuild · 2020-07-27T15:56:37Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-07-27T20:40:55Z

+1

jfernan2 · 2020-07-28T10:51:10Z

@francescobrivio we need a new PR in Master with
lastLumiFile = cms.untracked.string('last_lumi.txt'),
to make this PR working out of the box as commented in
#30696 (comment)
Thanks

Add DMQ FakeBeamMonitor plugin and clients

dfaab53

cmsbuild added this to the CMSSW_11_2_X milestone Jul 14, 2020

cmsbuild added code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Jul 14, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 14, 2020

cmsbuild added tests-started and removed tests-pending labels Jul 14, 2020

cmsbuild added tests-approved and removed tests-started labels Jul 14, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jul 15, 2020

francescobrivio closed this Jul 15, 2020

francescobrivio reopened this Jul 15, 2020

francescobrivio mentioned this pull request Jul 15, 2020

Add DMQ FakeBeamMonitor plugin and clients - backport to 11_1_X #30696

Merged

cmsbuild added tests-approved and removed tests-started labels Jul 17, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jul 17, 2020

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Jul 27, 2020

cmsbuild added orp-approved and removed orp-pending labels Jul 27, 2020

cmsbuild merged commit 2e95b4e into cms-sw:master Jul 27, 2020

francescobrivio mentioned this pull request Jul 28, 2020

Fix lastLumiFile parameter in fake BeamSpot clients #30943

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DMQ FakeBeamMonitor plugin and clients #30690

Add DMQ FakeBeamMonitor plugin and clients #30690

francescobrivio commented Jul 14, 2020 •

edited

Loading

cmsbuild commented Jul 14, 2020

cmsbuild commented Jul 14, 2020

cmsbuild commented Jul 14, 2020

jfernan2 commented Jul 14, 2020

cmsbuild commented Jul 14, 2020 •

edited

Loading

cmsbuild commented Jul 14, 2020

cmsbuild commented Jul 14, 2020

cmsbuild commented Jul 15, 2020

andrius-k commented Jul 15, 2020

francescobrivio commented Jul 15, 2020 •

edited

Loading

francescobrivio commented Jul 15, 2020

jfernan2 commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

francescobrivio commented Jul 17, 2020

andrius-k commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

francescobrivio commented Jul 17, 2020

andrius-k commented Jul 21, 2020

jfernan2 commented Jul 23, 2020

silviodonato commented Jul 27, 2020

gennai commented Jul 27, 2020

francescobrivio commented Jul 27, 2020

jfernan2 commented Jul 27, 2020 •

edited

Loading

gennai commented Jul 27, 2020

jfernan2 commented Jul 27, 2020

cmsbuild commented Jul 27, 2020

silviodonato commented Jul 27, 2020

jfernan2 commented Jul 28, 2020

Add DMQ FakeBeamMonitor plugin and clients #30690

Add DMQ FakeBeamMonitor plugin and clients #30690

Conversation

francescobrivio commented Jul 14, 2020 • edited Loading

PR description:

PR validation:

Backporting

cmsbuild commented Jul 14, 2020

cmsbuild commented Jul 14, 2020

cmsbuild commented Jul 14, 2020

jfernan2 commented Jul 14, 2020

cmsbuild commented Jul 14, 2020 • edited Loading

cmsbuild commented Jul 14, 2020

cmsbuild commented Jul 14, 2020

cmsbuild commented Jul 15, 2020

andrius-k commented Jul 15, 2020

francescobrivio commented Jul 15, 2020 • edited Loading

francescobrivio commented Jul 15, 2020

jfernan2 commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

francescobrivio commented Jul 17, 2020

andrius-k commented Jul 17, 2020

cmsbuild commented Jul 17, 2020

francescobrivio commented Jul 17, 2020

andrius-k commented Jul 21, 2020

jfernan2 commented Jul 23, 2020

silviodonato commented Jul 27, 2020

gennai commented Jul 27, 2020

francescobrivio commented Jul 27, 2020

jfernan2 commented Jul 27, 2020 • edited Loading

gennai commented Jul 27, 2020

jfernan2 commented Jul 27, 2020

cmsbuild commented Jul 27, 2020

silviodonato commented Jul 27, 2020

jfernan2 commented Jul 28, 2020

francescobrivio commented Jul 14, 2020 •

edited

Loading

cmsbuild commented Jul 14, 2020 •

edited

Loading

francescobrivio commented Jul 15, 2020 •

edited

Loading

jfernan2 commented Jul 27, 2020 •

edited

Loading