Standalone version of beamspot DIP client [12_3_X]

[sikler]

PR description:

The DQM client version was alive only for the duration of the run, and DIP consumers want a steady presence of the beamspot product. This way, the client is now transfered back to a standalone executable, running forever, polling every second some beamspot and tracker status txt files on /nfshome0. To be placed likely in the startup of a local P5 machine, from where both the CMS-related DIP machine (cmsdimns1.cern.ch, cmsdimns2.cern.ch) and /nfshome0 are both visible and accessible.

PR validation:

Tested successfully on fu-c2f11-11-03, by looking at DIP-related pages to check proper placement of publications:
https://cmsonline.cern.ch/webcenter/portal/cmsonline/pages_common/dcs/dipbrowser [under dip/CMS/...]
https://ccde.cern.ch/nxcals/device/313400?subscriptionId=29210033
https://ccde.cern.ch/nxcals/device/1731503?subscriptionId=29407975
https://timber.cern.ch/query?tab=Visualization&configuration=38326

PR issues:

The process is polling two files:
/nfshome0/dqmpro/BeamMonitorDQM/BeamFitResults.txt
/nfshome0/dqmpro/BeamMonitorDQM/BeamFitResults_TkStatus.txt
where the latter has a very old timestamp (from 2018), so very likely the file related to tracker status is not written by any client. Any ideas?

[cmsbuild]

A new Pull Request was created by @sikler (Ferenc Siklr) for CMSSW_12_3_X.

It involves the following packages:

• DQM/BeamMonitor (dqm)

@emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

• Backported from Standalone version of beamspot DIP client [12_4_X] #37835

[cmsbuild]

Pull request #37817 was updated. @emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

[pmandrik]

Hi, @sikler ,

the only place of *TkStatus.txt files creation I found is [0] called by beam client [1] where DIPFileName is 'BeamFitResultsOld.txt' at the moment. Probably beamhlt process workflow (where 'BeamFitResults.txt' is used) is needed to be updated with cms.EDAnalyzer("TKStatus").

[0] https://github.com/cms-sw/cmssw/blob/master/DQM/BeamMonitor/plugins/TKStatus.cc
[1]

cmssw/DQM/Integration/python/clients/beam_dqm_sourceclient-live_cfg.py

Line 240 in d302344

DIPFileName = process.dqmBeamMonitor.BeamFitter.DIPFileName

[emanueleusai]

please test

[cmsbuild]

Pull request #37817 was updated. @malbouis, @pmandrik, @emanueleusai, @ahmad3213, @tvami, @jfernan2, @ggovi, @francescobrivio, @micsucmed, @rvenditti can you please check and sign again.

[smuzaffar]

@sikler , is it already in 12.4.X/master Ib?

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e84e80/24478/summary.html
COMMIT: c20ae79
CMSSW: CMSSW_12_3_X_2022-05-05-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37817/24478/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3696087
• DQMHistoTests: Total failures: 13
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3696051
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 48 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 204 log files, 45 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[sikler]

@sikler , is it already in 12.4.X/master Ib?

Hi @smuzaffar, there is now a forward port to master at #37835.

[tvami]

backport of #37835

[emanueleusai]

testing at P5

[cmsbuild]

Pull request #37817 was updated. @malbouis, @pmandrik, @emanueleusai, @ahmad3213, @tvami, @cmsbuild, @jfernan2, @ggovi, @francescobrivio, @micsucmed, @rvenditti can you please check and sign again.

[cmsbuild]

-1

Failed Tests: AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e84e80/24536/summary.html
COMMIT: 306732b
CMSSW: CMSSW_12_3_X_2022-05-09-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37817/24536/install.sh to create a dev area with all the needed externals and cmssw changes.

AddOn Tests

• unknownAddOnTest might have timed out: FAILED - secs

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3696087
• DQMHistoTests: Total failures: 8
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3696057
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 204 log files, 45 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[francescobrivio]

Hi Ferenc (@sikler) could you add an .sh script to be used to steer this code (if you have it available already of course)? Maybe something similar to the old java application in https://github.com/cms-sw/cmssw/blob/master/DQM/BeamMonitor/test/dip/runtest.sh ?

[cmsbuild]

Pull request #37817 was updated. @malbouis, @pmandrik, @emanueleusai, @ahmad3213, @tvami, @cmsbuild, @jfernan2, @ggovi, @francescobrivio, @micsucmed, @rvenditti can you please check and sign again.

[sikler]

Hi Ferenc (@sikler) could you add an .sh script to be used to steer this code (if you have it available already of course)? Maybe something similar to the old java application in https://github.com/cms-sw/cmssw/blob/master/DQM/BeamMonitor/test/dip/runtest.sh ?

Hi @francescobrivio , runtest.sh added to this and to master. Now possible to pass parameters.

[tvami]

@cmsbuild , please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e84e80/24579/summary.html
COMMIT: f61ebc1
CMSSW: CMSSW_12_3_X_2022-05-10-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37817/24579/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3696087
• DQMHistoTests: Total failures: 13
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3696051
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -0.004 KiB( 48 files compared)
• DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
• Checked 204 log files, 45 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[francescobrivio]

+db

[emanueleusai]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_3_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_4_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1