Add ParticleFlow Client for Online DQM - GPUvsCPU comparison

[waredjeb]

This PR creates a new Online DQM client for Particle Flow, taking inspiration from the HCAL GPU client. Currently the PF client will be used for monitoring PFCluster@Alpaka, comparing the GPU with the CPU version from the DQMGPUvsCPU stream.

Test

Local test on lxplus with the following command, following instructions on the DQM Twiki
cmsRun DQM/Integration/python/clients/hcalgpu_dqm_sourceclient-live_cfg.py runInputDir=/eos/cms/store/group/comm_dqm/ runNumber=380649 runkey=pp_run scanOnce=True

Just for completeness, adding some of the plots produced by the client

Backport

Probably a backport to CMSSW_14_0_X will be needed

@missirol @swagata87 @stahlleiton @hatakeyamak @jsamudio

[cmsbuild]

cms-bot internal usage

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45079/40395

• This PR adds an extra 20KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-format:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45079/40395/code-format.patch
  e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45079/40395/code-format.patch | patch -p1
  You can also run scram build code-format to apply code format directly

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45079/40397

• This PR adds an extra 20KB to repository

[cmsbuild]

A new Pull Request was created by @waredjeb for master.

It involves the following packages:

• DQM/Integration (dqm)
• DQM/PFTasks (****)

The following packages do not have a category, yet:

DQM/PFTasks
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@syuvivida, @rvenditti, @nothingface0, @cmsbuild, @tjavaid, @antoniovagnerini can you please review it and eventually sign? Thanks.
@francescobrivio, @batinkov, @threus this is something you requested to watch as well.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[mmusich]

I think elsewhere the Host vs Device nomenclature was preferred

[mmusich]

could this (and other scatter plots) become a TH2I instead? (method book2I)

[swagata87]

type pf

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45079/40409

• This PR adds an extra 24KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-format:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45079/40409/code-format.patch
  e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45079/40409/code-format.patch | patch -p1
  You can also run scram build code-format to apply code format directly

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45079/40410

• This PR adds an extra 20KB to repository

[syuvivida]

Hi @mmusich
If we want the online shifters to check the pixel GPU clients results, the Tracker group needs to update/implement the instruction twiki page, and also include the plots in the shift page of DQMGUI.

[mmusich]

@syuvivida

If we want the online shifters to check the pixel GPU clients results, the Tracker group needs to update/implement the instruction twiki page, and also include the plots in the shift page of DQMGUI.

thanks but this doesn't answer the general question. E.g. for this client, does the PF group plan in providing such instructions etc.?

[waredjeb]

@syuvivida

If we want the online shifters to check the pixel GPU clients results, the Tracker group needs to update/implement the instruction twiki page, and also include the plots in the shift page of DQMGUI.

thanks but this doesn't answer the general question. E.g. for this client, does the PF group plan in providing such instructions etc.?

If updating the Twiki is required, we will provide these kinds of instructions.
@syuvivida, could you please send us the Twiki page that needs to be updated and maybe instructions for adding the plots in the DQMGUI? Thanks a lot

[syuvivida]

@syuvivida

If we want the online shifters to check the pixel GPU clients results, the Tracker group needs to update/implement the instruction twiki page, and also include the plots in the shift page of DQMGUI.

thanks but this doesn't answer the general question. E.g. for this client, does the PF group plan in providing such instructions etc.?

If updating the Twiki is required, we will provide these kinds of instructions. @syuvivida, could you please send us the Twiki page that needs to be updated and maybe instructions for adding the plots in the DQMGUI? Thanks a lot

Hello, I will send you these piece of information by email.

[mmusich]

Hello, I will send you these piece of information by email.

can you please keep me in the loop? thx.

[mmusich]

DQM/PFTasks (****)

@waredjeb you also need to make a PR to https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign the (DQM) category.

[waredjeb]

DQM/PFTasks (****)

@waredjeb you also need to make a PR to https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign the (DQM) category.

Sure, I was waiting for the merge of this PR.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-47eeef/39687/summary.html
COMMIT: b9a0564
CMSSW: CMSSW_14_1_X_2024-06-04-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/45079/39687/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 2 lines from the logs
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3338862
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3338836
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[mmusich]

Sure, I was waiting for the merge of this PR.

my understanding is that it needs to be done before merging the PR, so that the corresponding L2 maintainers of the new package can sign-off on that too.

[waredjeb]

Sure, I was waiting for the merge of this PR.

my understanding is that it needs to be done before merging the PR, so that the corresponding L2 maintainers of the new package can sign-off on that too.

Oh ok. I'll open it right now

[waredjeb]

Sure, I was waiting for the merge of this PR.

my understanding is that it needs to be done before merging the PR, so that the corresponding L2 maintainers of the new package can sign-off on that too.

Opened! cms-sw/cms-bot#2253

[mdelcourt]

@syuvivida

If we want the online shifters to check the pixel GPU clients results, the Tracker group needs to update/implement the instruction twiki page, and also include the plots in the shift page of DQMGUI.

thanks but this doesn't answer the general question. E.g. for this client, does the PF group plan in providing such instructions etc.?

If updating the Twiki is required, we will provide these kinds of instructions. @syuvivida, could you please send us the Twiki page that needs to be updated and maybe instructions for adding the plots in the DQMGUI? Thanks a lot

Hello, I will send you these piece of information by email.

Could you forward me this information too?

[tjavaid]

+1

• P5 tests OK
• @waredjeb , could you please create its backport PR for 14_0_X ? Thanks!

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @rappoccio, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

[antoniovilela]

+1