Add `SiPixelRawDataError` monitoring for `gpuValidation` setup #41920

mmusich · 2023-06-11T10:17:56Z

PR description:

Somewhat inspired by the discussion at #41715 (comment), the main goal of this PR is to add a DQMEDAnalyzer (SiPixelPhase1RawDataErrorComparator) designed to monitor the difference in terms of SiPixel Raw data errors in the CPU and GPU branches of the workflows running the gpuValidation modifier.
As there is no SoA-like data structure used in the CPU workflow (it reads directly the legacy digis) at variance with other heterogeneous data products used in the pixel workflow (recHits, tracks, vertices) I designed the module in order to read the legacy data format SiPixelRawDataError (converted from SoA in the GPU workflow and run directly in the CPU one).
I profit of this PR to fix a couple of bugs left from the portable data migration PR:

fix a mistake in one of the input tags in SiPixelCompareRecHitsSoA (commit d65a81a)
use in SiPixelCompareRecHitsSoA the convention Host -> CPU and Device -> GPU as used in other plugins of this package

PR validation:

Successfully run the worfklow runTheMatrix.py -l 10824.507 --command='-n 100'.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, but it might be backported in order to use in the 2023 HLT menus.

Cc:
@sroychow @AdrianoDee

- add comparison histograms - add warning messages, add total FED errors hisogram, add comments on input parametets - add 2D map of GPU/CPU error unbalance vs FEDid / FEDerror type

…onventions in this package

…the gpuValidation modifier is run

mmusich · 2023-06-11T10:20:36Z

test parameters:

enable_tests = gpu

cmsbuild · 2023-06-11T10:23:09Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41920/35878

This PR adds an extra 20KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/SiPixelHeterogeneous/plugins/SiPixelCompareRecHitsSoA.cc modified in PR(s): More Configurable Pixel Tracks and HIon Template #41632
- File DQM/SiPixelHeterogeneous/python/SiPixelHeterogenousDQM_FirstStep_cff.py modified in PR(s): More Configurable Pixel Tracks and HIon Template #41632

cmsbuild · 2023-06-11T10:23:33Z

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

DQM/SiPixelHeterogeneous (dqm)

@nothingface0, @emanueleusai, @cmsbuild, @pmandrik, @syuvivida, @tjavaid, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@idebruyn, @threus, @fioriNTU, @jandrea this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

mmusich · 2023-06-11T10:24:55Z

test parameters:

enable_tests = gpu
workflows_gpu = 11634.503, 11634.506, 11634.583, 11634.587, 10824.507

mmusich · 2023-06-11T10:25:00Z

please test

cmsbuild · 2023-06-11T17:16:47Z

-1

Failed Tests: RelVals-GPU
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2232eb/33084/summary.html
COMMIT: a7311f6
CMSSW: CMSSW_13_2_X_2023-06-11-0000/el8_amd64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41920/33084/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-GPU

ValueError: Undefined workflows: 10824.507

Comparison Summary

Summary:

You potentially added 10 lines to the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3220833
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3220808
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

mmusich · 2023-06-11T18:21:21Z

test parameters:

enable_tests = gpu
workflows_gpu = 10824.507
relvals_opt= -w upgrade
relvals_opt_gpu = -w upgrade

mmusich · 2023-06-12T03:37:46Z

@cmsbuild, please test

cmsbuild · 2023-06-12T06:45:17Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2232eb/33088/summary.html
COMMIT: a7311f6
CMSSW: CMSSW_13_2_X_2023-06-11-2300/el8_amd64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41920/33088/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3190910
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3190885
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

You potentially removed 22 lines from the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 56072
DQMHistoTests: Total failures: 36
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 56036
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 31796.708 KiB( 3 files compared)
DQMHistoSizes: changed ( 12434.587,... ): 15823.444 KiB SiPixelHeterogeneous/PixelErrorCompareGPUvsCPU
DQMHistoSizes: changed ( 12434.587,... ): -8648.893 KiB SiPixelHeterogeneous/PixelRecHitsCompareDevicevsHost
DQMHistoSizes: changed ( 12434.587,... ): 8648.654 KiB SiPixelHeterogeneous/PixelRecHitsCompareGPUvsCPU
DQMHistoSizes: changed ( 12434.587,... ): 37.574 KiB SiPixelHeterogeneous/PixelErrorsCPU
DQMHistoSizes: changed ( 12434.587,... ): 37.574 KiB SiPixelHeterogeneous/PixelErrorsGPU
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

emanueleusai · 2023-06-13T03:04:55Z

+1

cmsbuild · 2023-06-13T03:05:22Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2023-06-13T07:05:24Z

+1

mmusich added 4 commits June 11, 2023 12:09

Add comparator for SiPixel Errors

f4f9c22

- add comparison histograms - add warning messages, add total FED errors hisogram, add comments on input parametets - add 2D map of GPU/CPU error unbalance vs FEDid / FEDerror type

fix mistake on pixelHitsSrcCPU

18fe87e

rename Host -> CPU and Device -> GPU to comply with the rest of the c…

181a4cd

…onventions in this package

add SiPixelPhase1RawDataAnalyzer to the CPU and GPU branches in case …

a7311f6

…the gpuValidation modifier is run

cmsbuild added this to the CMSSW_13_2_X milestone Jun 11, 2023

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending code-checks-pending trk labels Jun 11, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 11, 2023

cmsbuild added tests-started and removed tests-pending labels Jun 11, 2023

cmsbuild added tests-rejected and removed tests-started labels Jun 11, 2023

cmsbuild added tests-started and removed tests-rejected labels Jun 12, 2023

cmsbuild added tests-approved and removed tests-started labels Jun 12, 2023

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Jun 13, 2023

cmsbuild added orp-approved and removed orp-pending labels Jun 13, 2023

cmsbuild merged commit 3197bcb into cms-sw:master Jun 13, 2023

mmusich deleted the sipixelphase1rawdatacompartor branch June 13, 2023 07:13

This was referenced Jun 13, 2023

[13.1.X] Add SiPixelRawDataError monitoring for gpuValidation setup #41938

Merged

[13.0.X] Add SiPixelRawDataError monitoring for gpuValidation setup #41939

Merged

cmsbuild mentioned this pull request Jun 13, 2023

Fix UBSAN warning in DataFormats/L1TParticleFlow #41941

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `SiPixelRawDataError` monitoring for `gpuValidation` setup #41920

Add `SiPixelRawDataError` monitoring for `gpuValidation` setup #41920

mmusich commented Jun 11, 2023

mmusich commented Jun 11, 2023

cmsbuild commented Jun 11, 2023

cmsbuild commented Jun 11, 2023

mmusich commented Jun 11, 2023

mmusich commented Jun 11, 2023

cmsbuild commented Jun 11, 2023

mmusich commented Jun 11, 2023

mmusich commented Jun 12, 2023

cmsbuild commented Jun 12, 2023

emanueleusai commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

perrotta commented Jun 13, 2023

Add SiPixelRawDataError monitoring for gpuValidation setup #41920

Add SiPixelRawDataError monitoring for gpuValidation setup #41920

Conversation

mmusich commented Jun 11, 2023

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

mmusich commented Jun 11, 2023

cmsbuild commented Jun 11, 2023

cmsbuild commented Jun 11, 2023

mmusich commented Jun 11, 2023

mmusich commented Jun 11, 2023

cmsbuild commented Jun 11, 2023

RelVals-GPU

Comparison Summary

mmusich commented Jun 11, 2023

mmusich commented Jun 12, 2023

cmsbuild commented Jun 12, 2023

Comparison Summary

GPU Comparison Summary

emanueleusai commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

perrotta commented Jun 13, 2023

Add `SiPixelRawDataError` monitoring for `gpuValidation` setup #41920

Add `SiPixelRawDataError` monitoring for `gpuValidation` setup #41920