Enable PU workflows for patatrack

[srimanob]

PR description:

This PR is to enable PU workflows for patatrack.

PR validation:

Configuration dump, and see
11634.596 TTbar_14TeV_TuneCP5_2021_GenSim+Digi_Patatrack_FullRecoTripletsGPU_2021+RecoNano_Patatrack_FullRecoTripletsGPU_2021+HARVESTNano_Patatrack_FullRecoTripletsGPU_2021
11834.596 TTbar_14TeV_TuneCP5_2021PU_GenSim+DigiPU_Patatrack_FullRecoTripletsGPU_2021PU+RecoNanoPU_Patatrack_FullRecoTripletsGPU_2021PU+HARVESTNanoPU_Patatrack_FullRecoTripletsGPU_2021PU

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

No need of backport.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40847/34300

• This PR adds an extra 24KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, Displaced regional tracking iteration #40643

[cmsbuild]

A new Pull Request was created by @srimanob (Phat Srimanobhas) for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)

@bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @Martin-Grunewald, @missirol, @fabiocos, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[srimanob]

test parameters:

• workflows = 11634.696,11834.596
• enable = gpu
• relvals_opt = --what cleanedupgrade,standard,highstats,pileup,generator,extendedgen,production,ged,machine,premix,nano

[srimanob]

@cmsbuild please test

[cmsbuild]

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e9d226/30809/summary.html
COMMIT: 96d700f
CMSSW: CMSSW_13_1_X_2023-02-21-2300/el8_amd64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40847/30809/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

ValueError: Undefined workflows: 11634.696

RelVals-INPUT

The relvals timed out after 4 hours.

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 9 differences found in the comparisons
• DQMHistoTests: Total files compared: 4
• DQMHistoTests: Total histograms compared: 19862
• DQMHistoTests: Total failures: 243
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 19619
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
• Checked 12 log files, 9 edm output root files, 4 DQM output files
• TriggerResults: found differences in 2 / 3 workflows

[srimanob]

test parameters:

[srimanob]

test parameters:

• workflows = 11634.596,11834.596
• enable = gpu
• relvals_opt = --what cleanedupgrade,standard,highstats,pileup,generator,extendedgen,production,ged,machine,premix,nano

[srimanob]

@cmsbuild please test

[cmsbuild]

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e9d226/30815/summary.html
COMMIT: 96d700f
CMSSW: CMSSW_13_1_X_2023-02-22-1100/el8_amd64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40847/30815/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

ValueError: Undefined workflows: 11634.596

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 4
• DQMHistoTests: Total histograms compared: 19862
• DQMHistoTests: Total failures: 59
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 19803
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
• Checked 12 log files, 9 edm output root files, 4 DQM output files
• TriggerResults: found differences in 1 / 3 workflows

[srimanob]

Hi @AdrianoDee @fwyzard
Do you know the proper way to call wf 11634.596 and 11834.596 for the test? My test parameters seem not to work as expected. But I can use runTheMatrix.py --what upgrade -l 11634.596,11834.596 --wm init.

Thx.

[gartung]

Workflow 11634.596 does not appear in the relval list and it segfaults in the RECO step.

[srimanob]

Workflow 11634.596 does not appear in the relval list and it segfaults in the RECO step.

Is this strange? As 11834.596 runs until the end. The difference is 11834.596 is PU workflow.

[AdrianoDee]

test parameters

• workflows_gpu = 11634.596,11834.596
• enable = gpu
• relvals_opt_gpu = --what upgrade,standard,highstats,pileup,generator,extendedgen,production,ged,machine,premix,nano
• pull_request = Fix cudaMemcpyAsync for localCoordToHostAsync #40869

[AdrianoDee]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e9d226/30877/summary.html
COMMIT: 57a1fb8
CMSSW: CMSSW_13_1_X_2023-02-23-2300/el8_amd64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40847/30877/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 6 lines from the logs
• Reco comparison results: 17 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3528709
• DQMHistoTests: Total failures: 123
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3528564
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 213 log files, 164 edm output root files, 49 DQM output files
• TriggerResults: no differences found

GPU Comparison Summary

Summary:

• You potentially removed 80 lines from the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 1142 differences found in the comparisons
• DQMHistoTests: Total files compared: 6
• DQMHistoTests: Total histograms compared: 334239
• DQMHistoTests: Total failures: 6152
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 328086
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 15673.778 KiB( 5 files compared)
• DQMHistoSizes: changed ( 11834.596 ): 8111.479 KiB Tracking/PixelTrackParameters
• DQMHistoSizes: changed ( 11834.596 ): 4543.241 KiB SiPixelHeterogeneous/PixelRecHitsSoA
• DQMHistoSizes: changed ( 11834.596 ): 2896.828 KiB OfflinePixelPV/Resolution
• DQMHistoSizes: changed ( 11834.596 ): 258.217 KiB OfflinePixelPV/pixelVertices
• DQMHistoSizes: changed ( 11834.596 ): -165.761 KiB Physics/NanoAODDQM
• DQMHistoSizes: changed ( 11834.596 ): 11.981 KiB SiPixelHeterogeneous/PixelTrackSoA
• DQMHistoSizes: changed ( 11834.596 ): 11.738 KiB OfflinePixelPV/Alignment
• DQMHistoSizes: changed ( 11834.596 ): 3.541 KiB OfflinePixelPV/offlineBeamSpot
• DQMHistoSizes: changed ( 11834.596 ): 2.451 KiB SiPixelHeterogeneous/PixelVertexSoA
• DQMHistoSizes: changed ( 11834.596 ): 0.031 KiB MessageLogger/Errors
• DQMHistoSizes: changed ( 11834.596 ): ...
• Checked 20 log files, 15 edm output root files, 6 DQM output files
• TriggerResults: found differences in 5 / 5 workflows

[srimanob]

+Upgrade

This PR adds the PU workflows for patatrack.

[srimanob]

@perrotta @rappoccio
I think this PR should get in, as PR itself just adds the workflows. It does not need the fix to some specific workflow, i.e. together, #40869, to be merged first. Added workflows will not run on any IB and short matrix test, but few which work will be useful for performance study, and debug any issues further.

Thanks.

[srimanob]

Kindly ping @cms-sw/pdmv-l2
Thx.

[sunilUIET]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1

[missirol]

by the way, do IBs really run all workflows from runTheMatrix.py -w gpu ? or just a subset ?

Fwiw, my understanding is that the GPU_X IBs run all the wfs of runTheMatrix.py -w gpu.