Enable concurrent GEN in Configuration/Generator/python #34710
Conversation
step 1 of the concurrent GEN updating plan: enable concurrent ExternalLHEProducer for all cfi & cff in Configuration/Generator/python/
step 2 of the concurrent GEN update plan: replace Pythia8GeneratorFilter with Pythia8ConcurrentGeneratorFilter if the module does not use ExternalDecays, for all cfi & cff in Configuration/Generator/python
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34710/24345
|
|
A new Pull Request was created by @colizz (Congqiao Li) for master. It involves the following packages:
@SiewYan, @mkirsano, @Saptaparna, @cmsbuild, @GurpreetSinghChahal, @agrohsje, @alberto-sanchez can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
@jordan-martins FYI. The new ReVal IDs are in (4), i.e. Please let me know if you have any comments :) I'll also backport to 10_6_X in #34527 when we agree on the change. |
colizz
force-pushed
the
dev-121X-enableConcurrentGen
branch
from
4aa1ba3 to
932a8f2
Compare
cmsbuild
added
code-checks-pending
pdmv-pending
upgrade-pending
and removed
code-checks-approved
labels
|
(forgot to push the changed file in Configuration/PyReleaseValidation/python. Now modified) |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34710/24350
|
|
Pull request #34710 was updated. @SiewYan, @mkirsano, @jordan-martins, @chayanit, @bbilin, @wajidalikhan, @Saptaparna, @cmsbuild, @GurpreetSinghChahal, @srimanob, @agrohsje, @kpedro88, @kskovpen, @alberto-sanchez can you please check and sign again. |
|
Please test |
|
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3fcfb0/17388/summary.html Comparison SummaryThe workflows 140.53 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons Summary:
|
|
Hi @colizz @cms-sw/generators-l2 |
|
+Upgrade This PR is to enable concurrent GEN. It includes new gen-only workflows as cases described in the PR description. Test of newly workflows and thread-enable runs fine. |
Backport from 2e7e1c0 in cms-sw#34710
|
@cms-sw/pdmv-l2 could you please have a look, and sign if you think so? |
|
+1 |
cmsbuild
added
fully-signed
pdmv-approved
and removed
pending-signatures
pdmv-pending
labels
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2) |
|
@cms-sw/pdmv-l2 @cms-sw/generators-l2 @rappoccio |
|
hi @qliphy , indeed. 19th of august might be no pdd but then 26th. |
Thanks @colizz Perfect！ PPD @rappoccio should be able to add you to the agenda. |
|
+1
|
|
I saw it is on the agenda of this Thursday's meeting: https://indico.cern.ch/event/1067353/#10-concurrent-gen-processing |
Backport of cms-sw#34710. Note that the requested events for workflow 537 has modified prior than this PR in cms-sw#34861
PR description:
Enable concurrent GEN configurations in the cfi/cff file in
Configuration/Generator/python.These concurrent GEN utilities improve the multithreading performance, and help to solve issue (for GEN modules): #25090 (comment).
As previously discussed with GEN and PdmV, the change is base on four rules documented in https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookGenMultithread#TL_DR, with details as follows.
Search over all fragments in
Configuration/Generator/python..relval_extendedgen.py.A summary of all changed cfi/cff files are shown in the below table.
Additional change: modify the number of events to 12 for workflow 537 to fix the issue in 4-thread concurrent mode. See #34531
A backport to 10_6_X is also needed.
PR validation:
Validated on all modified/added RelVal workflows. Here is a summary
Appendix: summary of all cfi/cff files in
Configuration/Generator/pythonenabled with concurrent GEN utilities.