Add globalBeginJob() interface for stream modules using GlobalCache #29585

makortel · 2020-04-29T14:42:33Z

PR description:

This PR adds an optional globalBeginJob() interface for stream modules using GlobalCache extension. It is motivated by #26438 (comment), i.e. to have a hook to do heavy or platform-dependent initialization after the before-beginJob module deletion (#29553) has been done, i.e. the job is known to run the module in question.

PR validation:

Framework unit tests run.

cmsbuild · 2020-04-29T14:43:02Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-04-29T14:48:25Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29585/14908

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/stream/EDAnalyzerAdaptorBase.h modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/interface/stream/ProducingModuleAdaptorBase.h modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/src/stream/EDAnalyzerAdaptorBase.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/src/stream/ProducingModuleAdaptorBase.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/test/stubs/TestStreamAnalyzers.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/test/stubs/TestStreamProducers.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/test/test_stream_modules_cfg.py modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553

cmsbuild · 2020-04-29T14:48:51Z

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

FWCore/Framework

@makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please review it and eventually sign? Thanks.
@wddgit this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2020-04-29T15:13:37Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-04-29T15:16:02Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-04-29T15:20:06Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29585/14910

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/stream/EDAnalyzerAdaptorBase.h modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/interface/stream/ProducingModuleAdaptorBase.h modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/src/stream/EDAnalyzerAdaptorBase.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/src/stream/ProducingModuleAdaptorBase.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/test/stubs/TestStreamAnalyzers.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/test/stubs/TestStreamProducers.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/test/test_stream_modules_cfg.py modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553

cmsbuild · 2020-04-29T15:20:34Z

Pull request #29585 was updated. @makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please check and sign again.

makortel · 2020-04-29T15:23:26Z

@cmsbuild, please test

cmsbuild · 2020-04-29T15:23:51Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5904/console Started: 2020/04/29 17:27

cmsbuild · 2020-04-29T16:23:46Z

-1

Tested at: 23dde92

CMSSW: CMSSW_11_1_X_2020-04-29-1100
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bfbd3/5904/summary.html

I found follow errors while testing this PR

Failed tests: ClangBuild

Clang:

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

cmsbuild · 2020-04-29T16:23:48Z

Comparison not run due to Build errors/Fireworks only changes/No short matrix requested (RelVals and Igprof tests were also skipped)

cmsbuild · 2020-04-29T16:51:39Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-04-29T16:58:30Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29585/14914

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/stream/EDAnalyzerAdaptorBase.h modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/interface/stream/ProducingModuleAdaptorBase.h modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/src/stream/EDAnalyzerAdaptorBase.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/src/stream/ProducingModuleAdaptorBase.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/test/stubs/TestStreamAnalyzers.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/test/stubs/TestStreamProducers.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553
- File FWCore/Framework/test/test_stream_modules_cfg.py modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553

cmsbuild · 2020-04-29T16:58:54Z

Pull request #29585 was updated. @makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please check and sign again.

makortel · 2020-04-29T17:01:22Z

@cmsbuild, please test

cmsbuild · 2020-04-29T17:01:48Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5910/console Started: 2020/04/29 19:15

cmsbuild · 2020-04-29T19:30:33Z

+1
Tested at: 150f1e7
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bfbd3/5910/summary.html
CMSSW: CMSSW_11_1_X_2020-04-29-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-04-29T19:30:36Z

Comparison job queued.

makortel · 2020-04-29T19:42:31Z

@Dr15Jones Please review.

Dr15Jones · 2020-04-29T20:00:44Z

Seems fine to me

Dr15Jones · 2020-04-29T20:00:49Z

+1

cmsbuild · 2020-04-29T20:01:23Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2020-04-29T20:57:50Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bfbd3/5910/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2696435
DQMHistoTests: Total failures: 26
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2696090
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

silviodonato · 2020-04-30T08:53:51Z

+1

cmsbuild added this to the CMSSW_11_1_X milestone Apr 29, 2020

cmsbuild added code-checks-pending comparison-pending core-pending orp-pending pending-signatures tests-pending labels Apr 29, 2020

makortel mentioned this pull request Apr 29, 2020

Prune non-consumed EDProducers that are only in Tasks #26438

Closed

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 29, 2020

makortel force-pushed the globalBeginJob branch from aff5cf8 to 9600ebe Compare April 29, 2020 15:13

cmsbuild added code-checks-pending and removed code-checks-approved labels Apr 29, 2020

makortel force-pushed the globalBeginJob branch from 9600ebe to 23dde92 Compare April 29, 2020 15:15

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 29, 2020

cmsbuild added tests-started and removed tests-pending labels Apr 29, 2020

cmsbuild removed the comparison-pending label Apr 29, 2020

cmsbuild added the tests-pending label Apr 29, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 29, 2020

cmsbuild added tests-started and removed tests-pending labels Apr 29, 2020

cmsbuild added tests-approved and removed tests-started labels Apr 29, 2020

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Apr 29, 2020

cmsbuild added comparison-available and removed comparison-pending labels Apr 29, 2020

cmsbuild added orp-approved and removed orp-pending labels Apr 30, 2020

cmsbuild merged commit ab17db7 into cms-sw:master Apr 30, 2020

makortel deleted the globalBeginJob branch April 30, 2020 13:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add globalBeginJob() interface for stream modules using GlobalCache #29585

Add globalBeginJob() interface for stream modules using GlobalCache #29585

makortel commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

makortel commented Apr 29, 2020

cmsbuild commented Apr 29, 2020 •

edited

Loading

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

makortel commented Apr 29, 2020

cmsbuild commented Apr 29, 2020 •

edited

Loading

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

makortel commented Apr 29, 2020

Dr15Jones commented Apr 29, 2020

Dr15Jones commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

silviodonato commented Apr 30, 2020

Add globalBeginJob() interface for stream modules using GlobalCache #29585

Add globalBeginJob() interface for stream modules using GlobalCache #29585

Conversation

makortel commented Apr 29, 2020

PR description:

PR validation:

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

makortel commented Apr 29, 2020

cmsbuild commented Apr 29, 2020 • edited Loading

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

makortel commented Apr 29, 2020

cmsbuild commented Apr 29, 2020 • edited Loading

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

makortel commented Apr 29, 2020

Dr15Jones commented Apr 29, 2020

Dr15Jones commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

cmsbuild commented Apr 29, 2020

silviodonato commented Apr 30, 2020

cmsbuild commented Apr 29, 2020 •

edited

Loading

cmsbuild commented Apr 29, 2020 •

edited

Loading