support global::OutputModule<ExternalWork> #34579

Dr15Jones · 2021-07-21T19:48:56Z

PR description:

Extend the framework to allow edm::global::OutputModuleedm::ExternalWork to function.

PR validation:

Newly added unit test passes. Also tested with a not-yet-committed OutputModule which uses the facility.

cmsbuild · 2021-07-21T19:56:29Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34579/24129

This PR adds an extra 52KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/src/Worker.h modified in PR(s): Add warning that lists legacy modules if any are configured. #34577
- File FWCore/Framework/src/WorkerT.cc modified in PR(s): Add warning that lists legacy modules if any are configured. #34577
- File FWCore/Framework/test/BuildFile.xml modified in PR(s): Make EventSetup get to throw if called without a token when EDModule consumed any ES product #31746

cmsbuild · 2021-07-21T19:56:53Z

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

FWCore/Framework (core)

@makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please review it and eventually sign? Thanks.
@makortel, @wddgit this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

Dr15Jones · 2021-07-21T19:57:22Z

please test

cmsbuild · 2021-07-22T03:01:53Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea9309/17070/summary.html
COMMIT: 796eba8
CMSSW: CMSSW_12_0_X_2021-07-20-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17070/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea9309/17070/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea9309/17070/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test test2026Geometry had ERRORS
---> test test2021Geometry had ERRORS
---> test GeometryDTGeometryBuilderTestDriver had ERRORS
---> test GeometryTrackerGeometryBuilderTestDriver had ERRORS
and more ...

Comparison Summary

The workflows 140.53 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

No significant changes to the logs found
Reco comparison results: 49129 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2995596
DQMHistoTests: Total failures: 202887
DQMHistoTests: Total nulls: 107
DQMHistoTests: Total successes: 2792580
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1691.389 KiB( 38 files compared)
DQMHistoSizes: changed ( 10024.0,... ): 117.572 KiB PixelPhase1/Tracks
DQMHistoSizes: changed ( 10224.0 ): 0.542 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 140.53 ): 44.531 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 140.53 ): 1.172 KiB RPC/DCSInfo
DQMHistoSizes: changed ( 250202.181 ): -0.117 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.416 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 7.3 ): -0.327 KiB SiStrip/MechanicalView
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: found differences in 10 / 38 workflows

makortel · 2021-07-22T14:27:41Z

@cmsbuild, please test

Let's try again if results would be cleaner

cmsbuild · 2021-07-22T18:33:43Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea9309/17091/summary.html
COMMIT: 796eba8
CMSSW: CMSSW_12_0_X_2021-07-21-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17091/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test_MuonGeometryDBConverter had ERRORS
---> test testDD4hepCompactView had ERRORS
---> test DetectorDescriptionDDCMSTestDriver had ERRORS
---> test testDD4hepFilteredView had ERRORS
and more ...

RelVals

11634.911

----- Begin Fatal Exception 22-Jul-2021 19:33:23 CEST-----------------------
An exception of category 'StdException' occurred while
   [0] Running EventSetup component DDDetectorESProducer/'
Exception Message:
A std::exception was thrown.
An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file Geometry/VeryForwardData/data/RP_Vertical_Device/2021/v1/RP_Vertical_Device.xml anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17091/CMSSW_12_0_X_2021-07-21-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17091/CMSSW_12_0_X_2021-07-21-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17091/CMSSW_12_0_X_2021-07-21-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-21-2300/src:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-21-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/11634.911_TTbar_14TeV+2021_DD4hep+TTbar_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST+ALCA

dd4hep: Error interpreting XML nodes of type <Include/>
dd4hep: Error interpreting XML nodes of type <IncludeSection/>
dd4hep: while parsing /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17091/CMSSW_12_0_X_2021-07-21-2300/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2021.xml
dd4hep: with plugin:DD4hep_CompactLoader
----- End Fatal Exception -------------------------------------------------

makortel · 2021-07-22T19:05:08Z

Maybe it is better to wait for next IB to get at least the matrix tests to run...

smuzaffar · 2021-07-22T19:27:17Z

CMSSW_12_0_X_2021-07-22-1700 IB with the fix should be available in 30 mins . Its Comparison Baseline is currently running https://cmssdt.cern.ch/jenkins/job/ib-run-baseline/22070/ , once baseline is available then you can start the tests

makortel · 2021-07-22T21:06:37Z

@cmsbuild, please test

cmsbuild · 2021-07-23T01:32:36Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea9309/17114/summary.html
COMMIT: 796eba8
CMSSW: CMSSW_12_0_X_2021-07-22-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17114/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test_MuonGeometryDBConverter had ERRORS
---> test testDD4hepCompactView had ERRORS
---> test testDD4hepFilteredViewFind had ERRORS
---> test testDD4hepFilteredView had ERRORS
and more ...

RelVals

11634.911

----- Begin Fatal Exception 23-Jul-2021 02:41:57 CEST-----------------------
An exception of category 'StdException' occurred while
   [0] Running EventSetup component DDDetectorESProducer/'
Exception Message:
A std::exception was thrown.
An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file Geometry/VeryForwardData/data/RP_Vertical_Device/2021/v1/RP_Vertical_Device.xml anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17114/CMSSW_12_0_X_2021-07-22-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17114/CMSSW_12_0_X_2021-07-22-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17114/CMSSW_12_0_X_2021-07-22-1100/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-22-1100/src:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-22-1100/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/11634.911_TTbar_14TeV+2021_DD4hep+TTbar_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST+ALCA

dd4hep: Error interpreting XML nodes of type <Include/>
dd4hep: Error interpreting XML nodes of type <IncludeSection/>
dd4hep: while parsing /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17114/CMSSW_12_0_X_2021-07-22-1100/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2021.xml
dd4hep: with plugin:DD4hep_CompactLoader
----- End Fatal Exception -------------------------------------------------

makortel · 2021-07-23T01:45:01Z

@cmsbuild, please test

How about now picking a newer IB?

cmsbuild · 2021-07-23T10:01:01Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea9309/17127/summary.html
COMMIT: 796eba8
CMSSW: CMSSW_12_0_X_2021-07-22-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17127/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test trackerMaterialAnalysisPlots had ERRORS
---> test materialBudgetTrackerPlots had ERRORS
---> test materialBudgetHGCalPlots had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2998564
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2998541
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

makortel · 2021-07-23T13:24:49Z

+1

The failing unit tests fail in the IBs.

cmsbuild · 2021-07-23T13:25:09Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy, @perrotta (and backports should be raised in the release meeting by the corresponding L2)

Dr15Jones · 2021-07-23T13:27:07Z

please test

cmsbuild · 2021-07-23T18:50:09Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ea9309/17148/summary.html
COMMIT: 796eba8
CMSSW: CMSSW_12_0_X_2021-07-22-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34579/17148/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test trackerMaterialAnalysisPlots had ERRORS
---> test materialBudgetTrackerPlots had ERRORS
---> test materialBudgetHGCalPlots had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2998564
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2998541
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

makortel · 2021-07-23T18:55:26Z

The unit tests fail already in the IBs (I would not hold this PR for those)

qliphy · 2021-07-24T02:13:10Z

+1

qliphy · 2021-07-24T02:18:51Z

merge

Dr15Jones added 2 commits July 21, 2021 13:34

Extend global::OutputModule to support ExternalWork

9199604

Add unit test for global::OutputModule<ExternalWork>

796eba8

cmsbuild added this to the CMSSW_12_0_X milestone Jul 21, 2021

cmsbuild added code-checks-pending core-pending orp-pending pending-signatures tests-pending labels Jul 21, 2021

Dr15Jones mentioned this pull request Jul 21, 2021

Extend functionality of global OutputModules cms-sw/framework-team#195

Closed

3 tasks

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 21, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 21, 2021

cmsbuild added tests-rejected and removed tests-started labels Jul 22, 2021

cmsbuild added tests-started and removed tests-rejected labels Jul 22, 2021

cmsbuild added tests-rejected and removed tests-started labels Jul 22, 2021

cmsbuild added tests-started and removed tests-rejected labels Jul 22, 2021

cmsbuild added tests-rejected and removed tests-started labels Jul 23, 2021

cmsbuild added tests-started and removed tests-rejected labels Jul 23, 2021

cmsbuild added tests-rejected and removed tests-started labels Jul 23, 2021

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Jul 23, 2021

cmsbuild added tests-started and removed tests-rejected labels Jul 23, 2021

cmsbuild mentioned this pull request Jul 23, 2021

Add warning that lists legacy modules if any are configured. #34577

Merged

cmsbuild added tests-rejected and removed tests-started labels Jul 23, 2021

cmsbuild added orp-approved and removed orp-pending labels Jul 24, 2021

cmsbuild merged commit 774b201 into cms-sw:master Jul 24, 2021

cmsbuild mentioned this pull request Jul 24, 2021

Updated root to tip of branch rootnext cms-sw/cmsdist#7160

Merged

Dr15Jones deleted the externalWorkGlobalOutputModule branch July 26, 2021 13:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support global::OutputModule<ExternalWork> #34579

support global::OutputModule<ExternalWork> #34579

Dr15Jones commented Jul 21, 2021

cmsbuild commented Jul 21, 2021

cmsbuild commented Jul 21, 2021

Dr15Jones commented Jul 21, 2021

cmsbuild commented Jul 22, 2021

makortel commented Jul 22, 2021

cmsbuild commented Jul 22, 2021

makortel commented Jul 22, 2021

smuzaffar commented Jul 22, 2021 •

edited

Loading

makortel commented Jul 22, 2021

cmsbuild commented Jul 23, 2021

makortel commented Jul 23, 2021

cmsbuild commented Jul 23, 2021

makortel commented Jul 23, 2021

cmsbuild commented Jul 23, 2021

Dr15Jones commented Jul 23, 2021

cmsbuild commented Jul 23, 2021

makortel commented Jul 23, 2021

qliphy commented Jul 24, 2021

qliphy commented Jul 24, 2021

support global::OutputModule<ExternalWork> #34579

support global::OutputModule<ExternalWork> #34579

Conversation

Dr15Jones commented Jul 21, 2021

PR description:

PR validation:

cmsbuild commented Jul 21, 2021

cmsbuild commented Jul 21, 2021

Dr15Jones commented Jul 21, 2021

cmsbuild commented Jul 22, 2021

Unit Tests

Comparison Summary

makortel commented Jul 22, 2021

cmsbuild commented Jul 22, 2021

Unit Tests

RelVals

makortel commented Jul 22, 2021

smuzaffar commented Jul 22, 2021 • edited Loading

makortel commented Jul 22, 2021

cmsbuild commented Jul 23, 2021

Unit Tests

RelVals

makortel commented Jul 23, 2021

cmsbuild commented Jul 23, 2021

Unit Tests

Comparison Summary

makortel commented Jul 23, 2021

cmsbuild commented Jul 23, 2021

Dr15Jones commented Jul 23, 2021

cmsbuild commented Jul 23, 2021

Unit Tests

Comparison Summary

makortel commented Jul 23, 2021

qliphy commented Jul 24, 2021

qliphy commented Jul 24, 2021

smuzaffar commented Jul 22, 2021 •

edited

Loading