Add Transformer ability to modules #38454

Dr15Jones · 2022-06-21T20:31:20Z

PR description:

Added ability to register tranform methods to modules. This allows a module to have a method which takes a data product already put into the Event by the same module and then create a new data product from it which is also put into the Event.

The plan is to use this ability to implement an automatic conversion from GPU to CPU data products.

PR validation:

Code compiles and new unit test passes.

cmsbuild · 2022-06-21T20:39:40Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38454/30666

This PR adds an extra 200KB to repository

cmsbuild · 2022-06-21T20:40:04Z

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

DataFormats/Provenance (core)
FWCore/Framework (core)
FWCore/Integration (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@makortel, @wddgit, @rovere this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

Dr15Jones · 2022-06-21T21:23:54Z

please test

cmsbuild · 2022-06-22T00:05:10Z

-1

Failed Tests: Build ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-49175f/25673/summary.html
COMMIT: 7346305
CMSSW: CMSSW_12_5_X_2022-06-21-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38454/25673/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-49175f/25673/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-49175f/25673/git-merge-result

Build

I found compilation error when building:

>> Leaving Package AnalysisAlgos/SiStripClusterInfoProducer
>> Package AnalysisAlgos/SiStripClusterInfoProducer built
Entering library rule at src/AnalysisAlgos/SiStripClusterInfoProducer/plugins
>> Compiling edm plugin /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-06-21-1100/src/AnalysisAlgos/SiStripClusterInfoProducer/plugins/SiStripProcessedRawDigiProducer.cc
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-06-21-1100/src/AnalysisAlgos/SiStripClusterInfoProducer/plugins/SiStripProcessedRawDigiProducer.cc: In member function 'void SiStripProcessedRawDigiProducer::pr_process(const edm::DetSetVector&, edm::DetSetVector&, const SiStripGain&)':
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-06-21-1100/src/AnalysisAlgos/SiStripClusterInfoProducer/plugins/SiStripProcessedRawDigiProducer.cc:126:5: error: 'virtual void edm::stream::EDProducerBase::transform(std::size_t, edm::EventForTransformer&) const' is private within this context
  126 |     transform(
      |     ^~~~~~~~~
In file included from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-06-21-1100/src/FWCore/Framework/interface/stream/implementors.h:31,
                 from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-06-21-1100/src/FWCore/Framework/interface/stream/AbilityToImplementor.h:25,
                 from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-06-21-1100/src/FWCore/Framework/interface/stream/EDProducer.h:22,

Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

>> Entering Package RecoMuon/MuonSeedGenerator
>> Entering Package RecoMuon/StandAloneMuonProducer
>> Entering Package SimG4Core/Application
>> Entering Package SimGeneral/MixingModule
>> Compile sequence completed for CMSSW CMSSW_12_5_X_2022-06-21-1100
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1
+ eval scram build outputlog '&&' '(python3' /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cms-bot/buildLogAnalyzer.py --ignoreWarning=Wdeprecated-declarations --logDir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-06-21-1100/tmp/el8_amd64_gcc10/cache/log/src '||' 'true)'
++ scram build outputlog
>> Entering Package DataFormats/Provenance
Entering library rule at DataFormats/Provenance
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-06-21-1100/src/DataFormats/Provenance/src/BranchChildren.cc

Dr15Jones · 2022-06-22T14:17:47Z

please test

cmsbuild · 2022-06-22T14:21:24Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38454/30682

This PR adds an extra 20KB to repository

cmsbuild · 2022-06-22T14:21:57Z

Pull request #38454 was updated. @makortel, @smuzaffar, @Dr15Jones can you please check and sign again.

cmsbuild · 2022-09-09T14:13:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38454/32060

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/global/EDFilterBase.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/interface/global/EDProducerBase.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/interface/global/implementors.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/interface/limited/EDFilterBase.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/interface/limited/EDProducerBase.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/interface/limited/implementors.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/interface/one/EDFilterBase.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/interface/one/EDProducerBase.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/interface/one/implementors.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/interface/stream/ProducingModuleAdaptorBase.h modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/src/ProductResolvers.cc modified in PR(s): Release dependent modules only after the worker has finished for scheduled modules #39245
- File FWCore/Framework/src/ProductResolvers.h modified in PR(s): Release dependent modules only after the worker has finished for scheduled modules #39245
- File FWCore/Framework/src/global/EDFilterBase.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/src/global/EDProducerBase.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/src/limited/EDFilterBase.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/src/limited/EDProducerBase.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/src/one/EDFilterBase.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/src/one/EDProducerBase.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/src/stream/ProducingModuleAdaptorBase.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/test/global_filter_t.cppunit.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/test/global_producer_t.cppunit.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/test/limited_filter_t.cppunit.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Framework/test/limited_producer_t.cppunit.cc modified in PR(s): Implement support for concurrent runs in the Framework #38801
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement support for concurrent runs in the Framework #38801

cmsbuild · 2022-09-09T14:13:46Z

Pull request #38454 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please check and sign again.

makortel · 2022-09-09T14:15:57Z

@cmsbuild, please test

cmsbuild · 2022-09-09T22:16:41Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-49175f/27448/summary.html
COMMIT: 0084b15
CMSSW: CMSSW_12_6_X_2022-09-09-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38454/27448/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestIOPoolInputNoParentDictionary had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3618326
DQMHistoTests: Total failures: 14
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3618290
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

makortel · 2022-09-10T00:11:14Z

So TestIOPoolInputNoParentDictionary is failing again with

===== Test "TestIOPoolInputNoParentDictionary" ====
WARNING: In non-interactive mode release checks e.g. deprecated releases, production architectures are disabled.
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_6_X_2022-09-09-1100/TestIOPoolInputNoParentDictionary/CMSSW_12_6_X_2022-09-09-1100/src /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_6_X_2022-09-09-1100/TestIOPoolInputNoParentDictionary
Failed to copy DataFormats/TestObjects from local or release area

---> test TestIOPoolInputNoParentDictionary had ERRORS

while in #38454 (comment) and #38454 (comment) the test succeeded. The failure is not caused by the developments of this PR, but gets somehow triggered based on what packages get checked out and if the IB is patch or full release (CMSSW_12_6_X_2022-09-09-1100 is a patch release). I suggest we go a head with this PR, and investigate this failure mode of TestIOPoolInputNoParentDictionary test separately.

makortel · 2022-09-10T00:11:17Z

+1

cmsbuild · 2022-09-10T00:11:41Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-09-10T09:32:53Z

+1

perrotta · 2022-09-10T09:33:27Z

merge

cmsbuild added this to the CMSSW_12_5_X milestone Jun 21, 2022

cmsbuild added code-checks-pending core-pending orp-pending pending-signatures tests-pending labels Jun 21, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 21, 2022

cmsbuild added tests-started tests-rejected and removed tests-pending tests-started labels Jun 21, 2022

Dr15Jones force-pushed the produceTransform branch from 7346305 to 998a628 Compare June 22, 2022 14:14

cmsbuild added code-checks-pending tests-pending and removed tests-rejected code-checks-approved labels Jun 22, 2022

cmsbuild added tests-started and removed tests-pending labels Jun 22, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 22, 2022

cmsbuild added tests-rejected and removed tests-started labels Jun 22, 2022

cmsbuild added code-checks-pending tests-pending and removed code-checks-approved labels Sep 9, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 9, 2022

cmsbuild added tests-started and removed tests-pending labels Sep 9, 2022

cmsbuild added tests-rejected and removed tests-started labels Sep 9, 2022

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Sep 10, 2022

cmsbuild added orp-approved and removed orp-pending labels Sep 10, 2022

cmsbuild merged commit 76c2158 into cms-sw:master Sep 10, 2022

cmsbuild mentioned this pull request Sep 10, 2022

Python 3.9.14 plus bulk update of many python package cms-sw/cmsdist#8077

Merged

Dr15Jones deleted the produceTransform branch September 16, 2022 15:26

makortel mentioned this pull request Sep 16, 2022

Evolution of the Alpaka "gpu framework" #39428

Merged

5 tasks

makortel mentioned this pull request Oct 14, 2022

Added transformAsync ability to modules #39557

Merged

makortel mentioned this pull request Feb 24, 2023

Exceptions not propagated through Transformer #40868

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Transformer ability to modules #38454

Add Transformer ability to modules #38454

Dr15Jones commented Jun 21, 2022

cmsbuild commented Jun 21, 2022

cmsbuild commented Jun 21, 2022

Dr15Jones commented Jun 21, 2022

cmsbuild commented Jun 22, 2022

Dr15Jones commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

cmsbuild commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

makortel commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

makortel commented Sep 10, 2022

makortel commented Sep 10, 2022

cmsbuild commented Sep 10, 2022

perrotta commented Sep 10, 2022

perrotta commented Sep 10, 2022

Add Transformer ability to modules #38454

Add Transformer ability to modules #38454

Conversation

Dr15Jones commented Jun 21, 2022

PR description:

PR validation:

cmsbuild commented Jun 21, 2022

cmsbuild commented Jun 21, 2022

Dr15Jones commented Jun 21, 2022

cmsbuild commented Jun 22, 2022

Build

Clang Build

Dr15Jones commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

cmsbuild commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

makortel commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

Unit Tests

Comparison Summary

makortel commented Sep 10, 2022

makortel commented Sep 10, 2022

cmsbuild commented Sep 10, 2022

perrotta commented Sep 10, 2022

perrotta commented Sep 10, 2022