HDF5 interface for LHE-HDF5 event files #30312

smrenna · 2020-06-19T16:51:29Z

PR description:

This is a copy of LHESource to handle event generator files in the HDF5 format.
This uses the header-only HighFive library for handling the HDF5 files and information.

PR validation:

I have used the two tests in the LHEInterface tests directory: testReader_cfg.py, which reads the hdf5 file and makes a root edm file and dumpLHE_cfg.py that prints out the stored information.

I can put some hdf5 files on lxplus for testing.

Before submitting your pull requests, make sure you followed this checklist:

verify that the PR is really intended for the chosen branch
verify that changes follow CMS Naming, Coding, And Style Rules
verify that the PR passes the basic test procedure suggested in the CMSSW PR instructions

cmsbuild · 2020-06-19T16:51:57Z

The code-checks are being triggered in jenkins.

smrenna · 2020-06-19T16:52:05Z

@bendavid might be interested in this

cmsbuild · 2020-06-19T16:59:54Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30312/16282

This PR adds an extra 60KB to repository
Found files with invalid states:
- GeneratorInterface/LHEInterface/src/HighFive/.gitmodules:
  - Added: 8104a8b
  - Deleted: 5a9447e
- GeneratorInterface/LHEInterface/src/HighFive/.travis.yml:
  - Added: 8104a8b
  - Deleted: 5a9447e
- GeneratorInterface/LHEInterface/src/HighFive/.gitignore:
  - Added: 8104a8b
  - Deleted: 5a9447e
- GeneratorInterface/LHEInterface/src/HighFive/.clang-format:
  - Added: 8104a8b
  - Deleted: 5a9447e

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30312/16282/code-checks.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30312/16282/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly
code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30312/16282/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30312/16282/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2020-06-19T17:11:26Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-19T17:13:17Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-19T17:17:38Z

-code-checks

ERROR: Build errors found during clang-tidy run.

GeneratorInterface/LHEInterface/plugins/LH5Source.cc:38:39: error: too few arguments to function call, single argument 'iCatalog' was not specified [clang-diagnostic-error]
      reader_(new LH5Reader(fileNames(),
                                      ^
/cvmfs/cms-ib.cern.ch/nweek-02633/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-06-19-1100/src/FWCore/Sources/interface/FromFiles.h:22:5: note: 'fileNames' declared here
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:128: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

smrenna · 2020-06-19T18:10:12Z

I need some help here. This is a direct copy of the structure of LHEReader, so why does one trigger a problem, but not the other?

Couldn't find the definition of iCatalog in CMSSW lxr.

cmsbuild · 2020-06-19T18:55:07Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-19T19:00:13Z

-code-checks

ERROR: Build errors found during clang-tidy run.

GeneratorInterface/LHEInterface/plugins/LH5Source.cc:38:39: error: too few arguments to function call, single argument 'iCatalog' was not specified [clang-diagnostic-error]
      reader_(new LH5Reader(fileNames(),params.getUntrackedParameter<unsigned int>("skipEvents", 0))),
                                      ^
/cvmfs/cms-ib.cern.ch/nweek-02633/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-06-19-1100/src/FWCore/Sources/interface/FromFiles.h:22:5: note: 'fileNames' declared here
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:128: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

cmsbuild · 2020-06-19T19:03:53Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-19T19:07:55Z

-code-checks

ERROR: Build errors found during clang-tidy run.

GeneratorInterface/LHEInterface/plugins/LHESource.cc:35:39: error: too few arguments to function call, single argument 'iCatalog' was not specified [clang-diagnostic-error]
      reader_(new LHEReader(fileNames(), params.getUntrackedParameter<unsigned int>("skipEvents", 0))),
                                      ^
/cvmfs/cms-ib.cern.ch/nweek-02633/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-06-19-1100/src/FWCore/Sources/interface/FromFiles.h:22:5: note: 'fileNames' declared here
--
GeneratorInterface/LHEInterface/plugins/LH5Source.cc:38:39: error: too few arguments to function call, single argument 'iCatalog' was not specified [clang-diagnostic-error]
      reader_(new LH5Reader(fileNames(),params.getUntrackedParameter<unsigned int>("skipEvents", 0))),
                                      ^
/cvmfs/cms-ib.cern.ch/nweek-02633/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-06-19-1100/src/FWCore/Sources/interface/FromFiles.h:22:5: note: 'fileNames' declared here
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:128: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

cmsbuild · 2020-08-07T08:59:40Z

-1

Tested at: 00b5cc6

Build:

I found compilation error when building:

ERROR: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc820/external/tensorflow-sources/2.1.0-e4ba21/tensorflow-2.1.0/tensorflow/tools/pip_package/BUILD:223:1 Executing genrule //tensorflow/python/keras/api:keras_python_api_gen_compat_v1 failed (Exit 1)
INFO: Elapsed time: 3080.834s, Critical Path: 206.47s
INFO: 9091 processes: 9091 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.6HZwWJ (%build)


RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.6HZwWJ (%build)

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e4ba21/8644/summary.html

cmsbuild · 2020-08-07T20:28:13Z

The tests are being triggered in jenkins.
Tested with other pull request(s) cms-sw/cmsdist#6160

CMSSW_11_2_X_2020-08-07-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/8659/console Started: 2020/08/07 22:30

cmsbuild · 2020-08-08T01:00:17Z

+1
Tested at: 00b5cc6
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3c5417/8659/summary.html
CMSSW: CMSSW_11_2_X_2020-08-07-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-08-08T01:00:20Z

Comparison job queued.

cmsbuild · 2020-08-08T02:25:23Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3c5417/8659/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2612401
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2612352
DQMHistoTests: Total skipped: 48
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

silviodonato · 2020-08-25T10:04:49Z

After having merged cms-sw/cmsdist#6160 the tests are ok. Do you have further comments @smrenna @cms-sw/generators-l2 ?

SiewYan · 2020-08-25T11:26:06Z

Hello, its looks all good to me. Unless @smrenna has other comments on this.

silviodonato · 2020-09-01T09:31:32Z

Kind reminder @cms-sw/generators-l2

agrohsje · 2020-09-01T11:59:56Z

Thanks @smrenna for the PR. As there was no additional feedback I am ok with the PR being merged.

agrohsje · 2020-09-01T12:00:03Z

+1

cmsbuild · 2020-09-01T12:00:19Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2020-09-01T13:02:25Z

+1

cmsbuild added this to the CMSSW_11_2_X milestone Jun 19, 2020

cmsbuild added code-checks-pending comparison-pending generators-pending orp-pending pending-signatures tests-pending labels Jun 19, 2020

cmsbuild added code-checks-rejected code-checks-pending and removed code-checks-pending code-checks-rejected labels Jun 19, 2020

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jun 19, 2020

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jun 19, 2020

cmsbuild added code-checks-rejected code-checks-pending and removed code-checks-pending code-checks-rejected labels Jun 19, 2020

cmsbuild added tests-started and removed tests-pending labels Aug 7, 2020

cmsbuild added tests-rejected and removed tests-started labels Aug 7, 2020

smuzaffar mentioned this pull request Aug 7, 2020

HDF5 toolfile cleanup and added openmpi deps cms-sw/cmsdist#6160

Merged

cmsbuild added tests-started and removed tests-rejected labels Aug 7, 2020

cmsbuild added tests-approved and removed tests-started labels Aug 8, 2020

cmsbuild added comparison-available and removed comparison-pending labels Aug 8, 2020

cmsbuild added fully-signed generators-approved and removed generators-pending pending-signatures labels Sep 1, 2020

cmsbuild added orp-approved and removed orp-pending labels Sep 1, 2020

cmsbuild merged commit 0f1252a into cms-sw:master Sep 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDF5 interface for LHE-HDF5 event files #30312

HDF5 interface for LHE-HDF5 event files #30312

smrenna commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

smrenna commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

smrenna commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Aug 7, 2020

cmsbuild commented Aug 7, 2020 •

edited

Loading

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

silviodonato commented Aug 25, 2020

SiewYan commented Aug 25, 2020

silviodonato commented Sep 1, 2020

agrohsje commented Sep 1, 2020

agrohsje commented Sep 1, 2020

cmsbuild commented Sep 1, 2020

qliphy commented Sep 1, 2020

HDF5 interface for LHE-HDF5 event files #30312

HDF5 interface for LHE-HDF5 event files #30312

Conversation

smrenna commented Jun 19, 2020

PR description:

PR validation:

cmsbuild commented Jun 19, 2020

smrenna commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

smrenna commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Jun 19, 2020

cmsbuild commented Aug 7, 2020

cmsbuild commented Aug 7, 2020 • edited Loading

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

silviodonato commented Aug 25, 2020

SiewYan commented Aug 25, 2020

silviodonato commented Sep 1, 2020

agrohsje commented Sep 1, 2020

agrohsje commented Sep 1, 2020

cmsbuild commented Sep 1, 2020

qliphy commented Sep 1, 2020

cmsbuild commented Aug 7, 2020 •

edited

Loading