Rivet 3.1.7 & YODA 1.9.7 #8109

mseidel42 · 2022-09-29T06:58:15Z

Rivet and YODA version bump

cmsbuild · 2022-09-29T06:58:37Z

A new Pull Request was created by @mseidel42 (Markus Seidel) for branch IB/CMSSW_12_6_X/master.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.
cms-bot commands are listed here

smuzaffar · 2022-09-29T09:25:16Z

please test

cmsbuild · 2022-09-29T09:48:40Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-265ba6/27839/summary.html
COMMIT: 70ef2e5
CMSSW: CMSSW_12_6_X_2022-09-28-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8109/27839/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ for x in external/gcc/10.3.0-84898dea653199466402e67d73657f10 external/zlib/1.2.11-71514f01d2850dfd0bcd391557259a58 external/bz2lib/1.0.6-86270889250d9cb8193d1963706a39cb external/expat/2.1.0-e2e74851cebffc0aa0988d120ae74fcd external/libuuid/2.34-fe91b21b8e54a48f2d45894d9da65f14 external/xz/5.2.5-d6fed2038c4e8d6e04531d1adba59f37 external/sqlite/3.36.0-cdaed8943a1c7d35c738a99ec764027b external/db6/6.2.32-0f34c5862bf3098cbedece4ce9240238 external/gdbm/1.10-d603980cbd3ec2df3505d5b82740ee09 external/libffi/3.4.2-482b99521caf507051a7a5dce0eb5926 external/python3/3.9.14-94e2a0b6aa9f396a5858e80362919be6 external/curl/7.79.0-497e9b528f6eb6d9394cc8387a9475d6 external/autotools/1.5-e0f14761fc5df7b05f8fe4bd51bd9290 external/libxml2/2.9.10-32fb6ccc7a5c1482269c150e9287ec24 external/cuda/11.5.2-2fdd815f4265b93fa1122d3d7f612379 external/rocm/5.0.2-95c215630c939706b0552e3eee38861c external/numactl/2.0.14-e8527c5132cef1ed03b7b2068c952729 external/libpciaccess/0.16-b067b0011e7283d86403eafb3159e181 external/hwloc/2.8.0-4d4cfa88fad0d153c066abad8510e5f5 external/py3-setuptools/63.4.3-5dae739833abcf953219cdcaf6a40c4d external/zstd/1.4.5-532bbea4f73669d8f80e2005b3a6e942 external/py3-pip/22.2.2-ba36c331a6b7d762520cb22a0a2edf3c external/pcre/8.43-28d54724578a9006cf0dca75629374e3 external/OpenBLAS/0.3.15-4b9e74563ec999425816b833afc2b93a external/py3-cython/0.29.32-d0eaca1ac25355ca455fae43f102e04d external/libpng/1.6.37-efbc28bcd11facd84b6eb25ead83f807 external/libjpeg-turbo/2.0.2-4fe5cd5a22d64ab2ef6b87949ed49b19 external/scitokens-cpp/0.7.0-652b557d24b1c5b9c5497cec97566cfe external/davix/0.8.1-ef1990aa6debd3de951f256764ba191e external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17 external/fftw3/3.3.8-6f0d9dffc39443df09636963a138f419 external/lz4/1.9.2-373b1f6c80ba13e93f436c77aa63c026 external/giflib/5.2.0-7f76394971aaf24e07b4b8369893078f external/gsl/2.6-ecfc7aef58d33908eb27131a8549a786 external/dcap/2.47.12-a77c8542c3e7a4c5572e17be1288579a external/libtiff/4.0.10-d77eb61821deee63a28db5d8c569b530 external/freetype/2.10.0-24d43b8f20cfbedd85c25c7fe8afff46 external/py3-wheel/0.37.1-3b6ba523e90d4ed8ad386e825a3a95ab external/py3-numpy/1.22.4-62e28bace1c65ef46c4a0988f2d65108 external/xrootd/5.5.0-a0410a6758fd4d07b495e4c473820f38 lcg/root/6.24.07-743e141b5605864d08d853874a243e48 external/python-python3/1.0-f3f06e2ecad3a92960349642c8d55e8b .
+ i=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc10/./etc/profile.d/init.sh
+ '[' -f /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc10/./etc/profile.d/init.sh ']'
+ sed -i -e 's|lPyROOT|lcppyyX.X|' ./pyext/setup.py.in
sed: can't read ./pyext/setup.py.in: No such file or directory
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.H8iBcO (%build)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+yoda+1.9.7-0dd2b2bb9f2fae69bcd93f84d6b97ca3
Macro expanded in comment on line 333: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

smuzaffar · 2022-09-29T10:00:57Z

please test

cmsbuild · 2022-09-29T10:01:00Z

Pull request #8109 was updated.

cmsbuild · 2022-09-29T10:50:40Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-265ba6/27841/summary.html
COMMIT: f28b664
CMSSW: CMSSW_12_6_X_2022-09-28-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8109/27841/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

make[2]: *** [Makefile:574: all-recursive] Error 1
make[2]: Leaving directory '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc10/external/sherpa/2.2.12-d9d171fbc6ffee8356a8acefc2ea5175/SHERPA-MC-2.2.12/AddOns'
make[1]: *** [Makefile:576: all-recursive] Error 1
make[1]: Leaving directory '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc10/external/sherpa/2.2.12-d9d171fbc6ffee8356a8acefc2ea5175/SHERPA-MC-2.2.12'
make: *** [Makefile:502: all] Error 2
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.aTGMFf (%build)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+sherpa+2.2.12-d9d171fbc6ffee8356a8acefc2ea5175
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.aTGMFf (%build)

smuzaffar · 2022-09-29T11:37:38Z

looks like rivet needs some fixing

Rivet/Tools/RivetSTL.hh:48:14: error: 'make_unique' has not been declared in 'std'

mseidel42 · 2022-09-29T11:45:55Z

Hi Malik,

Maybe it would be sufficient to compile Sherpa with C++14?

cmsdist/sherpa.spec

Line 50 in e0ddfdb

    
           CXXFLAGS="-fuse-cxa-atexit $ARCH_CMSPLATF -O2 -std=c++0x -I$LHAPDF_ROOT/include -I$BLACKHAT_ROOT/include -I$RIVET_ROOT/include" \

Let me try it out...

Best,
Markus

smuzaffar · 2022-09-30T12:40:10Z

@mseidel42 , I built sherpa with c++17 and it worked. So I would suggest to go with c++17 as rivet is already using it https://github.com/cms-sw/cmsdist/pull/8109/files#diff-5d526b7eadac085699ac3414d6f0dffefe4f218d6240d7e54a39eb86f6aa370bR39 .

Also the mcfc dependency in sherpa is unused, may be clean that up too

mseidel42 · 2022-09-30T12:53:25Z

Ok, I tried

time pkgtools/cmsBuild --new-scheduler -i a -a el8_amd64_gcc10 --builders 4 -j $(($(getconf _NPROCESSORS_ONLN) * 2)) build sherpa-toolfile

and it's somehow stuck at building openloops...

smuzaffar · 2022-10-09T16:22:51Z

please test

cmsbuild · 2022-10-09T16:23:08Z

Pull request #8109 was updated.

cmsbuild · 2022-10-10T01:13:34Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-265ba6/28141/summary.html
COMMIT: af72550
CMSSW: CMSSW_12_6_X_2022-10-09-0000/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8109/28141/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Python package update to fix the dependabot alerts #8117

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-265ba6/28141/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-265ba6/28141/git-merge-result

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-265ba6/41834.0_TTbar_14TeV+2026D94+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3392309
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3392287
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 49 edm output root files, 49 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-10-25T09:42:15Z

please test

cmsbuild · 2022-10-25T14:26:40Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-265ba6/28488/summary.html
COMMIT: af72550
CMSSW: CMSSW_12_6_X_2022-10-24-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8109/28488/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 7 differences found in the comparisons
Reco comparison had 6 failed jobs
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3378384
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3378353
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 201 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-10-26T08:40:58Z

@mseidel42 , can you please

delete %{relocateConfig}lib/python*/site-packages/rivet-*egg-info/SOURCES.txt line from rivet.spec as it is not needed any more
delete %{relocateConfig}lib/python*/site-packages/yoda*egg-info/SOURCES.txt frpm yoda.spec as it is not needed

cmsbuild · 2022-10-26T09:07:42Z

Pull request #8109 was updated.

smuzaffar · 2022-10-26T10:04:31Z

please test

cmsbuild · 2022-10-26T15:26:48Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-265ba6/28529/summary.html
COMMIT: 0b6554a
CMSSW: CMSSW_12_6_X_2022-10-25-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8109/28529/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
20834.911 step 3
4.6 step 3
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

No significant changes to the logs found
Reco comparison results: 11 differences found in the comparisons
Reco comparison had 6 failed jobs
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3384029
DQMHistoTests: Total failures: 96
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3383911
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 201 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-10-27T20:34:05Z

+externals

smuzaffar · 2022-11-01T16:50:47Z

@mseidel42 , looks like after this update we start getting crashes for couple of workflows ( see cms-sw/cmssw#39914 ). Any idea? should we revert this ?

mseidel42 · 2022-11-01T16:58:27Z

Hi @smuzaffar, ok, let's revert this for now.

I think we do not really need the Sherpa-internal Rivet interface in CMSSW, so I may try to disable that and see if things work fine. I got the complete build commands from one of the log files, so I can test it on cmsdev before making a new PR!

Rivet 3.1.7 & YODA 1.9.7

70ef2e5

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Sep 29, 2022

cmsbuild added tests-started and removed tests-pending labels Sep 29, 2022

cmsbuild added tests-rejected and removed tests-started labels Sep 29, 2022

cleanup : yoda latest version already has fix this patching

f28b664

cmsbuild added tests-started and removed tests-rejected labels Sep 29, 2022

cmsbuild added tests-rejected and removed tests-started labels Sep 29, 2022

build sherpa with c++17 ( same as rivet)

af72550

cmsbuild added tests-started and removed tests-rejected labels Oct 9, 2022

cmsbuild added tests-started and removed tests-approved labels Oct 25, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 25, 2022

Delete relocateConfig SOURCES.txt

0b6554a

cmsbuild added externals-pending tests-pending and removed tests-approved externals-approved labels Oct 26, 2022

smuzaffar mentioned this pull request Oct 26, 2022

make sure all tool files are readable #8148

Merged

cmsbuild added tests-started and removed tests-pending labels Oct 26, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 26, 2022

cmsbuild added externals-approved and removed externals-pending labels Oct 27, 2022

smuzaffar merged commit 22592a9 into cms-sw:IB/CMSSW_12_6_X/master Oct 27, 2022

makortel mentioned this pull request Oct 28, 2022

Crashes in workflows 534 and 536 (Sherpa) cms-sw/cmssw#39914

Closed

cmsbuild mentioned this pull request Nov 1, 2022

Add patch to disable builting Rivet documentation #8158

Merged

smuzaffar mentioned this pull request Nov 1, 2022

Revert "Rivet 3.1.7 & YODA 1.9.7" #8163

Merged

mseidel42 mentioned this pull request Nov 24, 2022

Rivet 3.1.7 & YODA 1.9.7, disentagle from Sherpa #8198

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rivet 3.1.7 & YODA 1.9.7 #8109

Rivet 3.1.7 & YODA 1.9.7 #8109

mseidel42 commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

smuzaffar commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

smuzaffar commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

smuzaffar commented Sep 29, 2022

mseidel42 commented Sep 29, 2022

smuzaffar commented Sep 30, 2022 •

edited

Loading

mseidel42 commented Sep 30, 2022

smuzaffar commented Oct 9, 2022

cmsbuild commented Oct 9, 2022

cmsbuild commented Oct 10, 2022

smuzaffar commented Oct 25, 2022

cmsbuild commented Oct 25, 2022

smuzaffar commented Oct 26, 2022

cmsbuild commented Oct 26, 2022

smuzaffar commented Oct 26, 2022

cmsbuild commented Oct 26, 2022

smuzaffar commented Oct 27, 2022

smuzaffar commented Nov 1, 2022

mseidel42 commented Nov 1, 2022

Rivet 3.1.7 & YODA 1.9.7 #8109

Rivet 3.1.7 & YODA 1.9.7 #8109

Conversation

mseidel42 commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

smuzaffar commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

External Build

smuzaffar commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

External Build

smuzaffar commented Sep 29, 2022

mseidel42 commented Sep 29, 2022

smuzaffar commented Sep 30, 2022 • edited Loading

mseidel42 commented Sep 30, 2022

smuzaffar commented Oct 9, 2022

cmsbuild commented Oct 9, 2022

cmsbuild commented Oct 10, 2022

Comparison Summary

smuzaffar commented Oct 25, 2022

cmsbuild commented Oct 25, 2022

Comparison Summary

smuzaffar commented Oct 26, 2022

cmsbuild commented Oct 26, 2022

smuzaffar commented Oct 26, 2022

cmsbuild commented Oct 26, 2022

Comparison Summary

smuzaffar commented Oct 27, 2022

smuzaffar commented Nov 1, 2022

mseidel42 commented Nov 1, 2022

smuzaffar commented Sep 30, 2022 •

edited

Loading