Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[13_0_X] Update run3 data GTs #41611

Merged

Conversation

francescobrivio
Copy link
Contributor

PR description:

Backport of #41609

This PR updates the run3 data GTs:

Important Note:
In CMSSW_13_0_X we still had the 126X data GTs (both for online and offline) - so, with respect to the master PR, the GT differences linked here also show the tag cleanup that was included in 13_1_X (at the time master branch) in #41076.

GT differences:

PR validation:

Code compiles - will use the bot to run the matrix tests

Backport:

Backport of #41609

@francescobrivio
Copy link
Contributor Author

backport of #41609

@cmsbuild
Copy link
Contributor

cmsbuild commented May 10, 2023

A new Pull Request was created by @francescobrivio for CMSSW_13_0_X.

It involves the following packages:

  • Configuration/AlCa (alca)

@cmsbuild, @tvami, @saumyaphor4252, @francescobrivio can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @mmusich, @fabiocos, @tocheng this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

@francescobrivio
Copy link
Contributor Author

test parameters:

  • workflow = 1002.3

@francescobrivio
Copy link
Contributor Author

@cmsbuild please test

@cmsbuild
Copy link
Contributor

-1

Failed Tests: RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78aa34/32532/summary.html
COMMIT: 7d2ab9b
CMSSW: CMSSW_13_0_X_2023-05-09-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

----- Begin Fatal Exception 10-May-2023 13:36:11 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/DeepFlavourV04_12X_training/DeepJet_Run3_122X.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.8311_RunJetHT2017FreMINIAOD


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Types.py(840): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Mixins.py(376): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Mixins.py(511): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Config.py(1212): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Config.py(1478): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 10-May-2023 13:36:11 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/DeepFlavourV04_12X_training/DeepJet_Run3_122X.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.7611_RunJetHT2016EreMINIAOD


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Types.py(840): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Mixins.py(376): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Mixins.py(511): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Config.py(1212): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Config.py(1478): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 10-May-2023 13:36:11 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/DeepFlavourV04_12X_training/DeepJet_Run3_122X.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.88811_RunJetHT2018DreMINIAODUL


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Types.py(840): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Mixins.py(376): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Mixins.py(511): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Config.py(1212): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src/FWCore/ParameterSet/python/Config.py(1478): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------
Expand to see more relval errors ...

RelVals-INPUT

  • 4.64.6_MinimumBias2010A/step2_MinimumBias2010A.log
  • 136.72411136.72411_RunJetHT2016B_reminiaodUL/step2_RunJetHT2016B_reminiaodUL.log
  • 136.72412136.72412_RunJetHT2016B_reminiaodUL/step2_RunJetHT2016B_reminiaodUL.log
Expand to see more relval errors ...

AddOn Tests

----- Begin Fatal Exception 10-May-2023 13:41:39 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/DeepFlavourV04_12X_training/DeepJet_Run3_122X.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-addon/addOnTests/hlt_mc_GRun
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 10-May-2023 13:42:50 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/DeepFlavourV04_12X_training/DeepJet_Run3_122X.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-addon/addOnTests/hlt_mc_GRun
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 10-May-2023 13:40:07 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/DeepFlavourV04_12X_training/DeepJet_Run3_122X.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32532/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02784/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_X_2023-05-09-2300/external/el8_amd64_gcc11/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-addon/addOnTests/hlt_data_GRun
----- End Fatal Exception -------------------------------------------------
Expand to see more addon errors ...

@mmusich
Copy link
Contributor

mmusich commented May 10, 2023

well, https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/ib/CMSSW_13_0_X are broken as well, seems cms-sw/cmsdist#8493 wasn't merged at the same time as #41434.
Hopefully next IB will be better ;)

@francescobrivio
Copy link
Contributor Author

well, https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/ib/CMSSW_13_0_X are broken as well, seems cms-sw/cmsdist#8493 wasn't merged at the same time as #41434. Hopefully next IB will be better ;)

Thanks Marco! Will this morning IB already work? Or should I wait for tonight to re-trigger the tests?

@mmusich
Copy link
Contributor

mmusich commented May 10, 2023

Thanks Marco! Will this morning IB already work? Or should I wait for tonight to re-trigger the tests?

I am not sure in which IB cms-sw/cmsdist#8493 was picked up. Maybe @smuzaffar can help clarifying that.

@tvami
Copy link
Contributor

tvami commented May 10, 2023

test parameters:

@tvami
Copy link
Contributor

tvami commented May 10, 2023

Or should I wait for tonight to re-trigger the tests?

no need to wait, we can just test together with the cms dist PR

@mmusich
Copy link
Contributor

mmusich commented May 10, 2023

from

Screenshot from 2023-05-10 15-14-02

the morning IB should be fine.

@tvami
Copy link
Contributor

tvami commented May 10, 2023

@cmsbuild , please test

@smuzaffar
Copy link
Contributor

yes, CMSSW_13_0_X_2023-05-10-1200 which is still in build phase contains it.

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78aa34/32538/summary.html
COMMIT: 7d2ab9b
CMSSW: CMSSW_13_0_X_2023-05-09-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41611/32538/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78aa34/32538/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78aa34/32538/git-merge-result

Comparison Summary

Summary:

  • You potentially added 2042 lines to the logs
  • Reco comparison results: 8 differences found in the comparisons
  • DQMHistoTests: Total files compared: 17
  • DQMHistoTests: Total histograms compared: 626330
  • DQMHistoTests: Total failures: 29
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 626293
  • DQMHistoTests: Total skipped: 8
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 16 files compared)
  • Checked 148 log files, 70 edm output root files, 17 DQM output files
  • TriggerResults: no differences found

@tvami
Copy link
Contributor

tvami commented May 10, 2023

+alca

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next CMSSW_13_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_13_2_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@rappoccio
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 41a4614 into cms-sw:CMSSW_13_0_X May 12, 2023
@francescobrivio francescobrivio deleted the alca-update_data_run3_GTs_13_0_X branch June 9, 2023 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants