Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[14.0.X] Update 2024 geometry and GEM alignment conditions in 2024 MC GTs #44512

Merged

Conversation

saumyaphor4252
Copy link
Contributor

@saumyaphor4252 saumyaphor4252 commented Mar 22, 2024

backport of #44497

Note that the GT versions here are different than the ones in master PR because descoping was done for #44598 (comment) which increased the version numbers of the GTs for this backport. But the net effect of the master and backport PR is the same, with just the geometry update.

PR description:

The PR updates the 2024 geometry and compatible GEM alignment conditions tags in the relevant MC GTs:

  • 140X_mcRun3_2024_design_v9
  • 140X_mcRun3_2024_realistic_v10
  • 140X_mcRun3_2024cosmics_realistic_deco_v10
  • 140X_mcRun3_2024cosmics_design_deco_v8

More details with the full list of tag updates and links to relevant presentations can be found in the CMS Talk request in [1].

[1] https://cms-talk.web.cern.ch/t/gt-mc-data-update-of-2024-gem-and-rpc-geomtery-conditions-for-data-and-mc/36739

GT Differences with the last ones are here:

PR validation:

See master PR #44497
See also the Full Track validation in https://its.cern.ch/jira/browse/CMSALCA-260

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Backport of #44497

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 22, 2024

A new Pull Request was created by @saumyaphor4252 for CMSSW_14_0_X.

It involves the following packages:

  • Configuration/AlCa (alca)

@cmsbuild, @perrotta, @consuegs, @saumyaphor4252 can you please review it and eventually sign? Thanks.
@tocheng, @yuanchao, @fabiocos, @missirol, @mmusich, @Martin-Grunewald, @rsreds this is something you requested to watch as well.
@antoniovilela, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 22, 2024

cms-bot internal usage

@saumyaphor4252
Copy link
Contributor Author

test parameters:

  • workflows = 13050.0,13049.0,13046.0,13040.0,12933.0,12841.0,12802.0,12824.0,12834.1,13052.0
  • relvals_opt = --what upgrade,standard

@saumyaphor4252
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1870ea/38344/summary.html
COMMIT: 2b8d06f
CMSSW: CMSSW_14_0_X_2024-03-21-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44512/38344/install.sh to create a dev area with all the needed externals and cmssw changes.

  • DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

  • You potentially removed 14 lines from the logs
  • Reco comparison results: 27494 differences found in the comparisons
  • DQMHistoTests: Total files compared: 59
  • DQMHistoTests: Total histograms compared: 4917074
  • DQMHistoTests: Total failures: 70391
  • DQMHistoTests: Total nulls: 54
  • DQMHistoTests: Total successes: 4846607
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 2489.296 KiB( 58 files compared)
  • DQMHistoSizes: changed ( 12802.0,... ): 111.859 KiB GEM/RecHits
  • DQMHistoSizes: changed ( 12802.0,... ): 75.433 KiB GEM/Digis
  • DQMHistoSizes: changed ( 12802.0,... ): 1.781 KiB GEM/DAQStatus
  • DQMHistoSizes: changed ( 12802.0,... ): 1.094 KiB GEM/EventInfo
  • DQMHistoSizes: changed ( 12802.0,... ): 0.614 KiB GEM/SimHits
  • DQMHistoSizes: changed ( 12802.0,... ): 0.531 KiB GEM/Pad
  • DQMHistoSizes: changed ( 12802.0,... ): 0.432 KiB GEM/PadCluster
  • DQMHistoSizes: changed ( 13040.0 ): -2.999 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 13046.0 ): -1.470 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 13050.0 ): 0.005 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 13052.0 ): ...
  • Checked 249 log files, 214 edm output root files, 59 DQM output files
  • TriggerResults: found differences in 11 / 57 workflows

@saumyaphor4252
Copy link
Contributor Author

The reason for holding this PR in 140X is some degradation and changes observed in muon reconstruction from the GT update in the ongoing Full Track Validation in https://its.cern.ch/jira/browse/CMSALCA-260, which are not fully understood and expected. See for eg:

image
image

Also FYI @bsunanda @cms-sw/muon-dpg-l2 @cms-sw/muon-pog-l2 (maybe also @cms-sw/upgrade-l2 )

We are going to do one last check with another run, but otherwise we will be considering to revert the changes until this is understood.

Also @cms-sw/muon-dpg-l2 @cms-sw/muon-pog-l2 it would be great if you could provide some feedback on how critical these changes would be by taking a look at the validation plots in https://its.cern.ch/jira/browse/CMSALCA-260.

@saumyaphor4252 saumyaphor4252 force-pushed the alca-140X-Update_GEM_geometry_2024 branch from 2b8d06f to 0af60d9 Compare April 22, 2024 20:45
@cmsbuild
Copy link
Contributor

Pull request #44512 was updated. @saumyaphor4252, @cmsbuild, @perrotta, @consuegs can you please check and sign again.

@cmsbuild
Copy link
Contributor

Pull request #44512 was updated. @consuegs, @saumyaphor4252, @cmsbuild, @perrotta can you please check and sign again.

@saumyaphor4252
Copy link
Contributor Author

@cmsbuild please test

Following on #44512 (comment): more details can be found in https://its.cern.ch/jira/browse/CMSALCA-260 but to summarize:

  • The degradation is highly reduced with the validation using a comparatively recent run 370776 and is understood from the ideal alignment scenario. The alignment will soon be updated with 2024 13.6Tev data to fix this (See this comment)
  • The change in the STA muon/Global muon chi2/ndof at negative eta is understood from the fact that the alignment corrections are larger in that region ( See this comment)

@cmsbuild
Copy link
Contributor

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1870ea/39010/summary.html
COMMIT: 166b518
CMSSW: CMSSW_14_0_X_2024-04-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1870ea/39010/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1870ea/39010/git-merge-result

  • DAS Queries: The DAS query tests failed, see the summary page for details.

RelVals

----- Begin Fatal Exception 23-Apr-2024 00:04:07 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/UParTAK4/PUPPI/V00/UParTAK4.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/CMSSW_14_0_X_2024-04-22-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/CMSSW_14_0_X_2024-04-22-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/CMSSW_14_0_X_2024-04-22-1100/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/external/el8_amd64_gcc12/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.8311_RunJetHT2017FreMINIAOD


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Types.py(881): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Mixins.py(381): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Mixins.py(516): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Config.py(1216): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Config.py(1489): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 23-Apr-2024 00:04:07 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/UParTAK4/PUPPI/V00/UParTAK4.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/CMSSW_14_0_X_2024-04-22-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/CMSSW_14_0_X_2024-04-22-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/CMSSW_14_0_X_2024-04-22-1100/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/external/el8_amd64_gcc12/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.7611_RunJetHT2016EreMINIAOD


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Types.py(881): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Mixins.py(381): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Mixins.py(516): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Config.py(1216): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Config.py(1489): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 23-Apr-2024 00:04:07 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/UParTAK4/PUPPI/V00/UParTAK4.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/CMSSW_14_0_X_2024-04-22-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/CMSSW_14_0_X_2024-04-22-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39010/CMSSW_14_0_X_2024-04-22-1100/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/external/el8_amd64_gcc12/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.88811_RunJetHT2018DreMINIAODUL


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Types.py(881): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Mixins.py(381): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Mixins.py(516): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Config.py(1216): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02834/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_X_2024-04-22-1100/src/FWCore/ParameterSet/python/Config.py(1489): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------
Expand to see more relval errors ...

RelVals-INPUT

  • 4.64.6_MinimumBias2010A/step2_MinimumBias2010A.log
  • 136.72411136.72411_RunJetHT2016B_reminiaodUL/step2_RunJetHT2016B_reminiaodUL.log
  • 136.72412136.72412_RunJetHT2016B_reminiaodUL/step2_RunJetHT2016B_reminiaodUL.log
Expand to see more relval errors ...

@perrotta
Copy link
Contributor

please test

  • Errors are related to the failing das queries (exception of category 'FileInPathError'): let try with the new IB now available

@mmusich
Copy link
Contributor

mmusich commented Apr 23, 2024

Errors are related to the failing das queries (exception of category 'FileInPathError'): let try with the new IB now available

not related to das queries, but rather to the fact that the last test dragged in #44660 without the corresponding cms-data update.

@perrotta
Copy link
Contributor

backport of #44594

Please notice that the following GTs are identical:

  • 140X_mcRun3_2024_design_v9 and 140X_mcRun3_2024_design_v7
  • 140X_mcRun3_2024_realistic_v10 and 140X_mcRun3_2024_realistic_v8
  • 140X_mcRun3_2024cosmics_realistic_deco_v10 and 140X_mcRun3_2024cosmics_realistic_deco_v8
  • 140X_mcRun3_2024cosmics_design_deco_v8 and 140X_mcRun3_2024cosmics_design_deco_v6

Therefore, this PR is backporting exactly what is currently in the master of cmssw, which was merged with #44594

@perrotta
Copy link
Contributor

Errors are related to the failing das queries (exception of category 'FileInPathError'): let try with the new IB now available

not related to das queries, but rather to the fact that the last test dragged in #44660 without the corresponding cms-data update.

Thank you @mmusich for the clarification.
In any case, tests must be rerun (and with the new IB which already contains #44660 and he corresponding cms-data update they are expected to succeed)

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1870ea/39017/summary.html
COMMIT: 166b518
CMSSW: CMSSW_14_0_X_2024-04-22-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44512/39017/install.sh to create a dev area with all the needed externals and cmssw changes.

  • DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

  • You potentially removed 2 lines from the logs
  • Reco comparison results: 27955 differences found in the comparisons
  • DQMHistoTests: Total files compared: 58
  • DQMHistoTests: Total histograms compared: 4917344
  • DQMHistoTests: Total failures: 69519
  • DQMHistoTests: Total nulls: 54
  • DQMHistoTests: Total successes: 4847751
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 2489.2959999999994 KiB( 57 files compared)
  • DQMHistoSizes: changed ( 12834.0,... ): 111.859 KiB GEM/RecHits
  • DQMHistoSizes: changed ( 12834.0,... ): 75.433 KiB GEM/Digis
  • DQMHistoSizes: changed ( 12834.0,... ): 1.781 KiB GEM/DAQStatus
  • DQMHistoSizes: changed ( 12834.0,... ): 1.094 KiB GEM/EventInfo
  • DQMHistoSizes: changed ( 12834.0,... ): 0.614 KiB GEM/SimHits
  • DQMHistoSizes: changed ( 12834.0,... ): 0.531 KiB GEM/Pad
  • DQMHistoSizes: changed ( 12834.0,... ): 0.432 KiB GEM/PadCluster
  • DQMHistoSizes: changed ( 13034.0 ): -0.980 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 13040.0 ): -2.999 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 13046.0 ): -1.470 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 13050.0 ): ...
  • Checked 246 log files, 213 edm output root files, 58 DQM output files
  • TriggerResults: found differences in 11 / 56 workflows

@perrotta
Copy link
Contributor

+alca

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next CMSSW_14_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_14_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

@antoniovilela
Copy link
Contributor

+alca

Hi Andrea, sorry if this was discussed yesterday and I missed. Is this PR meant for 14_0_6?
Thanks.

@perrotta
Copy link
Contributor

+alca

Hi Andrea, sorry if this was discussed yesterday and I missed. Is this PR meant for 14_0_6? Thanks.

Hi Antonio.
Yesterday I was traveling and I was not able to connect to the ORP.
In any case, this PR is not for data taking, as it only updates some MC GTs. Therefore it is not urgently needed in 14_0_6. It doesn't hurt also: therefore if you still have to build it, feel free to decide whether to include it or not.
Thank you

@antoniovilela
Copy link
Contributor

+alca

Hi Andrea, sorry if this was discussed yesterday and I missed. Is this PR meant for 14_0_6? Thanks.

Hi Antonio. Yesterday I was traveling and I was not able to connect to the ORP. In any case, this PR is not for data taking, as it only updates some MC GTs. Therefore it is not urgently needed in 14_0_6. It doesn't hurt also: therefore if you still have to build it, feel free to decide whether to include it or not. Thank you

Thanks.

@antoniovilela
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit ab9ed35 into cms-sw:CMSSW_14_0_X Apr 24, 2024
11 checks passed
@saumyaphor4252 saumyaphor4252 deleted the alca-140X-Update_GEM_geometry_2024 branch November 1, 2024 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants