DD4hep RelVal for Phase2: update geometry configuration and PyReleaseValidation

[fabiocos]

PR description:

This PR extends #30003 introducing DD4hep geometry configurations and corresponding RelVal test workflows for Phase2 scenarios. In the initial version of this PR only the geometry D49 configuration is explicitly added (while workflows are possibly built for any Phase2 scenario). If there is agreement on this strategy I may update the PR by adding configurations for any active scenario, in any case the generateGeometry.py script is instructed to produce whatever Phase2 DD4hep configuration required. I add a few fixes to the geometry dictionaries to keep in synchrony them with the latest version available (for 2021) and have the dd4hep modifier available for MTD (other sub-detectors will have to check).

PR validation:

Code compiles and produces the desired configurations, tested for 2021 and 2026D49 both as compressed and expanded python configuration. The workflow list shows workflows for all scenarios, but their building fails but for D49, as the geometry configurations are not provided for the time being.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31975/19436

• This PR adds an extra 40KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/Geometry/python/dict2026Geometry.py modified in PR(s): MTD geometry/reconstruction: define new ETL reco layer geometry, update LGAD pixel structure, activate workflows for scenarios D72 and D73 #31765, update the tracking material for Phase2 geometry D49 #30126
  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Patatrack integration - Pixel workflows (12/N) #31854, Adding ability to output PF Candidate information in JME NTuples #31795, MTD geometry/reconstruction: define new ETL reco layer geometry, update LGAD pixel structure, activate workflows for scenarios D72 and D73 #31765, Ecal phase2 new digitization 11 2 x #31726

[cmsbuild]

A new Pull Request was created by @fabiocos (Fabio Cossutti) for master.

It involves the following packages:

Configuration/Geometry
Configuration/PyReleaseValidation
Configuration/StandardSequences

@civanch, @Dr15Jones, @jordan-martins, @chayanit, @cvuosalo, @wajidalikhan, @ianna, @mdhildreth, @cmsbuild, @makortel, @franzoni, @silviodonato, @kpedro88, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@vargasa, @makortel, @felicepantaleo, @GiacomoSguazzoni, @rovere, @VinInn, @Martin-Grunewald, @lecriste, @mtosi, @dgulhan, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[fabiocos]

please test

[cmsbuild]

The tests are being triggered in jenkins.

• CMSSW_11_2_X_2020-10-27-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10359/console Started: 2020/10/28 16:27

[fabiocos]

16:26 farmui01 1668> runTheMatrix.py -n --what upgrade | grep DD4hep | grep D49
23207.911 2026D49_DD4hep+SingleMuPt10_Eta2p85_GenSimHLBeamSpot+DigiTrigger+RecoGlobal+HARVESTGlobal 
23234.911 2026D49_DD4hep+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal 
23250.911 2026D49_DD4hep+ZMM_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal 
23407.911 2026D49PU_DD4hep+SingleMuPt10_Eta2p85_GenSimHLBeamSpot+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU 
23434.911 2026D49PU_DD4hep+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU 
23450.911 2026D49PU_DD4hep+ZMM_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU 
16:26 farmui01 1669> runTheMatrix.py -n --what upgrade | grep DD4hep | wc -l
90

[kpedro88]

this condition is not complete (doesn't return false when the condition is not met)
I prefer avoiding the redundant if true return true construction:

        return ((fragment=='TTbar_13' or fragment=='ZMM_13' or fragment=='SingleMuPt10') and '2021' in key) \
            or ((fragment=='TTbar_14TeV' or fragment=='ZMM_14' or fragment=='SingleMuPt10') and '2026' in key)

[fabiocos]

right, I forgot the False condition, I'll update that

[kpedro88]

@fabiocos many thanks for following up on the discussion in #31902! the implementation here looks generally reasonable to me.

However, this PR will fail the unit test, which expects every available geometry cff to be generated. Therefore, all the configs should be added here.

[fabiocos]

@kpedro88 I was waiting for a comment, as said in the description I find reasonable to have all the possible scenarios available. I will update the PR, there is indeed a value in having all the scenarios available, apart for the unit test: for instance for MTD different parts of the code are probed when running different scenarios, and if we want to ensure that DD4hep is really working we need to run all of them (or at least a selection of them).

[cmsbuild]

+1
Tested at: 6d468df
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4de8e/10372/summary.html
CMSSW: CMSSW_11_2_X_2020-10-28-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4de8e/10372/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 35
• DQMHistoTests: Total histograms compared: 2544092
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2544063
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
• Checked 149 log files, 22 edm output root files, 35 DQM output files

[cvuosalo]

+1

[kpedro88]

+upgrade

[silviodonato]

+1

[chayanit]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.