[Do not merge] testing full build of externals on new VMs

[smuzaffar]

No description provided.

[smuzaffar]

test parameters:

• full_cmssw = true
• jenkins_node = cmsbuild916

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_14_2_X/master.

@aandvalenzuela, @cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

please test

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f8e7db/42648/summary.html
COMMIT: eee46e5
CMSSW: CMSSW_14_2_X_2024-11-07-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9509/42648/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 7 errors in the following unit tests:

---> test testPrintTkMaps had ERRORS
---> test GeometryMTDCommonDataTestDriver had ERRORS
---> test GeometryMTDGeometryBuilderTestDriver had ERRORS
and more ...

Comparison Summary

Summary:

• You potentially removed 129 lines from the logs
• Reco comparison results: 20909 differences found in the comparisons
• DQMHistoTests: Total files compared: 46
• DQMHistoTests: Total histograms compared: 3343138
• DQMHistoTests: Total failures: 23112
• DQMHistoTests: Total nulls: 8
• DQMHistoTests: Total successes: 3319998
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 45 files compared)
• DQMHistoSizes: changed ( 2024.000001 ): -0.008 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 2024.303001 ): 0.012 KiB JetMET/SUSYDQM
• Checked 195 log files, 172 edm output root files, 46 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f8e7db/42695/summary.html
COMMIT: eee46e5
CMSSW: CMSSW_14_2_X_2024-11-08-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9509/42695/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @fabiocos MTD geometry: update unit tests for default Phase2 scenario definition cmssw#46621
• @smuzaffar [UBSAN]Initialize data member to fix runtime error from ubsan cmssw#46637

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f8e7db/42695/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f8e7db/42695/git-merge-result

Comparison Summary

Summary:

• You potentially removed 167 lines from the logs
• Reco comparison results: 5 differences found in the comparisons
• DQMHistoTests: Total files compared: 46
• DQMHistoTests: Total histograms compared: 3343588
• DQMHistoTests: Total failures: 390
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3343178
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
• Checked 202 log files, 172 edm output root files, 46 DQM output files
• TriggerResults: no differences found

[smuzaffar]

New VMs based on c4.2xlarge.eph performs much better than our existing c3.2xlarge VMs.
This PR rebuilt all externals (including GCC and selected externals for multi-microarchs), full cmssw and ran all unit tests

• c4.3xlarge.eph: 9h45m
• c3.2xlarge: 19h