Using consistent Eigen alignment settings

[smuzaffar]

This is follow up on #9043. This change allows

• to use consistent build flags for project depending on eigen
• enable vectorization build for lwtnn, opencv and pytorch packages

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_14_1_X/master.

@iarspider, @aandvalenzuela, @cmsbuild, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

please test

[smuzaffar]

please test for CMSSW_14_1_MULTIARCHS_X/el8_amd64_gcc12

[makortel]

Thanks @smuzaffar. I have two questions

• In [DO NOT MERGE] Test making Eigen alignment setting consistent #9043 (comment) you discovered that lwtnn 2.13 overrides the CMAKE_CXX_FLAGS. Should this PR update lwtnn 2.14.1?
• Do we care about professor2? It depends on eigen, but as far as I can tell nothing in CMSSW depends on it. (I'm also fine if we deal with it later, given the urgency)

[cmsbuild]

Pull request #9060 was updated.

[smuzaffar]

ah my bad @makortel , yes lwtnn should be updated ( I already have updated the PR).
For professor2, I am still running some local tests. As it is not directly used in cmssw so I will update it in a separate PR

[smuzaffar]

please test

[smuzaffar]

please test for CMSSW_14_1_MULTIARCHS_X/el8_amd64_gcc12

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aaee32/38036/summary.html
COMMIT: a12ad75
CMSSW: CMSSW_14_1_X_2024-03-11-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9060/38036/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 44 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 181 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3297383
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3297360
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: found differences in 5 / 46 workflows

[smuzaffar]

+externals

[smuzaffar]

lets get this in for next 14.1.X IB.

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_14_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @antoniovilela, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[fwyzard]

enable vectorization build for lwtnn, opencv and pytorch packages

@smuzaffar it looks like those packages are still using only -msse3 ?

[smuzaffar]

Yes @fwyzard , for default (non multiarch build) IBs these packages will still use -msse3. This PR only adds these packages in to list of packages which can be build for multi-archs. #9061 PR actually build these packages for multi-archs (see logs https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b88baa/38046/externals/ )

[29/38] : && /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/g++ \
-fPIC -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -march=x86-64-v3 \
-Wall -pedantic -Wextra -Wcomment -Wunused-value -O3 -DNDEBUG  \
CMakeFiles/lwtnn-dump-config.dir/src/lwtnn-dump-config.cxx.o -o bin/lwtnn-dump-config  lib/liblwtnn-stat.a && :

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aaee32/38040/summary.html
COMMIT: a12ad75
CMSSW: CMSSW_14_1_MULTIARCHS_X_2024-03-10-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9060/38040/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @rovere Bring HLT Phase2 simplified timing menu variant up to date. cmssw#44041
• @swagata87 Hack EgammaHLTR9IDProducer to be able to run it in Phase2 HLT cmssw#44358
• @fwyzard Remove invalid tool from L1Trigger/L1TZDC cmssw#44361
• @rovere Sequence-based simplified HLT phase2 menu. cmssw#44025
• @NJManganelli Phase 2 GTT Fix ptbits z0bits of GTT ObjectWords cmssw#44334

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aaee32/38040/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aaee32/38040/git-merge-result

Comparison Summary

Summary:

• You potentially removed 589 lines from the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 68093 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3297383
• DQMHistoTests: Total failures: 147145
• DQMHistoTests: Total nulls: 272
• DQMHistoTests: Total successes: 3149946
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -4.849 KiB( 47 files compared)
• DQMHistoSizes: changed ( 13034.0 ): -0.854 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 140.063 ): -0.004 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 141.042 ): 0.004 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 141.044 ): -0.031 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 141.046 ): -0.012 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 250202.181 ): -0.421 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 25202.0 ): -0.225 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 7.3 ): 1.533 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 8.0 ): -4.839 KiB SiStrip/MechanicalView
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: found differences in 17 / 46 workflows

[fwyzard]

OK, thanks.

I see that the default build has

cmake ../lwtnn-2.14.1 -G Ninja -DCMAKE_CXX_COMPILER=g++ '-DCMAKE_CXX_FLAGS=-fPIC -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -msse3' ...

while the v3 build has

cmake ../lwtnn-2.14.1 -G Ninja -DCMAKE_CXX_COMPILER=g++ '-DCMAKE_CXX_FLAGS=-fPIC -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -march=x86-64-v3' ...

but, looking at the spec files, I don't see what makes the change between the two !

[smuzaffar]

@fwyzard , to avoid maintaining two spec files (e.g. lwtnn.spec and lwtnn_x86-64-v3.spec) cmsBuild has a way of auto generating <package>_<micro-arch>.spec from <package>.spec. For multi-arch packages, it uses https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_14_1_X/master/vectorization/cmsdist_packages.py to auto generate <package>_x86-64-v3.spec by replacing the default micro-arch flag (-msse3) with the selected micro-arch flag (-march=x86-64-v3)

[fwyzard]

Ah... I see, thanks.