[TF] Build with tensorflow_mkldnn_contraction_kernel=0

[smuzaffar]

see cms-sw/cmssw#42444 for details

[smuzaffar]

please test

[cmsbuild]

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_14_0_X/master.

@smuzaffar, @iarspider, @aandvalenzuela can you please review it and eventually sign? Thanks.
@sextonkennedy, @antoniovilela, @rappoccio you are the release manager for this.
cms-bot commands are listed here

[smuzaffar]

please test for el8_ppc64le_gcc12

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-155f43/36351/summary.html
COMMIT: 73d81e6
CMSSW: CMSSW_14_0_X_2023-12-06-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8861/36351/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 37 lines to the logs
• Reco comparison results: 538 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3430794
• DQMHistoTests: Total failures: 5842
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3424930
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 214 log files, 167 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[cmsbuild]

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-155f43/36355/summary.html
COMMIT: 73d81e6
CMSSW: CMSSW_14_0_X_2023-12-05-2300/el8_ppc64le_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8861/36355/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

• 24896.0A fatal system signal has occurred: segmentation violation
• 24900.0A fatal system signal has occurred: segmentation violation
• 24834.0A fatal system signal has occurred: segmentation violation

Expand to see more relval errors ...

• 23234.0
• 25034.999
• 24834.911

[smuzaffar]

please test for el8_aarch64_gcc12

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-155f43/36363/summary.html
COMMIT: 73d81e6
CMSSW: CMSSW_14_0_X_2023-12-06-2300/el8_aarch64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8861/36363/install.sh to create a dev area with all the needed externals and cmssw changes.

[smuzaffar]

please test for CMSSW_14_0_SKYLAKEAVX512_X

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-155f43/36375/summary.html
COMMIT: 73d81e6
CMSSW: CMSSW_14_0_SKYLAKEAVX512_X_2023-12-07-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8861/36375/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 415 lines from the logs
• Reco comparison results: 72516 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3430794
• DQMHistoTests: Total failures: 164238
• DQMHistoTests: Total nulls: 290
• DQMHistoTests: Total successes: 3266244
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -2.4320000000000004 KiB( 49 files compared)
• DQMHistoSizes: changed ( 10224.0 ): -0.127 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 12634.0 ): 2.589 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 141.044 ): -0.035 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 250202.181 ): -0.422 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 25202.0 ): 0.352 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 7.3 ): -2.610 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 8.0 ): -2.179 KiB SiStrip/MechanicalView
• Checked 214 log files, 167 edm output root files, 50 DQM output files
• TriggerResults: found differences in 22 / 48 workflows

[gartung]

@smuzaffar is there a way to make this available as a stand alone install for IB's. I would like to be able to setup this version of tensorflow, then run profiling.

[smuzaffar]

@gartung , only trivial way to do it is to setup a dedicated IB. I will do it once #8972 is merged ( which allows to make tensorflow_mkldnn_contraction_kernel a configurable build option)

[smuzaffar]

closing in favor of #8972

[smuzaffar]

@gartung , CMSSW_14_0_MKLDNN0_X_2024-01-28-2300 IB is now available where TF is build with tensorflow_mkldnn_contraction_kernel=0 option.