XGBoost producer bugfix and unit test fix for non-x86_64 (14_0_X) #44532

smorovic · 2024-03-25T11:12:38Z

It has been reported that inference in the unit test does not pass on ARM and PPC architectures, so, in agreement with HLT, we leavel the unit test check only for x86_64 (which is sufficient for HLT workloads).

Below is the log of tests which were failing on ARM64 and ppc 64 LE. This always with same numerical values on these two architecture. Cause of that will be investigated.

src/RecoEgamma/PhotonIdentification/test/test_PhotonMvaXgb.cc:44: FAILED:
  CHECK_THAT( xgbScore, Catch::Matchers::WithinAbs(mva_score_v1[i], 0.0001) )
with expansion:
  0.91074f is within 0.0001 of 0.9863399863

src/RecoEgamma/PhotonIdentification/test/test_PhotonMvaXgb.cc:44: FAILED:
  CHECK_THAT( xgbScore, Catch::Matchers::WithinAbs(mva_score_v1[i], 0.0001) )
with expansion:
  0.82656f is within 0.0001 of 0.9750099778

src/RecoEgamma/PhotonIdentification/test/test_PhotonMvaXgb.cc:44: FAILED:
  CHECK_THAT( xgbScore, Catch::Matchers::WithinAbs(mva_score_v1[i], 0.0001) )
with expansion:
  0.0f is within 0.0001 of 0.00179

src/RecoEgamma/PhotonIdentification/test/test_PhotonMvaXgb.cc:44: FAILED:
  CHECK_THAT( xgbScore, Catch::Matchers::WithinAbs(mva_score_v1[i], 0.0001) )
with expansion:
  0.93808f is within 0.0001 of 0.9837399721

UPDATE: another commit was pushed. This fixes another problem with the XGBoost photon producer. It was not scaling e2x2 variable by SC energy (s4). It should not impact the unit test.

PR validation:

Unit test unchanged for x86_64 and should pass on all architectures now.

… ARM and PPC architectures, so leaving the check only for X86_64

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

backport of #44531

Update: caused by behavior of abs. As in the master PR, a fix switching to std::abs is pushed.

cmsbuild · 2024-03-25T11:13:02Z

A new Pull Request was created by @smorovic for CMSSW_14_0_X.

It involves the following packages:

RecoEgamma/PhotonIdentification (reconstruction)

@mandrenguyen, @jfernan2, @cmsbuild can you please review it and eventually sign? Thanks.
@a-kapoor, @afiqaize, @Sam-Harper, @Prasant1993, @valsdav, @lgray, @ram1123, @sameasy, @jainshilpi, @sobhatta, @missirol, @varuns23 this is something you requested to watch as well.
@sextonkennedy, @antoniovilela, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Backported from XGBoost producer bugfix and unit test fix for non-x86_64 (14_1_X) #44531

cmsbuild · 2024-03-25T11:13:03Z

cms-bot internal usage

mmusich · 2024-03-25T12:52:59Z

@cmsbuild, please test

cmsbuild · 2024-03-25T20:31:28Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca99f7/38385/summary.html
COMMIT: 3fcd932
CMSSW: CMSSW_14_0_X_2024-03-25-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44532/38385/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

6.06.0_SingleMuPt1/step2_SingleMuPt1.log
4.64.6_MinimumBias2010A/step2_MinimumBias2010A.log
4.764.76_ZMuSkim2012D/step2_ZMuSkim2012D.log

Expand to see more relval errors ...

8.0
9.0

Comparison Summary

Summary:

You potentially removed 61 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 52 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3243082
DQMHistoTests: Total failures: 1352
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3241708
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 163 edm output root files, 48 DQM output files
TriggerResults: no differences found

… ARM and PPC architectures. Apparently "abs" is a floating point variant round only on x86_64, so converting it to std::abs

cmsbuild · 2024-03-25T22:50:50Z

Pull request #44532 was updated. @cmsbuild, @mandrenguyen, @jfernan2 can you please check and sign again.

smorovic · 2024-03-25T23:00:19Z

please test

cmsbuild · 2024-03-26T10:50:27Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca99f7/38411/summary.html
COMMIT: a0ed168
CMSSW: CMSSW_14_0_X_2024-03-25-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44532/38411/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 60 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 57 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3243082
DQMHistoTests: Total failures: 1355
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3241705
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 163 edm output root files, 48 DQM output files
TriggerResults: no differences found

cmsbuild · 2024-03-27T16:06:57Z

Pull request #44532 was updated. @jfernan2, @cmsbuild, @mandrenguyen can you please check and sign again.

smorovic · 2024-03-27T16:33:14Z

@cmsbuild please test

cmsbuild · 2024-03-27T19:20:25Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca99f7/38461/summary.html
COMMIT: b365265
CMSSW: CMSSW_14_0_X_2024-03-27-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44532/38461/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 40 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3346148
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3346126
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 205 log files, 166 edm output root files, 49 DQM output files
TriggerResults: no differences found

mandrenguyen · 2024-03-31T05:33:10Z

type bug-fix

mandrenguyen · 2024-03-31T05:33:37Z

backport of #44531

mandrenguyen · 2024-03-31T05:35:20Z

type egamma

mandrenguyen · 2024-03-31T05:35:27Z

+1

cmsbuild · 2024-03-31T05:35:39Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_14_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_14_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2024-04-01T17:32:33Z

+1

cmsbuild added this to the CMSSW_14_0_X milestone Mar 25, 2024

cmsbuild added reconstruction-pending pending-signatures tests-pending orp-pending labels Mar 25, 2024

cmsbuild added tests-started and removed tests-pending labels Mar 25, 2024

cmsbuild added tests-rejected and removed tests-started labels Mar 25, 2024

it has been reported that inference in the unit test does not pass on…

a0ed168

… ARM and PPC architectures. Apparently "abs" is a floating point variant round only on x86_64, so converting it to std::abs

smorovic force-pushed the 140x-fix-xgboost-unittest branch from 3fcd932 to a0ed168 Compare March 25, 2024 22:50

cmsbuild added tests-pending and removed tests-rejected labels Mar 25, 2024

smorovic changed the title ~~Unit test for XGBoost photon inference only on x86_64 (14_0_X)~~ Unit test for XGBoost photon inference fix for non-x86_64 (14_0_X) Mar 25, 2024

cmsbuild added tests-started and removed tests-pending labels Mar 25, 2024

cmsbuild added tests-approved and removed tests-started labels Mar 26, 2024

fix: s4 is correct variable to be used with the model, not e2x2

b365265

cmsbuild added tests-pending and removed tests-approved labels Mar 27, 2024

cmsbuild added tests-started and removed tests-pending labels Mar 27, 2024

smorovic changed the title ~~Unit test for XGBoost photon inference fix for non-x86_64 (14_0_X)~~ XGBoost producer bugfix and unit test fix for non-x86_64 (14_0_X) Mar 27, 2024

cmsbuild added tests-approved and removed tests-started labels Mar 27, 2024

cmsbuild added the bug-fix label Mar 31, 2024

cmsbuild added the backport label Mar 31, 2024

cmsbuild added reconstruction-approved fully-signed egamma and removed reconstruction-pending pending-signatures labels Mar 31, 2024

cmsbuild added backport-ok and removed backport labels Mar 31, 2024

cmsbuild added orp-approved and removed orp-pending labels Apr 1, 2024

cmsbuild merged commit 692b88a into cms-sw:CMSSW_14_0_X Apr 1, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XGBoost producer bugfix and unit test fix for non-x86_64 (14_0_X) #44532

XGBoost producer bugfix and unit test fix for non-x86_64 (14_0_X) #44532

smorovic commented Mar 25, 2024 •

edited

Loading

cmsbuild commented Mar 25, 2024 •

edited

Loading

cmsbuild commented Mar 25, 2024 •

edited

Loading

mmusich commented Mar 25, 2024

cmsbuild commented Mar 25, 2024

cmsbuild commented Mar 25, 2024

smorovic commented Mar 25, 2024

cmsbuild commented Mar 26, 2024

cmsbuild commented Mar 27, 2024

smorovic commented Mar 27, 2024

cmsbuild commented Mar 27, 2024

mandrenguyen commented Mar 31, 2024

mandrenguyen commented Mar 31, 2024

mandrenguyen commented Mar 31, 2024

mandrenguyen commented Mar 31, 2024

cmsbuild commented Mar 31, 2024

rappoccio commented Apr 1, 2024

XGBoost producer bugfix and unit test fix for non-x86_64 (14_0_X) #44532

XGBoost producer bugfix and unit test fix for non-x86_64 (14_0_X) #44532

Conversation

smorovic commented Mar 25, 2024 • edited Loading

PR validation:

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Mar 25, 2024 • edited Loading

cmsbuild commented Mar 25, 2024 • edited Loading

mmusich commented Mar 25, 2024

cmsbuild commented Mar 25, 2024

RelVals-INPUT

Comparison Summary

cmsbuild commented Mar 25, 2024

smorovic commented Mar 25, 2024

cmsbuild commented Mar 26, 2024

Comparison Summary

cmsbuild commented Mar 27, 2024

smorovic commented Mar 27, 2024

cmsbuild commented Mar 27, 2024

Comparison Summary

mandrenguyen commented Mar 31, 2024

mandrenguyen commented Mar 31, 2024

mandrenguyen commented Mar 31, 2024

mandrenguyen commented Mar 31, 2024

cmsbuild commented Mar 31, 2024

rappoccio commented Apr 1, 2024

smorovic commented Mar 25, 2024 •

edited

Loading

cmsbuild commented Mar 25, 2024 •

edited

Loading

cmsbuild commented Mar 25, 2024 •

edited

Loading