Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

iarspider · 2025-01-22T14:31:27Z

Additional changes :

PR Unit test job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-unittests) - add GPU_FLAVOR parameter, select node based on it.
PR Relval job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-relvals/) - select node based on TEST_FLAVOR
Monitoring job (https://cmssdt.cern.ch/jenkins/job/jenkins-monitor-queue/) needs to be changed - kill pr test jobs that are stuck due to lumi node being offline
If ROCm job is killed due to node being offline, mark commit status!

cmsbuild · 2025-01-22T14:31:53Z

cms-bot internal usage

iarspider · 2025-01-22T14:55:00Z

please test with cms-sw/cmssw#46579

to check that cpu tests are not broken

cmsbuild · 2025-01-22T15:00:14Z

-1

Failed Tests: ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43916/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43916/install.sh to create a dev area with all the needed externals and cmssw changes.

CMS deprecated warnings: 1 CMS deprecated warnings found, see summary page for details.

Clang Build

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

iarspider · 2025-01-22T15:07:25Z

please test with cms-sw/cmssw#47163

cmsbuild · 2025-01-22T17:13:41Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43917/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43917/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 1664 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3819085
DQMHistoTests: Total failures: 149
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3818916
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 47 workflows

iarspider · 2025-01-23T08:39:49Z

test parameters:

enable = gpu
workflows_gpu = 141.044406,141.044408,141.044412,141.044414,141.044422,141.044424,160.03502

iarspider · 2025-01-23T08:39:53Z

please test

cmsbuild · 2025-01-23T17:05:19Z

Pull request #2418 was updated.

iarspider · 2025-01-23T17:19:57Z

please test

smuzaffar · 2025-03-01T12:10:42Z

@iarspider , links are broken on https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44749/summary.html page. I guess unittest/relval data is not uploaded

iarspider · 2025-03-01T21:38:02Z

please test

cmsbuild · 2025-03-01T21:38:30Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44758/summary.html
COMMIT: a51eebd
CMSSW: CMSSW_15_1_X_2025-02-28-1100/el8_amd64_gcc12
Additional Tests: CUDA
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2418/44758/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 3 lines to the logs
Reco comparison results: 17 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 114
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920166
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 47 workflows

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 1
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 1 DQM output files

cmsbuild · 2025-03-01T21:38:32Z

Pull request #2418 was updated.

cmsbuild · 2025-03-02T09:28:06Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44760/summary.html
COMMIT: a51eebd
CMSSW: CMSSW_15_1_X_2025-03-01-1100/el8_amd64_gcc12
Additional Tests: CUDA
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2418/44760/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 3 lines to the logs
Reco comparison results: 19 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 49
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920231
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 47 workflows

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 1
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 1 DQM output files

iarspider · 2025-03-03T10:21:40Z

please test

cmsbuild · 2025-03-03T10:21:56Z

Pull request #2418 was updated.

iarspider · 2025-03-03T12:40:25Z

please abort

iarspider · 2025-03-03T12:45:27Z

please test

cmsbuild · 2025-03-03T12:45:41Z

Pull request #2418 was updated.

cmsbuild · 2025-03-03T15:14:07Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44767/summary.html
COMMIT: 1d0c56c
CMSSW: CMSSW_15_1_X_2025-03-03-1100/el8_amd64_gcc12
Additional Tests: CUDA
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/44767/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 14 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 98
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920182
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 47 workflows

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 10
DQMHistoTests: Total histograms compared: 57865
DQMHistoTests: Total failures: 1239
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 56626
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 9 files compared)
Checked 50 log files, 48 edm output root files, 10 DQM output files
TriggerResults: no differences found

iarspider · 2025-03-03T15:17:13Z

All links are fixed now. Reminder: 1d0c56c should be reverted before merging.

iarspider · 2025-03-10T10:03:44Z

please test

cmsbuild · 2025-03-10T10:24:03Z

Pull request #2418 was updated.

cmsbuild added orp-pending pending-signatures externals-pending tests-pending labels Jan 22, 2025

cmsbuild added tests-started requires-external and removed tests-pending labels Jan 22, 2025

cmsbuild added tests-rejected and removed tests-started labels Jan 22, 2025

cmsbuild added tests-started and removed tests-rejected labels Jan 22, 2025

cmsbuild added tests-approved and removed tests-started labels Jan 22, 2025

cmsbuild added tests-started tests-pending and removed requires-external tests-approved tests-started labels Jan 23, 2025

cmsbuild added tests-started and removed tests-pending tests-started labels Jan 23, 2025

cmsbuild added tests-approved and removed tests-rejected labels Mar 1, 2025

Fix comparision URL

f67b598

cmsbuild added tests-started and removed tests-approved labels Mar 3, 2025

cmsbuild added tests-pending and removed tests-started labels Mar 3, 2025

[HACK] Use correct branch for running run-ib-pr-matrix.sh

1d0c56c

iarspider force-pushed the pr-rocm-tests branch from 17fc727 to 1d0c56c Compare March 3, 2025 12:44

cmsbuild added tests-started and removed tests-pending labels Mar 3, 2025

cmsbuild added tests-approved and removed tests-started labels Mar 3, 2025

Fix grouping of unit tests (CPU/GPU)

e217f79

cmsbuild added tests-started and removed tests-approved labels Mar 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

iarspider commented Jan 22, 2025 •

edited

Loading

cmsbuild commented Jan 22, 2025 •

edited

Loading

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

iarspider commented Jan 23, 2025 •

edited

Loading

iarspider commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

iarspider commented Jan 23, 2025

smuzaffar commented Mar 1, 2025

iarspider commented Mar 1, 2025

cmsbuild commented Mar 1, 2025

cmsbuild commented Mar 1, 2025

cmsbuild commented Mar 2, 2025

iarspider commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

iarspider commented Mar 3, 2025

iarspider commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

iarspider commented Mar 3, 2025

iarspider commented Mar 10, 2025

cmsbuild commented Mar 10, 2025

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

Are you sure you want to change the base?

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

Conversation

iarspider commented Jan 22, 2025 • edited Loading

cmsbuild commented Jan 22, 2025 • edited Loading

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

Clang Build

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

Comparison Summary

iarspider commented Jan 23, 2025 • edited Loading

iarspider commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

iarspider commented Jan 23, 2025

smuzaffar commented Mar 1, 2025

iarspider commented Mar 1, 2025

cmsbuild commented Mar 1, 2025

Comparison Summary

CUDA Comparison Summary

cmsbuild commented Mar 1, 2025

cmsbuild commented Mar 2, 2025

Comparison Summary

CUDA Comparison Summary

iarspider commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

iarspider commented Mar 3, 2025

iarspider commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

cmsbuild commented Mar 3, 2025

Comparison Summary

CUDA Comparison Summary

iarspider commented Mar 3, 2025

iarspider commented Mar 10, 2025

cmsbuild commented Mar 10, 2025

iarspider commented Jan 22, 2025 •

edited

Loading

cmsbuild commented Jan 22, 2025 •

edited

Loading

iarspider commented Jan 23, 2025 •

edited

Loading