nano test scripts in CMSSW #39765

vlimant · 2022-10-19T08:24:29Z

PR description:

as discussed in #39452 the testing script for nano are being put in CMSSW to enable testing in PR integration

PR validation:

ran
`runTheMatrix.py --what nano -l 2500.601 --site "" --job-reports
./inspectNanoFile.py 2500.311_mc106Xul17v2+TTbarMINIAOD10.6_UL17v2+NANO_mc10.6ul17v2+HRV_NANO_mc/step2.root -j 2500.311-size.json
./inspectNanoFile.py 2500.601_mc126X+TTBarMINIAOD12.6+NANO_mc12.6+HRV_NANO_mc/step2.root -j 2500.601-size.json

./get_timing_from_jobreport.py 2500.311_mc106Xul17v2+TTbarMINIAOD10.6_UL17v2+NANO_mc10.6ul17v2+HRV_NANO_mc/JobReport2.xml 2500.311-timing.json
./get_timing_from_jobreport.py 2500.601_mc126X+TTBarMINIAOD12.6+NANO_mc12.6+HRV_NANO_mc/JobReport2.xml 2500.601-timing.json

./compare_sizes_json.py -H -f md 2500.601 2500.311 --ref "./" --base "{}-size.json,{}-timing.json" >> NANO_report.md`

to get a markdown print out that should be displayed in the summary.
N.B: --ref should point to a directory where we have reference files for comparison

cmsbuild · 2022-10-19T08:30:51Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39765/32628

This PR adds an extra 12KB to repository

cmsbuild · 2022-10-19T08:31:16Z

A new Pull Request was created by @vlimant (vlimant) for master.

It involves the following packages:

PhysicsTools/NanoAOD (xpog)

@cmsbuild, @swertz, @vlimant can you please review it and eventually sign? Thanks.
@gpetruc this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

smuzaffar · 2022-10-19T08:32:15Z

thanks @vlimant , I will update bot to run these and include the results in PR summary.

smuzaffar · 2022-10-19T10:36:38Z

@vlimant , if PhysicsTools/NanoAOD/test/runtests.sh is not needed then please also remove PhysicsTools/NanoAOD/test/BuildFile.xml ( which is using this runtests.sh script) and PhysicsTools/NanoAOD/test/runtestPhysicsToolsNanoAOD.cpp

cmsbuild · 2022-10-19T10:49:03Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39765/32636

This PR adds an extra 16KB to repository

cmsbuild · 2022-10-19T10:49:22Z

Pull request #39765 was updated. @cmsbuild, @swertz, @vlimant can you please check and sign again.

smuzaffar · 2022-10-19T11:05:30Z

@vlimant , looks like PhysicsTools/NanoAOD/test/dqmQualityTests.xml file is needed see cms-bot PR test results

----- Begin Fatal Exception 19-Oct-2022 12:50:50 CEST-----------------------
An exception of category 'StdException' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=QualityTester label='nanoDQMQTester'
Exception Message:
A std::exception was thrown.
/cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/1867/28357/CMSSW_12_6_X_2022-10-18-2300/poison/PhysicsTools/NanoAOD/test/dqmQualityTests.xml(1): expected <
----- End Fatal Exception -------------------------------------------------

cmsbuild · 2022-10-19T11:56:14Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39765/32639

This PR adds an extra 16KB to repository
Found files with invalid states:
- PhysicsTools/NanoAOD/test/dqmQualityTests.xml:
  - Deleted: 2766dc8
  - Added: 354f3fc

cmsbuild · 2022-10-19T11:56:39Z

Pull request #39765 was updated. @cmsbuild, @swertz, @vlimant can you please check and sign again.

smuzaffar · 2022-10-20T05:15:16Z

test parameters:

enable = nano
pull_request = Added nano relval comparison cms-bot#1867

smuzaffar · 2022-10-20T05:15:42Z

please test

cmsbuild · 2022-10-20T08:32:24Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7c3bc5/28386/summary.html
COMMIT: 354f3fc
CMSSW: CMSSW_12_6_X_2022-10-19-2300/el8_amd64_gcc10
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39765/28386/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
Reco comparison had 6 failed jobs
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3391158
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3391130
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 201 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-7c3bc5/2500.001_data80X+MuonEG2016HMINIAOD8.0+NANO_data8.0+HRV_NANO_data
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-7c3bc5/2500.0_mc80X+TTbarMINIAOD8.0+NANO_mc8.0+HRV_NANO_mc
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-7c3bc5/2500.101_mc94Xv2+TTbarMINIAOD9.4v2+NANO_mc9.4v2+HRV_NANO_mc
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-7c3bc5/2500.11_data94X2016+MuonEG2016MINIAOD9.4v2+NANO_2016data9.4v2+HRV_NANO_data
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-7c3bc5/2500.1_mc94X2016+TTbar2016MINIAOD9.4v2+NANO_2016mc9.4v2+HRV_NANO_mc

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 6
DQMHistoTests: Total histograms compared: 5223
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 5223
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 5 files compared)
Checked 12 log files, 0 edm output root files, 6 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.0	2.165	2.165	0.000 ( +0.0% )	7.59	7.79	-2.6%	1.445	1.841
2500.001	1.126	1.126	0.000 ( +0.0% )	11.16	11.61	-3.8%	1.450	1.836
2500.1	2.291	2.291	0.000 ( +0.0% )	5.81	5.97	-2.6%	1.807	2.357
2500.101	2.346	2.346	0.000 ( +0.0% )	4.93	5.09	-3.2%	1.954	2.396
2500.11	1.219	1.219	0.000 ( +0.0% )	8.87	9.24	-4.0%	1.657	2.052

vlimant · 2022-10-20T08:48:03Z

+1
excellent ! many thanks @smuzaffar

cmsbuild · 2022-10-20T08:48:24Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-10-20T09:14:29Z

@vlimant @smuzaffar do I understand correctly that this PR can get merged even before cms-sw/cms-bot#1867 is finalized? (I see there are still ongoing discussions in it)

smuzaffar · 2022-10-20T09:21:26Z

yes @perrotta this PR can be merged. If we needed any changes then those will go in to cms-bot.

vlimant · 2022-10-20T09:21:37Z

from the scripts POV in this PR, things are final indeed.

perrotta · 2022-10-20T09:24:20Z

+1

vlimant added 2 commits October 19, 2022 09:46

cleanup and add scripts from gitlab

2766dc8

executable. add file naming options

f81a295

cmsbuild added this to the CMSSW_12_6_X milestone Oct 19, 2022

cmsbuild added code-checks-pending orp-pending pending-signatures tests-pending xpog-pending labels Oct 19, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 19, 2022

smuzaffar mentioned this pull request Oct 19, 2022

Added nano relval comparison cms-sw/cms-bot#1867

Merged

add a branch to branch comparison script

e843c8e

cmsbuild added code-checks-pending and removed code-checks-approved labels Oct 19, 2022

remove unused files

3f59dcb

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 19, 2022

add dqm quality test back

354f3fc

cmsbuild added code-checks-pending and removed code-checks-approved labels Oct 19, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 19, 2022

cmsbuild added the requires-external label Oct 20, 2022

cmsbuild added tests-started and removed tests-pending labels Oct 20, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 20, 2022

cmsbuild added fully-signed xpog-approved and removed pending-signatures xpog-pending labels Oct 20, 2022

cmsbuild added orp-approved and removed orp-pending labels Oct 20, 2022

cmsbuild merged commit b3fa723 into cms-sw:master Oct 20, 2022

vlimant deleted the nano_scripts_from_gitlab branch November 22, 2022 10:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nano test scripts in CMSSW #39765

nano test scripts in CMSSW #39765

vlimant commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

smuzaffar commented Oct 19, 2022

smuzaffar commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

smuzaffar commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

smuzaffar commented Oct 20, 2022

smuzaffar commented Oct 20, 2022

cmsbuild commented Oct 20, 2022

vlimant commented Oct 20, 2022

cmsbuild commented Oct 20, 2022

perrotta commented Oct 20, 2022

smuzaffar commented Oct 20, 2022 •

edited

Loading

vlimant commented Oct 20, 2022 •

edited

Loading

perrotta commented Oct 20, 2022

nano test scripts in CMSSW #39765

nano test scripts in CMSSW #39765

Conversation

vlimant commented Oct 19, 2022

PR description:

PR validation:

cmsbuild commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

smuzaffar commented Oct 19, 2022

smuzaffar commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

smuzaffar commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

smuzaffar commented Oct 20, 2022

smuzaffar commented Oct 20, 2022

cmsbuild commented Oct 20, 2022

Comparison Summary

NANO Comparison Summary

vlimant commented Oct 20, 2022

cmsbuild commented Oct 20, 2022

perrotta commented Oct 20, 2022

smuzaffar commented Oct 20, 2022 • edited Loading

vlimant commented Oct 20, 2022 • edited Loading

perrotta commented Oct 20, 2022

smuzaffar commented Oct 20, 2022 •

edited

Loading

vlimant commented Oct 20, 2022 •

edited

Loading