Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added nano relval comparison #1867

Merged
merged 9 commits into from
Oct 23, 2022
Merged

Added nano relval comparison #1867

merged 9 commits into from
Oct 23, 2022

Conversation

smuzaffar
Copy link
Contributor

No description provided.

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch master.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.
cms-bot commands are listed here

@smuzaffar
Copy link
Contributor Author

test parameters:

@smuzaffar
Copy link
Contributor Author

please test

@smuzaffar
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

Pull request #1867 was updated.

@cmsbuild
Copy link
Contributor

Pull request #1867 was updated.

@smuzaffar
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

Pull request #1867 was updated.

@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 19, 2022 •
edited by smuzaffar
Loading

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74a38f/28376/summary.html
COMMIT: 430f27c
CMSSW: CMSSW_12_6_X_2022-10-19-1100/el8_amd64_gcc10
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/1867/28376/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 4 differences found in the comparisons
  • Reco comparison had 6 failed jobs
  • DQMHistoTests: Total files compared: 48
  • DQMHistoTests: Total histograms compared: 3391158
  • DQMHistoTests: Total failures: 6
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3391130
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
  • Checked 201 log files, 48 edm output root files, 48 DQM output files
  • TriggerResults: no differences found

NANO Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.001_data80X+MuonEG2016HMINIAOD8.0+NANO_data8.0+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.0_mc80X+TTbarMINIAOD8.0+NANO_mc8.0+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.101_mc94Xv2+TTbarMINIAOD9.4v2+NANO_mc9.4v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.11_data94X2016+MuonEG2016MINIAOD9.4v2+NANO_2016data9.4v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.1_mc94X2016+TTbar2016MINIAOD9.4v2+NANO_2016mc9.4v2+HRV_NANO_mc

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 6
  • DQMHistoTests: Total histograms compared: 5223
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 5223
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 5 files compared)
  • Checked 12 log files, 0 edm output root files, 6 DQM output files
  • Nano Size comparison results
Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.0 2.165 2.165 0.000 ( +0.0% ) 7.70 7.59 +1.4% 1.854 1.420
2500.001 1.126 1.126 0.000 ( +0.0% ) 11.39 11.15 +2.2% 1.769 1.386
2500.1 2.291 2.291 0.000 ( +0.0% ) 5.92 5.84 +1.4% 2.386 1.657
2500.101 2.346 2.346 0.000 ( +0.0% ) 5.06 5.02 +0.7% 2.404 1.748
2500.11 1.219 1.219 0.000 ( +0.0% ) 9.11 9.06 +0.7% 2.001 1.456

@vlimant
Copy link
Contributor

vlimant commented Oct 20, 2022

will you extend this to all workflows in the --what nano table ?
N.B we are soon going to remove several of those workflows

@vlimant
Copy link
Contributor

vlimant commented Oct 20, 2022

somehow the DQM does not contain the proper directory for nano and plots.
also, https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisonsNANO/CMSSW_12_6_X_2022-10-19-2300+74a38f/53552/ looks failed, and does not have the relmon directories

@smuzaffar
Copy link
Contributor Author

will you extend this to all workflows in the --what nano table ? N.B we are soon going to remove several of those workflows

Although I can dynamically find all the nano workflows and run them during PR nano tests but I was hesitating to do that as then we do not have any control during the PR tests and we might end up running a large number of tests during PR test time.

This is also true that if we hard-code the nano workflow numbers in cms-bot then any change in nano workflows will also need a cms-bot update . I prefer this as this way we do have some control on what we run for PR tests

@smuzaffar
Copy link
Contributor Author

somehow the DQM does not contain the proper directory for nano and plots. also, https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisonsNANO/CMSSW_12_6_X_2022-10-19-2300+74a38f/53552/ looks failed, and does not have the relmon directories

Note that DQM comparison generates a lot data ( in past for normal PR tests it was 250K files with over few GB of data) and that takes a lot of resources (upload time which increases the PR test time too, disk on server etc.). So we only upload those directories where there are any results with Failures, Nulls or Skipped e.g. for normal PR comparison results there were only 3 workflows where there were differences and only those were uploaded.

@vlimant
Copy link
Contributor

vlimant commented Oct 20, 2022

Note that DQM comparison generates a lot data ( in past for normal PR tests it was 250K files with over few GB of data) and that takes a lot of resources (upload time which increases the PR test time too, disk on server etc.). So we only upload those directories where there are any results with Failures, Nulls or Skipped e.g. for normal PR comparison results there were only 3 workflows where there were differences and only those were uploaded.

I see, so no relmon directory means no differences ; good.

@vlimant
Copy link
Contributor

vlimant commented Oct 20, 2022

we need to run more than the 5 workflows, best would be all 28 (currently) of them when @cms-sw/xpog or else enables nano. this number will go down to 20 soonish.
or at least can we run min[over Y](2500.XY) and max[over Y](2500.XY) for each X=1,..,9 ? that's 12 workflows

@smuzaffar
Copy link
Contributor Author

Note that DQM comparison generates a lot data ( in past for normal PR tests it was 250K files with over few GB of data) and that takes a lot of resources (upload time which increases the PR test time too, disk on server etc.). So we only upload those directories where there are any results with Failures, Nulls or Skipped e.g. for normal PR comparison results there were only 3 workflows where there were differences and only those were uploaded.

I see, so no relmon directory means no differences ; good.

correct, basically you should check the Comparison NANO failed link on the summary page. This should contains the links for failed DQM workflows.

@vlimant vlimant mentioned this pull request Oct 20, 2022
Merged
3 tasks
@smuzaffar
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

Pull request #1867 was updated.

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74a38f/28402/summary.html
COMMIT: 7559370
CMSSW: CMSSW_12_6_X_2022-10-20-1100/el8_amd64_gcc10
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/1867/28402/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 4 differences found in the comparisons
  • Reco comparison had 6 failed jobs
  • DQMHistoTests: Total files compared: 48
  • DQMHistoTests: Total histograms compared: 3391158
  • DQMHistoTests: Total failures: 3
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3391133
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
  • Checked 201 log files, 48 edm output root files, 48 DQM output files
  • TriggerResults: no differences found

NANO Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.001_data80X+MuonEG2016HMINIAOD8.0+NANO_data8.0+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.0_mc80X+TTbarMINIAOD8.0+NANO_mc8.0+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.101_mc94Xv2+TTbarMINIAOD9.4v2+NANO_mc9.4v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.11_data94X2016+MuonEG2016MINIAOD9.4v2+NANO_2016data9.4v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.12_data94Xv2+MuonEG2017MINIAOD9.4v2+NANO_data9.4v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.1_mc94X2016+TTbar2016MINIAOD9.4v2+NANO_2016mc9.4v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.21_data102X+MuonEG2018MINIAOD10.2+NANO_data10.2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.2_mc102X+TTbarMINIAOD10.2+NANO_mc10.2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.301_mc106Xul17+TTbarMINIAOD10.6_UL17v1+NANO_mc10.6ul17v1+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.302_mc106Xul18+TTbarMINIAOD10.6_UL18v1+NANO_mc10.6ul18v1+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.311_mc106Xul17v2+TTbarMINIAOD10.6_UL17v2+NANO_mc10.6ul17v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.312_mc106Xul17v2+TTbarMINIAOD10.6_UL18v2+NANO_mc10.6ul18v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.31_mc106Xul17v2+TTbarMINIAOD10.6_UL16v2+NANO_mc10.6ul16v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.321_data106Xul17+MuonEG2017MINIAOD10.6v1+NANO_data10.6ul17v1+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.322_data106Xul18+MuonEG2018MINIAOD10.6v1+NANO_data10.6ul18v1+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.32_data106Xul16+MuonEG2016MINIAOD10.6v1+NANO_data10.6ul16v1+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.331_data106Xul17v2+MuonEG2017MINIAOD10.6v2+NANO_data10.6ul17v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.332_data106Xul18v2+MuonEG2018MINIAOD10.6v2+NANO_data10.6ul18v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.33_data106Xul16v2+MuonEG2016MINIAOD10.6v2+NANO_data10.6ul16v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.3_mc106Xul16+TTbarMINIAOD10.6_UL16v1+NANO_mc10.6ul16v1+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.401_mc122Xrun3+TTbarMINIAOD12.2+NANO_mc12.2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.4_mc122Xrun3_v10+TTbarMINIAOD12.2+NANO_mc12.2_v10+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.501_mc124Xrun3+TTbarMINIAOD12.4+NANO_mc12.4+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.511_data124Xrun3+MuonEG2022MINIAOD12.4+NANO_data12.4+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.51_data124Xrun3_v10+MuonEG2022MINIAOD12.4+NANO_data12.4_v10+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.5_mc124Xrun3_v10+TTbarMINIAOD12.4+NANO_mc12.4_v10+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-74a38f/2500.6_mc126X_v10+TTBarMINIAOD12.6+NANO_mc12.6_v10+HRV_NANO_mc

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 29
  • DQMHistoTests: Total histograms compared: 28879
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 28879
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 28 files compared)
  • Checked 66 log files, 0 edm output root files, 29 DQM output files

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.0 2.165 2.165 0.000 ( +0.0% ) 7.02 6.55 +7.2% 1.298 1.358
2500.001 1.126 1.126 0.000 ( +0.0% ) 10.22 9.69 +5.4% 1.259 1.371
2500.1 2.291 2.291 0.000 ( +0.0% ) 5.32 5.02 +6.0% 1.666 1.728
2500.101 2.346 2.346 0.000 ( +0.0% ) 4.47 4.27 +4.6% 1.747 1.878
2500.11 1.219 1.219 0.000 ( +0.0% ) 8.15 7.87 +3.5% 1.459 1.589
2500.12 1.318 1.318 0.000 ( +0.0% ) 6.07 5.80 +4.5% 1.640 1.784
2500.2 2.189 2.189 0.000 ( +0.0% ) 5.24 4.98 +5.2% 1.715 1.860
2500.21 1.208 1.208 0.000 ( +0.0% ) 7.78 7.35 +5.9% 1.668 1.731
2500.3 2.347 2.347 0.000 ( +0.0% ) 5.11 4.85 +5.4% 1.642 1.721
2500.301 2.452 2.452 0.000 ( +0.0% ) 4.65 4.42 +5.2% 1.427 1.497
2500.302 2.404 2.404 0.000 ( +0.0% ) 4.75 4.54 +4.5% 1.700 1.785
2500.31 2.232 2.232 0.000 ( +0.0% ) 8.63 8.12 +6.2% 1.410 1.444
2500.311 2.322 2.322 0.000 ( +0.0% ) 8.33 7.84 +6.3% 1.798 1.804
2500.312 2.272 2.272 0.000 ( +0.0% ) 8.33 7.88 +5.6% 1.789 1.800
2500.32 1.085 1.085 0.000 ( +0.0% ) 8.94 8.47 +5.6% 1.413 1.580
2500.321 1.453 1.453 0.000 ( +0.0% ) 5.99 5.67 +5.7% 1.635 1.785
2500.322 1.225 1.225 0.000 ( +0.0% ) 7.44 7.26 +2.4% 1.507 1.507
2500.33 1.099 1.099 0.000 ( +0.0% ) 19.94 19.07 +4.6% 1.476 1.476
2500.331 1.389 1.389 0.000 ( +0.0% ) 14.82 14.14 +4.8% 1.633 1.634
2500.332 1.316 1.316 0.000 ( +0.0% ) 16.52 15.75 +4.9% 1.598 1.568
2500.4 2.109 2.109 0.000 ( +0.0% ) 9.52 9.25 +3.0% 1.324 1.307
2500.401 2.109 2.109 0.000 ( +0.0% ) 9.64 9.34 +3.3% 1.432 1.343
2500.5 0.302 0.302 0.000 ( +0.0% ) 48.83 45.99 +6.2% 1.186 1.193
2500.501 0.302 0.302 0.000 ( +0.0% ) 47.66 45.05 +5.8% 1.206 1.047
2500.51 1.080 1.080 0.000 ( +0.0% ) 27.89 26.72 +4.4% 1.314 1.314
2500.511 1.080 1.080 0.000 ( +0.0% ) 28.10 27.47 +2.3% 1.317 1.312
2500.6 1.366 1.366 0.000 ( +0.0% ) 23.29 22.82 +2.1% 1.080 1.029
2500.601 1.367 1.367 0.000 ( +0.0% ) 24.02 23.71 +1.3% 1.113 1.106

@swertz swertz mentioned this pull request Oct 20, 2022
Merged
@vlimant
Copy link
Contributor

vlimant commented Oct 20, 2022

awesome!

@smuzaffar
Copy link
Contributor Author

@vlimant , we also need to update https://github.com/cms-sw/cms-bot/blob/master/comparisons/matrix_RE.txt to include nano workflows 2500.xx . Normally @slava77 does this . If you want to have complate comparsion for nano workflows then better to update matrix_RE.txt too

vlimant
vlimant reviewed Oct 20, 2022
View reviewed changes
cmssw-pr-test-config
@@ -2,4 +2,4 @@ PR_TEST_MATRIX_EXTRAS=1306.0,101.0,9.0,25202.0,10224.0,250202.181
PR_TEST_MATRIX_EXTRAS_GPU=11634.506,11634.512,11634.522
PR_TEST_MATRIX_EXTRAS_PROFILING=39634.21,21034.21,11834.21,136.889
PR_TEST_MATRIX_EXTRAS_HIGH_STATS=35034.0
PR_TEST_MATRIX_EXTRAS_NANO=2500.0
PR_TEST_MATRIX_EXTRAS_NANO=2500.0,2500.001,2500.1,2500.101,2500.11,2500.12,2500.2,2500.21,2500.3,2500.301,2500.302,2500.31,2500.311,2500.312,2500.32,2500.321,2500.322,2500.33,2500.331,2500.332,2500.4,2500.401,2500.5,2500.501,2500.51,2500.511,2500.6,2500.601
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no way to have this being a wild-card 2500.* ?
the nano workflows are done dynamically ; maybe I need to make them static

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no wild card does not work. It think static wf are better which can easily match entires in matrix_RE.txt otherwise we have to keep changing matrix_RE.txt for every change in workflow number

@smuzaffar
Copy link
Contributor Author

+externals
merging it for now, will check if I can get the nano relvals dynamically

@smuzaffar smuzaffar merged commit e0404ea into master Oct 23, 2022
@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@vlimant
Copy link
Contributor

vlimant commented Oct 24, 2022

thanks @smuzaffar ! we keep in touch for the numbering. I might change it to static, no big deal.

@smuzaffar
Copy link
Contributor Author

@vlimant , #1869 should allow to dynamically find all nano workflows and run them. I would still like to control how many workflows we should during PR tests

@smuzaffar smuzaffar deleted the nano-comparison branch October 27, 2022 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vlimant vlimant vlimant left review comments

Assignees
No one assigned
Labels
externals-approved fully-signed orp-pending requires-external tests-approved
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@smuzaffar @cmsbuild @vlimant