Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove redundant customizations in HcalNano #45561

Merged
merged 2 commits into from
Jul 31, 2024

Conversation

jhakala
Copy link
Contributor

@jhakala jhakala commented Jul 25, 2024

PR description:

This is a code cleanup/simplification for HcalNano, plus a one-line change in a related runTheMatrix workflow. The change in PhysicsTools/NanoAOD/python/autoNANO.py is because of a removal of a duplicate function -- the change is to point it to the preserved version of the function.

PR validation:

I ran the related workflows: 2500.313_hcalDPGCalibNANO130Xrun3 and 2500.312_hcalDPGNANO130Xrun3 (which until recently were numbered 1060.1 and 1060.2). They looked fine.

No backport is needed.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 25, 2024

cms-bot internal usage

@jhakala jhakala changed the title remove redundant customizations remove redundant customizations in HcalNano Jul 25, 2024
@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @jhakala for master.

It involves the following packages:

  • DPGAnalysis/HcalNanoAOD (xpog)
  • PhysicsTools/NanoAOD (xpog)

@cmsbuild, @ftorrresd, @hqucms, @vlimant can you please review it and eventually sign? Thanks.
@AnnikaStein, @gpetruc, @mmusich this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@hqucms
Copy link
Contributor

hqucms commented Jul 25, 2024

enable nano

@hqucms
Copy link
Contributor

hqucms commented Jul 25, 2024

please test

@cmsbuild
Copy link
Contributor

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd8c6c/40621/summary.html
COMMIT: 614faec
CMSSW: CMSSW_14_1_X_2024-07-25-1100/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45561/40621/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd8c6c/40621/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd8c6c/40621/git-merge-result

  • DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

  • You potentially added 7 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 75 differences found in the comparisons
  • DQMHistoTests: Total files compared: 44
  • DQMHistoTests: Total histograms compared: 3312343
  • DQMHistoTests: Total failures: 12218
  • DQMHistoTests: Total nulls: 14
  • DQMHistoTests: Total successes: 3300091
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 61.702000000000005 KiB( 43 files compared)
  • DQMHistoSizes: changed ( 12834.0,... ): 8.774 KiB HLT/Filters
  • DQMHistoSizes: changed ( 12834.0,... ): 0.070 KiB HLT/TAU
  • Checked 191 log files, 161 edm output root files, 44 DQM output files
  • TriggerResults: found differences in 6 / 42 workflows

NANO Comparison Summary

Summary:

  • You potentially removed 463 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 21
  • DQMHistoTests: Total histograms compared: 54901
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 54901
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 20 files compared)
  • Checked 100 log files, 57 edm output root files, 21 DQM output files
  • TriggerResults: no differences found

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.001 2.783 2.783 0.000 ( +0.0% ) 3.30 3.32 -0.5% 5.942 6.661
2500.002 2.897 2.897 0.000 ( +0.0% ) 2.94 2.95 -0.4% 6.297 7.019
2500.003 2.843 2.843 0.000 ( +0.0% ) 3.04 3.10 -1.8% 6.257 6.991
2500.011 1.446 1.446 0.000 ( +0.0% ) 5.62 5.80 -3.1% 2.281 2.403
2500.012 1.906 1.906 0.000 ( +0.0% ) 3.11 3.15 -1.1% 2.439 2.593
2500.013 1.761 1.761 0.000 ( +0.0% ) 4.49 4.57 -1.9% 2.348 2.496
2500.021 0.022 0.022 0.000 ( +0.0% ) 0.92 0.97 -4.9% 2.168 2.200
2500.022 0.022 0.022 0.000 ( +0.0% ) 0.89 0.93 -4.2% 2.170 2.202
2500.023 0.022 0.022 0.000 ( +0.0% ) 0.88 0.90 -2.0% 2.112 2.173
2500.024 0.022 0.022 0.000 ( +0.0% ) 0.70 0.71 -1.7% 2.277 2.311
2500.031 0.035 0.035 0.000 ( +0.0% ) 0.84 0.86 -1.6% 2.247 2.312
2500.032 0.036 0.036 0.000 ( +0.0% ) 0.87 0.89 -2.3% 2.194 2.256
2500.033 0.037 0.037 0.000 ( +0.0% ) 0.77 0.80 -4.5% 2.302 2.355
2500.034 0.036 0.036 0.000 ( +0.0% ) 0.80 0.81 -2.2% 2.276 2.337
2500.101 2.642 2.642 0.000 ( +0.0% ) 8.74 8.95 -2.4% 6.188 6.887
2500.111 1.327 1.327 0.000 ( +0.0% ) 19.31 20.77 -7.0% 2.090 2.215
2500.112 1.731 1.731 0.000 ( +0.0% ) 15.26 14.37 +6.2% 2.075 2.125
2500.131 5.194 5.194 0.000 ( +0.0% ) 15.66 16.00 -2.1% 1.392 1.538
2500.201 2.475 2.475 0.000 ( +0.0% ) 7.17 7.47 -4.0% 5.432 6.134
2500.211 1.589 1.589 0.000 ( +0.0% ) 17.23 17.38 -0.9% 2.046 2.096
2500.212 2.030 2.030 0.000 ( +0.0% ) 13.73 13.57 +1.1% 2.094 2.142
2500.221 2.006 2.006 0.000 ( +0.0% ) 7.54 7.45 +1.2% 2.225 2.265
2500.222 3.071 3.071 0.000 ( +0.0% ) 7.36 7.53 -2.3% 2.286 2.324
2500.223 8.884 8.884 0.000 ( +0.0% ) 2.75 2.81 -2.0% 2.313 2.317
2500.224 5.511 5.511 0.000 ( +0.0% ) 1.08 1.11 -2.7% 1.934 2.555
2500.225 5.530 5.530 0.000 ( +0.0% ) 1.01 1.02 -0.6% 1.924 2.403
2500.226 2.970 2.970 0.000 ( +0.0% ) 7.41 7.57 -2.1% 2.285 2.288
2500.227 8.972 8.972 0.000 ( +0.0% ) 10.08 10.18 -1.0% 1.340 1.377
2500.231 1.407 1.407 0.000 ( +0.0% ) 13.68 13.12 +4.2% 1.934 1.942
2500.232 2.145 2.145 0.000 ( +0.0% ) 13.42 13.06 +2.8% 1.988 2.031
2500.233 4.667 4.667 0.000 ( +0.0% ) 4.91 4.85 +1.1% 1.742 2.038
2500.234 3.304 3.304 0.000 ( +0.0% ) 1.49 1.49 +0.0% 1.785 2.062
2500.235 3.314 3.314 0.000 ( +0.0% ) 1.38 1.41 -1.6% 1.801 2.085
2500.236 2.082 2.082 0.000 ( +0.0% ) 13.35 13.86 -3.7% 2.009 2.053
2500.241 9.405 9.405 0.000 ( +0.0% ) 3.62 3.68 -1.7% 1.760 1.764
2500.242 10.331 10.331 0.000 ( +0.0% ) 0.88 0.91 -3.4% 1.597 1.646
2500.243 2.712 2.712 0.000 ( +0.0% ) 8.21 8.23 -0.2% 1.067 1.066
2500.244 485.976 485.976 0.000 ( +0.0% ) 0.55 0.57 -4.1% 1.544 1.674
2500.245 823.224 823.158 0.066 ( +0.0% ) 0.75 0.72 +3.5% 1.532 1.642
2500.901 1.749 1.749 0.000 ( +0.0% ) 21.27 20.79 +2.3% 1.755 1.764
2500.902 1.598 1.598 0.000 ( +0.0% ) 20.43 21.77 -6.2% 1.690 1.691
2500.911 13.931 13.931 0.000 ( +0.0% ) 1.41 2.42 -41.8% 1.027 1.072
2500.912 0.171 0.240 -0.069 ( -28.9% ) 0.79 1.30 -39.1% 0.942 0.966
2500.913 0.110 0.110 0.000 ( +0.0% ) 1.17 1.22 -3.5% 0.951 0.971

@@ -35,25 +35,7 @@ def customiseHcalCalib(process):
process.raw2digi_step = cms.Path(process.hcalCalibDigiSequence, process.RawToDigiTask)

# Insert the HLT filter at start of user path and nanoaod endpath
process.user_step.insert(0, process.hcalCalibHLTFilter)
process.nanoAOD_step.insert(0, process.hcalCalibHLTFilter)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a safer way, IMO, would be to list of path of the process and insert the filter in first position, instead of calling explicitly nanoAOD_step and NANOAODoutput_step without checking their existence in the process

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to work when I replace that line with:

process.paths.values()[0].insert(0, process.hcalCalibHLTFilter)

Does that seem reasonable to you? Or is there a more preferred way to do this? (I don't see this trick anywhere else in CMSSW with an LXR search...)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something like

for path in self.process.paths:
     path.insert(0,process.hcalCalibHLTFilter)

I think. and verify that it does prepend it to the output path too at the next line

@jhakala
Copy link
Contributor Author

jhakala commented Jul 29, 2024

@vlimant I added a commit that (I think) takes care of your suggestion. The output path was an EndPath, so I added the filter to the beginning of everything in process.paths and process.endpaths. Let me know if it looks OK now.

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Pull request #45561 was updated. @cmsbuild, @ftorrresd, @hqucms, @vlimant can you please check and sign again.

@vlimant
Copy link
Contributor

vlimant commented Jul 29, 2024

please test

@cmsbuild
Copy link
Contributor

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd8c6c/40677/summary.html
COMMIT: 510fdf2
CMSSW: CMSSW_14_1_X_2024-07-28-2300/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/45561/40677/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

NANO Comparison Summary

Summary:

  • You potentially removed 1666 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 21
  • DQMHistoTests: Total histograms compared: 54901
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 54901
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 20 files compared)
  • Checked 100 log files, 57 edm output root files, 21 DQM output files
  • TriggerResults: no differences found

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.001 2.783 2.783 0.000 ( +0.0% ) 3.33 3.34 -0.1% 6.024 5.972
2500.002 2.897 2.897 0.000 ( +0.0% ) 2.96 2.93 +1.3% 6.372 6.307
2500.003 2.843 2.843 0.000 ( +0.0% ) 3.07 3.08 -0.4% 6.336 6.272
2500.011 1.446 1.446 0.000 ( +0.0% ) 5.76 5.75 +0.2% 2.407 2.306
2500.012 1.906 1.906 0.000 ( +0.0% ) 3.15 3.13 +0.8% 2.582 2.448
2500.013 1.761 1.761 0.000 ( +0.0% ) 4.55 4.49 +1.5% 2.386 2.366
2500.021 0.022 0.022 0.000 ( +0.0% ) 0.98 0.98 +0.3% 2.197 2.162
2500.022 0.022 0.022 0.000 ( +0.0% ) 0.93 0.95 -1.9% 2.196 2.164
2500.023 0.022 0.022 0.000 ( +0.0% ) 0.93 0.96 -2.3% 2.139 2.115
2500.024 0.022 0.022 0.000 ( +0.0% ) 0.72 0.72 +0.1% 2.317 2.279
2500.031 0.035 0.035 0.000 ( +0.0% ) 0.85 0.90 -5.8% 2.288 2.255
2500.032 0.036 0.036 0.000 ( +0.0% ) 0.90 0.90 -0.5% 2.225 2.199
2500.033 0.037 0.037 0.000 ( +0.0% ) 0.80 0.81 -1.4% 2.326 2.300
2500.034 0.036 0.036 0.000 ( +0.0% ) 0.80 0.79 +1.6% 2.309 2.280
2500.101 2.642 2.642 0.000 ( +0.0% ) 9.03 8.91 +1.4% 6.259 6.268
2500.111 1.327 1.327 0.000 ( +0.0% ) 19.92 19.81 +0.6% 2.224 2.099
2500.112 1.731 1.731 0.000 ( +0.0% ) 15.12 15.18 -0.4% 2.078 2.072
2500.131 5.194 5.194 0.000 ( +0.0% ) 15.89 15.79 +0.7% 1.443 1.424
2500.201 2.475 2.475 0.000 ( +0.0% ) 7.47 7.35 +1.7% 5.481 5.463
2500.211 1.589 1.589 0.000 ( +0.0% ) 15.54 17.40 -10.7% 2.048 2.035
2500.212 2.030 2.030 0.000 ( +0.0% ) 13.39 13.74 -2.5% 2.098 2.096
2500.221 2.006 2.006 0.000 ( +0.0% ) 7.78 7.69 +1.2% 2.238 2.224
2500.222 3.071 3.071 0.000 ( +0.0% ) 7.55 7.33 +3.0% 2.302 2.284
2500.223 8.884 8.884 0.000 ( +0.0% ) 2.78 2.79 -0.3% 2.075 2.001
2500.224 5.511 5.511 0.000 ( +0.0% ) 1.10 1.10 +0.2% 1.930 1.876
2500.225 5.530 5.530 0.000 ( +0.0% ) 1.02 1.00 +1.7% 1.880 1.968
2500.226 2.970 2.970 0.000 ( +0.0% ) 7.50 7.40 +1.4% 1.912 2.275
2500.227 8.972 8.972 0.000 ( +0.0% ) 10.19 10.05 +1.5% 1.281 1.119
2500.231 1.407 1.407 0.000 ( +0.0% ) 12.90 13.61 -5.2% 1.669 1.932
2500.232 2.145 2.145 0.000 ( +0.0% ) 13.83 13.70 +1.0% 2.007 1.998
2500.233 4.667 4.667 0.000 ( +0.0% ) 4.92 4.80 +2.5% 1.738 2.010
2500.234 3.304 3.304 0.000 ( +0.0% ) 1.52 1.48 +2.6% 1.779 2.043
2500.235 3.314 3.314 0.000 ( +0.0% ) 1.40 1.40 -0.2% 1.747 2.067
2500.236 2.082 2.082 0.000 ( +0.0% ) 13.09 13.48 -2.9% 2.011 2.015
2500.241 9.405 9.405 0.000 ( +0.0% ) 3.85 3.22 +19.5% 1.762 1.884
2500.242 10.331 10.331 0.000 ( +0.0% ) 0.92 0.88 +4.3% 1.647 1.647
2500.243 2.712 2.712 0.000 ( +0.0% ) 8.22 8.22 +0.1% 1.067 1.068
2500.244 485.976 485.976 0.000 ( +0.0% ) 0.55 0.57 -3.6% 1.624 1.581
2500.245 823.224 823.158 0.066 ( +0.0% ) 0.76 0.76 +0.4% 1.582 1.565
2500.901 1.749 1.749 0.000 ( +0.0% ) 18.28 20.48 -10.7% 1.779 1.766
2500.902 1.598 1.598 0.000 ( +0.0% ) 21.30 21.82 -2.4% 1.709 1.692
2500.911 13.931 13.931 0.000 ( +0.0% ) 2.98 1.70 +75.4% 1.030 1.042
2500.912 0.171 0.171 -0.001 ( -0.4% ) 1.21 0.84 +42.9% 0.965 0.946
2500.913 0.110 0.110 0.000 ( +0.0% ) 1.19 1.17 +2.1% 0.969 0.953

@vlimant
Copy link
Contributor

vlimant commented Jul 30, 2024

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @sextonkennedy, @rappoccio, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

@mandrenguyen
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit a3b4557 into cms-sw:master Jul 31, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants