Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

edmPluginHelp bug fix, crash would occur if helper plugin contains itself recursively #43126

Merged
merged 1 commit into from
Nov 1, 2023

Conversation

wddgit
Copy link
Contributor

@wddgit wddgit commented Oct 26, 2023

PR description:

Bug fix in edmPluginHelp. A crash would occur if a module plugin contained a helper plugin that contained a helper plugin of the same category recursively. At a point in the hierarchy where a helper plugin is contained, edmPluginHelp prints the descriptions of all possible plugins in the category. If one of the possibilities is a plugin that recursively contains a helper plugin of the same category the infinite recursion would occur. Containment could be direct or indirect. Then either memory or the stack limit would be exceeded eventually.

This fixes the crash reported in Issue #42988. It also fixes the format issues with the printout that Chris reported in the same issue.

This situation is somewhat rare. It was noticed when printing for this module edmPluginHelp -p CkfTrackCandidateMaker. That was the only one exhibiting the problem that I am aware of.

I added some additional modifications to the printout related PluginDescription to make it easier to understand. There is some extra content to explain things. Also one level of numbering of sections is deleted. The section number hierarchy will now match the hierarchy of the ParameterSet, which is what we are trying to document, instead of the hierarchy of the description which has an extra level. It also makes this complex printout slightly simpler. The PluginDescription printout is still complex, but I don't see how to simplify it further. It is describing something complex.

Also adds to existing unit tests to cover these modifications. The extended unit tests would have detected the reported bug.

PR validation:

This should only affect the output of edmPluginHelp for modules with helper plugins and ParameterSet unit tests in FWCore/Integration. It shouldn't affect anything else.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43126/37390

  • This PR adds an extra 52KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @wddgit (W. David Dagenhart) for master.

It involves the following packages:

  • FWCore/Integration (core)
  • FWCore/ParameterSet (core)

@cmsbuild, @makortel, @Dr15Jones, @smuzaffar can you please review it and eventually sign? Thanks.
@makortel, @missirol this is something you requested to watch as well.
@sextonkennedy, @rappoccio, @antoniovilela you are the release manager for this.

cms-bot commands are listed here

@wddgit
Copy link
Contributor Author

wddgit commented Oct 26, 2023

please test

@wddgit
Copy link
Contributor Author

wddgit commented Oct 26, 2023

abort test

Also improves and fixes the printout of edmPluginHelp in the
sections related PluginDescriptions. Add to existing unit test
to cover these modifications.
@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43126/37396

  • This PR adds an extra 56KB to repository

@cmsbuild
Copy link
Contributor

Pull request #43126 was updated. @smuzaffar, @makortel, @cmsbuild, @Dr15Jones can you please check and sign again.

@wddgit
Copy link
Contributor Author

wddgit commented Oct 26, 2023

please test

The changes since the previous test run only implement Chris's suggestion. The significant changes are in the 3 FWCore/ParameterSet files, plus the extra text shows up in the reference files for the edmPluginHelp output.

@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-db51db/35449/summary.html
COMMIT: f6200bd
CMSSW: CMSSW_13_3_X_2023-10-26-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43126/35449/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test testTauEmbeddingProducers had ERRORS

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

Summary:

  • You potentially added 2 lines to the logs
  • Reco comparison results: 24 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 3357400
  • DQMHistoTests: Total failures: 1067
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3356311
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 214 log files, 167 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@wddgit
Copy link
Contributor Author

wddgit commented Oct 27, 2023

The unit test that failed is also failing in the IBs already so that is not related to this PR. I can see the fix to the unit test is merged but not in any IB release yet. I could try again in a day or two if anyone thinks it is worth getting the test status green.

Is it normal for RelVals to timeout these days? I cannot see how this PR could cause that.

@smuzaffar
Copy link
Contributor

Is it normal for RelVals to timeout these days? I cannot see how this PR could cause that.

not really, let me re-start the input relvals test

@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-db51db/35449/summary.html
COMMIT: f6200bd
CMSSW: CMSSW_13_3_X_2023-10-26-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43126/35449/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test testTauEmbeddingProducers had ERRORS

RelVals-INPUT

  • 4.764.76_ZMuSkim2012D/step2_ZMuSkim2012D.log

Comparison Summary

Summary:

  • You potentially added 2 lines to the logs
  • Reco comparison results: 24 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 3357400
  • DQMHistoTests: Total failures: 1067
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3356311
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 214 log files, 167 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@wddgit
Copy link
Contributor Author

wddgit commented Oct 27, 2023

Both the unit test and RelVal test that failed are also failing in the IB.

@makortel
Copy link
Contributor

+core

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @sextonkennedy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@wddgit
Copy link
Contributor Author

wddgit commented Oct 30, 2023

please test

Try again and see if the tests pass now. At least one of the bug fixes is in (bugs were unrelated to this PR).

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-db51db/35507/summary.html
COMMIT: f6200bd
CMSSW: CMSSW_13_3_X_2023-10-30-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43126/35507/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially removed 8 lines from the logs
  • Reco comparison results: 26 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 3362691
  • DQMHistoTests: Total failures: 1070
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3361599
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 214 log files, 167 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@makortel
Copy link
Contributor

Comparison differences are related to #39803

@antoniovilela
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit ed10991 into cms-sw:master Nov 1, 2023
@wddgit wddgit deleted the bugFixEdmPluginHelp branch March 11, 2024 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants