[14_0_X] Another backports to fix memory leak

[srimanob]

PR description:

To fix memory leak reported in #46901
I need to backport more PRs, to make it consistent with master.
This PR replaces #46942

This PR includes:

• [14.1.X] use ReadPrescalesFromFile=False in the GenericTriggerEventFlag of a bunch of DQM modules #46591
• Add GEN-SIM information to resonant di-muon TkAl ALCARECO producers #45357
• ObjectSelectorBase add option to not throw on missing input #45385
• Rework Muon candidate selection in few tracker ALCARECO  #46574

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

[cmsbuild]

A new Pull Request was created by @srimanob for CMSSW_14_0_X.

It involves the following packages:

• DQM/SiStripMonitorClient (dqm)
• DQMOffline/Trigger (dqm)

@antoniovagnerini, @cmsbuild, @rseidita can you please review it and eventually sign? Thanks.
@Fedespring, @HuguesBrun, @arossi83, @cericeci, @fioriNTU, @idebruyn, @jandrea, @jhgoh, @missirol, @mmusich, @mtosi, @rociovilar, @sroychow, @threus, @trocino, @venturia this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[cmsbuild]

Pull request #46954 was updated. @antoniovagnerini, @atpathak, @cmsbuild, @consuegs, @perrotta, @rseidita can you please check and sign again.

[srimanob]

assign alca

[cmsbuild]

Pull request #46954 was updated. @antoniovagnerini, @atpathak, @cmsbuild, @consuegs, @perrotta, @rseidita can you please check and sign again.

[srimanob]

@cmsbuild please test

[srimanob]

@cmsbuild abort

[srimanob]

hold

[cmsbuild]

Pull request has been put on hold by @srimanob
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

[cmsbuild]

Pull request #46954 was updated. @antoniovagnerini, @atpathak, @cmsbuild, @consuegs, @perrotta, @rseidita can you please check and sign again.

[cmsbuild]

Pull request #46954 was updated. @antoniovagnerini, @atpathak, @cmsbuild, @consuegs, @jfernan2, @mandrenguyen, @perrotta, @rseidita can you please check and sign again.

[cmsbuild]

Pull request #46954 was updated. @antoniovagnerini, @atpathak, @cmsbuild, @consuegs, @jfernan2, @mandrenguyen, @perrotta, @rseidita can you please check and sign again.

[srimanob]

@cmsbuild please test

[cmsbuild]

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-852b27/43458/summary.html
COMMIT: 8abe08c
CMSSW: CMSSW_14_0_X_2024-12-15-0000/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46954/43458/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 89 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 46 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3349879
• DQMHistoTests: Total failures: 391
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3349467
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -0.016 KiB( 47 files compared)
• DQMHistoSizes: changed ( 1000.0 ): -0.008 KiB MessageLogger/Errors
• DQMHistoSizes: changed ( 1000.0 ): -0.008 KiB MessageLogger/Warnings
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: found differences in 1 / 46 workflows

[jfernan2]

+1

[perrotta]

@srimanob as far as I can see #46574 was never backported to CMSSW_14_1_X. Would you mind preparing also that backport, in order to avoid holes in the release cycles?

[perrotta]

+alca

• Verbatim backport of the four PRs listed in the PR descriptions
• As I already wrote in [14_0_X] Backports of 46591,45357, and 46574 to fix memory leak (add update ALCARECO) #46942 (comment) I think that also Add GEN-SIM information to resonant di-muon TkAl ALCARECO producers #45357 is liable to be backported, as it only adds new products to GEN-SIM to the ALCARECOTkAlJpsiMuMu output
• I'm signing this PR now because I know that it is needed to cut a CMSSW_14_0_X version for the re-reco, and I am going to leave for vacations: but I would recommend release managers to wait for the backport of Rework Muon candidate selection in few tracker ALCARECO  #46574 in CMSSW_14_1_X being at least submitted, so that it does not get forgotten

[srimanob]

Hi @perrotta
Here is the backport of 46574,
#46968

[srimanob]

@mmusich Would you mind to re-check if I miss something, i.e. bug fixes that came later? Thanks very much.

[mmusich]

@srimanob I have several problems with this PR:

• is it really necessary? I am not sure all the fixes in master got tested / used yet
• the commit history is kind of messy (there is a revert in it)
• the original author of the commit doesn't appear in the git history

[mmusich]

to be clear about my comments earlier today at the ORP meeting for this PR.

I think actually

• Add GEN-SIM information to resonant di-muon TkAl ALCARECO producers #45357
• ObjectSelectorBase add option to not throw on missing input #45385

could be useful to have -- but they have nothing to do with memory leaks or memory reduction.
The only point that I am wary about it is

#46942 (comment), or quoting Andrea:

on the other hand, 2024 MC production has already started with it, as far as I know, and it could be that some of the samples will finally add that info, and some (those processed with releases prior to the one in which this PR will get merged) will not. This is propably acceptable (either you have those ectra GEN.SIM infos or not, but nothing vhanges otherwise), but I'd let it to be decided by you (O&C) and PPD.

Could @cms-sw/ppd-l2 chime in about potentially inconsistent (ALCARECO) event contents in the same campaign?

What I am bit more cautious about is the backport of #46574. That offers some advantage in terms of overall memory usage, but I would feel more confident about it if it was passed first through a full release validation.

[srimanob]

@antoniovilela @mandrenguyen
New PR with only backport of #46530 can be found at
#46978

[malbouis]

Could @cms-sw/ppd-l2 chime in about potentially inconsistent (ALCARECO) event contents in the same campaign?

Hi @mmusich , I am not sure how this PR for MC AlCaReco would improve the memory consumption for the ReRECO. So in principle I'd say let's not include it, just because it relates to MC and it doesn't seem to relate to memory consumption. As for your concern, I think the best would be to not add a non-validated PR to a running MC production release indeed, although this one would only concern AlCaRecos which have a limited MC production, afaik.

What I am bit more cautious about is the backport of #46574. That offers some advantage in terms of overall memory usage, but I would feel more confident about it if it was passed first through a full release validation.

Was there a Replay to test it?

In any case, I see @srimanob already made a new PR with just the backport of 46530.

[mmusich]

I am not sure how this PR for MC AlCaReco would improve the memory consumption for the ReRECO. So in principle I'd say let's not include it, just because it relates to MC and it doesn't seem to relate to memory consumption.

@malbouis I think there is a misunderstanding
#45357 + #45385 (as I wrote above) are not going to improve memory consumption at all (thus they don't belong to this PR).
On the other hand they are just adding some GEN level collections to the ALCARECO data-tier in MC and those added collections are in principle quite useful for both a Tracker Alignment point of view and also in view of precision electroweak physics analysis using early Run3 data - assuming MC samples are going to be created in 14.0.X for 2022, 2023, 2024 for the foreseeable future (adding them now would avoid to have user-specific campaigns to create samples to be used for that later on). So all in all I would be in favor of backporting them - in another PR. The question (to you as PPD) is if you think that's acceptable or not as some of the ALCARECO samples within the current campaign would have a slightly different event content.

Was there a Replay to test it?

I don't think so (as far as I know no replay in 14_2_X occurred yet).

[malbouis]

Hi @mmusich, since this PR was about memory consumption, I assumed the questions were related to that only. But I see now there is this additional request of backporting this AlCaReco PR for other reasons.
In principle, as I mentioned above, since it only touches MC AlCaRecos, I wouldn't see a problem, and as far as I know, no AlCaRecos have been produced with official Summer24 production yet, so we'd be fine wrt to the content. It's a pity this backport wasn't requested with the original PR as it could have entered the new patch release just created. I'd say please go ahead, submit the backport when you have the chance and we will discuss in the PR. In principle, I see no big problem apart from not undergoing the validation campaign, but considering this touches only the MC alcarecos which have a very reduced usage in comparison to the standard MC production, I'd think it's ok. We could do a small validation within the TK group, with a couple of RelVal samples. What do you think?

[mmusich]

Hi Helena, @malbouis,

in principle, as I mentioned above, since it only touches MC AlCaRecos, I wouldn't see a problem, and as far as I know, no AlCaRecos have been produced with official Summer24 production yet, so we'd be fine wrt to the content.

thanks for the confirmation: I opened #47016 to that effect.

It's a pity this backport wasn't requested with the original PR as it could have entered the new patch release just created.

I agree and I apologize for the oversight, I originally meant to backport it to 14.0.X long time ago (as in the original PR description #45357 (comment), but somehow I managed to forget about it).

We could do a small validation within the TK group, with a couple of RelVal samples. What do you think?

sounds perfect to me. Thanks!