update of hltDumpStream for Run-3 HLT menus

[missirol]

resolves #37762

PR description:

This PR updates the hltDumpStream script, adapting it to the most recent versions of the HLT menus.

Testing is still in progress, but the review could already start.

FYI: @fwyzard

Short summary of the updates below.

• Take into account that stream OutputModules can be in FinalPaths (as opposed to EndPaths like in Run 2).
• Take into account that, since the introduction of DatasetPaths, a dataset itself can be prescaled, and smart-prescales can be applied at dataset level.
• Continue to take into account that smart-prescales could be applied at stream level, and streams could be prescaled (was the case in Run-2 menus, is not the case in 2022 menus partly due to the use of FinalPaths, might potentially be the case again in the future).
• Add some basic support to take wildcards into account in the lookup of smart-prescales.
• Try and reduce reliance on the "datasets" and "streams" PSets, to improve support for 'partial' configurations (e.g. hltGetConfiguration --paths [..]).
• In addition, the definition of "trigger seed" is slightly changed:
  • restricting to modules that precede the HLT prescaler (if any), and
  • considering also TriggerResultsFilter, as opposed to only HLTL1TSeed (we currently have corner cases like HLT_EphemeralPhysics, where TriggerResultsFilter is used to select on L1T seeds, and DST_HLTMuon_Run3_PFScoutingPixelTracking, where TriggerResultsFilter is used to seed the HLT path with events selected by the Muon PD, i.e. Dataset_Muon).

A few command-line options are added, see hltDumpStream --help below:

usage: ./hltDumpStream [-h] [-m MODE] [-p COLUMN [COLUMN ...] | --no-prescales] [--no-seeds] [MENU]

hltDumpStream: print information on streams, primary datasets, triggers and (HLT) prescales of a HLT menu

examples:
  # run hltDumpStream on HLT menu used in run 123456 (output in CSV format)
  hltConfigFromDB --runNumber 123456 | hltDumpStream --mode csv
  # run hltDumpStream on local cfg file hlt.py (output in text format)
  hltDumpStream hlt.py --mode text

positional arguments:
  MENU                  path to local cmsRun cfg file (if not specified, stdin is used)

optional arguments:
  -h, --help            show this help message and exit
  -m MODE, --mode MODE  format of output file (must be "text", or "csv") [default: "text"]
  -p COLUMN [COLUMN ...], --prescale-columns COLUMN [COLUMN ...]
                        list of patterns, passed to fnmatch, to select prescale columns [default: ["*"]]
  --no-prescales        do not show information on prescales
  --no-seeds            do not show the seed of a HLT path [default: False]

PR validation:

Private tests.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

N/A

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40223/33263

• This PR adds an extra 16KB to repository

[cmsbuild]

A new Pull Request was created by @missirol (Marino Missiroli) for master.

It involves the following packages:

• HLTrigger/Configuration (hlt)

@cmsbuild, @missirol, @Martin-Grunewald can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @silviodonato this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[fwyzard]

  -p PSCOLNAMEREGEXPS [PSCOLNAMEREGEXPS ...], --prescale-columns PSCOLNAMEREGEXPS [PSCOLNAMEREGEXPS ...]
                        list of regular expressions to select prescale columns [default: [".*"]]

I'd suggest renaming PSCOLNAMEREGEXPS to simply COLUMN, and using glob expressions instead of regular expressions -- unless you have some examples in mind where a regex is better suited, of course.

[missirol]

I'd suggest renaming PSCOLNAMEREGEXPS to simply COLUMN, and using glob expressions instead of regular expressions -- unless you have some examples in mind where a regex is better suited, of course.

Thanks for having a look. Agreed (for clarity I added 290d569, eventually I will squash).

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40223/33264

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #40223 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40223/33312

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #40223 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40223/33317

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #40223 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40223/33318

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #40223 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

[missirol]

please test

Tested on a few more configs (Run-2 and Run-3 ones), and made a few small fixes. As of now, I don't have other changes planned for this PR.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-47e38b/29523/summary.html
COMMIT: d50b038
CMSSW: CMSSW_13_0_X_2022-12-08-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40223/29523/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 25 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3421214
• DQMHistoTests: Total failures: 1189
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3420003
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 206 log files, 158 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[missirol]

+hlt

Spurious differences in wf 11634.7 are a known issue (see #39803), unrelated to this PR.

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[rappoccio]

+1