Update autoSkim for 2023 data taking [13_0_X]

[youyingli]

PR description:

Update new PDs to support the skim for 2023 data taking. The update follows the following:

• CMSHLT-2596 : Split JetMET, EGamma and Muon into 2 stream
• CMSHLT-2707 : Move HLT_CscCluster and HLT_L1CSCShower_* triggers from DisplacedJet PD to Muon PD
• Disappearing Muon skim is added : Added skimming filter for disappearing muons #40666
• Still waiting for LLP Jet skim added and its backport

Update the skim workflow to fit the 2023 skim config

PR validation:

N/A

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

backport of #41145

[cmsbuild]

A new Pull Request was created by @youyingli (You-Ying Li) for CMSSW_13_0_X.

It involves the following packages:

• Configuration/Skimming (pdmv)

@cmsbuild, @bbilin, @sunilUIET, @kskovpen can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @fabiocos this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

• Backported from Update autoSkim for 2023 data taking [13_1_X] #41145

[youyingli]

backport of #41145

[perrotta]

backport of #41145

[kskovpen]

please test

[cmsbuild]

-1

Failed Tests: UnitTests RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17b18a/31802/summary.html
COMMIT: 57ffe12
CMSSW: CMSSW_13_0_X_2023-04-03-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41146/31802/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test-runTheMatrix-interactive had ERRORS

RelVals

RelVals-INPUT

[francescobrivio]

@youyingli just to be clear, if these changes need to go in Tier0 (i.e. affect prompt reconstruction), you should actually change the Tier0 configuration. e.g. see this line for example for JetMET:
https://github.com/dmwm/T0/blob/ca4ab8e66f971294d77b209b96de65b5f3ba6794/etc/ProdOfflineConfiguration.py#L654-L667

You can scroll through the rest of the file and look for physics_skims to double check that all the PDs are correctly covered.

Cheers,
Francesco as ORM

[francescobrivio]

@youyingli just to be clear, if these changes need to go in Tier0 (i.e. affect prompt reconstruction), you should actually change the Tier0 configuration. e.g. see this line for example for JetMET: https://github.com/dmwm/T0/blob/ca4ab8e66f971294d77b209b96de65b5f3ba6794/etc/ProdOfflineConfiguration.py#L654-L667

You can scroll through the rest of the file and look for physics_skims to double check that all the PDs are correctly covered.

Cheers, Francesco as ORM

FYI @cms-sw/pdmv-l2 in case something needs to be changed in the Tier0 config, you can request it at the JointOps meeting...there is a dedicated PdmV section for cases like this 😄

[youyingli]

@francescobrivio , thanks for the reminder. Yes, this should be updated to T0 config. I will submit a ticket to T0 JIRA for a request.

[youyingli]

@francescobrivio , thanks for the reminder. Yes, this should be updated to T0 config. I will submit a ticket to T0 JIRA for a request.

The ticket has been created in CMSTZ-1031.

[francescobrivio]

The error from this log shows:

ERROR reading file: relval_standard 'SKIMJETMETRUN3_reHLT'
Traceback (most recent call last):
  File "/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41146/31802/CMSSW_13_0_X_2023-04-03-2300/bin/el8_amd64_gcc11/runTheMatrix.py", line 633, in <module>
    ret = runSelected(opt)
  File "/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41146/31802/CMSSW_13_0_X_2023-04-03-2300/bin/el8_amd64_gcc11/runTheMatrix.py", line 23, in runSelected
    mrd.prepare(opt.useInput, opt.refRel, opt.fromScratch)
  File "/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41146/31802/CMSSW_13_0_X_2023-04-03-2300/src/Configuration/PyReleaseValidation/python/MatrixReader.py", line 509, in prepare
    self.readMatrix(matrixFile, useInput, refRel, fromScratch)
  File "/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41146/31802/CMSSW_13_0_X_2023-04-03-2300/src/Configuration/PyReleaseValidation/python/MatrixReader.py", line 255, in readMatrix
    if self.relvalModule.steps[stepName] is None:
KeyError: 'SKIMJETMETRUN3_reHLT'

@cms-sw/pdmv-l2 you might know where this comes from.

[youyingli]

Hi @francescobrivio , thanks for the clarification. Indeed, the wf should be updated to fit the correct skim config here, not T0. I will update them.

[cmsbuild]

Pull request #41146 was updated. @perrotta, @rappoccio, @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET, @fabiocos, @davidlange6 can you please check and sign again.

[kskovpen]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17b18a/31864/summary.html
COMMIT: 74a3f53
CMSSW: CMSSW_13_0_X_2023-04-05-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41146/31864/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 5 lines from the logs
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3553952
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3553930
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 213 log files, 164 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[youyingli]

@kskovpen , @srimanob, @perrotta , could you please sign it as the test was passed?

[kskovpen]

+pdmv

[srimanob]

+Upgrade

[perrotta]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_13_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_13_1_X is complete. This pull request will be automatically merged.