update of `hltIntegrationTests` and path of Run-3 Data file for TSG tests [`12_3_X`] #37304

missirol · 2022-03-22T14:18:42Z

backport of #37283

PR description:

Exact backport of #37283. See the original PR for a complete description. This PR includes

an update of the hltIntegrationTests script, mainly to support Dataset Paths in the HLT menus;
a fix to the path of the EDM file used for TSG tests on Run-3 Data; this fixes the current issue with the "HLT-Validation" tests in IBs.

PR validation:

Relies on the tests done for the original PR.

If this PR is a backport, please specify the original PR and why you need to backport that PR:

#37283

The backport is needed to support new features (e.g. Dataset Paths) in the upcoming HLT menu for 12_3_X, plus fixing a problem with file access in the "HLT-Validation" tests of the IBs.

cmsbuild · 2022-03-22T14:19:10Z

A new Pull Request was created by @missirol (Marino Missiroli) for CMSSW_12_3_X.

It involves the following packages:

HLTrigger/Configuration (hlt)

@cmsbuild, @missirol, @Martin-Grunewald can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @silviodonato this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

Backported from update of hltIntegrationTests and path of Run-3 Data file for TSG tests #37283

missirol · 2022-03-22T14:19:31Z

type bugfix

Because of the EDM-file fix for the "HLT-Validation" tests running in IBs.

missirol · 2022-03-22T14:19:44Z

please test

cmsbuild · 2022-03-22T22:10:56Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f104db/23281/summary.html
COMMIT: b8ccfd1
CMSSW: CMSSW_12_3_X_2022-03-22-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37304/23281/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695161
DQMHistoTests: Total failures: 17
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3695122
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

The "hltListPaths" script is updated to support FinalPaths, give the option to ignore paths by name (by means of regular expressions), and give the option to ignore paths that depend on the result of other paths in the same job. The "hltIntegrationTests" script is updated to ignore Paths that depend on the result of other Paths in the same job, as by construction those cannot be tested without other Paths.

cmsbuild · 2022-03-22T22:31:59Z

Pull request #37304 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

missirol · 2022-03-22T22:38:58Z

please test

Propagated the update of the EDM file-path to the addOnTests.

cmsbuild · 2022-03-23T02:39:47Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f104db/23297/summary.html
COMMIT: f29fa19
CMSSW: CMSSW_12_3_X_2022-03-22-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37304/23297/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695161
DQMHistoTests: Total failures: 20
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3695119
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

missirol · 2022-03-23T07:56:48Z

+hlt

cmsbuild · 2022-03-23T07:57:12Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_3_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_4_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-03-23T20:16:27Z

+1

perrotta · 2022-03-24T07:58:53Z

@missirol HLT tests are failing in the IB also after the merging of this PR, see CMSSW_12_3_X_2022-03-23-2300
I did not look at the details of such a failure: is it something that should have been fixed by this PR?

Martin-Grunewald · 2022-03-24T08:20:04Z

This is unexpected - only the PRef MC integration test fails, all other integration tests are ok - weird!

Martin-Grunewald · 2022-03-24T14:45:22Z

Hmm, I can not reproduce this error in a local development area.

Martin-Grunewald · 2022-03-24T15:20:38Z

A second run also does not give any error, so for now I just wait for the next IB to see whether it was a glitch.

missirol · 2022-03-24T20:39:23Z

Sorry for the late reply.

It does look like a glitch. The same HLT tests in the next IB (12_3_X_2022-03-24-1100) came back clean, see here.

Maybe related (maybe not), the 12_4_X IB from last night (12_4_X_2022-03-23-2300) also had a very strange failure in the HLT tests, but a completely different one, see here.

Copying the error messages here for reference, as the links will eventually vanish.

[1]

https://cmssdt.cern.ch/SDT/jenkins-artifacts/HLT-Validation/CMSSW_12_3_X_2022-03-23-2300/slc7_amd64_gcc10/HLT_Integration_PRef_MC_hlt.log

----- Begin Fatal Exception 24-Mar-2022 05:58:27 UTC-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named hlt.py
Exception Message:
 unknown python problem occurred.
NameError: name 'cms' is not defined

At:
  hlt.py(10): <module>

----- End Fatal Exception -------------------------------------------------

https://cmssdt.cern.ch/SDT/jenkins-artifacts/HLT-Validation/CMSSW_12_3_X_2022-03-23-2300/slc7_amd64_gcc10/HLT_Integration_PRef_MC.log

stty: standard input: Inappropriate ioctl for device
Will run 35 HLT paths over 100 events, with 4 jobs in parallel
Extracting full menu dump
HLT menu: hltGetConfiguration /dev/CMSSW_12_3_0/PRef/V52 --full --offline --mc --input file:../RelVal_Raw_PRef_MC.root --unprescale --process TEST20220324055823 --max-events 100 --globaltag=auto:run3_mc_PRef --type=PRef
Traceback (most recent call last):
  File "/pool/condor/dir_18518/jenkins/workspace/ib-run-HLT/CMSSW_12_3_X_2022-03-23-2300/bin/slc7_amd64_gcc10/hltCheckPrescaleModules", line 25, in <module>
    exec(open(name).read(), globals(), menu.__dict__)
  File "<string>", line 10, in <module>
NameError: name 'cms' is not defined
Traceback (most recent call last):
  File "/cvmfs/cms-ib.cern.ch/nweek-02725/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-03-20-0000/bin/slc7_amd64_gcc10/edmConfigDump", line 25, in <module>
    loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "hlt.py", line 10, in <module>
    process.source = cms.Source( "PoolSource",
NameError: name 'cms' is not defined

[2]

https://cmssdt.cern.ch/SDT/jenkins-artifacts/HLT-Validation/CMSSW_12_4_X_2022-03-23-2300/slc7_amd64_gcc10/HLT_Integration_GRun_MC_HLT_Photon75_R9Id90_HE10_IsoM_EBOnly_CaloMJJ300_PFJetsMJJ400DEta3_v5.log

%MSG-i ThreadStreamSetup:  (NoModuleName) 24-Mar-2022 11:49:31 UTC pre-events
setting # threads 4
setting # streams 4
%MSG
In file included from input_line_2:1:
/cvmfs/cms-ib.cern.ch/nweek-02725/slc7_amd64_gcc10/lcg/root/6.24.07-1af4b6b72252977b8fe2fde94ca5d475/etc/cling/Interpreter/RuntimeUniverse.h:53:11: error: redefinition of 'InterpreterGeneratedCodeDiagnosticsMaybeIncorrect'
      int InterpreterGeneratedCodeDiagnosticsMaybeIncorrect;
          ^
/cvmfs/cms-ib.cern.ch/nweek-02725/slc7_amd64_gcc10/lcg/root/6.24.07-1af4b6b72252977b8fe2fde94ca5d475/etc//cling/Interpreter/RuntimeUniverse.h:53:11: note: previous definition is here
      int InterpreterGeneratedCodeDiagnosticsMaybeIncorrect;
          ^
Replaced symbol atexit cannot be found in JIT!
Replaced symbol at_quick_exit cannot be found in JIT!
IncrementalExecutor::executeFunction: symbol '_ZN5cling7runtime6gClingE' unresolved while linking [cling interface function]!
You are probably missing the definition of cling::runtime::gCling
Maybe you need to load the corresponding shared library?

[..]

24-Mar-2022 11:58:44 UTC  Closed file file:../RelVal_Raw_GRun_MC.root
----- Begin Fatal Exception 24-Mar-2022 11:58:44 UTC-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=EgammaHLTClusterShapeProducer label='hltEgammaClusterShape'
   [2] While initializing meta data for branch: recoRecoEcalCandidatesToValuefloatAssociation_hltEgammaClusterShape__TEST20220324073045.
   Additional Info:
      [a] Fatal Root Error: @SUB=TInterpreter::AutoParse
Error parsing payload code for class reco::RecoEcalCandidate with content:

cmsbuild added this to the CMSSW_12_3_X milestone Mar 22, 2022

cmsbuild added backport hlt-pending orp-pending pending-signatures tests-pending labels Mar 22, 2022

cmsbuild added the bug-fix label Mar 22, 2022

cmsbuild added tests-started and removed tests-pending labels Mar 22, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 22, 2022

missirol and others added 2 commits March 22, 2022 23:30

updated path to EDM file for TSG tests on Run-3 Data

cbd71c9

missirol force-pushed the devel_hltIntegTests_123X branch from b8ccfd1 to f29fa19 Compare March 22, 2022 22:31

cmsbuild added tests-pending and removed tests-approved labels Mar 22, 2022

cmsbuild added tests-started and removed tests-pending labels Mar 22, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 23, 2022

cmsbuild removed hlt-pending pending-signatures labels Mar 23, 2022

cmsbuild added the fully-signed label Mar 23, 2022

cmsbuild added hlt-approved backport-ok and removed backport labels Mar 23, 2022

cmsbuild added orp-approved and removed orp-pending labels Mar 23, 2022

cmsbuild merged commit 5940100 into cms-sw:CMSSW_12_3_X Mar 23, 2022

missirol deleted the devel_hltIntegTests_123X branch March 24, 2022 20:23

missirol mentioned this pull request Apr 17, 2022

hltIntegrationTests tests failing randomly in IBs #37598

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update of `hltIntegrationTests` and path of Run-3 Data file for TSG tests [`12_3_X`] #37304

update of `hltIntegrationTests` and path of Run-3 Data file for TSG tests [`12_3_X`] #37304

missirol commented Mar 22, 2022

cmsbuild commented Mar 22, 2022 •

edited

Loading

missirol commented Mar 22, 2022

missirol commented Mar 22, 2022

cmsbuild commented Mar 22, 2022

cmsbuild commented Mar 22, 2022

missirol commented Mar 22, 2022

cmsbuild commented Mar 23, 2022

missirol commented Mar 23, 2022

cmsbuild commented Mar 23, 2022

perrotta commented Mar 23, 2022

perrotta commented Mar 24, 2022

Martin-Grunewald commented Mar 24, 2022

Martin-Grunewald commented Mar 24, 2022

Martin-Grunewald commented Mar 24, 2022 •

edited

Loading

missirol commented Mar 24, 2022

update of hltIntegrationTests and path of Run-3 Data file for TSG tests [12_3_X] #37304

update of hltIntegrationTests and path of Run-3 Data file for TSG tests [12_3_X] #37304

Conversation

missirol commented Mar 22, 2022

PR description:

PR validation:

If this PR is a backport, please specify the original PR and why you need to backport that PR:

cmsbuild commented Mar 22, 2022 • edited Loading

missirol commented Mar 22, 2022

missirol commented Mar 22, 2022

cmsbuild commented Mar 22, 2022

Comparison Summary

cmsbuild commented Mar 22, 2022

missirol commented Mar 22, 2022

cmsbuild commented Mar 23, 2022

Comparison Summary

missirol commented Mar 23, 2022

cmsbuild commented Mar 23, 2022

perrotta commented Mar 23, 2022

perrotta commented Mar 24, 2022

Martin-Grunewald commented Mar 24, 2022

Martin-Grunewald commented Mar 24, 2022

Martin-Grunewald commented Mar 24, 2022 • edited Loading

missirol commented Mar 24, 2022

update of `hltIntegrationTests` and path of Run-3 Data file for TSG tests [`12_3_X`] #37304

update of `hltIntegrationTests` and path of Run-3 Data file for TSG tests [`12_3_X`] #37304

cmsbuild commented Mar 22, 2022 •

edited

Loading

Martin-Grunewald commented Mar 24, 2022 •

edited

Loading