Add bx skim and corresponding RelVal

[kfjack]

PR description:

This PR includes the proposal of BunchCrossingSkim PR #26419 and the corresponding updates for RelVal. This is for the master branch.

PR validation:

RelVal workflows are added to 2018E HLTPhy and ZeroBias. Given the special situation of the BunchCrossingSkim the RelVal has only SKIM part, without RECO and other steps.
The corresponding workflows are
136.899
136.900

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26476/9323

• This PR adds an extra 56KB to repository

[cmsbuild]

A new Pull Request was created by @kfjack (Kai-Feng Chen) for master.

It involves the following packages:

Configuration/PyReleaseValidation
DPGAnalysis/Skims

@pgunnell, @zhenhu, @prebello, @cmsbuild, @kpedro88 can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[fabiocos]

please test workflow 136.899,136.900

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34254/console Started: 2019/04/17 17:38

[cmsbuild]

+1
Tested at: 1214bcd
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26476/34254/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26476/34254/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-26476/136.899_RunHLTPhy2018E+RunHLTPhy2018E+HLTDR2_2018+RECODR2_2018reHLT_skimHLTPhy_Prompt+HARVEST2018_Prompt
• /build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-26476/136.9_RunZeroBias2018E+RunZeroBias2018E+HLTDR2_2018+RECODR2_2018reHLT_skimZeroBias_Prompt+HARVEST2018_Prompt

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3142783
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3142585
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 133 log files, 14 edm output root files, 32 DQM output files

[kfjack]

Hi all, it seems that I cannot remove RECO steps from the relval (the comparison cannot be performed), even it is not really needed for this special skim. Will try to add them back and see if it is functioning...

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/418/console Started: 2019/05/22 15:06

[cmsbuild]

-1

Tested at: 53ac3e6

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3d7b1b/418/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

• RelVals:

When I ran the RelVals I found an error in the following workflows:
136.899 step3

runTheMatrix-results/136.899_RunHLTPhy2018E+RunHLTPhy2018E+HLTDR2_2018+RECODR2_2018reHLT_skimHLTPhy_Prompt+HARVEST2018_Prompt/step3_RunHLTPhy2018E+RunHLTPhy2018E+HLTDR2_2018+RECODR2_2018reHLT_skimHLTPhy_Prompt+HARVEST2018_Prompt.log

136.9 step3

runTheMatrix-results/136.9_RunZeroBias2018E+RunZeroBias2018E+HLTDR2_2018+RECODR2_2018reHLT_skimZeroBias_Prompt+HARVEST2018_Prompt/step3_RunZeroBias2018E+RunZeroBias2018E+HLTDR2_2018+RECODR2_2018reHLT_skimZeroBias_Prompt+HARVEST2018_Prompt.log

[cmsbuild]

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

[kpedro88]

For future reference, the error is:

----- Begin Fatal Exception 22-May-2019 15:37:54 CEST-----------------------
An exception of category 'NoProxyException' occurred while
   [0] Processing  Event run: 325594 lumi: 1 event: 16371 stream: 0
   [1] Running path 'Flag_METFilters'
   [2] Prefetching for module BooleanFlagFilter/'HBHENoiseFilter'
   [3] Prefetching for module HBHENoiseFilterResultProducer/'HBHENoiseFilterResultProducer'
   [4] Prefetching for module HcalNoiseInfoProducer/'hcalnoise'
   [5] Prefetching for module FastjetJetProducer/'ak4PFJets'
   [6] Prefetching for module PFLinker/'particleFlow'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module GsfTrackProducer/'electronGsfTracks'
   [11] Prefetching for module MeasurementTrackerEventProducer/'MeasurementTrackerEvent'
   [12] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [13] Prefetching for module PrimaryVertexProducer/'firstStepPrimaryVerticesPreSplitting'
   [14] Calling method for module TrackProducer/'initialStepTracksPreSplitting'
   [15] Using EventSetup component TkTransientTrackingRecHitBuilderESProducer/'TTRHBuilderAngleAndTemplate' to make data TransientTrackingRecHitBuilder/'WithAngleAndTemplate' in record TransientRecHitRecord
   [16] Using EventSetup component PixelCPEClusterRepairESProducer/'templates2' to make data PixelClusterParameterEstimator/'PixelCPEClusterRepair' in record TkPixelCPERecord
   [17] Using EventSetup component SiPixel2DTemplateDBObjectESProducer/'siPixel2DTemplateDBObjectESProducer' to make data SiPixel2DTemplateDBObject/'' in record SiPixel2DTemplateDBObjectESProducerRcd
Exception Message:
No data of type "SiPixel2DTemplateDBObject" with label "0T" in record "SiPixel2DTemplateDBObjectRcd"
 Please add an ESSource or ESProducer to your job which can deliver this data.
----- End Fatal Exception -------------------------------------------------

It could take significant work to make the full RECO step cooperate with this special scenario. To make comparison plots, really it's the DQM step that is needed, but I doubt this can run easily without the preceding RECO step.

If there is sufficient interest in studying the data from these runs, we can try to introduce a specific Modifier like run2_2018_highPU, in order to remove/modify pieces of the RECO step that don't work. But I am not sure exactly how much work this would take.

[fabiocos]

@kfjack what is the plan about this PR? Is there activity, or it can be just closed for the time being?

[kfjack]

Hi @fabiocos I think we shall close this PR for the time being as well. If all agree I'll close it.