L1T re-emulation on 90X MC does not get the correct HCAL TP Digis

[davignon]

@rekovic @bortigno

Using the standard recipe for L1Ntuples production:
cmsDriver.py -s RAW2DIGI --python_filename=l1NtupleMcMaker2017_RAW2DIGI.py -n 50 --no_output --no_exec --era=Run2_2016 --mc --conditions=auto:run2_mc --customise=L1Trigger/Configuration/customiseReEmul.L1TReEmulFromRAW --customise=L1Trigger/L1TNtuples/customiseL1Ntuple.L1NtupleRAWEMU --customise=L1Trigger/Configuration/customiseSettings.L1TSettingsToCaloStage2Params_2017_v1_4 --filein=/store/mc/PhaseISpring17DR/VBFHToTauTau_M125_13TeV_powheg_pythia8/GEN-SIM-RAW/FlatPU28to62HcalNZSRAW_HIG07_90X_upgrade2017_realistic_v20-v1/100000/004D4A52-A62C-E711-B518-848F69FD2853.root

I observed the HCAL TPs are empty, which in turn creates towers without an hadronic component.
It seems L1Trigger/Configuration/python/customiseReEmul.py is not reading the correct HCAL TPs in 90X.

We tried to solve but did not manage. Can someone have a look?

Cheers,

• Olivier

[fwyzard]

might be related to https://its.cern.ch/jira/browse/CMSLITDPG-112 ?

[davignon]

@fwyzard

Thanks.

It may very well be related: I also see empty HCAL TPs/Towers in the L1Ntuples using the latest data command of CMSLITDPG-112:
cmsDriver.py l1Ntuple -s RAW2DIGI --era=Run2_2017 --customise=L1Trigger/L1TNtuples/customiseL1Ntuple.L1NtupleRAWEMU --customise=L1Trigger/Configuration/customiseReEmul.L1TReEmulFromRAW --conditions=92X_dataRun2_Prompt_v2 -n 100 --data --no_exec --no_output --filein=/store/data/Run2017A/ZeroBias/RAW/v1/000/295/315/00000/A82679F4-B542-E711-B148-02163E013395.root --customise=L1Trigger/Configuration/customiseSettings.L1TSettingsToCaloStage2Params_2017_v1_5

Cheers,

• Olivier

[bortigno]

Hi @davignon,
At the trigger primitives meeting yesterday
https://docs.google.com/presentation/d/113-AKQZKnhzfZrcNCJhl1v-oT6D3Nx_arOsCL8bpbUY/edit#slide=id.g225956b814_0_169
HCAL experts reported that they are aware of a problem that our TEA shifter noticed in the last release validation with the the HCAL TPs not appearing in the latest release. They should have a fix for it in PR cms-sw#19047.

Not sure if it will fix your problem, but this is all I know for the moment about a possible issue with HCAL TPs.

Thank you,
Pier

[davignon]

Problem with empty HCAL TPs in data is "solved" when changing this:

cmsDriver.py l1Ntuple -s RAW2DIGI --era=Run2_2017 --customise=L1Trigger/L1TNtuples/customiseL1Ntuple.L1NtupleRAWEMU --customise=L1Trigger/Configuration/customiseReEmul.L1TReEmulFromRAW --conditions=92X_dataRun2_Prompt_v2 -n 100 --data --no_exec --no_output --filein=/store/data/Run2017A/ZeroBias/RAW/v1/000/295/315/00000/A82679F4-B542-E711-B148-02163E013395.root --customise=L1Trigger/Configuration/customiseSettings.L1TSettingsToCaloStage2Params_2017_v1_5

to

cmsDriver.py l1Ntuple -s RAW2DIGI --era=Run2_2016 --customise=L1Trigger/L1TNtuples/customiseL1Ntuple.L1NtupleRAWEMU --customise=L1Trigger/Configuration/customiseReEmul.L1TReEmulFromRAW --conditions=92X_dataRun2_Prompt_v2 -n 100 --data --no_exec --no_output --filein=/store/data/Run2017A/ZeroBias/RAW/v1/000/295/315/00000/A82679F4-B542-E711-B148-02163E013395.root --customise=L1Trigger/Configuration/customiseSettings.L1TSettingsToCaloStage2Params_2017_v1_5

Problem on MC 90X remains... So far I had tried with era=Run2_2016. I just tried with era=Run2_2017, leading to a crash this time:

----- Begin Fatal Exception 02-Jun-2017 15:07:47 CEST-----------------------
An exception of category 'Conditions mismatch' occurred while
   [0] Processing  Event run: 1 lumi: 515 event: 484449 stream: 0
   [1] Running path 'L1TReEmulPath'
   [2] Calling method for module HcalTrigPrimDigiProducer/'simHcalTriggerPrimitiveDigis'
Exception Message:
Requested conditions of type HcalTPChannelParameters for cell (0x49307401) (HF -29,1,3) got conditions for cell (0x49107801) (HF -30,1,1)
----- End Fatal Exception -------------------------------------------------

I then tried to update the customiseReEmul to:

    import EventFilter.HcalRawToDigi.HcalRawToDigi_cfi
    process.unpackHcal = EventFilter.HcalRawToDigi.HcalRawToDigi_cfi.hcalDigis.clone(
        InputLabel = cms.InputTag( 'rawDataCollector', processName=cms.InputTag.skipCurrentProcess()))

    process.simHcalTriggerPrimitiveDigis.inputLabel = cms.VInputTag(
        cms.InputTag('unpackHcal'),
        cms.InputTag('unpackHcal')
        )
    
    process.simHcalTriggerPrimitiveDigis.inputUpgradeLabel = cms.VInputTag(
        cms.InputTag('unpackHcal'),
        cms.InputTag('unpackHcal')
    )

with era2017, leading to the following error:

----- Begin Fatal Exception 02-Jun-2017 15:11:42 CEST-----------------------
An exception of category 'NoRecord' occurred while
   [0] Processing  Event run: 1 lumi: 515 event: 484449 stream: 0
   [1] Running path 'L1TReEmulPath'
   [2] Calling method for module CSCTriggerPrimitivesProducer/'simCscTriggerPrimitiveDigis'
   [3] Using EventSetup component GEMGeometryESModule/'' to make data GEMGeometry/'' in record MuonGeometryRecord
   [4] While getting dependent Record from Record MuonGeometryRecord
Exception Message:
No "GEMRecoGeometryRcd" record found in the EventSetup for synchronization value
Run: 1 LuminosityBlock: 515 Event: 0 Time: 1
 Please add an ESSource or ESProducer that delivers such a record.
----- End Fatal Exception -------------------------------------------------

So it now concerns muons. Maybe related to #546? @bortigno , what do you think?

[rekovic]

@davignon
On MC RAW one should be using L1TReEmulMCFromRAW and not L1TReEmulFromRAW customisation option.

For 2017 MC, one should use --era=Run2_2017.

This example command with L1TReEmulMCFromRAW and --era=Run2_2017 on a Phase1Spring17DR MC, produces L1TNtuples with filled emulated HCAL TPs see here.

cmsDriver.py -s RAW2DIGI --python_filename=l1NtupleMcMaker2017_RAW2DIGI_v2.py -n 10 --no_output --no_exec --era=Run2_2017 --mc --conditions=92X_upgrade2017_TSG_For90XSamples_V1 --customise=L1Trigger/Configuration/customiseReEmul.L1TReEmulMCFromRAW --customise=L1Trigger/L1TNtuples/customiseL1Ntuple.L1NtupleRAWEMU --customise=L1Trigger/Configuration/customiseSettings.L1TSettingsToCaloStage2Params_2017_v1_4 --filein=/store/mc/PhaseISpring17DR/VBFHToTauTau_M125_13TeV_powheg_pythia8/GEN-SIM-RAW/FlatPU28to62HcalNZSRAW_HIG07_90X_upgrade2017_realistic_v20-v1/100000/004D4A52-A62C-E711-B518-848F69FD2853.root

Note

If the above command is used with --era=Run2_2016, the emulate HCAL TPs are missing in forward region here

[davignon]

@rekovic

Thanks for looking into it.

Rather than switching from L1TReEmulFromRAW to L1TReEmulMCFromRAW (which doesn't change a thing to the calo settings and output), comparing our cmsDriver commands, the problem is solved in two ways:

• Using --era=Run2_2017 (if you use Run2_2016, it will plain crash like I mentioned before).
• Using --conditions=92X_upgrade2017_TSG_For90XSamples_V1 instead of --conditions=auto:run2_mc which seems to be failing (do you know why?).

I tried both Run2_2017 and 92X_upgrade2017_TSG_For90XSamples_V1 before, but not at the same time...

Do you mind if I update the recipe on the TWiki?

Anyway, we can close the issue. Thanks Vladimir!

Cheers,

• Olivier

[rekovic]

Yes, I forgot to say that in my example I used the GT --conditions=92X_upgrade2017_TSG_For90XSamples_V1.
This is needed to pick up the L1T Menu prescales/masks from DB.

[rekovic]

@davignon
To be kosher, you should use the ReEmulMCFromRAW, as it does not rely on CSCTF digis, in case
anybody else apart from Calo people look at your ntuples.

Now, it is correct that Calo part is the same between ReEmulMCFromRAW and ReEmuFromRAW,
so you are free to use either of the two.

[rekovic]

@davignon
Sure, please feel free to update the twiki.

[davignon]

@rekovic

I updated the TWiki.
Yes I will use ReEmulMCFromRAW.
About the GlobalTag, I was wondering why auto is 'broken'?

[fwyzard]

In general "auto" GTs assume a consistent sample, where (possible GENSIM), DIGI, RAW and RECO are done with the same release and "auto" tag.
If you are running with 9.2.x over 83x/90x samples there are likely differences and incompatibilities between the releases and GTs used to produce the samples, and the current ones.
That's why we need specific GTs to mix these samples with the current releases (see https://twiki.cern.ch/twiki/bin/viewauth/CMS/SpecialGlobalTagsForTSG )

I have not checked the specific case, though.

[fwyzard]

@rekovic , rather than keeping two recipes, would it make sense to migrate the recipe for making ntuples to use the L1REPACK step ?

[rekovic]

@fwyzard
Yes, I think that is a good idea, as both emulated and unpacked L1T data are available.

[rekovic]

I am re-opening this issue as some development was done on this front.

While discussing consistent derivation of Calo Layer1 & Layer2 calibrations from 90x MC RAW samples, it was concluded:

1. Layer1 calibrations were done by re-emulating HCalTPs starting with simHcalUnsuppressedDigis as inputs, as there were problems accessing unpacked hcalDigis from MC 90X RAW samples.
2. Layer2 calibrations derivation, to be absolutely consistent with Layer1 calibrations need to use L1TReEmulMCFrom90xRAWSimHcalTP, now available in l1t-integration-v95.13, like in the example command (make sure you use correct caloParams customisation.)

cmsDriver.py -s RAW2DIGI --python_filename=l1NtupleMcMaker2017_RAW2DIGI_reEmu_HCAL_TPs.py -n 100 --no_output --no_exec --era=Run2_2017 --mc --conditions=92X_upgrade2017_TSG_For90XSamples_V1 --customise=L1Trigger/Configuration/customiseReEmul.L1TReEmulMCFrom90xRAWSimHcalTP --customise=L1Trigger/L1TNtuples/customiseL1Ntuple.L1NtupleRAWEMU --customise=L1Trigger/Configuration/customiseSettings.L1TSettingsToCaloStage2Params_2017_v1_9_inconsistent --filein=/store/mc/PhaseISpring17DR/VBFHToTauTau_M125_13TeV_powheg_pythia8/GEN-SIM-RAW/FlatPU28to62HcalNZSRAW_HIG07_90X_upgrade2017_realistic_v20-v1/100000/004D4A52-A62C-E711-B518-848F69FD2853.root

This version of ReEmul customisation uses simHcalUnsuppressedDigis available in 90x MC RAW as inputs to re-emulation of HCAL TPs, just like in procedure used for derivation of Layer1 calibrations. .

3. Previously reported spike distribution of HcalTPs are understood, as they were obtained with L1TReEmulMCFrom90xRAWSimHcalTP. This is due to mix-use of hcalDigis and default simHcalUnsuppressedDigis for simHcalTriggerPrimitiveDigis.inputLabel and inputUpgradeLabel, respectively. The reason these were not both set to hcalDigis (like in L1REPACK:FullMC) is the crash in the re-emulation of HCAL TPs :

An exception of category 'BadAlloc' occurred while
   [0] Processing  Event run: 1 lumi: 515 event: 484449 stream: 0
   [1] Running path 'L1TReEmulPath'
   [2] Calling method for module HcalTrigPrimDigiProducer/'simHcalTriggerPrimitiveDigis'
Exception Message:

This crash in re-emulation was observed in various workflows (reported by Layer1 group, HCAL DPG) and is not understood up to this moment. It is interesting that it does not happen in L1REPACK` workflows.

4. Users should note use L1REPACK:FullMC as a step to derive Layer2 calibration, as that step used
   unpacked HCAL digis for Upgrade is not available in CMSSW_9_2_0. Plus, it has not yet been validated on 90x MC RAW samples.

[davignon]

@rekovic
Thanks for clarifying this. I think we have a working recipe that should be enough for now.

@fwyzard
Thanks for the explanation about global tags.