dump_python does not work with Nano step

[srimanob]

While investigating on Nano sequence, I see that I can't dump config file if Nano step is a part of the driver. The error message is

RuntimeError: Task has a label, but using its label to get an attribute from the process yields a different object or None
label = electronTask

cmsDriver I am using with CMSSW_12_5_0_pre3:
cmsDriver.py step3 -s NANO --conditions auto:phase1_2022_realistic --datatier NANOAODSIM -n -1 --eventcontent NANOEDMAODSIM --geometry DB:Extended --era Run3 --python Nano_2021_dump.py --no_exec --filein file:step1_inMINIAODSIM.root --fileout file:step3.root --dump_python

It seems the dump_python can't get the electronTask right with
https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/NanoAOD/python/nano_cff.py#L228
(Just a guess as I comment it out, this step will pass. And then go to the next issues of photonTask, egmPhotonIDTask).

By the way, how can I get a full config of
cmsDriver.py step3 -s NANO --conditions auto:phase1_2022_realistic --datatier NANOAODSIM -n -1 --eventcontent NANOEDMAODSIM --geometry DB:Extended --era Run3 --python Nano_2021_dump.py --no_exec --filein file:step1_inMINIAODSIM.root --fileout file:step3.root
without --dump_python? I am checking why Nano can't work with FastSim if they are in the same sequence, but work when they are separated.

Thanks.

[cmsbuild]

A new Issue was created by @srimanob Phat Srimanobhas.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[srimanob]

I see one recommendation to run in
https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuidePythonTips#Running_cmsRun_from_the_script_i

[Dr15Jones]

You could have cmsDriver write the standard cfg file and then add print(process.dumpPython()) at the very end of the file. Then do python3 the_cfg.py

[mariadalfonso]

assign xpog

[cmsbuild]

New categories assigned: xpog

@mariadalfonso,@gouskos,@fgolf you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

edmConfigDump Nano_2021_dump.py seems to work as well, pointing the culprit to lie somewhere in ConfigBuilder.

[makortel]

assign operations

[cmsbuild]

New categories assigned: operations

@fabiocos,@qliphy,@davidlange6,@rappoccio,@perrotta you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mariadalfonso]

@srimanob I prepared a fix for your python dump problem.
Can you test locally on your side.

[srimanob]

Hi @mariadalfonso
Thanks very much.

I try with the following cmsDriver (need to split GEN / FastSim as discussed in #37351)

Step-1
cmsDriver.py TTbar_14TeV_TuneCP5_cfi -s GEN -n 200 --conditions auto:phase1_2022_realistic --beamspot Run3RoundOptics25ns13TeVLowSigmaZ --datatier GEN --eventcontent FEVTDEBUG --geometry DB:Extended --era Run3_FastSim --fast --python TTbar_14TeV_TuneCP5_2021_Gen.py --no_exec --fileout file:step1_GEN.root --nThreads 8

Step-2 (without --dump_python)
cmsDriver.py step2 -s SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,PAT,NANO -n 50 --conditions auto:phase1_2022_realistic --beamspot Run3RoundOptics25ns13TeVLowSigmaZ --datatier MINIAODSIM,NANOAODSIM --eventcontent MINIAODSIM,NANOEDMAODSIM --geometry DB:Extended --era Run3_FastSim --fast --python TTbar_14TeV_TuneCP5_2021_Run3FSNano.py --no_exec --fileout file:step2.root --filein file:step1_GEN.root --nThreads 8

Step-2 (with --dump_python)
cmsDriver.py step2 -s SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,PAT,NANO -n 50 --conditions auto:phase1_2022_realistic --beamspot Run3RoundOptics25ns13TeVLowSigmaZ --datatier MINIAODSIM,NANOAODSIM --eventcontent MINIAODSIM,NANOEDMAODSIM --geometry DB:Extended --era Run3_FastSim --fast --python TTbar_14TeV_TuneCP5_2021_Run3FSNano_Dump.py --no_exec --fileout file:step2_dump.root --filein file:step1_GEN.root --nThreads 8 --dump_python

WIth your PR (#38749), the config can be dumped now. However, it can't run. What I see is that
(1) this info

customising the process with miniAOD_customizeAllMCFastSim from PhysicsTools/PatAlgos/slimming/miniAOD_tools
	--- egmGsfElectronIDs:cutBasedElectronID-Fall17-94X-V1-loose added to patElectrons
	--- egmGsfElectronIDs:cutBasedElectronID-Fall17-94X-V1-medium added to patElectrons
	--- egmGsfElectronIDs:cutBasedElectronID-Fall17-94X-V1-tight added to patElectrons

is on top of

customising the process with nanoAOD_customizeMC from PhysicsTools/NanoAOD/nano_cff
	--- egmGsfElectronIDs:cutBasedElectronID-Fall17-94X-V1-loose added to patElectrons
	--- egmGsfElectronIDs:cutBasedElectronID-Fall17-94X-V1-medium added to patElectrons
	--- egmGsfElectronIDs:cutBasedElectronID-Fall17-94X-V1-tight added to patElectrons

and (2) when I try to run:

----- Begin Fatal Exception 15-Jul-2022 14:40:46 CEST-----------------------
An exception of category 'ValueNameAlreadyExists' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 5 stream: 2
   [1] Running path 'MINIAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'MINIAODSIMoutput'
   [3] Calling method for module PATElectronSlimmer/'slimmedElectrons'
Exception Message:
Trying to add new UserInt = heepElectronID_HEEPV71 failed because it already exists and you didnt specify to override it (set in the config overrideExistingValues=cms.bool(True) )
----- End Fatal Exception -------------------------------------------------

[swertz]

Hi @srimanob , I've tried to run the above commands and everything works fine now, can you confirm the issue is fixed?

I've noticed that when running without --dump-python, the messages about adding the ID to ele/photons only appear for Nano, but appear for both Mini and Nano when running with --dump-python... any idea why this could happen? Regardless, running the config works in both cases.

[srimanob]

Hi @swertz
It is the same observation as in #38686 (comment). Kindly ping @cms-sw/xpog-l2 in parallel, to have a look if possible.

[swertz]

Yes but the exception (2) above does not happen anymore when running the config, despite the duplicate messages.

[swertz]

Looking into it, the messages only appear once, for the Nano customization, not the Mini. The reason they appear a second time when dumping the python is that edmConfigDump executes the config, thereby calling nanoAOD_customizeMC(process) again.

The EGM VID in https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/NanoAOD/python/nano_cff.py#L223-L240 could be cleaned up a bit but things seem to be working fine as they are, so could you please clarify what the problem is?

[srimanob]

Hi @swertz
Thanks. I think I misunderstood your message here. The answer is "Yes", the issue was fixed by #39127. There was a discussion on how things are loaded, and behavior of config file in general. At the end, we chose that dump_python will use edmConfigDump instead of dumpPython.

[srimanob]

This issue can be closed if there is no more issue to discuss. Thanks.

[swertz]

+xpog

Ok, thanks for confirming!