-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clean up temporary 2017 workflows and eras, add "2017 design" workflow #15994
Clean up temporary 2017 workflows and eras, add "2017 design" workflow #15994
Conversation
A new Pull Request was created by @makortel (Matti Kortelainen) for CMSSW_8_1_X. It involves the following packages: Configuration/Eras @cvuosalo, @franzoni, @fabozzi, @cmsbuild, @srimanob, @slava77, @hengne, @davidlange6 can you please review it and eventually sign? Thanks. cms-bot commands are list here #13028 |
@cmsbuild , please test |
The tests are being triggered in jenkins. |
-1 Tested at: 49a5a10 You can see the results of the tests here: I found follow errors while testing this PR Failed tests: UnitTests RelVals AddOn
I found errors in the following unit tests: ---> test TestConfigDP had ERRORS
When I ran the RelVals I found an error in the following worklfows: runTheMatrix-results/5.1_TTbar+TTbarFS+HARVESTFS/step1_TTbar+TTbarFS+HARVESTFS.log4.22 step2 runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log135.4 step1 runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step1_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log140.53 step2 runTheMatrix-results/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log101.0 step1 runTheMatrix-results/101.0_SingleElectronE120EHCAL+SingleElectronE120EHCAL/step1_SingleElectronE120EHCAL+SingleElectronE120EHCAL.log8.0 step3 runTheMatrix-results/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/step3_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS.log1000.0 step2 runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log1001.0 step2 runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log1003.0 step2 runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log1306.0 step3 runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15.log10021.0 step3 runTheMatrix-results/10021.0_TenMuE_0_200+TenMuE_0_200_pythia8_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017/step3_TenMuE_0_200+TenMuE_0_200_pythia8_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log1330.0 step3 runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15/step3_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15.log136.731 step3 runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step3_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log9.0 step3 runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log25.0 step3 runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log20024.0 step3 runTheMatrix-results/20024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D1_GenSimFull+DigiFull_2023D1+RecoFullGlobal_2023D1+HARVESTFullGlobal_2023D1/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D1_GenSimFull+DigiFull_2023D1+RecoFullGlobal_2023D1+HARVESTFullGlobal_2023D1.log10024.0 step3 runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log50202.0 step3 runTheMatrix-results/50202.0_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50/step3_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50.log25202.0 step3 runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25.log20824.0 step3 runTheMatrix-results/20824.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D3_GenSimFull+DigiFull_2023D3+RecoFullGlobal_2023D3+HARVESTFullGlobal_2023D3/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D3_GenSimFull+DigiFull_2023D3+RecoFullGlobal_2023D3+HARVESTFullGlobal_2023D3.log
I found errors in the following addon tests: cmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,HLT:@Fake,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Tue Sep 27 14:03:16 2016-date Tue Sep 27 14:03:14 2016 s - exit: 256 |
Didn't clean up my developer area before running tests, and missed this import to a removed file. |
@cmsbuild please test |
The tests are being triggered in jenkins. |
The changes in this PR look fine to me from the HCAL side. (At some point we need to get the "final" 2017 geometry into the database and GT, but that can be handled separately once #15985 is resolved.) |
The tests are being triggered in jenkins. |
+1 |
be submitted shortly... On Mon, 17 Oct 2016, Marco Musich wrote:
|
so this will not be included in pre13 after all? #16232 |
seems not: making conditions management a nightmare... |
Comparison is ready @slava77 comparisons for the following workflows were not done due to missing matrix map:
|
@mmusich Out of curiosity, would it help at all if we get this PR merged soon? Anyway, I guess for pre13 RelVals we're interested only in |
@makortel indeed it would help, but the problem is that since this PR is changing the "meaning" of the
that I will need to roll back and forth each time I will need to update the queue for any other reason (in the "current" meaning of |
+1 |
+1 |
#2017 WFs to run in IB (TenMuE_0_200, TTbar, ZEE, MinBias, TTbar PU, ZEE PU) | ||
numWFIB = [10021.0,10024.0,10025.0,10026.0,10023.0,10224.0,10225.0] | ||
#2017 WFs to run in IB (TenMuE_0_200, TTbar, ZEE, MinBias, TTbar PU, ZEE PU, TTbar design) | ||
numWFIB = [10021.0,10024.0,10025.0,10026.0,10023.0,10224.0,10225.0,10424.0] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@makortel , In 81X IBs, this workflow 10424.0 is failing as it is trying to get /RelValTTbar_13/1/GEN-SIM data. Can you please look why it is asking for wrong dataset?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
my guess is tht https://github.com/makortel/cmssw/blob/77eecdbe8a344f294ced034a29773c8d8a6d0d27/Configuration/PyReleaseValidation/python/relval_steps.py#L1876 is where it is setting invalid dataset for TTbar_13TeV_TuneCUETP8M1_2017Design_GenSimFull
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Umm... I need to take a look. Thanks for letting me know. Apparently the workflow has never worked with -i all
...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@smuzaffar it's a combination of the line you referenced and https://github.com/makortel/cmssw/blob/77eecdbe8a344f294ced034a29773c8d8a6d0d27/Configuration/PyReleaseValidation/python/relval_steps.py#L1652-L1653
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it make sense on
https://github.com/makortel/cmssw/blob/77eecdbe8a344f294ced034a29773c8d8a6d0d27/Configuration/PyReleaseValidation/python/relval_steps.py#L1875
that instead of "temporarily excluding 2023" with an explicit if, enable the input query only if defaultDataSets
is not empty for that upgradeKey?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would be better if you can make it work consistently... the actual use of defaultDataSets
is a bit complicated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I noticed. I'm giving a try anyway (but it won't simplify the overall complexity).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#16460 should fix the issue.
@@ -62,33 +61,24 @@ | |||
|
|||
upgradeProperties[2017] = { | |||
'2017' : { | |||
'Geom' : 'DB:Extended', | |||
'Geom' : 'Extended2017new', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@makortel - this does not exist any more, please, use DB:Extended
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ianna don't worry, this is an old PR
'ScenToRun' : ['GenSimFull','DigiFull','RecoFull','ALCAFull','HARVESTFull'], | ||
}, | ||
'2017AllNew' : { | ||
'2017Design' : { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@makortel - same here
This PR in an attempt to clean up all the temporary 2017 workflows and eras introduced in pre12. The old "2017AllNew" workflow is made as the new "2017" workflow. It is possible that I missed something, but at least everything in this PR needs to be done.
In addition, a "2017 design" workflow is added for further detailed (tracking) studies with the new geometry (also after 810pre13).
Tested in 8_1_0_pre12 (rebased on top of
CMSSW_8_1_X_2016-09-28-2300CMSSW_8_1_X_2016-10-02-2300CMSSW_8_1_X_2016-10-06-1100CMSSW_8_1_X_2016-10-12-1100CMSSW_8_1_X_2016-10-16-1100), no changes expected between IB 108XX.0, and IB+PR 100XX.0 (or phase0/phase2). Between IB 100XX.0 and IB+PR 100XX.0 changes can be expected everywhere depending on tracker or HCAL (details being a topic of the validation of pre12).@schneiml @veszpv @kpedro88