start branch for 13_0_0 replay

[davidlange6]

Replay Request

Requestor
ORM

Describe the configuration

• Release: CMSSW_13_0_0
• Run: 364158, 350966, 359691
• GTs:
  • expressGlobalTag: 130X_dataRun3_Express_Candidate_2023_03_07_09_53_07
  • promptrecoGlobalTag: 130X_dataRun3_Prompt_Candidate_2023_03_09_09_47_16
• Additional changes: still need to incorporate an alca matrix update

Purpose of the test
First test of CMSSW_13_0_0

[cmsdmwmbot]

Can one of the admins verify this patch?

[davidlange6]

@francescobrivio - I started this PR - I can pick up GTs and Alca matrix when you have time ready (either via a PR to my branch, or a new PR, or other..)

[francescobrivio]

Hi @davidlange6 thanks for preparing this!
You can cherry-pick commit francescobrivio@223c427 to get the alca updates for 13_0_0, namely:

• GTs are updated to:
  • Express: 130X_dataRun3_Express_Candidate_2023_03_07_09_53_07
  • Prompt: 130X_dataRun3_Prompt_Candidate_2023_03_07_09_53_07
  • They are actually identical to the current 126X GTs deployed online, but we want to update the names to be in sync with the release and avoid confusion in the future
• JetMET, EGamma and Muon PDs have been split in 2 PDs (e.g. "JetMET0, JetMET1") (see CMSHLT-2596)
  • I did not remove the original PDs (e.g. "JetMET") since last year we had only the single PDs and that's what we need to use for the replay
• Added the AlCaPCCRandom alcareco to the AlCaLumiPixelsCountsPrompt PD (see CMSHLT-2595)
  • Altough this alcareco runs on a trigger (AlCa_LumiPixelsCounts_Random_v) which last year was not included in the AlCaLumiPixelsCountsPrompt dataset, so it won't produce anything I think

[francescobrivio]

@germanfgv how is the Express processing going to work with this replay?
Last time (see #4793 (comment)) you had to repack in 12_4_X and perform the Express reco step with 12_6_X. So I guess you'll have to do the same trick here right?

[francescobrivio]

test syntax please

[davidlange6]

Thx. I've added @francescobrivio's commit to this PR and removed the draft label. I think we can try a test from this config..

[davidlange6]

seems the test failed, does someone know how to find the actual failure amongst the 100 warnings?

[germanfgv]

The syntax check is broken... I'll check why it's not working, but we can ignore it for now

@germanfgv how is the Express processing going to work with this replay? Last time (see #4793 (comment)) you had to repack in 12_4_X and perform the Express reco step with 12_6_X. So I guess you'll have to do the same trick here right?

Yes, I'll manually run this. The mismatch in Overrides Catalogs prevents us from smoothly using both releases. We will use 12_4_X for Repack and first step of Express, and 13_0_0 for Prompt and the reco step of Express.

[germanfgv]

I started a replay with the current configuration. You can monitor it's progress here:
https://monit-grafana.cern.ch/d/t_jr45h7k/cms-tier0-replayid-monitoring?orgId=11&var-Bin=5m&var-ReplayID=230308190233&var-JobType=All&var-WorkflowType=All&refresh=1m

I included #4796 in the replay agent @francescobrivio

[germanfgv]

I missed the fact that 13_0_0 was only released with el8_amd64_gcc11. Most 12_4_X releases are not available in this arch, so I had to make a change to the configuration. I'm re-launching the test.

[francescobrivio]

Given the issue with the Prompt GT reported in Mattermost [*], we have prepared a new Prompt Candidate GT to be used in this replay:

• New Prompt Candidate GT: 130X_dataRun3_Prompt_Candidate_2023_03_09_09_47_16
  • As expected the difference wrt the previous Candidate GT is only in the trigger bits tag (link to CondDB)
  • @germanfgv This change will affect only the AlCaLumiPixelsCountsPrompt stream in case you want to re-launch only that one

[*] Error log:

An exception of category 'Configuration' occurred while
   [0] Processing  Event run: 359691 lumi: 84 event: 112342348 stream: 6
   [1] Running path 'pathALCARECOAlCaPCCRandom'
   [2] Calling method for module HLTHighLevel/'ALCARECORandomHLT'
Exception Message:
 HLTHighLevel [instance: ALCARECORandomHLT - path: pathALCARECOAlCaPCCRandom]: No triggerList with key AlCaPCCRandom in AlCaRecoTriggerBitsRcd

[germanfgv]

This is the monitoring for the new replay.

https://monit-grafana.cern.ch/d/t_jr45h7k/cms-tier0-replayid-monitoring?orgId=11&var-Bin=5m&var-ReplayID=230309152235&var-JobType=All&var-WorkflowType=All&refresh=1m&from=now-6h&to=now

[germanfgv]

The replay finished succesfully

[francescobrivio]

AlCaDB checklist for replays:

• PCL conditions upload - OK (CondDB link)
• Datasets produced - OK
  • ALCARECOs DAS link
  • ALCAPROMPTs DAS link
• DQM GUI upload - OK (see attached screenshot)

From the AlCaDB point of view we consider this replay validated.

[germanfgv]

I'm closing this PR. The configuration changes will be added to production here #4798