coverage of 2023 wfs in runTheMatrix, PR tests, IBs

[missirol]

This issue is meant to clarify (to me) if there are "enough" 2023 wfs implemented in runTheMatrix, and exercised in PR tests and IBs.

Taking PR tests in 13_1_X as an example, I think the "relvals" test during PR reviews runs 48 wfs, of which 2 use the 2023 HLT menu (139.001 for Data, 12434.0 for MC), while 5 wfs use the 2022 HLT menu (which will soon be replaced by a fake HLT menu).

I would expect that in 13_X_Y cycles the number of 2023 wfs we test should increase (e.g. more 2023 wfs than 2022 wfs).

• Right now, I think we have less wfs testing the HLT menu used for data-taking this year [1], compared to last year [2].

• Regarding what we are running IBs, I'm not sure, but I suspect a similar argument applies [*].

• What do @cms-sw/pdmv-l2 and others think?

FYI: @cms-sw/hlt-l2 @silviodonato @fwyzard

[1] "this year" meaning wfs in 13_X_Y testing the 2023 HLT menu.
[2] "last year" meaning wfs in 12_4_Y testing the 2022 HLT menu.

[*] If I check the IB tests in CMSSW_13_1 2023-04-25-2300 (el8_amd64_gcc11), I see 1018 RelVal wfs. I don't know how these are chosen by the following returns exactly 1018 wfs.

runTheMatrix.py -nel all -w standard,highstats,pileup,generator,extendedgen,production,ged,2017,2016,machine,premix,nano,2026

That command returns 42 MC wfs using the 2022 HLT menu, and only 6 MC wfs using the 2023 HLT menu. For data, it looks okay: there are 0 wfs using the 2022 HLT menu, and 60 wfs using the 2023 HLT menu.

[cmsbuild]

A new Issue was created by @missirol Marino Missiroli.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[fwyzard]

By the way, from my point of view we could drop the 2018 GPU workflows from the IB and PR tests.

[makortel]

assign pdmv

[cmsbuild]

New categories assigned: pdmv

@bbilin,@sunilUIET,@kskovpen you have been requested to review this Pull request/Issue and eventually sign? Thanks

[missirol]

By the way, from my point of view we could drop the 2018 GPU workflows from the IB and PR tests.

@fwyzard , does missirol@14ce88d look like what you meant, or not ?

[fwyzard]

Yes, it looks good (without actually running it) - thanks.

[sunilUIET]

Sorry for the delayed response. I hope my comment is still applicable :) I agree that there should be increasing number of 2023 WFs testing. But I am not sure how it goes in PR testing etc. One problem we may face in PR testing with 2023 of finding input for PU wfs and other similar WFs. PdmV still need to validate 2023 WFs against 2022 WFs before making 2023 WFs default in release validation. Hopeful very soon.

[silviodonato]

Hi @sunilUIET @cms-sw/pdmv-l2 ,
do you need something from our side (TSG)?
The pp data taking of 2023 has finished, so I think it is pointless still running 2022 HLT menu in 5 wfs and 2023 in only 2 wfs.

[sunilUIET]

Hi @silviodonato

  We haven't explicitly removed the 2022 WFs from runTheMarix. We will check for the possibility of the addition of other 2023 WFs such as PU and fastsim on the lines 2022 WF.  If you any specific WF(s) in mind, please let us know

https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/scripts/runTheMatrix.py#L94-L99

[silviodonato]

Hi @sunilUIET ,
you can find here the list of wfs of the limited matrix and of the full matrix.

In particular we have:

Limited matrix:

• 5 wfs with HLT:@relval2022
• 2 wfs with HLT:@relval2023

Full matrix:

• 107 wfs with HLT:@relval2022
• 55 wfs with HLT:@relval2023

We would like to have more workflows running HLT:@relval2023 then HLT:@relval2022.
Basically we would like that at least all workflows runnning HLT:@relval2022 have the equivalent version running HLT:@relval2023

[sunilUIET]

Hi @silviodonato

As already mentioned, we will work on updating the limited matrix.
We are not missing WFs but WFs with different scenarios such as postEE and design. I don't think adding postEE WFs will be useful. As far as design is concerned, that can be considered if useful.

https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py#L2577-L2631

[sunilUIET]

Hi @silviodonato

I created a PR to update WFs in limited matrix #42588
Please let me know if you have an opinion to add any other WF here.

[Martin-Grunewald]

Just to comment that since long, HLT is not running with/on FastSim.

[mmusich]

A clear case of missing test workflows for 2023 is the failures uncovered at #43300 (comment).

@cms-sw/pdmv-l2 can we think about adding a couple of 2023 workflows in the short matrix (run in PR tests) and remove the 2021 pilot beam (that at this point I think is of very limited interest)?

[AdrianoDee]

A clear case of missing test workflows for 2023 is the failures uncovered at #43300 (comment).

@cms-sw/pdmv-l2 can we think about adding a couple of 2023 workflows in the short matrix (run in PR tests) and remove the 2021 pilot beam (that at this point I think is of very limited interest)?

Sure, will have a look during the week.

[AdrianoDee]

+pdmv

[cmsbuild]

cms-bot internal usage

[cmsbuild]

This issue is fully signed and ready to be closed.

[AdrianoDee]

@cmsbuild please close