Allow multiple "--command" in runTheMatrix

[srimanob]

PR description:

As discussed in #34599 (comment)
This PR considers --command as a list, and join them together. This allows multiple --command(s) to be used in the PR test.

PR validation:

runTheMatrix.py --what upgrade -l 34634.0 --wm init --command '--customise Validation/Performance/TimeMemoryJobReport.customiseWithTimeMemoryJobReport' --command '--procModifiers fineCalo'
gives the expected output.

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not a backport, and no need of backport.

[perrotta]

@smuzaffar

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34722/24361

• This PR adds an extra 24KB to repository

[cmsbuild]

A new Pull Request was created by @srimanob (Phat Srimanobhas) for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)

@jordan-martins, @chayanit, @bbilin, @wajidalikhan, @kpedro88, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

[smuzaffar]

@srimanob , by default opt.command is None and the above will fail if --command option is not used. I would suggest to revert this change and instead add the following

    if opt.command: opt.command = ' '.join(opt.command)

after https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/scripts/runTheMatrix.py#L307

[srimanob]

Thanks @smuzaffar
I've updated the PR.

[srimanob]

test parameters:

• relvals_opt = --command '--customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_3000'

[srimanob]

test parameters:

• relvals_opt = --command '--procModifiers fineCalo'

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34722/24363

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #34722 was updated. @jordan-martins, @chayanit, @bbilin, @wajidalikhan, @kpedro88, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please check and sign again.

[smuzaffar]

@srimanob , / was not recognized by bot that is why #34722 (comment) was rejected. I have updated bot to recognize / too ( cms-sw/cms-bot#1604 ) . Feel free to try relvals_opt = --command '--customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_3000' now

[srimanob]

test parameters:

[srimanob]

test parameters:

• relvals_opt = --command '--customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_3000'

[srimanob]

@cmsbuild please test

Thanks @smuzaffar

[cmsbuild]

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd036a/17501/summary.html
COMMIT: 072fcb3
CMSSW: CMSSW_12_1_X_2021-08-03-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34722/17501/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

The relvals timed out after 4 hours.

[srimanob]

Hi @smuzaffar
How can I spot the fail relvals without going through them one by one? Thanks.

[smuzaffar]

@srimanob , I case of timeout runTheMatrix.py do not generate the summary so yes it is hard to findout which one fail. I look at the logs and it is workflow 140.56 which started at 9h26 and did not finish in 4 hours.

[srimanob]

Thanks @smuzaffar
140.56 is the HI real data, we should not care about it with the modifier. Since it applies to all workflows during PR test, we may need to use commands with does not effect much on all of them.

[smuzaffar]

@srimanob , this PR itself is good to go in. This shows that we can now run PR tests with extra command "option". If needed, you can reset the test parameters and re-run the tests to get test-approved label

[srimanob]

test parameters:

[srimanob]

Please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd036a/17535/summary.html
COMMIT: 072fcb3
CMSSW: CMSSW_12_1_X_2021-08-03-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34722/17535/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2998541
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[tvami]

@boudoul RelVal-INPUT issue is related to #34633
As far as I can see, there is not yet a solution coded in a PR...

Hi @boudoul in the meantime the following PR was proposed to fix the time-out issue: #34779

[srimanob]

+Upgrade

[chayanit]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy, @perrotta (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1