Add warning if WF name will be too long for submission #29621

srimanob · 2020-05-03T11:11:27Z

PR description:

The character length of computing workflow is limited to 100 characters. The workflow name composes of 3 main parts:

username
RequestString
data/time of submission; This part is added during the submission, max. characters are 19, e.g. _200425_064221_5181.

This PR is to check if len(username_RequestString) is longer than 81 characters or not, then print the warning at the end when --wm init is used.

The goal is not to break the --wm init process, as one may need to just dump config for testing using any release (including IB). This PR just provides the warning in case someone would like to inject workflows later. Without this test, the injection will break when --wm force is used. However, some of the workflows will be submitted if they come before problematic workflows.

PR validation:

Test with 11_1_0_pre6:
runTheMatrix.py --what standard -l 136.886,136.7611,136.756 -t 8 -b 'DQM8CoresData' --label 'DQM8CoresData1234567890' --wm init

*** FIX NEEDED BEFORE INJECTION: 2 candidate workflows have too long name (>81 characters) ***
srimanob_RVCMSSW_11_1_0_pre6RunJetHT2016E_reminiaod__DQM8CoresData1234567890_RelVal_2016Ermaod (94 characters) 
srimanob_RVCMSSW_11_1_0_pre6RunZeroBias2016D__DQM8CoresData1234567890_RelVal_2016D (82 characters)

workflow 136.866 is OK in this test,

srimanob_RVCMSSW_11_1_0_pre6RunEGamma2018D__DQM8CoresData1234567890_RelVal_2018D (80 characters long)

During injection, 136.866 will pass the injection, then break when 136.7611 injection is done.

if this PR is a backport please specify the original PR and why you need to backport that PR:

No need to backport.

cmsbuild · 2020-05-03T11:11:52Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-05-03T11:16:28Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29621/14971

This PR adds an extra 20KB to repository

cmsbuild · 2020-05-03T11:16:52Z

A new Pull Request was created by @srimanob (Phat Srimanobhas) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@chayanit, @cmsbuild, @wajidalikhan, @pgunnell, @kpedro88 can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

chayanit · 2020-05-04T05:51:17Z

please test

cmsbuild · 2020-05-04T05:51:35Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5965/console Started: 2020/05/04 07:52

cmsbuild · 2020-05-04T07:01:02Z

+1
Tested at: fc805c6
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bc6d8d/5965/summary.html
CMSSW: CMSSW_11_1_X_2020-05-03-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-05-04T07:01:05Z

Comparison job queued.

cmsbuild · 2020-05-04T08:34:28Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bc6d8d/5965/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2696239
DQMHistoTests: Total failures: 48
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2695872
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

Configuration/PyReleaseValidation/python/MatrixInjector.py

kpedro88 · 2020-05-04T14:45:36Z

Configuration/PyReleaseValidation/python/MatrixInjector.py

-
-
+        if self.testMode and self.countLongWFName>0:
+            print("\n*** FIX NEEDED BEFORE INJECTION: "+str(self.countLongWFName)+" candidate workflows have too long name (>81 characters) ***")


raise an exception?

Fix the text. This is for warning only that the workflows can't be submitted later.

Is there a situation where we want to allow those long workflow names? If not, I still think an exception would be better. That way, any PR creating an unnecessarily long workflow name must be fixed.

As mentioned in the PR description, if someone would like to use "--wm init" to dump the configuration to run local-test only, we should not break it. This warning is targeted for people (like PdmV) who will submit relvals later, using "--wm submit, force" after "--wm init".

Configuration/PyReleaseValidation/python/MatrixInjector.py

cmsbuild · 2020-05-04T15:34:26Z

Pull request #29621 was updated. @chayanit, @cmsbuild, @wajidalikhan, @pgunnell, @kpedro88 can you please check and sign again.

chayanit · 2020-05-04T15:36:16Z

please test

cmsbuild · 2020-05-04T15:36:42Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5976/console Started: 2020/05/04 17:38

cmsbuild · 2020-05-04T16:49:15Z

+1
Tested at: 0236929
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bc6d8d/5976/summary.html
CMSSW: CMSSW_11_1_X_2020-05-04-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-05-04T16:49:19Z

Comparison job queued.

cmsbuild · 2020-05-04T18:19:29Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bc6d8d/5976/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2696239
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2695919
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

silviodonato · 2020-05-05T13:30:04Z

@kpedro88 does the last version of the code address your comments?
@chayanit @pgunnell @wajidalikhan do you have any comments?

chayanit · 2020-05-05T13:53:28Z

+1

kpedro88 · 2020-05-05T14:03:52Z

+upgrade

cmsbuild · 2020-05-05T14:04:15Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-05-05T14:16:30Z

+1

Add warning if WF name (without date/time from computing) is too long

fc805c6

cmsbuild added this to the CMSSW_11_1_X milestone May 3, 2020

cmsbuild added code-checks-pending comparison-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels May 3, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels May 3, 2020

cmsbuild added tests-started and removed tests-pending labels May 4, 2020

cmsbuild added tests-approved and removed tests-started labels May 4, 2020

cmsbuild added comparison-available and removed comparison-pending labels May 4, 2020

kpedro88 reviewed May 4, 2020

View reviewed changes

Update to list for long workflow name counting

0236929

cmsbuild removed code-checks-approved comparison-available tests-approved labels May 4, 2020

cmsbuild added tests-started and removed tests-pending labels May 4, 2020

cmsbuild added tests-approved and removed tests-started labels May 4, 2020

cmsbuild added comparison-available and removed comparison-pending labels May 4, 2020

cmsbuild added pdmv-approved and removed pdmv-pending labels May 5, 2020

cmsbuild added fully-signed upgrade-approved and removed pending-signatures upgrade-pending labels May 5, 2020

cmsbuild added orp-approved and removed orp-pending labels May 5, 2020

cmsbuild merged commit 939f5ab into cms-sw:master May 5, 2020

This was referenced May 5, 2020

Clean up BuildFiles under RecoLocalTracker/ #29705

Merged

Clean up BuildFiles under RecoTracker/ #29716

Merged

Clean up BuildFiles under L1Trigger/ #29685

Merged

Clean up BuildFiles under Geometry/ #29675

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add warning if WF name will be too long for submission #29621

Add warning if WF name will be too long for submission #29621

srimanob commented May 3, 2020 •

edited

Loading

cmsbuild commented May 3, 2020

cmsbuild commented May 3, 2020

cmsbuild commented May 3, 2020

chayanit commented May 4, 2020

cmsbuild commented May 4, 2020 •

edited

Loading

cmsbuild commented May 4, 2020

cmsbuild commented May 4, 2020

cmsbuild commented May 4, 2020

kpedro88 May 4, 2020

srimanob May 4, 2020

kpedro88 May 4, 2020

srimanob May 4, 2020 •

edited

Loading

cmsbuild commented May 4, 2020

chayanit commented May 4, 2020

cmsbuild commented May 4, 2020 •

edited

Loading

cmsbuild commented May 4, 2020

cmsbuild commented May 4, 2020

cmsbuild commented May 4, 2020

silviodonato commented May 5, 2020

chayanit commented May 5, 2020

kpedro88 commented May 5, 2020

cmsbuild commented May 5, 2020

silviodonato commented May 5, 2020



		if self.testMode and self.countLongWFName>0:
		print("\n* FIX NEEDED BEFORE INJECTION: "+str(self.countLongWFName)+" candidate workflows have too long name (>81 characters) *")

Add warning if WF name will be too long for submission #29621

Add warning if WF name will be too long for submission #29621

Conversation

srimanob commented May 3, 2020 • edited Loading

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented May 3, 2020

cmsbuild commented May 3, 2020

cmsbuild commented May 3, 2020

chayanit commented May 4, 2020

cmsbuild commented May 4, 2020 • edited Loading

cmsbuild commented May 4, 2020

cmsbuild commented May 4, 2020

cmsbuild commented May 4, 2020

kpedro88 May 4, 2020

Choose a reason for hiding this comment

srimanob May 4, 2020

Choose a reason for hiding this comment

kpedro88 May 4, 2020

Choose a reason for hiding this comment

srimanob May 4, 2020 • edited Loading

Choose a reason for hiding this comment

cmsbuild commented May 4, 2020

chayanit commented May 4, 2020

cmsbuild commented May 4, 2020 • edited Loading

cmsbuild commented May 4, 2020

cmsbuild commented May 4, 2020

cmsbuild commented May 4, 2020

silviodonato commented May 5, 2020

chayanit commented May 5, 2020

kpedro88 commented May 5, 2020

cmsbuild commented May 5, 2020

silviodonato commented May 5, 2020

srimanob commented May 3, 2020 •

edited

Loading

cmsbuild commented May 4, 2020 •

edited

Loading

srimanob May 4, 2020 •

edited

Loading

cmsbuild commented May 4, 2020 •

edited

Loading