Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dropping phedex args #11181

Merged
merged 2 commits into from
Aug 30, 2022
Merged

Dropping phedex args #11181

merged 2 commits into from
Aug 30, 2022

Conversation

jhonatanamado
Copy link
Contributor

@jhonatanamado jhonatanamado commented Jun 13, 2022

Fixes #11145 and partially #9639 (Only T0 system)

Status

Tested with a replay

Description

Changes in several wfs arguments, some DAOs and RucioInjector to completely drop most of PhEDEx arguments in dataset subscriptions and use some of those arguments as an additional information for Rucio subscriptions
To wrap-up the following modifications are included in this PR:

  • Delete from the databases the following entries. move, phedex_group
  • Delete from the subscriptionInfo and wmspec CustodialSubType , NonCustodialSubType , CustodialGroup ,NonCustodialGroup and AutoApproveSubscriptionSites
  • Map the priority from the subscriptionInfo to priority for Rucio Container rules

Is it backward compatible (if not, which system it affects?)

No. Database schema change with this dev

Related PRs

T0#4679

External dependencies / deployment changes

<Does it require deployment changes? Does it rely on third-party libraries?>
No

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 13 new failures
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 56 warnings and errors that must be fixed
    • 12 warnings
    • 558 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 115 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13310/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor

test this please

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 12 new failures
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 56 warnings and errors that must be fixed
    • 12 warnings
    • 558 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 115 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13340/artifact/artifacts/PullRequestReport.html

@jhonatanamado jhonatanamado force-pushed the dropping_PhEDEx_args branch from 96ed16c to 8aace28 Compare June 15, 2022 16:16
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 12 warnings
    • 596 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 145 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13341/artifact/artifacts/PullRequestReport.html

@jhonatanamado jhonatanamado force-pushed the dropping_PhEDEx_args branch from 8aace28 to 4cc7319 Compare June 16, 2022 15:01
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 128 warnings and errors that must be fixed
    • 28 warnings
    • 1153 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 545 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13343/artifact/artifacts/PullRequestReport.html

@jhonatanamado jhonatanamado force-pushed the dropping_PhEDEx_args branch 2 times, most recently from c629255 to aeb77d4 Compare June 17, 2022 05:05
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 14 new failures
    • 2 tests no longer failing
    • 4 changes in unstable tests
  • Python3 Pylint check: failed
    • 128 warnings and errors that must be fixed
    • 28 warnings
    • 1153 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 545 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13351/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 5 tests no longer failing
    • 5 changes in unstable tests
  • Python3 Pylint check: failed
    • 77 warnings and errors that must be fixed
    • 12 warnings
    • 596 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 145 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13352/artifact/artifacts/PullRequestReport.html

@jhonatanamado jhonatanamado force-pushed the dropping_PhEDEx_args branch from aeb77d4 to 86a0925 Compare June 17, 2022 05:51
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 3 tests no longer failing
    • 5 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 12 warnings
    • 596 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 145 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13353/artifact/artifacts/PullRequestReport.html

@jhonatanamado jhonatanamado deleted the dropping_PhEDEx_args branch June 17, 2022 07:27
@jhonatanamado jhonatanamado restored the dropping_PhEDEx_args branch June 17, 2022 07:34
@jhonatanamado jhonatanamado reopened this Jun 17, 2022
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 3 tests no longer failing
    • 4 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 12 warnings
    • 596 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 145 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13354/artifact/artifacts/PullRequestReport.html

@jhonatanamado jhonatanamado force-pushed the dropping_PhEDEx_args branch from 86a0925 to 7e349c2 Compare June 17, 2022 13:07
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 14 tests no longer failing
    • 7 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 620 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 146 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13355/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 2 tests deleted
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 624 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 154 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13500/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 12 new failures
    • 2 tests deleted
    • 1 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 624 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 154 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13501/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 1 tests deleted
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 625 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 153 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13502/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 2 tests deleted
    • 1 tests no longer failing
    • 4 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 624 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 153 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13503/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 2 tests deleted
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 625 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 154 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13523/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 2 tests deleted
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 623 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 154 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13528/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 3 new failures
    • 2 tests deleted
    • 1 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 623 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 154 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13529/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 2 tests deleted
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 623 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 154 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13530/artifact/artifacts/PullRequestReport.html

@jhonatanamado
Copy link
Contributor Author

test this please

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 2 tests deleted
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 623 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 154 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13538/artifact/artifacts/PullRequestReport.html

@jhonatanamado
Copy link
Contributor Author

Hi @amaltaro , Alan seems that WMCore_t.WorkQueue_t.WorkQueue_t. is not working with Jenkins. I run the same test and this is the output. Any ideas?
Meanwhile I will move to update the ReqMgr templates.

.
----------------------------------------------------------------------
Ran 35 tests in 998.198s

OK
Exception ignored in: <function MockPlugin.__del__ at 0x7fe5432cf280>
Traceback (most recent call last):
  File "/home/dmwm/wmcore_unittest/WMCore/src/python/WMCore/BossAir/Plugins/MockPlugin.py", line 268, in __del__
  File "/home/dmwm/wmcore_unittest/WMCore/src/python/WMCore/BossAir/Plugins/MockPlugin.py", line 248, in close
AttributeError: 'NoneType' object has no attribute 'debug'

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 tests deleted
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 76 warnings and errors that must be fixed
    • 18 warnings
    • 623 comments to review
  • Pylint py3k check: failed
    • 3 errors and warnings that should be fixed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 154 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13541/artifact/artifacts/PullRequestReport.html

@jhonatanamado jhonatanamado requested a review from amaltaro August 24, 2022 06:58
@amaltaro
Copy link
Contributor

@jhonatanamado as we quickly discussed this morning, it turns out that we have already been running the oracle jenkins with this patch in.
I fear I am not able to say whether it's failing new unit tests, but it's close to the number of failures we have in our standard/MySQL jenkins unit tests (around 32 failures in MySQL vs 54 in Oracle, but I know some were already failing in Oracle).
I am going to review it a bit later.

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhonatanamado Jhonatan, these changes are looking great for me. I left a couple of comments along the code though.
In addition to that, could you please squash all these commits into only 2 commits:

  1. for src/* changes
  2. for test/* changes.

If you are unsure about this step, please let me know because we do not want to lose these changes ;)

src/python/WMCore/WMSpec/WMWorkloadTools.py Outdated Show resolved Hide resolved
@jhonatanamado
Copy link
Contributor Author

@amaltaro , Thanks Alan. What about the modifications on the JSON templates, Should those be squashed with src/* changes?

@amaltaro
Copy link
Contributor

@amaltaro , Thanks Alan. What about the modifications on the JSON templates, Should those be squashed with src/* changes?

@jhonatanamado no, those changes go together with unit tests (2nd commit)

Dropping PhEDEx args_StdBase
Dropping PhEDEx args_WMTask
Dropping PhEDEx args_WMWorkload


Dropping PhEDEx DAOs

Dropping phedex MySQL_Create
Dropping phedex MySQL_NewSubscription
Dropping phedex Oracle_Create
Dropping phedex Oracle_NewSubscription


Dropping PhEDEx Rucio Component

Dropping phedex UnsubscribedDts
Dropping phedex RucioInjectorPoller
Assign RUCIO_SUBSCRIPTION_PRIORITIES for containerRules 

Add RUCIO_RULES_PRIORITY

Add dict to map PhEDEx priority to Rucio priority

Change RUCIO_RULES_PRIORITY  from Rucio to RucioUtils


Add RUCIO_RULES_PRIORITY to RucioInjectorPoller


Delete PhEDEx component


Drop for loop from custodial


Change to RUCIO_RULES_PRIORITY


Dropping AutoApproveSubscriptionSites


Delete auto_approve-AutoApproveSites NewSubscriptions


Delete autoApproveSites from WMTask


Deletete AutoApproval from WMWorkload


Delete autoapprove from Oracle and create.py


Delete autoapprove from RucioInjector-GetUnsubscr-Dtsets


Delete validatePhEDExSubscription method
Dropping phedex StepChain
Dropping phedex WMTask
Dropping phedex WMWorkoad
Dropping phedex WMBSHelper

Delete PhEDEx component unit test


Delete AutoApproval from unit-test


Delete testValidatePhEDExSubscription from WMWorkload_t 


Updating ReqMgr json templates
@amaltaro
Copy link
Contributor

amaltaro commented Aug 30, 2022

Tests didn't come back - jenkins seem to be broken - but last unit tests were successful and the latest changes should have been safe enough. Merging it to get a new release out and deployed in testbed.
Thank you very much, Jhonatan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Repurpose Tier0 workflow output data placement to deal with rucio rules.
4 participants