Backport codes for adding new data catalogs from Rucio storage description (RucioCatalog) and use it by default instead of trivial data catalogs (TrivialCatalog) #37278, and related bug fixes

[nhduongvn]

PR description:

Backport to 12_5_X:
#37278
#39314
#39727
#40961

[cmsbuild]

A new Pull Request was created by @nhduongvn for CMSSW_12_5_X.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)
• FWCore/Catalog (core)
• FWCore/Services (core)

@smuzaffar, @Dr15Jones, @makortel, @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET can you please review it and eventually sign? Thanks.
@makortel, @wddgit, @Martin-Grunewald, @missirol, @kpedro88, @fwyzard, @fabiocos, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[rappoccio]

please test

[rappoccio]

@nhduongvn please explain the rationale for a backport for this very old feature to be back ported to 12_5?

[makortel]

please explain the rationale for a backport for this very old feature to be back ported to 12_5?

The motivation was outlined in #37278 (comment) (quoting @stlammel)

Is there any need to backport this to earlier release cycles?

yes, it would be desirable to port this to the latest versions of all CMSSW_X releases so
storage.xml can be phased out (at some point).

[nhduongvn]

Yes, there was a plan to backport this "old" feature to previous releases (up to 12_5_X if I remember correctly). I have done it now so that the feature can be fully tested and following up fixes can be included. It looks to me things are OK now and I will continue to backport to other releases.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-73c960/31911/summary.html
COMMIT: 63ca0fd
CMSSW: CMSSW_12_5_X_2023-04-09-0000/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41309/31911/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 10 lines to the logs
• Reco comparison results: 6 differences found in the comparisons
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3724047
• DQMHistoTests: Total failures: 8
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3724017
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 216 log files, 167 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[makortel]

up to 12_5_X if I remember correctly

Actually these should go back to "all actively used release cycles". Defining this set of release cycles has been challenging in the past, partially because we tend to keep release cycles open in the IBs longer than they are actually being used (because figuring out what is actually being used is not that easy).

Maybe a reasonable starting point would be (in addition of this PR, and 12_6_X and 13_0_X that need only #40961)

• 12_4_X (2022 data taking release)
• 10_6_X (Run 2 ultra-legacy)
• 9_4_X (HLT in Run 2 UL)
• 8_0_X (HLT in Run 2 UL)

For the other release cycles (10_2_X, 10_3_X, 11_0_X, 11_1_X, 11_2_X, 11_3_X, 12_0_X, 12_1_X) we (@cms-sw/orp-l2 @cms-sw/pdmv-l2) should figure out if they still need to be kept open or could be closed. For those that need to be kept open (for, say, more than a year), the Rucio catalog should backported to them.

5_3_X (Run 1 legacy) we can probably leave out

[makortel]

Comparison differences in workflows 39496.0 and 39500.0 seem point to an old issue #39214

[makortel]

+core

The commits diffs are identical to the original PRs

[AdrianoDee]

+upgrade

Identical backport of #37278, #39314, #39727, #40961.

[sunilUIET]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_5_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_13_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[rappoccio]

+1