Fix fetching of datasets in relval wfs

[kskovpen]

PR description:

Following #36771, specifies explicitly Rucio URL to fetch datasets that are available at T2_CH_CERN. Temporary fix - can be removed when dasgoclient is fully fixed.

PR validation:

Tested on 4.76.

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not a backport.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36787/27927

• This PR adds an extra 16KB to repository

[cmsbuild]

A new Pull Request was created by @kskovpen for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)

@jordan-martins, @bbilin, @wajidalikhan, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @Martin-Grunewald, @missirol, @fabiocos, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[tvami]

@cmsbuild , please test

[tvami]

@cmsbuild , ping

[mmusich]

please test

[smuzaffar]

dasgoclinet has been updated to use RUCIO_URL=http://cms-rucio.cern.ch ( dmwm/das2go@cba4d3e ) if it is not defined. New dasgoclient will be available in tonight's IB. So this PR can be closed.

[tvami]

seems like the test is still failing, no?

[cmsbuild]

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ffd8ef/21958/summary.html
COMMIT: 448e58e
CMSSW: CMSSW_12_3_X_2022-01-23-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36787/21958/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @makortel Modernize most legacy EDModules in TopQuarkAnalysis package #36692
• @mmusich get rid of CMSDEPRECATED_X warnings from TrackPropagation #36757
• @civanch Fixed static analyzer warnings in SimG4Core/Notification and clean-up Alignment/LaserAlignmentSimulation #36737
• @jfernan2 Remove Event Interpretation (EI) from relval steps #36465

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ffd8ef/21958/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ffd8ef/21958/git-merge-result

RelVals-INPUT

• 4.764.76_ZMuSkim2012D+ZMuSkim2012D+HLTDSKIM2+RECODR1reHLT2+HARVESTDR1reHLT/step2_ZMuSkim2012D+ZMuSkim2012D+HLTDSKIM2+RECODR1reHLT2+HARVESTDR1reHLT.log

Comparison Summary

Summary:

• No significant changes to the logs found
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 1037 differences found in the comparisons
• DQMHistoTests: Total files compared: 43
• DQMHistoTests: Total histograms compared: 3449324
• DQMHistoTests: Total failures: 4539
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3444763
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 42 files compared)
• Checked 181 log files, 42 edm output root files, 43 DQM output files
• TriggerResults: no differences found

[kskovpen]

seems like the test is still failing, no?

From running the dasgoclient query locally:

(RUCIO_URL=http://cms-rucio.cern.ch dasgoclient --limit 0 --query 'file dataset=/SingleMu/Run2012D-ZMu-15Apr2014-v1/RAW-RECO run=208307 site=T2_CH_CERN' RUCIO_URL=http://cms-rucio.cern.ch dasgoclient --limit 0 --query 'file dataset=/SingleMu/Run2012D-ZMu-15Apr2014-v1/RAW-RECO run=207454 site=T2_CH_CERN') | Utilities/General/ibeos/ibeos-lfn-sort -u

as mentioned here:

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ffd8ef/21958/runTheMatrixINPUT-results/4.76_ZMuSkim2012D+ZMuSkim2012D+HLTDSKIM2+RECODR1reHLT2+HARVESTDR1reHLT/cmdLog

I get the following list:

/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/0A3C1F91-2ECD-E311-8E64-90E6BA442F3F.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/10FC6D15-31CD-E311-B9FF-90E6BA442F1A.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/125446BC-34CD-E311-AE2B-E0CB4EA0A931.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/205431CF-46CD-E311-8696-20CF305B04FB.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/2411DF9C-32CD-E311-91DD-E0CB4EA0A937.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/36A3529F-54CD-E311-878A-00261834B52B.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/420A02F8-47CD-E311-B9D4-E0CB4E29C504.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/4A3C4AF2-57CD-E311-9F4E-20CF3056170B.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/4C105176-56CD-E311-BA44-90E6BA442F43.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/5EA45770-50CD-E311-A739-0025907B4EF2.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/604629D5-48CD-E311-83A1-485B39800C2D.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/64D04C3C-2BCD-E311-8309-00259073E2F2.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/86AFCF5C-31CD-E311-9FAB-BCAEC50971F9.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/965DC07D-43CD-E311-A750-90E6BA442F0D.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/9E8532D7-33CD-E311-9B6F-E0CB4E553666.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/A27A4528-2CCD-E311-8CF9-0025907B4F9A.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/C08BA23E-2FCD-E311-92A0-00261834B5A7.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/C8D271CA-39CD-E311-AC6B-90E6BA442F32.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/CCF07524-46CD-E311-8608-E0CB4E1A11A4.root
/store/data/Run2012D/SingleMu/RAW-RECO/ZMu-15Apr2014-v1/00000/F4FE749C-2FCD-E311-B4CC-90E6BA442F3F.root

which contains files that are all present at T2_CH_CERN.

The wf in Jenkins, however, produces a different list, where there are files not present at CERN. Maybe someone knows if there are any additional das-related queries that are executed in Jenkins tests?

[kskovpen]

dasgoclinet has been updated to use RUCIO_URL=http://cms-rucio.cern.ch ( dmwm/das2go@cba4d3e ) if it is not defined. New dasgoclient will be available in tonight's IB. So this PR can be closed.

Sure, will close it.

@tvami , I guess we can try tomorrow again?

[tvami]

dasgoclinet has been updated to use RUCIO_URL=http://cms-rucio.cern.ch ( dmwm/das2go@cba4d3e ) if it is not defined. New dasgoclient will be available in tonight's IB. So this PR can be closed.

Sure, will close it.

@tvami , I guess we can try tomorrow again?

Yes, I hope it will be ok indeed!

[smuzaffar]

The wf in Jenkins, however, produces a different list, where there are files not present at CERN. Maybe someone knows if there are any additional das-related queries that are executed in Jenkins tests?

@kskovpen , to avoid das glitches during PR/IB tests we cache the dasgoclinet results and refresh these results every 3 days. I will force update the das cache reusults using new dasgoclient which should be available for next IB. Let hope new dasgiclient results do not break otherworkflows

[kskovpen]

The wf in Jenkins, however, produces a different list, where there are files not present at CERN. Maybe someone knows if there are any additional das-related queries that are executed in Jenkins tests?

@kskovpen , to avoid das glitches during PR/IB tests we cache the dasgoclinet results and refresh these results every 3 days. I will force update the das cache reusults using new dasgoclient which should be available for next IB. Let hope new dasgiclient results do not break otherworkflows

Thanks a lot for explanations!