Fix site-local-config.xml parsing when subsite tag has no element #39727

nhduongvn · 2022-10-13T16:34:43Z

PR description:

This PR fixes site-local-config.xml parsing when subsite tag has no element:
<site name=...>
<subsite name=.../>
...
<site/>
This PR is related to #37278 where subsite tag is extracted to locate storage description, storage.json. The extraction here uses subsite tag containing all elements (<data-access>, <local-stage-out>....).

The symbolic links in path to the storage.json is also resolved using filesystem::path::canonical

PR validation:

All tests in FWCore/Services/test are performed (scram b runtests)

cmsbuild · 2022-10-13T16:35:05Z

@nhduongvn, CMSSW_12_6_X branch is closed for direct updates. cms-bot is going to move this PR to master branch.
In future, please use cmssw master branch to submit your changes.

cmsbuild · 2022-10-13T16:42:41Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39727/32566

This PR adds an extra 20KB to repository

cmsbuild · 2022-10-13T16:43:10Z

A new Pull Request was created by @nhduongvn for master.

It involves the following packages:

FWCore/Services (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@makortel, @missirol, @wddgit, @fwyzard this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

makortel · 2022-10-13T17:04:11Z

@cmsbuild, please test

cmsbuild · 2022-10-13T20:34:06Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2b4b57/28242/summary.html
COMMIT: efe8149
CMSSW: CMSSW_12_6_X_2022-10-13-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39727/28242/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-2b4b57/20834.0_TTbar_14TeV+2026D88+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-2b4b57/20834.75_TTbar_14TeV+2026D88_HLT75e33+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HLT75e33+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-2b4b57/20896.0_CloseByPGun_CE_E_Front_120um+2026D88+CE_E_Front_120um_GenSimHLBeamSpotHGCALCloseBy+DigiTrigger+RecoGlobal+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-2b4b57/20900.0_CloseByPGun_CE_H_Coarse_Scint+2026D88+CE_H_Coarse_Scint_GenSimHLBeamSpotHGCALCloseBy+DigiTrigger+RecoGlobal+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-2b4b57/21034.999_TTbar_14TeV+2026D88PU_PMXS1S2PR+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-2b4b57/23234.0_TTbar_14TeV+2026D94+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3392309
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3392287
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 49 edm output root files, 49 DQM output files
TriggerResults: no differences found

makortel · 2022-10-14T15:06:57Z

FWCore/Services/src/SiteLocalConfigService.cc

-          filename_storage = siteconfig_path + "/../../" + aDataCatalog.storageSite + "/storage.json";
+          filename_storage = siteconfig_path + "/../../" + aDataCatalog.storageSite;
+      }
+      filename_storage = filename_storage.string() + "/storage.json";


How about

Suggested change

filename_storage = filename_storage.string() + "/storage.json";

filename_storage /= "storage.json";

?

yes, it is better

makortel · 2022-10-14T15:07:53Z

FWCore/Services/src/SiteLocalConfigService.cc

+        cms::Exception ex("SiteLocalConfigService");
+        ex << "Fail to convert path to the storage description, " << filename_storage.string()
+           << ", to the canonical absolute path"
+           << ". Path exists?";


Would it be useful to include e.what() in the exception message?

I see e.what() just says "std::exception" when path does not exist, which try catches. I see no benefit of including e.what()

makortel · 2022-10-14T15:13:14Z

FWCore/Services/src/SiteLocalConfigService.cc

@@ -341,7 +350,7 @@ namespace edm {
      // The Site Config has the following format
      // <site-local-config>
      // <site name="FNAL">
-      // <subsite name="FNAL_SUBSITE">
+      //   <subsite name="FNAL_SUBSITE"/>


(commenting in somewhat random place)

How about adding a check that the subsite element has no children?

Yes, we can add extra guard for wrong format of site-local-config.xml

cmsbuild · 2022-10-17T17:43:07Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39727/32602

This PR adds an extra 16KB to repository

cmsbuild · 2022-10-17T17:43:33Z

Pull request #39727 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please check and sign again.

makortel · 2022-10-17T18:04:32Z

@cmsbuild, please test

cmsbuild · 2022-10-18T18:26:06Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39727/32617

This PR adds an extra 8KB to repository

cmsbuild · 2022-10-18T18:26:31Z

Pull request #39727 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please check and sign again.

makortel · 2022-10-18T19:05:00Z

@cmsbuild, please test

cmsbuild · 2022-10-18T22:17:11Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2b4b57/28343/summary.html
COMMIT: 9202779
CMSSW: CMSSW_12_6_X_2022-10-18-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39727/28343/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
Reco comparison had 6 failed jobs
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3391158
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3391130
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 201 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

makortel · 2022-10-19T06:16:48Z

+1

cmsbuild · 2022-10-19T06:17:13Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-10-19T06:45:11Z

+1

cmsbuild changed the base branch from CMSSW_12_6_X to master October 13, 2022 16:35

cmsbuild added this to the CMSSW_12_6_X milestone Oct 13, 2022

cmsbuild added code-checks-pending core-pending orp-pending pending-signatures tests-pending labels Oct 13, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 13, 2022

cmsbuild added tests-started and removed tests-pending labels Oct 13, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 13, 2022

makortel reviewed Oct 14, 2022

View reviewed changes

nhduongvn force-pushed the fix-site-local-config-subsite-tag-bug branch from efe8149 to 848f930 Compare October 17, 2022 17:35

cmsbuild added code-checks-pending tests-pending and removed tests-approved code-checks-approved labels Oct 17, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 17, 2022

cmsbuild removed the tests-pending label Oct 17, 2022

fix site-local-config.xml parsing when subsite tag has no children

9202779

nhduongvn force-pushed the fix-site-local-config-subsite-tag-bug branch from 848f930 to 9202779 Compare October 18, 2022 18:18

cmsbuild added code-checks-pending tests-pending and removed tests-approved code-checks-approved labels Oct 18, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 18, 2022

cmsbuild added tests-started and removed tests-pending labels Oct 18, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 18, 2022

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Oct 19, 2022

makortel mentioned this pull request Oct 19, 2022

Add new data catalogs from Rucio storage description (RucioCatalog) and use it by default instead of trivial data catalogs (TrivialCatalog) #37278

Merged

cmsbuild added orp-approved and removed orp-pending labels Oct 19, 2022

cmsbuild merged commit 849e200 into cms-sw:master Oct 19, 2022

nhduongvn mentioned this pull request Oct 21, 2022

DQM/Integration unit tests are failing in all releases but 12_6_X #39669

Closed

nhduongvn mentioned this pull request Jan 26, 2023

Backport codes for adding new data catalogs from Rucio storage description (RucioCatalog) and use it by default instead of trivial data catalogs (TrivialCatalog) #37278, and related bug fixes #40628

Closed

nhduongvn mentioned this pull request Apr 10, 2023

Backport codes for adding new data catalogs from Rucio storage description (RucioCatalog) and use it by default instead of trivial data catalogs (TrivialCatalog) #37278, and related bug fixes #41309

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix site-local-config.xml parsing when subsite tag has no element #39727

Fix site-local-config.xml parsing when subsite tag has no element #39727

nhduongvn commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

makortel commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

makortel Oct 14, 2022

nhduongvn Oct 14, 2022

makortel Oct 14, 2022

nhduongvn Oct 14, 2022 •

edited

Loading

makortel Oct 15, 2022

makortel Oct 14, 2022

nhduongvn Oct 14, 2022

cmsbuild commented Oct 17, 2022

cmsbuild commented Oct 17, 2022

makortel commented Oct 17, 2022

cmsbuild commented Oct 18, 2022

cmsbuild commented Oct 18, 2022

makortel commented Oct 18, 2022

cmsbuild commented Oct 18, 2022

makortel commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

perrotta commented Oct 19, 2022

	filename_storage = filename_storage.string() + "/storage.json";
	filename_storage /= "storage.json";

Fix site-local-config.xml parsing when subsite tag has no element #39727

Fix site-local-config.xml parsing when subsite tag has no element #39727

Conversation

nhduongvn commented Oct 13, 2022

PR description:

PR validation:

cmsbuild commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

makortel commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

Comparison Summary

makortel Oct 14, 2022

Choose a reason for hiding this comment

nhduongvn Oct 14, 2022

Choose a reason for hiding this comment

makortel Oct 14, 2022

Choose a reason for hiding this comment

nhduongvn Oct 14, 2022 • edited Loading

Choose a reason for hiding this comment

makortel Oct 15, 2022

Choose a reason for hiding this comment

makortel Oct 14, 2022

Choose a reason for hiding this comment

nhduongvn Oct 14, 2022

Choose a reason for hiding this comment

cmsbuild commented Oct 17, 2022

cmsbuild commented Oct 17, 2022

makortel commented Oct 17, 2022

cmsbuild commented Oct 18, 2022

cmsbuild commented Oct 18, 2022

makortel commented Oct 18, 2022

cmsbuild commented Oct 18, 2022

Comparison Summary

makortel commented Oct 19, 2022

cmsbuild commented Oct 19, 2022

perrotta commented Oct 19, 2022

nhduongvn Oct 14, 2022 •

edited

Loading