[8_0_X] Support vector<string> data products in friendly class name #36580

makortel · 2021-12-23T02:38:11Z

PR description:

This PR backports

Allow shared_ptr and unique_ptr in top level data products #16221 (from 8_1_X)
FriendlyName: add "String" options for C++11 ABI #17139 (from 9_0_X)
The new gcc ABI for C++11 uses different namespace #17371 (from 9_0_X)

as a possible solution for problem described in https://its.cern.ch/jira/browse/CMSCOMPPR-22029. In short, there a 10_6_X release is used for GEN, SIM, and DIGI steps, and 8_0_33_UL is used for the HLT step. The SIM step adds a product std::vector<std::string> that the HLT step is unable to read because the friendly class name (that is part of the TTree branch name) is different. The friendly class name for vector<string> in 8_0_X is currently charcharstdchar_traitsstdbasic_strings, which can not be used for persistent data products because of the underscore.

An alternative solution would be to redesign the workflow to give only the RAW input to the HLT step, and using two-file solution in the RECO step to read in DIGI and HLT outputs. I'm making a backport anyway in case it would be useful (since it was rather easy to do).

PR validation:

Local test of HLT step of https://its.cern.ch/jira/browse/CMSCOMPPR-22029 succeeds. FWCore/Utilities unit tests pass.

Added special cases to the friendly name generator to handle std::shared_ptr and std::unique_ptr as items being held by collections which are used as data products. This avoids having an extra underscore be in friendly class name which is not allowed.

This is needed for C++11 (and above) ABI, which is available starting GCC 5. Signed-off-by: David Abdurachmanov <[email protected]>

The gcc ABI for C++11 internally uses the namespace std::__cxx11 for the standard library containers. The friendly class name generator now knows about that naming in order to keep the branch names the same as before the ABI change.

cmsbuild · 2021-12-23T02:38:35Z

A new Pull Request was created by @makortel (Matti Kortelainen) for CMSSW_8_0_X.

It involves the following packages:

FWCore/Utilities (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@felicepantaleo, @wddgit this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

makortel · 2021-12-23T02:39:15Z

@perrotta @qliphy I let @cms-sw/pdmv-l2 and @tvami to describe the urgency and if this would be the preferred solution.

makortel · 2021-12-23T02:39:33Z

@cmsbuild, please test

(although this will likely fail in absence of 8_0_X IBs)

makortel · 2021-12-23T02:41:15Z

cms/36580/unknown/release — Unable to find CMSSW IB for CMSSW_8_0_X.

Yup.

makortel · 2021-12-23T02:42:28Z

Note that if we really backport this, I'd consider backporting #36458 too.

tvami · 2021-12-23T15:10:44Z

I let @cms-sw/pdmv-l2 and @tvami to describe the urgency and if this would be the preferred solution.

I think this is a good solution! About the urgency, I guess we can come back to it in January (provided that all the other years work fine for the MC production, which I think they do)

re-enable 80X IBs , see cms-sw/cmssw#36580

qliphy · 2022-01-04T13:15:53Z

please test
80X IB now re-opened

cmsbuild · 2022-01-04T13:57:19Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8676f1/21526/summary.html
COMMIT: 583a2b2
CMSSW: CMSSW_8_0_X_2022-01-03-2300/slc6_amd64_gcc530
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36580/21526/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

10224.0

----- Begin Fatal Exception 04-Jan-2022 14:38:42 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
   [2] Calling RootFileSequenceBase::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root'
   Additional Info:
      [a] XrdCl::File::Open(name='root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] Unable to open file /eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root; No such file or directory
' (errno=3011, code=400). No additional data servers were found.
      [b] Last URL tried: root://eoscms.cern.ch:1094//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root?tried=
      [c] Problematic data server: eoscms.cern.ch:1094
      [d] Disabled source: eoscms.cern.ch:1094
      [e] Input file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root could not be opened.
Fallback Input file root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root also could not be opened.
Original exception info is above; fallback exception info is below.
      [f] XrdCl::File::Open(name='root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [g] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root?tried=
      [h] Problematic data server: cms-xrd-global.cern.ch:1094
      [i] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

makortel · 2022-01-05T16:00:47Z

I'd consider backporting #36458 too.

This backport is now done in #36632.

makortel · 2022-01-05T18:37:12Z

The dataset /RelValMinBias_13/CMSSW_8_0_0_patch1-80X_upgrade2017_design_v4_UPG17-v1/GEN-SIM appears to be deleted, but according to DAS (rucio) the specific file in question, /store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root, would still be on T0_CH_CERN_Tape. How should we proceed?

tvami · 2022-01-05T18:46:25Z

@smuzaffar can you please move that file to the IB EOS area? Thanks!

smuzaffar · 2022-01-11T16:37:28Z

/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root is available now in ib-eos cache area. I have restarted the relval test so see if it can find it

makortel · 2022-01-12T13:58:45Z

@smuzaffar I see the bot added tests-approved label, but the test results are not shown here?

cmsbuild · 2022-01-12T15:19:12Z

+1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8676f1/21526/summary.html
COMMIT: 583a2b2
CMSSW: CMSSW_8_0_X_2022-01-03-2300/slc6_amd64_gcc530
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36580/21526/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

10224.0

----- Begin Fatal Exception 04-Jan-2022 14:38:42 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
   [2] Calling RootFileSequenceBase::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root'
   Additional Info:
      [a] XrdCl::File::Open(name='root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] Unable to open file /eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root; No such file or directory
' (errno=3011, code=400). No additional data servers were found.
      [b] Last URL tried: root://eoscms.cern.ch:1094//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root?tried=
      [c] Problematic data server: eoscms.cern.ch:1094
      [d] Disabled source: eoscms.cern.ch:1094
      [e] Input file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root could not be opened.
Fallback Input file root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root also could not be opened.
Original exception info is above; fallback exception info is below.
      [f] XrdCl::File::Open(name='root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [g] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root?tried=
      [h] Problematic data server: cms-xrd-global.cern.ch:1094
      [i] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

Comparison Summary

There are some workflows for which there are errors in the baseline:
10224.0 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 14
DQMHistoTests: Total histograms compared: 1020390
DQMHistoTests: Total failures: 822
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 1019460
DQMHistoTests: Total skipped: 108
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -13 KiB( 13 files compared)
Checked 60 log files, 11 edm output root files, 14 DQM output files

smuzaffar · 2022-01-12T15:20:14Z

please test
looks like bit does not behave properly if only one of the tests were re-run. I have restarted the tests

cmsbuild · 2022-01-12T16:10:47Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8676f1/21680/summary.html
COMMIT: 583a2b2
CMSSW: CMSSW_8_0_X_2022-01-09-0000/slc6_amd64_gcc530
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36580/21680/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

10224.0

----- Begin Fatal Exception 12-Jan-2022 17:00:44 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
   [2] Calling RootFileSequenceBase::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root'
   Additional Info:
      [a] XrdCl::File::Open(name='root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] Unable to open file /eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root; No such file or directory
' (errno=3011, code=400). No additional data servers were found.
      [b] Last URL tried: root://eoscms.cern.ch:1094//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root?tried=
      [c] Problematic data server: eoscms.cern.ch:1094
      [d] Disabled source: eoscms.cern.ch:1094
      [e] Input file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root could not be opened.
Fallback Input file root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root also could not be opened.
Original exception info is above; fallback exception info is below.
      [f] XrdCl::File::Open(name='root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [g] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/F2B970E8-76DD-E511-BDB4-0025905B8610.root?tried=
      [h] Problematic data server: cms-xrd-global.cern.ch:1094
      [i] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

makortel · 2022-01-12T16:48:05Z

@cmsbuild, please test

We're getting bitten by randomness in the MixingModule constructor for the initial file open that checks the contents (or something).

cmsbuild · 2022-01-12T18:25:30Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8676f1/21682/summary.html
COMMIT: 583a2b2
CMSSW: CMSSW_8_0_X_2022-01-09-0000/slc6_amd64_gcc530
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36580/21682/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
10224.0 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 14
DQMHistoTests: Total histograms compared: 1020390
DQMHistoTests: Total failures: 821
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 1019461
DQMHistoTests: Total skipped: 108
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -13 KiB( 13 files compared)
Checked 60 log files, 11 edm output root files, 14 DQM output files

makortel · 2022-01-12T18:40:30Z

The tests look ok. Comparisons show differences in

4.53: HLT/Higgs/PhotonJet/trigvsnvtx (looks visually empty, perhaps NaNs?)
5.1: SimHitsValidationHcal/HcalHitsV(visually look like some technical issue)

but they look more like "technical issue", and this PR could not really have that kind of effect.

makortel · 2022-01-13T18:58:53Z

(just to remind that this PR was signed already)

qliphy · 2022-01-14T00:37:24Z

+1

Dr15Jones and others added 3 commits December 22, 2021 23:02

FriendlyName: add "String" options for C++11 ABI

334b638

This is needed for C++11 (and above) ABI, which is available starting GCC 5. Signed-off-by: David Abdurachmanov <[email protected]>

The new gcc ABI for C++11 uses different namespace

583a2b2

The gcc ABI for C++11 internally uses the namespace std::__cxx11 for the standard library containers. The friendly class name generator now knows about that naming in order to keep the branch names the same as before the ABI change.

cmsbuild added this to the CMSSW_8_0_X milestone Dec 23, 2021

cmsbuild added core-pending orp-pending pending-signatures tests-pending labels Dec 23, 2021

cmsbuild added tests-started tests-pending and removed tests-pending tests-started labels Dec 23, 2021

smuzaffar added a commit to cms-sw/cms-bot that referenced this pull request Jan 4, 2022

re-enable 80X IBs , see cms-sw/cmssw#36580

37fccbf

smuzaffar added a commit to cms-sw/cms-bot that referenced this pull request Jan 4, 2022

Merge pull request #1683 from cms-sw/smuzaffar-patch-2

72d10a8

re-enable 80X IBs , see cms-sw/cmssw#36580

cmsbuild added tests-started and removed tests-pending labels Jan 4, 2022

cmsbuild added tests-rejected and removed tests-started labels Jan 4, 2022

makortel mentioned this pull request Jan 5, 2022

[8_0_X] Protect storage accounting UDP messages from NaN, and Use StatisticsSenderService for all framework files #36632

Merged

cmsbuild added the tests-started label Jan 11, 2022

cmsbuild added tests-approved and removed tests-started labels Jan 11, 2022

cmsbuild added tests-started and removed tests-approved labels Jan 12, 2022

cmsbuild added tests-approved and removed tests-started labels Jan 12, 2022

cmsbuild added tests-started and removed tests-approved labels Jan 12, 2022

cmsbuild added tests-rejected and removed tests-started labels Jan 12, 2022

cmsbuild added tests-started and removed tests-rejected labels Jan 12, 2022

cmsbuild added tests-approved and removed tests-started labels Jan 12, 2022

cmsbuild added orp-approved and removed orp-pending labels Jan 14, 2022

cmsbuild merged commit a3b252c into cms-sw:CMSSW_8_0_X Jan 14, 2022

qliphy mentioned this pull request Jan 14, 2022

Build CMSSW_8_0_36_patch1 #36703

Closed

makortel deleted the backportFriendlyName branch January 14, 2022 02:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8_0_X] Support vector<string> data products in friendly class name #36580

[8_0_X] Support vector<string> data products in friendly class name #36580

makortel commented Dec 23, 2021

cmsbuild commented Dec 23, 2021

makortel commented Dec 23, 2021

makortel commented Dec 23, 2021

makortel commented Dec 23, 2021

makortel commented Dec 23, 2021

tvami commented Dec 23, 2021

qliphy commented Jan 4, 2022

cmsbuild commented Jan 4, 2022

makortel commented Jan 5, 2022

makortel commented Jan 5, 2022

tvami commented Jan 5, 2022

smuzaffar commented Jan 11, 2022

makortel commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

smuzaffar commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

makortel commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

makortel commented Jan 12, 2022

makortel commented Jan 13, 2022

qliphy commented Jan 14, 2022

[8_0_X] Support vector<string> data products in friendly class name #36580

[8_0_X] Support vector<string> data products in friendly class name #36580

Conversation

makortel commented Dec 23, 2021

PR description:

PR validation:

cmsbuild commented Dec 23, 2021

makortel commented Dec 23, 2021

makortel commented Dec 23, 2021

makortel commented Dec 23, 2021

makortel commented Dec 23, 2021

tvami commented Dec 23, 2021

qliphy commented Jan 4, 2022

cmsbuild commented Jan 4, 2022

RelVals

makortel commented Jan 5, 2022

makortel commented Jan 5, 2022

tvami commented Jan 5, 2022

smuzaffar commented Jan 11, 2022

makortel commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

RelVals

Comparison Summary

smuzaffar commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

RelVals

makortel commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

Comparison Summary

makortel commented Jan 12, 2022

makortel commented Jan 13, 2022

qliphy commented Jan 14, 2022