Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix error code returned by cmsRun on a FallbackFileOpenError #42249

Merged
merged 1 commit into from
Jul 24, 2023

Conversation

wddgit
Copy link
Contributor

@wddgit wddgit commented Jul 12, 2023

PR description:

Fix error code returned by cmsRun on a FallbackFileOpenError. There was a bug where it returns FileOpenError even in the fallback case.

There is more discussion and more information related to this bug in issue #42040. FYI @stlammel

Also added a unit test so this bug will not recur.

Also I noticed a case where a misconfigured storage.json file could cause a seg fault. I added a check for that case and now it throws an exception with an informative error message (remains fatal as before).

PR validation:

New unit test passes and existing unit tests pass. This should only affect the behavior after an exception or seg fault occurs. It should not affect the output of successful cmsRun processes.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-42249/36267

  • This PR adds an extra 24KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @wddgit (W. David Dagenhart) for master.

It involves the following packages:

  • FWCore/Catalog (core)
  • IOPool/Input (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@makortel, @missirol this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

@wddgit
Copy link
Contributor Author

wddgit commented Jul 12, 2023

please test

@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2a7d00/33667/summary.html
COMMIT: c41f603
CMSSW: CMSSW_13_2_X_2023-07-12-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/42249/33667/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test_cmsLHEtoEOSManager had ERRORS

RelVals

----- Begin Fatal Exception 12-Jul-2023 20:19:44 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2018D/JetHT/AOD/12Nov2019_UL2018-v4/110007/02F69212-64ED-6B4B-BAC4-D9C2D53C8C57.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2018D/JetHT/AOD/12Nov2019_UL2018-v4/110007/02F69212-64ED-6B4B-BAC4-D9C2D53C8C57.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2018D/JetHT/AOD/12Nov2019_UL2018-v4/110007/02F69212-64ED-6B4B-BAC4-D9C2D53C8C57.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2018D/JetHT/AOD/12Nov2019_UL2018-v4/110007/02F69212-64ED-6B4B-BAC4-D9C2D53C8C57.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2018D/JetHT/AOD/12Nov2019_UL2018-v4/110007/02F69212-64ED-6B4B-BAC4-D9C2D53C8C57.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.�
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2018D/JetHT/AOD/12Nov2019_UL2018-v4/110007/02F69212-64ED-6B4B-BAC4-D9C2D53C8C57.root?tried=+1213xrootd-cms-redir-int.cr.cnaf.infn.it,&xrdcl.requuid=79ff36ec-b40a-4224-bbe5-19194247bd06
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 12-Jul-2023 20:20:41 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.�
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root?tried=+1213llrxrd-redir.in2p3.fr,&xrdcl.requuid=a9dea61b-7b43-47f9-af3e-bad136940d65
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 12-Jul-2023 20:20:41 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2017C/DoubleEG/RAW/v1/000/301/998/00000/0073E73C-B68C-E711-9BF8-02163E019B6C.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2017C/DoubleEG/RAW/v1/000/301/998/00000/0073E73C-B68C-E711-9BF8-02163E019B6C.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2017C/DoubleEG/RAW/v1/000/301/998/00000/0073E73C-B68C-E711-9BF8-02163E019B6C.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2017C/DoubleEG/RAW/v1/000/301/998/00000/0073E73C-B68C-E711-9BF8-02163E019B6C.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2017C/DoubleEG/RAW/v1/000/301/998/00000/0073E73C-B68C-E711-9BF8-02163E019B6C.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.�
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2017C/DoubleEG/RAW/v1/000/301/998/00000/0073E73C-B68C-E711-9BF8-02163E019B6C.root?tried=+1213llrxrd-redir.in2p3.fr,&xrdcl.requuid=934e85e2-5cb8-494f-8352-b271d2311741
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------
Expand to see more relval errors ...

RelVals-INPUT

  • 4.424.42_RunMET2012A/step2_RunMET2012A.log
  • 4.374.37_ZMuSkim2011B/step2_ZMuSkim2011B.log
  • 4.174.17_RunMinBias2011A/step2_RunMinBias2011A.log
Expand to see more relval errors ...

AddOn Tests

----- Begin Fatal Exception 12-Jul-2023 20:12:18 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
      [c] Input file root://xrootd-cms.infn.it//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root', flags=0x10, permissions=0660) => error '[FATAL] Redirect limit has been reached' (errno=0, code=306). No additional data servers were found.
      [e] Last URL tried: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=&xrdcl.requuid=180babf4-1030-4b3b-96fb-6500f04ee5f4
      [f] Problematic data server: xrootd-cms.infn.it:1094
      [g] Disabled source: eoscms.cern.ch:1094
      [h] Disabled source: xrootd-cms.infn.it:1094
      [i] Disabled source: eoscmsgsi.cern.ch:1094
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 12-Jul-2023 19:59:42 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2012A/MuEG/RAW/v1/000/191/718/14932935-E289-E111-830C-5404A6388697.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2012A/MuEG/RAW/v1/000/191/718/14932935-E289-E111-830C-5404A6388697.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2012A/MuEG/RAW/v1/000/191/718/14932935-E289-E111-830C-5404A6388697.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2012A/MuEG/RAW/v1/000/191/718/14932935-E289-E111-830C-5404A6388697.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2012A/MuEG/RAW/v1/000/191/718/14932935-E289-E111-830C-5404A6388697.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.�
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2012A/MuEG/RAW/v1/000/191/718/14932935-E289-E111-830C-5404A6388697.root?tried=+1213xrootd-cms-redir-int.cr.cnaf.infn.it,&xrdcl.requuid=2957be88-72dc-41d7-88cb-5e3e22966235
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 12-Jul-2023 19:59:49 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling File::sysopen()
Exception Message:
Failed to open the file 'RelVal_Raw_Fake_DATA.root'
   Additional Info:
      [a] Input file file:RelVal_Raw_Fake_DATA.root could not be opened.
      [b] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------
Expand to see more addon errors ...

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-42249/36268

  • This PR adds an extra 24KB to repository

@cmsbuild
Copy link
Contributor

Pull request #42249 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please check and sign again.

@wddgit
Copy link
Contributor Author

wddgit commented Jul 13, 2023

please test

Try tests again. Maybe the failures are some kind of transient glitch unrelated to my code changes. At first glance I thought the test failures might be related to my changes because they had FileOpenError and FallbackFileOpen in the exception messages. But on closer examination, I can't see how my changes could cause all the errors before the exception. Maybe it is even confirmation the changes are working. For example:

12-Jul-2023 19:50:18 CEST  Initiating request to open file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 19:51:03.103170 +0200][Error  ][AsyncSock         ] [eoscms.cern.ch:1094.0] Socket error while handshaking: [ERROR] Socket timeout
[2023-07-12 19:51:03.103365 +0200][Error  ][PostMaster        ] [eoscms.cern.ch:1094] elapsed = 45, pConnectionWindow = 31 seconds.
[2023-07-12 19:51:48.157917 +0200][Error  ][AsyncSock         ] [eoscms.cern.ch:1094.0] Socket error while handshaking: [ERROR] Socket timeout
[2023-07-12 19:51:48.158015 +0200][Error  ][PostMaster        ] [eoscms.cern.ch:1094] elapsed = 45, pConnectionWindow = 31 seconds.
[2023-07-12 19:53:18.439997 +0200][Error  ][XRootD            ] [eoscms.cern.ch:1094] Unable to get the response to request kXR_prepare (flags: kXR_stage , priority: 0, paths: /eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root)
[2023-07-12 19:53:18.440078 +0200][Error  ][XRootD            ] [eoscms.cern.ch:1094] Unable to get the response to request kXR_query (code: kXR_Qconfig, arg length: 9)
[2023-07-12 19:53:33.441497 +0200][Error  ][XRootD            ] [eoscms.cern.ch:1094] Unable to get the response to request kXR_open (file: /eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat )
%MSG-w XrdAdaptorInternal:  file_open 12-Jul-2023 19:53:33 CEST pre-events
Failed to open file at URL root://eoscms.cern.ch:1094//eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?xrdcl.requuid=2d67fb07-5d9e-42ad-b257-dee1822cc167.
%MSG
[2023-07-12 19:56:33.507382 +0200][Error  ][XRootD            ] [eoscms.cern.ch:1094] Unable to get the response to request kXR_open (file: /eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat )
%MSG-w XrdAdaptorInternal:  file_open 12-Jul-2023 19:56:33 CEST pre-events
Failed to open file at URL root://eoscms.cern.ch:1094//eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=&xrdcl.requuid=50c6160a-1a75-48aa-b925-d9fd4a43dfed.
%MSG
12-Jul-2023 19:56:33 CEST  Initiating request to open file root://cms-xrd-global.cern.ch//eos/cms/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 19:59:48.544744 +0200][Error  ][XRootD            ] [eoscms.cern.ch:1094] Unable to get the response to request kXR_open (file: /eos/cms/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat )
%MSG-w XrdAdaptorInternal:  file_open 12-Jul-2023 19:59:48 CEST pre-events
Failed to open file at URL root://eoscms.cern.ch:1094//eos/cms/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?xrdcl.requuid=78717ede-8aac-4481-948b-a5f172df2d0c.
%MSG
[2023-07-12 20:01:49.572443 +0200][Error  ][PostMaster        ] [eoscmshlt-ip565.cern.ch:1098] Forcing error on disconnect: [ERROR] Operation interrupted.
[2023-07-12 20:01:50.573906 +0200][Error  ][PostMaster        ] [eoscmshlt.cern.ch:1094] Forcing error on disconnect: [ERROR] Operation interrupted.
[2023-07-12 20:02:43.580037 +0200][Error  ][XRootD            ] [eoscmsgsi.cern.ch:1094] Unable to get the response to request kXR_open (file: /eos/cms/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=, mode: 0460, flags: kXR_open_read kXR_async kXR_retstat )
%MSG-w XrdAdaptorInternal:  file_open 12-Jul-2023 20:02:43 CEST pre-events
Failed to open file at URL root://eoscmsgsi.cern.ch:1094//eos/cms/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=&xrdcl.requuid=dda285ef-7098-4257-be37-3227bf9ad2ba.
%MSG
[2023-07-12 20:05:03.829794 +0200][Error  ][PostMaster        ] [eoscmsgsi-ip556.cern.ch:1098] Forcing error on disconnect: [ERROR] Operation interrupted.
[2023-07-12 20:05:48.604595 +0200][Error  ][XRootD            ] [eoscms.cern.ch:1094] Unable to get the response to request kXR_open (file: /eos/cms/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat )
%MSG-w XrdAdaptorInternal:  file_open 12-Jul-2023 20:05:48 CEST pre-events
Failed to open file at URL root://eoscms.cern.ch:1094//eos/cms/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=&xrdcl.requuid=a13cf01f-acd6-4565-912e-a05c571b88c6.
%MSG
12-Jul-2023 20:05:48 CEST  Initiating request to open file root://xrootd-cms.infn.it//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:06:17.744314 +0200][Error  ][AsyncSock         ] [eoscms.cern.ch:1094.0] Socket error encountered: [ERROR] Socket error: connection reset by peer
[2023-07-12 20:08:12.662915 +0200][Error  ][AsyncSock         ] [eoscms.cern.ch:1094.0] Socket error while handshaking: [ERROR] Socket timeout
[2023-07-12 20:08:12.663057 +0200][Error  ][PostMaster        ] [eoscms.cern.ch:1094] elapsed = 45, pConnectionWindow = 31 seconds.
[2023-07-12 20:08:57.710125 +0200][Error  ][AsyncSock         ] [eoscms.cern.ch:1094.0] Socket error while handshaking: [ERROR] Socket timeout
[2023-07-12 20:08:57.710320 +0200][Error  ][PostMaster        ] [eoscms.cern.ch:1094] elapsed = 45, pConnectionWindow = 31 seconds.
[2023-07-12 20:09:03.632145 +0200][Error  ][XRootD            ] [eoscms.cern.ch:1094] Impossible to send message kXR_open (file: /store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=eoscmshlt.cern.ch&triedrc=srverr, mode: 0460, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
[2023-07-12 20:09:03.632218 +0200][Error  ][XRootD            ] [eoscms.cern.ch:1094] Unable to get the response to request kXR_open (file: /store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=eoscmshlt.cern.ch&triedrc=srverr, mode: 0460, flags: kXR_open_read kXR_async kXR_retstat )
%MSG-w XrdAdaptorInternal:  file_open 12-Jul-2023 20:09:03 CEST pre-events
Failed to open file at URL root://eoscms.cern.ch:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?xrdcl.requuid=5f951e41-6530-493f-a1bc-eb570a4687a0.
%MSG
[2023-07-12 20:09:42.660314 +0200][Error  ][AsyncSock         ] [eoscms.cern.ch:1094.0] Socket error while handshaking: [ERROR] Socket timeout
[2023-07-12 20:09:42.660419 +0200][Error  ][PostMaster        ] [eoscms.cern.ch:1094] elapsed = 45, pConnectionWindow = 31 seconds.
[2023-07-12 20:10:29.688175 +0200][Error  ][AsyncSock         ] [eoscms.cern.ch:1094.0] Socket error while handshaking: [ERROR] Socket timeout
[2023-07-12 20:10:29.688390 +0200][Error  ][PostMaster        ] [eoscms.cern.ch:1094] elapsed = 47, pConnectionWindow = 31 seconds.
[2023-07-12 20:10:29.688506 +0200][Error  ][PostMaster        ] [eoscms.cern.ch:1094] Unable to recover: [ERROR] Socket timeout.
[2023-07-12 20:10:29.688520 +0200][Error  ][XRootD            ] [eoscms.cern.ch:1094] Impossible to send message kXR_open (file: /store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=, mode: 0460, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
[2023-07-12 20:11:04.650192 +0200][Error  ][PostMaster        ] [eoscmshlt-ip565.cern.ch:1098] Forcing error on disconnect: [ERROR] Operation interrupted.
[2023-07-12 20:11:05.651920 +0200][Error  ][PostMaster        ] [eoscmshlt.cern.ch:1094] Forcing error on disconnect: [ERROR] Operation interrupted.
[2023-07-12 20:11:59.659635 +0200][Error  ][XRootD            ] [eoscmsgsi.cern.ch:1094] Unable to get the response to request kXR_open (file: /store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=eoscmsgsi.cern.ch&triedrc=srverr, mode: 0460, flags: kXR_open_read kXR_async kXR_retstat )
%MSG-w XrdAdaptorInternal:  file_open 12-Jul-2023 20:11:59 CEST pre-events
Failed to open file at URL root://eoscmsgsi.cern.ch:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=&xrdcl.requuid=826d7396-510d-4534-89bb-cd0ea95f27f3.
%MSG
[2023-07-12 20:12:16.842198 +0200][Warning][XRootD            ] [xrootd-cms.infn.it:1094] Redirect limit has been reached for message kXR_open (file: /store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=eoscms-ns-ip563.cern.ch,eoscmsgsi.cern.ch,eoscmsgsi.cern.ch,eoscmshlt.cern.ch,eoscms-ns-ip563.cern.ch,eoscmshlt.cern.ch&triedrc=srverr,srverr,srverr,srverr,srverr,srverr, mode: 0460, flags: kXR_open_read kXR_async kXR_retstat ), the last known error is: [FATAL] Socket timeout
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] Redirect trace-back:
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	0. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscms-ns-ip563.cern.ch:1098/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	1. Redirected from: root://eoscms-ns-ip563.cern.ch:1098/ to: root://eoscms.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	2. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	3. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscmsgsi-ip556.cern.ch:1098/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	4. Redirected from: root://eoscmsgsi-ip556.cern.ch:1098/ to: root://eoscmsgsi.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	5. Waited at server request. Resending: root://eoscmsgsi.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	6. Waited at server request. Resending: root://eoscmsgsi.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	7. Waited at server request. Resending: root://eoscmsgsi.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	8. Redirected from: root://eoscmsgsi.cern.ch:1094/ to: root://eoscms.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	9. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	10. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscmsgsi-ip556.cern.ch:1098/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	11. Redirected from: root://eoscmsgsi-ip556.cern.ch:1098/ to: root://eoscmsgsi.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	12. Redirected from: root://eoscmsgsi.cern.ch:1094/ to: root://eoscms.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	13. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	14. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscmshlt-ip565.cern.ch:1098/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	15. Redirected from: root://eoscmshlt-ip565.cern.ch:1098/ to: root://eoscmshlt.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	16. Redirected from: root://eoscmshlt.cern.ch:1094/ to: root://eoscms.cern.ch:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	17. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	18. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscms-ns-ip563.cern.ch:1098/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	19. Redirected from: root://eoscms-ns-ip563.cern.ch:1098/ to: root://eoscms.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	20. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	21. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscmshlt-ip565.cern.ch:1098/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	22. Redirected from: root://eoscmshlt-ip565.cern.ch:1098/ to: root://eoscmshlt.cern.ch:1094/
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	23. Redirected from: root://eoscmshlt.cern.ch:1094/ to: root://eoscms.cern.ch:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:16.842466 +0200][Warning][XRootD            ] 	24. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
%MSG-w XrdAdaptorInternal:  file_open 12-Jul-2023 20:12:16 CEST pre-events
Failed to open file at URL root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=&xrdcl.requuid=b690b2af-1087-45b6-8075-74768ae35b1d.
%MSG
[2023-07-12 20:12:18.051102 +0200][Warning][XRootD            ] [xrootd-cms.infn.it:1094] Redirect limit has been reached for message kXR_open (file: /store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=eoscmsgsi.cern.ch,eoscmsgsi.cern.ch,eoscms-ns-ip563.cern.ch,eoscmshlt.cern.ch,eoscms-ns-ip563.cern.ch,eoscmsgsi.cern.ch&triedrc=srverr,srverr,srverr,srverr,srverr,srverr, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat ), the last known error is: [FATAL] Socket timeout
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] Redirect trace-back:
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	0. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscmsgsi-ip556.cern.ch:1098/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	1. Redirected from: root://eoscmsgsi-ip556.cern.ch:1098/ to: root://eoscmsgsi.cern.ch:1094/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	2. Redirected from: root://eoscmsgsi.cern.ch:1094/ to: root://eoscms.cern.ch:1094/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	3. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	4. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscmsgsi-ip556.cern.ch:1098/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	5. Redirected from: root://eoscmsgsi-ip556.cern.ch:1098/ to: root://eoscmsgsi.cern.ch:1094/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	6. Redirected from: root://eoscmsgsi.cern.ch:1094/ to: root://eoscms.cern.ch:1094/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	7. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	8. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscms-ns-ip563.cern.ch:1098/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	9. Redirected from: root://eoscms-ns-ip563.cern.ch:1098/ to: root://eoscms.cern.ch:1094/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	10. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	11. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscmshlt-ip565.cern.ch:1098/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	12. Redirected from: root://eoscmshlt-ip565.cern.ch:1098/ to: root://eoscmshlt.cern.ch:1094/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	13. Redirected from: root://eoscmshlt.cern.ch:1094/ to: root://eoscms.cern.ch:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	14. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	15. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscms-ns-ip563.cern.ch:1098/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	16. Redirected from: root://eoscms-ns-ip563.cern.ch:1098/ to: root://eoscms.cern.ch:1094/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	17. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	18. Redirected from: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root to: root://eoscmsgsi-ip556.cern.ch:1098/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	19. Redirected from: root://eoscmsgsi-ip556.cern.ch:1098/ to: root://eoscmsgsi.cern.ch:1094/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	20. Redirected from: root://eoscmsgsi.cern.ch:1094/ to: root://eoscms.cern.ch:1094/
[2023-07-12 20:12:18.051249 +0200][Warning][XRootD            ] 	21. Retrying: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
%MSG-w XrdAdaptorInternal:  file_open 12-Jul-2023 20:12:18 CEST pre-events
Failed to open file at URL root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=&xrdcl.requuid=180babf4-1030-4b3b-96fb-6500f04ee5f4.
%MSG
----- Begin Fatal Exception 12-Jul-2023 20:12:18 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
      [c] Input file root://xrootd-cms.infn.it//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root', flags=0x10, permissions=0660) => error '[FATAL] Redirect limit has been reached' (errno=0, code=306). No additional data servers were found.
      [e] Last URL tried: root://xrootd-cms.infn.it:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=&xrdcl.requuid=180babf4-1030-4b3b-96fb-6500f04ee5f4
      [f] Problematic data server: xrootd-cms.infn.it:1094
      [g] Disabled source: eoscms.cern.ch:1094
      [h] Disabled source: xrootd-cms.infn.it:1094
      [i] Disabled source: eoscmsgsi.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-42249/36352

  • This PR adds an extra 20KB to repository

@cmsbuild
Copy link
Contributor

Pull request #42249 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please check and sign again.

@wddgit
Copy link
Contributor Author

wddgit commented Jul 20, 2023

please test

Done. I implemented the suggestion from @stlammel. I also rebased. And force pushed it. Should be good to go if the tests pass, but let me know if there is anything else that needs modifying or to be discussed.

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2a7d00/33810/summary.html
COMMIT: 03acdd0
CMSSW: CMSSW_13_3_X_2023-07-20-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/42249/33810/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 6 lines to the logs
  • Reco comparison results: 10 differences found in the comparisons
  • DQMHistoTests: Total files compared: 48
  • DQMHistoTests: Total histograms compared: 3197149
  • DQMHistoTests: Total failures: 7
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3197120
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
  • Checked 207 log files, 159 edm output root files, 48 DQM output files
  • TriggerResults: no differences found

@wddgit
Copy link
Contributor Author

wddgit commented Jul 24, 2023

@Dr15Jones I think this PR is ready for approval again. I implemented the handling of the exception as Stephan suggested. Let me know if there is anything more I need to do.

@Dr15Jones
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@rappoccio
Copy link
Contributor

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants