[8_0_X] Protect storage accounting UDP messages from NaN, and Use StatisticsSenderService for all framework files #36632

makortel · 2022-01-05T16:00:22Z

PR description:

This PR is a combined backport of #35362 and #35505, following requests in #29412 and #36349. It also contains small part of #26789 and #26729 to make the cherry-picks easier, and #36403 (fully) as further cleanup.

Making backport now because #36580 opens door for a new 8_0_X release to be actually used in production.

PR validation:

Unit tests run.

Backport from 9fc3f03

Backport from f49614b

Backport from 08cd101

NaN's were being reported from the values computed using sqrt. This most likely was from the different variables not being updated atomically together.

Previously, each try to open the file using a different PFN would report an open attempt for the same LFN. This meant we could have multiple opens but only one close for a given LFN.

When sending information to the StatisticsSenderService, the file LFN or URL must be supplied.

Send statistics for primary, secondary, and embedded files. The aggregate file statistics are only reset on primary file close boundaries to keep the behavior the same as previous. Changed all calls to closeFile_() to be the new closeFile()

Now broadcasts how the file is used.

Needed to compile tbb::concurrent_unordered_map<X, FileInfo>

cmsbuild · 2022-01-05T16:00:50Z

A new Pull Request was created by @makortel (Matti Kortelainen) for CMSSW_8_0_X.

It involves the following packages:

IOPool/Input (core)
IOPool/SecondaryInput (core)
IOPool/TFileAdaptor (core)
Utilities/StorageFactory (core)
Utilities/XrdAdaptor (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@wddgit this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

makortel · 2022-01-05T16:01:41Z

@cmsbuild, please test

cmsbuild · 2022-01-05T16:41:33Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-025e10/21538/summary.html
COMMIT: 0c6aae7
CMSSW: CMSSW_8_0_X_2022-01-03-2300/slc6_amd64_gcc530
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36632/21538/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

10224.0

----- Begin Fatal Exception 05-Jan-2022 17:23:55 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
   [2] Calling RootFileSequenceBase::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/2006F97B-77DD-E511-BC51-0CC47A745298.root'
   Additional Info:
      [a] XrdCl::File::Open(name='root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/2006F97B-77DD-E511-BC51-0CC47A745298.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] Unable to open file /eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/2006F97B-77DD-E511-BC51-0CC47A745298.root; No such file or directory
' (errno=3011, code=400). No additional data servers were found.
      [b] Last URL tried: root://eoscms.cern.ch:1094//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/2006F97B-77DD-E511-BC51-0CC47A745298.root?tried=
      [c] Problematic data server: eoscms.cern.ch:1094
      [d] Disabled source: eoscms.cern.ch:1094
      [e] Input file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/2006F97B-77DD-E511-BC51-0CC47A745298.root could not be opened.
Fallback Input file root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/2006F97B-77DD-E511-BC51-0CC47A745298.root also could not be opened.
Original exception info is above; fallback exception info is below.
      [f] XrdCl::File::Open(name='root://cms-xrd-global.cern.ch//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/2006F97B-77DD-E511-BC51-0CC47A745298.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [g] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/relval/CMSSW_8_0_0_patch1/RelValMinBias_13/GEN-SIM/80X_upgrade2017_design_v4_UPG17-v1/10000/2006F97B-77DD-E511-BC51-0CC47A745298.root?tried=
      [h] Problematic data server: cms-xrd-global.cern.ch:1094
      [i] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

smuzaffar · 2022-01-12T08:22:04Z

please test

cmsbuild · 2022-01-12T09:27:38Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-025e10/21662/summary.html
COMMIT: 0c6aae7
CMSSW: CMSSW_8_0_X_2022-01-09-0000/slc6_amd64_gcc530
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36632/21662/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
10224.0 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 14
DQMHistoTests: Total histograms compared: 1020424
DQMHistoTests: Total failures: 839
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 1019477
DQMHistoTests: Total skipped: 108
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -13 KiB( 13 files compared)
Checked 60 log files, 11 edm output root files, 14 DQM output files

makortel · 2022-01-12T18:41:13Z

+core

Comparison differences are same as in #36580 (comment)

cmsbuild · 2022-01-12T18:41:31Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_3_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2022-01-13T00:43:55Z

+1

makortel and others added 12 commits January 5, 2022 09:54

Code format

1be243c

Backport from 9fc3f03

Clang-tidy

6a3fc4c

Backport from f49614b

Clang-Format

668ce6d

Backport from 08cd101

Protect storage accounting UDP messages from NaN

b1a7fa5

NaN's were being reported from the values computed using sqrt. This most likely was from the different variables not being updated atomically together.

Report only once when trying to open file

5c8dd83

Previously, each try to open the file using a different PFN would report an open attempt for the same LFN. This meant we could have multiple opens but only one close for a given LFN.

StatisticsSenderService now requires LFN or URL

d8240fa

When sending information to the StatisticsSenderService, the file LFN or URL must be supplied.

Changed SecondaryProducer to be a one::EDProducer

3e25ab4

Added unit test for StatisticsSenderService

00321cb

Pass InputType to StatisticsSenderService

7560c16

Now broadcasts how the file is used.

Silence file closuring printouts in StatisticsSenderService by default

e806fd2

Add copy constructor for FileInfo

0c6aae7

Needed to compile tbb::concurrent_unordered_map<X, FileInfo>

makortel mentioned this pull request Jan 5, 2022

Backport recent StatisticsSenderService changes to earlier releases cms-sw/framework-team#329

Closed

11 tasks

cmsbuild added this to the CMSSW_8_0_X milestone Jan 5, 2022

makortel mentioned this pull request Jan 5, 2022

[8_0_X] Support vector<string> data products in friendly class name #36580

Merged

cmsbuild added core-pending orp-pending pending-signatures tests-pending labels Jan 5, 2022

cmsbuild added tests-started and removed tests-pending labels Jan 5, 2022

cmsbuild added tests-rejected and removed tests-started labels Jan 5, 2022

cmsbuild added tests-started and removed tests-rejected labels Jan 12, 2022

cmsbuild added tests-approved and removed tests-started labels Jan 12, 2022

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Jan 12, 2022

cmsbuild added orp-approved and removed orp-pending labels Jan 13, 2022

cmsbuild merged commit b0755ee into cms-sw:CMSSW_8_0_X Jan 13, 2022

makortel deleted the backportStatisticsSenderService_80x branch January 13, 2022 00:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8_0_X] Protect storage accounting UDP messages from NaN, and Use StatisticsSenderService for all framework files #36632

[8_0_X] Protect storage accounting UDP messages from NaN, and Use StatisticsSenderService for all framework files #36632

makortel commented Jan 5, 2022

cmsbuild commented Jan 5, 2022

makortel commented Jan 5, 2022

cmsbuild commented Jan 5, 2022

smuzaffar commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

makortel commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

qliphy commented Jan 13, 2022

[8_0_X] Protect storage accounting UDP messages from NaN, and Use StatisticsSenderService for all framework files #36632

[8_0_X] Protect storage accounting UDP messages from NaN, and Use StatisticsSenderService for all framework files #36632

Conversation

makortel commented Jan 5, 2022

PR description:

PR validation:

cmsbuild commented Jan 5, 2022

makortel commented Jan 5, 2022

cmsbuild commented Jan 5, 2022

RelVals

smuzaffar commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

Comparison Summary

makortel commented Jan 12, 2022

cmsbuild commented Jan 12, 2022

qliphy commented Jan 13, 2022