Question on how `TFileService` is supposed to interact with `eos` #46024

mmusich · 2024-09-17T15:31:22Z

I have a naive question concerning the expected behavior of TFileService when it's configured to (over-)write files on eos.
While trying to re-run some alignment related jobs @henriettepetersen reported a segmentation fault in SplitVertexResolution, stack trace below:

Thread 1 (Thread 0x7f0c5f6c8640 (LWP 947722) "cmsRun"):
#0  0x00007f0c5e9019ff in poll () from /lib64/libc.so.6
#1  0x00007f0c5a5bf09f in full_read.constprop () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_14/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00007f0c5a5744ec in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_14/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00007f0c5a574670 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_14/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f0c5ff8ba47 in TStreamerInfo::ForceWriteInfo(TFile*, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_14/external/el9_amd64_gcc12/lib/libRIO.so
#6  0x00007f0c607642ae in TTree::BuildStreamerInfo(TClass*, void*, bool) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_14/external/el9_amd64_gcc12/lib/libTree.so
#7  0x00007f0c60775f72 in TTree::BronchExec(char const*, char const*, void*, bool, int, int) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_14/external/el9_amd64_gcc12/lib/libTree.so
#8  0x00007f0bffa4acf5 in SplitVertexResolution::beginJob() () from /afs/cern.ch/cms/CAF/CMSALCA/ALCA_TRACKERALIGN/data/commonValidation/legacy_2024_releases/CMSSW_14_0_14/lib/el9_amd64_gcc12/pluginAlign
mentOfflinevalidationPlugins.so
#9  0x00007f0c60c1d322 in edm::Worker::beginJob() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_14/lib/el9_amd64_gcc12/libFWCoreFramework.so
#10 0x00007f0c60c21a59 in edm::WorkerManager::beginJob(edm::ProductRegistry const&, edm::eventsetup::ESRecordsToProductResolverIndices const&, edm::ProcessBlockHelperBase const&) () from /cvmfs/cms.cern.
ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_14/lib/el9_amd64_gcc12/libFWCoreFramework.so
#11 0x00007f0c60b4079f in edm::EventProcessor::beginJob() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_0_14/lib/el9_amd64_gcc12/libFWCoreFramework.so
#12 0x000000000040746c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#13 0x00007f0c5fd8096d in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_3-el9_amd64_gcc12/build/CMSSW_14_0_3-build/BUILD/el9_amd64_gcc
12/external/tbb/v2021.9.0-d33db04d4520c6ff791eab900054e986/tbb-v2021.9.0/src/tbb/arena.cpp:688
#14 0x0000000000408ee2 in main::{lambda()#1}::operator()() const ()
#15 0x000000000040517c in main ()

(a reproducer is available at /afs/cern.ch/work/h/hpeterse/public/splitV_seg_fault, by copying locally the folder in any recent cmssw release and then running cmsRun validation_cfg.py config=validation.json).

The issue seems to be related to the fact that the file that we're trying to write already exists with the same name at the same location.
In particular the segmentation fault originates here:

cmssw/Alignment/OfflineValidation/plugins/SplitVertexResolution.cc

Line 970 in e54f434

tree_->Branch("event", &event_, 64000, 2);

I can circumvent the issue by commenting that line, but then when running I see the following warning:

Warning in <TStorageFactoryFile::Write>: file root://eoscms.cern.ch//eos/cms/store/group/alca_trackeralign/AlignmentValidation/AlignmentValidation/2024_CDE_ReReco_mp3949_splitV_379525/SplitV/single/GT/compare2024/379525/SplitV.root not opened in write mode

What's somehow puzzling to me, is that when the address of the output file is local (e.g. the $PWD) even if the file is already existing there, there is no issue whatsoever.
Also I would have thought that due to this:

cmssw/CommonTools/UtilAlgos/src/TFileService.cc

Line 22 in 5e10089

    
           tFileDirectory_ = TFileDirectory("", "", TFile::Open(fileName_.c_str(), "RECREATE"), "");

the file would have been overwritten anyway.
Also when trying to prepare a reproducer via a simple ROOT script:

#include "TFile.h"
#include "TTree.h"
#include <iostream>
#include "Alignment/OfflineValidation/src/pvTree.h"
#include "PhysicsTools/FWLite/interface/TFileService.h"
#include <vector>
#include <string>

int test_TTreeEOS() {
  // Define the file path to EOS (replace with your EOS path)
  const std::string eosFilePath = "/eos/cms/store/group/alca_trackeralign/musich/test.root";

  fwlite::TFileService outfile_ = fwlite::TFileService(eosFilePath);
    
  // Create a TTree and a branch  
  pvEvent event_;
  event_.pvs.clear();
  event_.nVtx = -1;

  TTree* tree_ = outfile_.make<TTree>("pvTree", "pvTree");
  tree_->Branch("event", &event_, 64000, 2);
  
  return 0;
}

I have found out that with this I can overwrite the remote file as many times as I want.
Am I missing something trivial ?

Cc: @TomasKello

The text was updated successfully, but these errors were encountered:

cmsbuild · 2024-09-17T15:31:43Z

cms-bot internal usage

cmsbuild · 2024-09-17T15:31:44Z

A new Issue was created by @mmusich.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel · 2024-09-17T15:35:56Z

assign CommonTools/UtilAlgos

cmsbuild · 2024-09-17T15:36:13Z

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel · 2024-09-17T18:10:00Z

Since

cmssw/CommonTools/UtilAlgos/src/TFileService.cc

Line 22 in 5e10089

tFileDirectory_ = TFileDirectory("", "", TFile::Open(fileName_.c_str(), "RECREATE"), "");

calls TFile::Open(), the I/O gets rerouted through our StorageFactory layer, which is also visible in the error message

Warning in <TStorageFactoryFile::Write>: file root://eoscms.cern.ch//eos/cms/store/group/alca_trackeralign/AlignmentValidation/AlignmentValidation/2024_CDE_ReReco_mp3949_splitV_379525/SplitV/single/GT/compare2024/379525/SplitV.root not opened in write mode

With the StorageFactory, root:// URLs lead to our XrdAdaptor layer to be used for the actual I/O. On a quick look the XrdAdaptor code looks like it should be able to deal with writing files too, but I'd guess the writing part hasn't been tested much (since we use xrootd predominantly for reading data).

It may be worth of noting here that writing to (CERN) EOS through the FUSE mount has an "interesting" behavior as well #44369 (ROOT internally transforms the local-looking path into a root:// URL, while the StorageFactory layer continues act like the file would be local.

makortel · 2024-09-17T18:10:04Z

assign core

cmsbuild · 2024-09-17T18:10:19Z

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel · 2024-09-17T18:11:35Z

Would you be able to try if adding

process.add_(cms.Service("AdaptorConfig", native=cms.untracked.vstring("root")))

to the job configuration would impact the behavior? (this prevents CMSSW to register the StorageFactory + XrdAdaptor for root protocol)

makortel · 2024-09-17T18:11:46Z

type root

makortel · 2024-09-17T18:13:00Z

@pcanal Could there be some error condition (or other assumption) in TStreamerInfo::ForceWriteInfo() that could lead it to segfault instead of reporting an error when writing to CERN EOS through xrootd?

pcanal · 2024-09-17T19:00:46Z

I can see two possibility. One is that the ROOT build being used does not have the code from root-project/root#13842.

The other, more likely, is that writing in a file open in read-only mode might not be failing elegantly .... i.e.

Warning in <TStorageFactoryFile::Write>: file root://eoscms.cern.ch//eos/....SplitV.root not opened in write mode

when/if the file was open with "RECREATE" indicates that something 'bad' happened during the TFile::Open (and undefined behavior might be a consequence thereof).

One possibility is that the file is seen/thought-of as non-writeable (for example issue with permissions) and that some part of the logic in or around TFile::Open is silently falling back to opening the file in read-only mode.

mmusich · 2024-09-18T07:13:22Z

Would you be able to try if adding
process.add_(cms.Service("AdaptorConfig", native=cms.untracked.vstring("root")))
to the job configuration would impact the behavior? (this prevents CMSSW to register the StorageFactory + XrdAdaptor for root protocol)

indeed, adding this line in the configuration file, the segmentation fault is prevented. Thank you.

jfernan2 · 2024-10-30T15:53:38Z

@mmusich is this issue solved? Thanks

mmusich · 2024-10-30T15:56:45Z

@jfernan2

is this issue solved?

I am not sure. With the workaround at #46024 (comment) this particular instance of the problem is solved, though I can't say if that's a design feature or a bug.

cmsbuild added the pending-assignment label Sep 17, 2024

cmsbuild added reconstruction-pending pending-signatures and removed pending-assignment labels Sep 17, 2024

cmsbuild added the core-pending label Sep 17, 2024

cmsbuild added the root label Sep 17, 2024

github-project-automation bot added this to ROOT prioritization Sep 17, 2024

github-project-automation bot moved this to New in ROOT prioritization Sep 17, 2024

makortel moved this from New to Work in CMS in ROOT prioritization Nov 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Question on how `TFileService` is supposed to interact with `eos` #46024

Question on how `TFileService` is supposed to interact with `eos` #46024

mmusich commented Sep 17, 2024 •

edited

Loading

cmsbuild commented Sep 17, 2024 •

edited

Loading

cmsbuild commented Sep 17, 2024

makortel commented Sep 17, 2024

cmsbuild commented Sep 17, 2024

makortel commented Sep 17, 2024

makortel commented Sep 17, 2024

cmsbuild commented Sep 17, 2024

makortel commented Sep 17, 2024

makortel commented Sep 17, 2024

makortel commented Sep 17, 2024

pcanal commented Sep 17, 2024

mmusich commented Sep 18, 2024

jfernan2 commented Oct 30, 2024

mmusich commented Oct 30, 2024

Question on how TFileService is supposed to interact with eos #46024

Question on how TFileService is supposed to interact with eos #46024

Comments

mmusich commented Sep 17, 2024 • edited Loading

cmsbuild commented Sep 17, 2024 • edited Loading

cmsbuild commented Sep 17, 2024

makortel commented Sep 17, 2024

cmsbuild commented Sep 17, 2024

makortel commented Sep 17, 2024

makortel commented Sep 17, 2024

cmsbuild commented Sep 17, 2024

makortel commented Sep 17, 2024

makortel commented Sep 17, 2024

makortel commented Sep 17, 2024

pcanal commented Sep 17, 2024

mmusich commented Sep 18, 2024

jfernan2 commented Oct 30, 2024

mmusich commented Oct 30, 2024

Question on how `TFileService` is supposed to interact with `eos` #46024

Question on how `TFileService` is supposed to interact with `eos` #46024

mmusich commented Sep 17, 2024 •

edited

Loading

cmsbuild commented Sep 17, 2024 •

edited

Loading