Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock in Tensorflow in ARM after exception in job #40996

Open
Dr15Jones opened this issue Mar 8, 2023 · 9 comments
Open

Deadlock in Tensorflow in ARM after exception in job #40996

Dr15Jones opened this issue Mar 8, 2023 · 9 comments
Labels
core-pending l1-pending pending-signatures

Comments

@Dr15Jones
Copy link
Contributor

So the job

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_aarch64_gcc11/CMSSW_13_1_X_2023-03-07-2300/pyRelValMatrixLogs/run/136.771_RunDoubleMuon2016H/step3_RunDoubleMuon2016H.log#/

threw an 'out of memory' exception

%MSG-w MemoryCheck:  PoolOutputModule:RECOoutput 08-Mar-2023 07:38:36 CET  Run: 283877 Event: 10870746
MemoryCheck: module PoolOutputModule:RECOoutput VSIZE 5577.25 0 RSS 3567.69 2.375
%MSG
----- Begin Fatal Exception 08-Mar-2023 07:38:36 CET-----------------------
An exception of category 'BadAlloc' occurred while
   [0] Processing  Event run: 283877 lumi: 7 event: 10870746 stream: 3
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module L1TMuonEndCapTrackProducer/'valEmtfStage2Digis'
Exception Message:
A std::bad_alloc exception was thrown.
The job has probably exhausted the virtual memory available to the process.
----- End Fatal Exception -------------------------------------------------
%MSG-w MemoryCheck:  SeedingLayersEDProducer:initialStepSeedLayersPreSplitting  08-Mar-2023 07:38:36 CET Run: 283877 Event: 11624573
MemoryCheck: module SeedingLayersEDProducer:initialStepSeedLayersPreSplitting VSIZE 5577.25 0 RSS 3568.81 1.125
%MSG

but then the job was killed for taking too long. The log file shows that it wrote to the log for the first 5 minutes of the job and then after nearly 4 hours it was killed for taking too long. The stack traces are

Thread 8 (Thread 0x4000cbc99250 (LWP 2736855) "cmsRun"):
[Eigen thread from tensorflow]
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000040006c5e63a0 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
[cut]

Thread 7 (Thread 0x4000cba89250 (LWP 2736854) "cmsRun"):
[another Eigen thread from tensorflow]
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000040006c5e63a0 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2

Thread 6 (Thread 0x4000cb879250 (LWP 2736853) "cmsRun"):
[yet another Eigen thread from tensorflow]
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x000040006c5e63a0 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2

Thread 5 (Thread 0x400092a09250 (LWP 2736431) "cmsRun"):
[a waiting TBB thread]

Thread 4 (Thread 0x40008eb79250 (LWP 2736430) "cmsRun"):
#7  0x000040007379f0ec in nsync::nsync_mu_lock(nsync::nsync_mu_s_*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#8  0x000040007318733c in tensorflow::DirectSessionFactory::NewSession(tensorflow::SessionOptions const&, tensorflow::Session**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#9  0x000040008456527c in tensorflow::NewSession(tensorflow::SessionOptions const&, tensorflow::Session**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#10 0x00004000697aa040 in tensorflow::createSession(tensorflow::SessionOptions&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#11 0x00004000697aa3d4 in tensorflow::createSession(tensorflow::GraphDef const*, tensorflow::SessionOptions&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#12 0x00004000697ad730 in tensorflow::createSession(tensorflow::GraphDef const*, tensorflow::Backend, int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#13 0x00004000afea404c in PtAssignmentEngineDxy::configure(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#14 0x00004000afe9deb4 in PtAssignment::configure(PtAssignmentEngine*, PtAssignmentEngineDxy*, int, int, int, int, bool, bool, bool, bool, bool, bool, bool, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#15 0x00004000afea7f5c in SectorProcessor::process_single_bx(int, std::vector<L1TMuon::TriggerPrimitive, std::allocator<L1TMuon::TriggerPrimitive> > const&, std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >&, std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&, std::deque<std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >, std::allocator<std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> > > >&, std::deque<std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >, std::allocator<std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> > > >&, std::map<std::array<int, 3ul>, int, std::less<std::array<int, 3ul> >, std::allocator<std::pair<std::array<int, 3ul> const, int> > >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#16 0x00004000afea91f0 in SectorProcessor::process(edm::EventID const&, std::vector<L1TMuon::TriggerPrimitive, std::allocator<L1TMuon::TriggerPrimitive> > const&, std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >&, std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#17 0x00004000afeb3e10 in TrackFinder::process(edm::Event const&, edm::EventSetup const&, std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >&, std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#18 0x00004000b3f12b84 in L1TMuonEndCapTrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/pluginL1TriggerL1TMuonEndCapPlugins.so
#19 0x000040003d143e7c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
[cut]

Thread 3 (Thread 0x40008e169250 (LWP 2736429) "cmsRun"):
[waiting TBB thread]

Thread 2 (Thread 0x400048749250 (LWP 2736236) "cmsRun"):
[stacktrace helper thread]

Thread 1 (Thread 0x40003ebdd070 (LWP 2735080) "cmsRun"):
#6  0x00004000737a11d0 in nsync::nsync_mu_semaphore_p(nsync::nsync_semaphore_s_*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#7  0x000040007379efd4 in nsync::nsync_mu_lock_slow_(nsync::nsync_mu_s_*, nsync::waiter*, unsigned int, nsync::lock_type_s*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#8  0x000040007379f0ec in nsync::nsync_mu_lock(nsync::nsync_mu_s_*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#9  0x000040007318733c in tensorflow::DirectSessionFactory::NewSession(tensorflow::SessionOptions const&, tensorflow::Session**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#10 0x000040008456527c in tensorflow::NewSession(tensorflow::SessionOptions const&, tensorflow::Session**) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-07-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#11 0x00004000697aa040 in tensorflow::createSession(tensorflow::SessionOptions&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#12 0x00004000697aa3d4 in tensorflow::createSession(tensorflow::GraphDef const*, tensorflow::SessionOptions&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#13 0x00004000697ad730 in tensorflow::createSession(tensorflow::GraphDef const*, tensorflow::Backend, int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#14 0x00004000afea404c in PtAssignmentEngineDxy::configure(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#15 0x00004000afe9deb4 in PtAssignment::configure(PtAssignmentEngine*, PtAssignmentEngineDxy*, int, int, int, int, bool, bool, bool, bool, bool, bool, bool, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#16 0x00004000afea7f5c in SectorProcessor::process_single_bx(int, std::vector<L1TMuon::TriggerPrimitive, std::allocator<L1TMuon::TriggerPrimitive> > const&, std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >&, std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&, std::deque<std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >, std::allocator<std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> > > >&, std::deque<std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >, std::allocator<std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> > > >&, std::map<std::array<int, 3ul>, int, std::less<std::array<int, 3ul> >, std::allocator<std::pair<std::array<int, 3ul> const, int> > >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#17 0x00004000afea91f0 in SectorProcessor::process(edm::EventID const&, std::vector<L1TMuon::TriggerPrimitive, std::allocator<L1TMuon::TriggerPrimitive> > const&, std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >&, std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#18 0x00004000afeb3e10 in TrackFinder::process(edm::Event const&, edm::EventSetup const&, std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >&, std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libL1TriggerL1TMuonEndCap.so
#19 0x00004000b3f12b84 in L1TMuonEndCapTrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/pluginL1TriggerL1TMuonEndCapPlugins.so
#20 0x000040003d143e7c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
[cut]
#28 0x000040003d02ec84 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#29 0x000040003d03bcd8 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#30 0x000040003d03c260 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02775/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-06-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
[cut]
#33 0x000000000040f73c in main::{lambda()#1}::operator()() const ()
#34 0x0000000000406fc4 in main ()

Current Modules:

Module: L1TMuonEndCapTrackProducer:valEmtfStage2Digis (crashed)
Module: none
Module: none
Module: L1TMuonEndCapTrackProducer:valEmtfStage2Digis

One can see that thread 1 and 4 are both waiting to get a lock within tensorflow but no thread has any stack which is holding that lock. It looks like when the exception happened, during the stack unwinding of the exception the tensorflow lock was never released.
@Dr15Jones
Copy link
Contributor Author

assign l1, core

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 8, 2023

New categories assigned: core,l1

@epalencia,@Dr15Jones,@smuzaffar,@aloeliger,@makortel,@cecilecaillol you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 8, 2023

A new Issue was created by @Dr15Jones Chris Jones.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor

makortel commented Mar 8, 2023

Umm, why is L1TMuonEndCapTrackProducer::produce() creating new TensorFlow sessions during the event processing?

@makortel
Copy link
Contributor

makortel commented Mar 8, 2023

So the call chain from L1TMuonEndCapTrackProducer::produce() is

cmssw/L1Trigger/L1TMuonEndCap/plugins/L1TMuonEndCapTrackProducer.cc

Line 23 in e7caf24

track_finder_->process(iEvent, iSetup, *out_hits_tmp, *out_tracks);

cmssw/L1Trigger/L1TMuonEndCap/src/TrackFinder.cc

Lines 97 to 105 in e7caf24

for (int endcap = emtf::MIN_ENDCAP; endcap <= emtf::MAX_ENDCAP; ++endcap) {
for (int sector = emtf::MIN_TRIGSECTOR; sector <= emtf::MAX_TRIGSECTOR; ++sector) {
const int es = (endcap - emtf::MIN_ENDCAP) * (emtf::MAX_TRIGSECTOR - emtf::MIN_TRIGSECTOR + 1) +
(sector - emtf::MIN_TRIGSECTOR);
sector_processors_.at(es).configure(&setup_, verbose_, endcap, sector);
sector_processors_.at(es).process(iEvent.id(), muon_primitives, out_hits, out_tracks);
}
}

cmssw/L1Trigger/L1TMuonEndCap/src/SectorProcessor.cc

Lines 156 to 171 in e7caf24

PtAssignment pt_assign;
pt_assign.configure(pt_assign_engine_,
pt_assign_engine_dxy_,
verbose_,
endcap_,
sector_,
bx,
cfg.readPtLUTFile_,
cfg.fixMode15HighPt_,
cfg.bug9BitDPhi_,
cfg.bugMode7CLCT_,
cfg.bugNegPt_,
cfg.bugGMTPhi_,
cfg.promoteMode7_,
cfg.modeQualVer_,
cfg.pbFileName_);

cmssw/L1Trigger/L1TMuonEndCap/src/PtAssignment.cc

Line 35 in e7caf24

pt_assign_engine_dxy_->configure(verbose_, pbFileName);

cmssw/L1Trigger/L1TMuonEndCap/src/PtAssignmentEngineDxy.cc

Lines 18 to 37 in e7caf24

void PtAssignmentEngineDxy::configure(int verbose, const std::string pbFileNameDxy) {
verbose_ = verbose;
pbFileNameDxy_ = pbFileNameDxy;
std::string pbFilePathDxy_ = "L1Trigger/L1TMuon/data/emtf_luts/" + pbFileNameDxy_;
inputNameDxy_ = "input1";
outputNamesDxy_ = {"Identity"};
if (graphDefDxy_ == nullptr) {
graphDefDxy_ = tensorflow::loadGraphDef(edm::FileInPath(pbFilePathDxy_).fullPath());
}
emtf_assert(graphDefDxy_ != nullptr);
if (sessionDxy_ == nullptr) {
sessionDxy_ = tensorflow::createSession(graphDefDxy_);
}
emtf_assert(sessionDxy_ != nullptr);
}

In the end it looks only like a "lazy initialization", but it would be nevertheless be better to read the TF graph and create the session around the module construction time (given that the model file seems to be specified in the configuration, and the PtAssignmentEngineDxy seems to be constructed at that time already).

@aloeliger
Copy link
Contributor

@eyigitba I may be mistaken, but is this the model you were discussing/worked on/with?

@aloeliger aloeliger mentioned this issue Mar 8, 2023
Open
@makortel
Copy link
Contributor

makortel commented Mar 8, 2023

I found we've seen unreliable behavior of TensorFlow's nsync mutex on ARM before in conjunction of ONNX

@eyigitba
Copy link
Contributor

Hi @aloeliger , sorry for replying late, but yes this is the NN module that I worked on. Please let me know if there is something that needs to be done on my end

@aloeliger
Copy link
Contributor

@eyigitba perhaps @makortel can clarify, but I think that the way the tensorflow session is created for running this network is less than optimal. I don't know that that's related to the deadlock seen, but should likely be fixed if there is an active developer on this project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core-pending l1-pending pending-signatures
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@makortel @Dr15Jones @cmsbuild @aloeliger @eyigitba