Assertion failure in edm::service::MessageLogger::unEstablishModule() #34520

makortel · 2021-07-16T17:58:12Z

In PR test #34517 (comment) TestDQMServicesDemo test, more specifically cmsRun DQMServices/Demo/test/run_analyzers_cfg.py outfile=huge.root numberEventsInRun=300 numberEventsInLuminosityBlock=100 nEvents=600 nThreads=10 failed with

Begin processing the 410th record. Run 2, Event 110, LumiSection 2 on stream 3 at 16-Jul-2021 17:08:02.171 CEST
cmsRun: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/src/FWCore/MessageService/plugins/MessageLogger.cc:469: void edm::service::MessageLogger::unEstablishModule(const edm::ModuleCallingContext&, const char*): Assertion `tran == GlobalContext::Transition::kBeginProcessBlock || tran == GlobalContext::Transition::kAccessInputProcessBlock || tran == GlobalContext::Transition::kEndProcessBlock' failed.

Thread 11 (Thread 0x2b6985a02700 (LWP 13012)):
#2  0x00002b69677d5590 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6960724d19 in syscall () from /lib64/libc.so.6
#5  0x00002b695f609f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b6966cbfe2c) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 10 (Thread 0x2b6983600700 (LWP 13011)):
#2  0x00002b69677d5590 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6960724d19 in syscall () from /lib64/libc.so.6
#5  0x00002b695f609f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b6966cbfdac) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 9 (Thread 0x2b6984e01700 (LWP 13010)):
#3  0x00002b69677d995b in sig_dostack_then_abort () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b69606623d7 in raise () from /lib64/libc.so.6
#6  0x00002b6960663ac8 in abort () from /lib64/libc.so.6
#7  0x00002b696065b1a6 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00002b696065b252 in __assert_fail () from /lib64/libc.so.6
#9  0x00002b69665bcaaf in edm::service::MessageLogger::unEstablishModule(edm::ModuleCallingContext const&, char const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreMessageServicePlugins.so
#10 0x00002b695dee928b in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#11 0x00002b695dede39d in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002b695de3cc35 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#13 0x00002b695de3cded in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002b695de3d0f6 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#15 0x00002b695de392c6 in edm::Path::finished(std::__exception_ptr::exception_ptr, edm::StreamContext const*, edm::EventTransitionInfo const&, edm::StreamID const&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#16 0x00002b695de39a6e in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#17 0x00002b695de39d51 in edm::FunctorWaitingTask<edm::Path::runNextWorkerAsync(unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#18 0x00002b695db32335 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
#19 0x00002b695f610d0b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (this=0x2b6963487380, t=0x2b6963463b00, waiter=...) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.h:396
#20 0x00002b695f60d7e5 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x2b6963487380) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.cpp:178
#21 tbb::detail::r1::task_dispatcher::execute_and_wait (t=0x0, wait_ctx=..., w_ctx=...) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.cpp:168
#22 0x00002b6960a049cd in ROOT::Experimental::TTaskGroup::Wait() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_0_X_2021-07-15-2300/external/slc7_amd64_gcc900/lib/libImt.so
#23 0x00002b695e313a83 in TTree::Fill() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_0_X_2021-07-15-2300/external/slc7_amd64_gcc900/lib/libTree.so
#24 0x00002b697e5072cf in (anonymous namespace)::TreeHelper<TProfile2D>::doFill(dqm::legacy::MonitorElement*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-14-2300/lib/slc7_amd64_gcc900/pluginDQMServicesFwkIOPlugins.so
#25 0x00002b697e505a58 in DQMRootOutputModule::writeLuminosityBlock(edm::LuminosityBlockForOutput const&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-14-2300/lib/slc7_amd64_gcc900/pluginDQMServicesFwkIOPlugins.so
#26 0x00002b695defd380 in edm::one::OutputModuleBase::doWriteLuminosityBlock(edm::LuminosityBlockPrincipal const&, edm::ModuleCallingContext const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002b695de371a8 in void edm::SerialTaskQueueChain::actionToRun<edm::OutputModuleCommunicatorT<edm::one::OutputModuleBase>::writeLumiAsync(edm::WaitingTaskHolder, edm::LuminosityBlockPrincipal const&, edm::ProcessContext const*, edm::ActivityRegistry*)::{lambda()#1}&>(edm::OutputModuleCommunicatorT<edm::one::OutputModuleBase>::writeLumiAsync(edm::WaitingTaskHolder, edm::LuminosityBlockPrincipal const&, edm::ProcessContext const*, edm::ActivityRegistry*)::{lambda()#1}&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#28 0x00002b695de37551 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::OutputModuleCommunicatorT<edm::one::OutputModuleBase>::writeLumiAsync(edm::WaitingTaskHolder, edm::LuminosityBlockPrincipal const&, edm::ProcessContext const*, edm::ActivityRegistry*)::{lambda()#1}>(tbb::detail::d1::task_group&, edm::OutputModuleCommunicatorT<edm::one::OutputModuleBase>::writeLumiAsync(edm::WaitingTaskHolder, edm::LuminosityBlockPrincipal const&, edm::ProcessContext const*, edm::ActivityRegistry*)::{lambda()#1}&&)::{lambda()#1}>::execute() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#29 0x00002b695db302c9 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
#30 0x00002b695f5fac53 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (this=0x2b6963487380, t=0x2b696340fd00, t@entry=0x0, waiter=...) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.h:396
#31 0x00002b695f5f6982 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=..., t=0x0, this=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/governor.h:147

Thread 8 (Thread 0x2b6984001700 (LWP 13009)):
#2  0x00002b69677d5590 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6960724d19 in syscall () from /lib64/libc.so.6
#5  0x00002b695f609f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b6966cc002c) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 7 (Thread 0x2b6981c01700 (LWP 13008)):
#2  0x00002b69677d5590 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6960724d19 in syscall () from /lib64/libc.so.6
#5  0x00002b695f609f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b6966cbffac) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 6 (Thread 0x2b6982602700 (LWP 13007)):
#2  0x00002b69677d5590 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6960724d19 in syscall () from /lib64/libc.so.6
#5  0x00002b695f609f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b6966cbff2c) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 5 (Thread 0x2b6981200700 (LWP 13006)):
#2  0x00002b69677d5590 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6960724d19 in syscall () from /lib64/libc.so.6
#5  0x00002b695f609f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b6966cbfeac) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 4 (Thread 0x2b697f946700 (LWP 13005)):
#1  0x00002b69606f1784 in sleep () from /lib64/libc.so.6
#2  0x00002b69677d5590 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6960724d19 in syscall () from /lib64/libc.so.6
#5  0x00002b695f609f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b6966cc012c) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 3 (Thread 0x2b697ef45700 (LWP 13004)):
#2  0x00002b69677d5590 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6960724d19 in syscall () from /lib64/libc.so.6
#5  0x00002b695f609f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b6966cc00ac) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 2 (Thread 0x2b697dff6700 (LWP 12997)):
#0  0x00002b696041f1d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00002b69677d5747 in edm::service::cmssw_stacktrace_fork() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#2  0x00002b69677d63da in edm::service::InitRootHandlers::stacktraceHelperThread() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServ

Thread 1 (Thread 0x2b69627a6180 (LWP 12960)):
#2  0x00002b69677d5590 in sig_pause_for_stacktrace () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6967f9a183 in std::_Rb_tree<dqm::legacy::MonitorElement*, dqm::legacy::MonitorElement*, std::_Identity<dqm::legacy::MonitorElement*>, dqm::impl::MonitorElement::MEComparison, std::allocator<dqm::legacy::MonitorElement*> >::find(dqm::legacy::MonitorElement* const&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libDQMServicesCore.so
#5  0x00002b6967f9cabc in dqm::legacy::MonitorElement* dqm::implementation::DQMStore::findME<dqm::legacy::MonitorElement*>(dqm::legacy::MonitorElement* const&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libDQMServicesCore.so
#6  0x00002b6967f94983 in dqm::implementation::DQMStore::enterLumi(unsigned int, unsigned int, unsigned long) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libDQMServicesCore.so
#7  0x00002b697e44f4b8 in DQMOneEDAnalyzer<>::accumulate(edm::Event const&, edm::EventSetup const&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/pluginDQMServicesDemoTestPlugins.so
#8  0x00002b695def9bdb in edm::one::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#9  0x00002b695dedde9d in edm::WorkerT<edm::one::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#10 0x00002b695de3cc35 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#11 0x00002b695de3cded in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002b695de3d0f6 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#13 0x00002b695de3f6e0 in void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002b695de3f921 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#15 0x00002b695db302c9 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-07-15-2300/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so

Current Modules:
Module: PathStatusInserter:p (crashed)
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: TestDQMOneEDAnalyzer:testone

The text was updated successfully, but these errors were encountered:

makortel · 2021-07-16T17:58:18Z

assign core

cmsbuild · 2021-07-16T17:58:29Z

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild · 2021-07-16T17:58:31Z

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel · 2021-07-16T17:59:58Z

So the thread 9 is running DQMRootOutputModule::writeLuminosityBlock() and then ends up running edm::Path::workerFinished(). Can we be missing tbb::this_task_arena::isolate() in DQMRootOutputModule?

Dr15Jones · 2021-07-16T18:01:01Z

Can we be missing tbb::this_task_arena::isolate() in DQMRootOutputModule?

seems so.

Dr15Jones · 2021-07-16T18:02:22Z

@makortel I had thought that now ROOT had put an arena call around all of its internal uses of TBB. Guess that isn't the case :(.

makortel · 2021-07-16T18:04:26Z

I had thought that now ROOT had put an arena call around all of its internal uses of TBB. Guess that isn't the case :(.

Maybe @pcanal could comment?

makortel · 2021-07-17T01:40:58Z

There is another one in workflow 4.28 step 3 in CMSSW_12_0_X_2021-07-16-1100

Begin processing the 20th record. Run 172802, Event 460861990, LumiSection 551 on stream 3 at 16-Jul-2021 18:36:12.449 CEST
%MSG-w BSFitter:  AlcaBeamMonitor:AlcaBeamMonitor@endLumi  16-Jul-2021 18:36:15 CEST Run: 172802 Lumi: 551
need at least 150 tracks to run beamline fitter.
%MSG
cmsRun: /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/22d893ec6026b2e3d97e97225270bf20/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/src/FWCore/MessageService/plugins/MessageLogger.cc:469: void edm::service::MessageLogger::unEstablishModule(const edm::ModuleCallingContext&, const char*): Assertion `tran == GlobalContext::Transition::kBeginProcessBlock || tran == GlobalContext::Transition::kAccessInputProcessBlock || tran == GlobalContext::Transition::kEndProcessBlock' failed.

Thread 5 (Thread 0x2b2dff000700 (LWP 360)):
#2  0x00002b2dbc153590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b2db628154d in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00002b2db627ceb6 in _L_lock_941 () from /lib64/libpthread.so.0
#6  0x00002b2db627cdaf in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00002b2dbb90cc7b in dqm::implementation::DQMStore::leaveLumi(unsigned int, unsigned int, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libDQMServicesCore.so
#8  0x00002b2e104d2118 in DQMEDAnalyzer::endLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginDQMSiStripMonitorCluster.so
#9  0x00002b2db3d66a33 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndLuminosityBlock(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#10 0x00002b2db3d39460 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#11 0x00002b2db3c36002 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002b2db3c3624d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#13 0x00002b2db3c36430 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002b2db3c365aa in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so

Thread 4 (Thread 0x2b2dfe098700 (LWP 359)):
#2  0x00002b2dbc153590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b2db54c6d85 in ?? () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libz.so.1
#5  0x00002b2db54c70b5 in ?? () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libz.so.1
#6  0x00002b2db54c7c8c in ?? () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libz.so.1
#7  0x00002b2db54beee7 in ?? () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libz.so.1
#8  0x00002b2db54bf451 in deflate () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libz.so.1
#9  0x00002b2db4be2c15 in R__zipMultipleAlgorithm () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libCore.so
#10 0x00002b2db40f210a in TBasket::WriteBuffer() () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libTree.so
#11 0x00002b2db40ff617 in std::_Function_handler<void (), ROOT::Internal::TBranchIMTHelper::Run<TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*)::{lambda()#1}>(TBranch::WriteBasketImpl(TBasket*, int, ROOT::Internal::TBranchIMTHelper*)::{lambda()#1} const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libTree.so
#12 0x00002b2db6868b2f in tbb::detail::d1::function_task<std::function<void ()> >::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libImt.so

Thread 3 (Thread 0x2b2dfd697700 (LWP 358)):
#2  0x00002b2dbc153590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b2db628154d in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00002b2db627ceb6 in _L_lock_941 () from /lib64/libpthread.so.0
#6  0x00002b2db627cdaf in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00002b2dbb90cc7b in dqm::implementation::DQMStore::leaveLumi(unsigned int, unsigned int, unsigned long) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libDQMServicesCore.so
#8  0x00002b2e13086fd8 in DQMEDAnalyzer::endLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginDQMSiStripMonitorTrack.so
#9  0x00002b2db3d66a33 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndLuminosityBlock(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#10 0x00002b2db3d39460 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#11 0x00002b2db3c36002 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002b2db3c3624d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#13 0x00002b2db3c36430 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002b2db3c365aa in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so

Thread 1 (Thread 0x2b2db8609a40 (LWP 32566)):
#3  0x00002b2dbc15795b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b2db64c63d7 in raise () from /lib64/libc.so.6
#6  0x00002b2db64c7ac8 in abort () from /lib64/libc.so.6
#7  0x00002b2db64bf1a6 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00002b2db64bf252 in __assert_fail () from /lib64/libc.so.6
#9  0x00002b2db86d4aaf in edm::service::MessageLogger::unEstablishModule(edm::ModuleCallingContext const&, char const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginFWCoreMessageServicePlugins.so
#10 0x00002b2db3c36033 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#11 0x00002b2db3c3624d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002b2db3c36430 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#13 0x00002b2db3c365aa in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002b2db5473d0b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (this=0x2b2db93ef300, t=0x2b2ee68a5a00, waiter=...) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.h:396
#15 0x00002b2db54707e5 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x2b2db93ef300) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.cpp:178
#16 tbb::detail::r1::task_dispatcher::execute_and_wait (t=0x0, wait_ctx=..., w_ctx=...) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.cpp:168
#17 0x00002b2db68689cd in ROOT::Experimental::TTaskGroup::Wait() () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libImt.so
#18 0x00002b2db4176a83 in TTree::Fill() () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libTree.so
#19 0x00002b2e227892cf in (anonymous namespace)::TreeHelper<TProfile2D>::doFill(dqm::legacy::MonitorElement*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginDQMServicesFwkIOPlugins.so
#20 0x00002b2e22787a58 in DQMRootOutputModule::writeLuminosityBlock(edm::LuminosityBlockForOutput const&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginDQMServicesFwkIOPlugins.so
#21 0x00002b2db3d5f380 in edm::one::OutputModuleBase::doWriteLuminosityBlock(edm::LuminosityBlockPrincipal const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#22 0x00002b2db3c991a8 in void edm::SerialTaskQueueChain::actionToRun<edm::OutputModuleCommunicatorT<edm::one::OutputModuleBase>::writeLumiAsync(edm::WaitingTaskHolder, edm::LuminosityBlockPrincipal const&, edm::ProcessContext const*, edm::ActivityRegistry*)::{lambda()#1}&>(edm::OutputModuleCommunicatorT<edm::one::OutputModuleBase>::writeLumiAsync(edm::WaitingTaskHolder, edm::LuminosityBlockPrincipal const&, edm::ProcessContext const*, edm::ActivityRegistry*)::{lambda()#1}&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#23 0x00002b2db3c99551 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::OutputModuleCommunicatorT<edm::one::OutputModuleBase>::writeLumiAsync(edm::WaitingTaskHolder, edm::LuminosityBlockPrincipal const&, edm::ProcessContext const*, edm::ActivityRegistry*)::{lambda()#1}>(tbb::detail::d1::task_group&, edm::OutputModuleCommunicatorT<edm::one::OutputModuleBase>::writeLumiAsync(edm::WaitingTaskHolder, edm::LuminosityBlockPrincipal const&, edm::ProcessContext const*, edm::ActivityRegistry*)::{lambda()#1}&&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#24 0x00002b2db3a112c9 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so
#25 0x00002b2db5473d0b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (this=0x2b2db93ef300, t=0x2b2ee682be00, waiter=...) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.h:396
#26 0x00002b2db54707e5 in tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x2b2db93ef300) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.cpp:178
#27 tbb::detail::r1::task_dispatcher::execute_and_wait (t=0x0, wait_ctx=..., w_ctx=...) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/task_dispatcher.cpp:168
#28 0x00002b2db3c0e85f in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so

Current Modules:
Module: SegmentTrackAnalyzer:staMuonSegmentAnalyzer (crashed)
Module: SiStripMonitorCluster:SiStripCalZeroBiasMonitorCluster
Module: none
Module: SiStripMonitorTrack:SiStripMonitorTrackIB

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_0_X_2021-07-16-1100/pyRelValMatrixLogs/run/4.28_HighMet2011A+HighMet2011A+HLTDSKIM+RECODR1reHLT+HARVESTDR1reHLT/step3_HighMet2011A+HighMet2011A+HLTDSKIM+RECODR1reHLT+HARVESTDR1reHLT.log#/

makortel · 2021-07-17T01:42:19Z

assign dqm

cmsbuild · 2021-07-17T01:42:40Z

New categories assigned: dqm

@jfernan2,@kmaeshima,@rvenditti,@andrius-k,@ErnestaP,@ahmad3213 you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel · 2021-07-17T01:49:36Z

Another one also in DQMServices/Demo unit test in CMSSW_12_0_X_2021-07-16-1100

makortel · 2021-07-17T02:04:13Z

#34532 protects all TTree::Fill() calls with tbb::this_task_arena::isolate() in DQMRootOutputModule.

makortel · 2021-07-17T02:32:51Z

On the other hand, these failures started to appear after #34506, that added this assertion, was merged. @wddgit, could also you take a look and comment if the assertion failures make sense in this context?

dan131riley · 2021-07-17T15:40:55Z

@makortel I had thought that now ROOT had put an arena call around all of its internal uses of TBB. Guess that isn't the case :(.

ROOT has isolation calls around the TThreadExecutor parallel for/reduce/map/etc. However, the TTree (and RNTuple) IMT compression uses TTaskGroup, and the way that is structured it isn't possible to do the arena isolation internally to the TTaskGroup, and doing it externally would mean breaking the TTaskGroup abstraction--so it ends up being left to the caller to decide what to do. During the RNTuple work there was some discussion with @jblomer about this but I don't think we reached any conclusion on whether it would be possible to use one of the TThreadExecutor operations instead of TTaskGroup.

jfernan2 · 2021-07-19T13:25:20Z

+1
For the records

wddgit · 2021-07-19T15:13:29Z

I am looking at this now carefully trying to understand if I should remove those new asserts.

wddgit · 2021-07-19T22:05:10Z

See pull request #34557. Hopefully this handles this better.

cmsbuild added core-pending pending-signatures labels Jul 16, 2021

makortel mentioned this issue Jul 16, 2021

use integer/floor division in IgProfInfo.py #34517

Merged

cmsbuild added the dqm-pending label Jul 17, 2021

makortel mentioned this issue Jul 17, 2021

Protect all TTree::Fill() calls with tbb::this_task_arena::isolate() in DQMRootOutputModule #34532

Merged

makortel mentioned this issue Jul 17, 2021

Protect all TTree::Fill() calls with tbb::this_task_arena::isolate() in DQMRootOutputModule cms-sw/framework-team#194

Closed

qliphy mentioned this issue Jul 18, 2021

Run4-hgx289 Update the validation scripts for HGCal to test the V16 version #34522

Merged

jfernan2 mentioned this issue Jul 19, 2021

Configurations and analyzer module to unpack and analyze B904 GEM trigger data #34478

Merged

qliphy mentioned this issue Jul 19, 2021

PPS geometry for Run3 #34499

Merged

cmsbuild closed this as completed in #34532 Jul 19, 2021

cmsbuild added dqm-approved and removed dqm-pending labels Jul 19, 2021

qliphy mentioned this issue Jul 19, 2021

Build files cleaning #34544

Merged

jfernan2 mentioned this issue Jul 19, 2021

Clean/rename FromMultiCl HGCal egamma collections #34439

Merged

wddgit mentioned this issue Jul 19, 2021

Improve MessageLogger context handling #34557

Merged

jpata mentioned this issue Jul 21, 2021

Re-enable suppression of CSC digis without matching pretriggers and triggers #34435

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assertion failure in edm::service::MessageLogger::unEstablishModule() #34520

Assertion failure in edm::service::MessageLogger::unEstablishModule() #34520

makortel commented Jul 16, 2021

makortel commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

makortel commented Jul 16, 2021

Dr15Jones commented Jul 16, 2021

Dr15Jones commented Jul 16, 2021

makortel commented Jul 16, 2021

makortel commented Jul 17, 2021

makortel commented Jul 17, 2021

cmsbuild commented Jul 17, 2021

makortel commented Jul 17, 2021

makortel commented Jul 17, 2021

makortel commented Jul 17, 2021

dan131riley commented Jul 17, 2021

jfernan2 commented Jul 19, 2021

wddgit commented Jul 19, 2021

wddgit commented Jul 19, 2021

Assertion failure in edm::service::MessageLogger::unEstablishModule() #34520

Assertion failure in edm::service::MessageLogger::unEstablishModule() #34520

Comments

makortel commented Jul 16, 2021

makortel commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

makortel commented Jul 16, 2021

Dr15Jones commented Jul 16, 2021

Dr15Jones commented Jul 16, 2021

makortel commented Jul 16, 2021

makortel commented Jul 17, 2021

makortel commented Jul 17, 2021

cmsbuild commented Jul 17, 2021

makortel commented Jul 17, 2021

makortel commented Jul 17, 2021

makortel commented Jul 17, 2021

dan131riley commented Jul 17, 2021

jfernan2 commented Jul 19, 2021

wddgit commented Jul 19, 2021

wddgit commented Jul 19, 2021