msgtime:2023-05-07 13:29:21 doc_type:cmsswlog date:2023-05-07T11:29:21.709Z run:367112 host:fu-c2b03-32-01 pid:1770918 doctype:stacktrace severity:FATAL severityVal:4 instance:global lexicalId:549852445 message:A fatal system signal has occurred: segmentation violation The following is the call stack containing the origin of the signal. Sun May 7 13:28:58 CEST 2023 Thread 44 (Thread 0x7f59e29ff700 (LWP 1772297) "cmsRun"): #0 0x00007f5b9e3bacd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0 #1 0x00007f5b9e3badc8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007f5b80d6b0a2 in ?? () from /lib64/libcuda.so.1 #3 0x00007f5b80d80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f5a46bff700 (LWP 1772274) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0d7b58 in usleep () from /lib64/libc.so.6 #2 0x00007f5b92f8cf5c in FedRawDataInputSource::readSupervisor() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #3 0x00007f5b9ea3e9b4 in std::execute_native_thread_routine (__p=0x7f5a587e54e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f5a4a7fe700 (LWP 1772202) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b40eb12d5 in RectangularEtaPhiTrackingRegion::checkTracks(std::vector > const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkTrackingRegions.so #5 0x00007f5b3dc47ec0 in TrackSelectorByRegion::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerFinalTrackSelectorsPlugins.so #6 0x00007f5ba0b0d9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5ba0b075d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #11 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f6173200, waiter=..., this=0x7f5b99014580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #12 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #13 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #14 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #15 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #17 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #18 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f5a4b5ff700 (LWP 1772201) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b9e143c59 in __memset_avx2_unaligned_erms () from /lib64/libc.so.6 #5 0x00007f5b96f28abe in FEDRawData::resize(unsigned long) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsFEDRawData.so #6 0x00007f5b92f7fe53 in FedRawDataInputSource::fillFEDRawDataCollection(FEDRawDataCollection&, bool&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #7 0x00007f5b92f82201 in FedRawDataInputSource::read(edm::EventPrincipal&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #8 0x00007f5ba0a7e5a7 in void edm::callWithTryCatchAndPrint(std::function, char const*, bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a7ea0b in edm::InputSource::readEvent(edm::EventPrincipal&, edm::StreamContext&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a297e6 in edm::EventProcessor::readEvent(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0a29c3b in edm::EventProcessor::readNextEventForStream(edm::WaitingTaskHolder const&, unsigned int, edm::LuminosityBlockProcessingStatus&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0a2bb16 in edm::EventProcessor::handleNextEventForStreamAsync(edm::WaitingTaskHolder, unsigned int)::{lambda()#1}::operator()() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a4c548 in edm::SerialTaskQueue::QueuedTask(tbb::detail::d1::task_group&, edm::EventProcessor::handleNextEventForStreamAsync(edm::WaitingTaskHolder, unsigned int)::{lambda()#1}&&)::{lambda()#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba07eb099 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a4bd64d00, waiter=..., this=0x7f5b99014500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f5a4c9ff700 (LWP 1772200) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b8c714818 in HelixArbitraryPlaneCrossing2Order::pathLength(Plane const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #5 0x00007f5b8c717119 in HelixArbitraryPlaneCrossing::pathLength(Plane const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #6 0x00007f5b8c713454 in AnalyticalPropagator::propagateWithHelixCrossing(HelixPlaneCrossing&, Plane const&, float, Point3DBase&, Vector3DBase&, double&) const [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #7 0x00007f5b8c71e6ea in AnalyticalPropagator::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #8 0x00007f5b8c713365 in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #9 0x00007f5b40e51a72 in PropagatorWithMaterial::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #10 0x00007f5aa0c6d70d in BaseCkfTrajectoryBuilder::seedMeasurements(TrajectorySeed const&, TempTrajectory&, bool) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #11 0x00007f5aa0c6dfd2 in BaseCkfTrajectoryBuilder::createStartingTrajectory(TrajectorySeed const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #12 0x00007f5aa0cbb7a3 in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector >&, unsigned int&, TrajectoryFilter const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so #13 0x00007f5aa0c75263 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #14 0x00007f5aa0c76ceb in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #15 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #20 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f5db1a00, waiter=..., this=0x7f5b99014480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #24 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #26 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #27 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f5a4d7ff700 (LWP 1772199) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5ba07ec238 in edm::WaitingTaskList::add(edm::WaitingTaskHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #5 0x00007f5ba0aac04a in edm::PuttableProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5ba0a9a8e1 in edm::Principal::prefetchAsync(edm::WaitingTaskHolder, unsigned int, bool, edm::ServiceToken const&, edm::ModuleCallingContext const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5ba0afebb0 in edm::Worker::edPrefetchAsync(edm::WaitingTaskHolder, edm::ServiceToken const&, edm::Principal const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0b364f2 in void edm::Worker::prefetchAsync >(edm::WaitingTaskHolder, edm::ServiceToken const&, edm::ParentContext const&, edm::OccurrenceTraits::TransitionInfoType const&, edm::Transition) [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a96eff in void edm::Worker::doWorkAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a975d7 in edm::Path::runNextWorkerAsync(unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0a8ead1 in edm::Path::processOneOccurrenceAsync(edm::WaitingTaskHolder, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0ae860c in edm::StreamSchedule::processOneEventAsync(edm::WaitingTaskHolder, edm::EventTransitionInfo&, edm::ServiceToken const&, std::vector >, std::allocator > > >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0acc332 in edm::Schedule::processOneEventAsync(edm::WaitingTaskHolder, unsigned int, edm::EventTransitionInfo&, edm::ServiceToken const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba0a2ee1d in edm::waiting_task::detail::WaitingTaskChain >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder) [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5ba0a2c5de in edm::EventProcessor::processEventAsyncImpl(edm::WaitingTaskHolder, unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f5ba0a4fc2c in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a4dba8200, waiter=..., this=0x7f5b99014400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f5a4e5fe700 (LWP 1772198) "cmsRun"): #0 0x00007f5b9e0d4a71 in poll () from /lib64/libc.so.6 #1 0x00007f5b96fd846f in full_read.constprop () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f5b96fa3b6c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f5b96fa433b in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #4 #5 0x00007f5b40ed8777 in void std::__introsort_loop<__gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #6 0x00007f5b40edabf5 in RecHitsSortedInPhi::RecHitsSortedInPhi(std::vector > const&, Point3DBase const&, DetLayer const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #7 0x00007f5b40eded11 in LayerHitMapCache::operator()(SeedingLayerSetsHits::SeedingLayer const&, TrackingRegion const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #8 0x00007f5b40edf07a in HitPairGeneratorFromLayerPair::doublets(TrackingRegion const&, edm::Event const&, edm::EventSetup const&, SeedingLayerSetsHits::SeedingLayer const&, SeedingLayerSetsHits::SeedingLayer const&, LayerHitMapCache&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #9 0x00007f5a971b9d97 in (anonymous namespace)::Impl<(anonymous namespace)::DoNothing, (anonymous namespace)::ImplIntermediateHitDoublets, (anonymous namespace)::RegionsLayersSeparate>::produce(bool, edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkHitPairsPlugins.so #10 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f64e2600, waiter=..., this=0x7f5b99014380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f5a4f1ff700 (LWP 1772197) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a97907b6e in KDTreeLinkerTrackHcal::searchLinks() () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFProducerPlugins_kdtrees.so #5 0x00007f5a979a405b in PFBlockAlgo::findBlocks() () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/libRecoParticleFlowPFProducer.so #6 0x00007f5a97a2e9b5 in PFBlockProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFProducerPlugins.so #7 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a4f534200, waiter=..., this=0x7f5b99014300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f5a501ff700 (LWP 1772196) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b3ea93403 in PixelCPEBase::setTheClu(PixelCPEBase::DetParam const&, PixelCPEBase::ClusterParam&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #5 0x00007f5b3ea93f38 in PixelClusterParameterEstimator::getParameters(SiPixelCluster const&, GeomDet const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #6 0x00007f5b4470f5be in TkClonerImpl::makeShared(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTransientTrackingRecHit.so #7 0x00007f5b4470d854 in SiPixelRecHit::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTransientTrackingRecHit.so #8 0x00007f5b448f9b57 in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector, std::allocator > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrackFitters.so #9 0x00007f5aa0c866c7 in TransientInitialStateEstimator::innerState(Trajectory const&, bool) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #10 0x00007f5aa0c77723 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #11 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #16 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f6279a00, waiter=..., this=0x7f5b99014280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f5a50fff700 (LWP 1772195) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b3eab2907 in PixelCPEGeneric::localPosition(PixelCPEBase::DetParam const&, PixelCPEBase::ClusterParam&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #5 0x00007f5b3ea93f62 in PixelClusterParameterEstimator::getParameters(SiPixelCluster const&, GeomDet const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #6 0x00007f5b4470f5be in TkClonerImpl::makeShared(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTransientTrackingRecHit.so #7 0x00007f5b4470d854 in SiPixelRecHit::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTransientTrackingRecHit.so #8 0x00007f5b448f9b57 in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector, std::allocator > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrackFitters.so #9 0x00007f5aa0c866c7 in TransientInitialStateEstimator::innerState(Trajectory const&, bool) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #10 0x00007f5aa0c77723 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #11 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #16 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f6554c00, waiter=..., this=0x7f5b99014200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f5a521ff700 (LWP 1772194) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b40e4f33a in EnergyLossUpdator::compute(TrajectoryStateOnSurface const&, PropagationDirection, materialEffect::Effect&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #5 0x00007f5b40e4f919 in MaterialEffectsUpdator::updateStateInPlace(TrajectoryStateOnSurface&, PropagationDirection) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #6 0x00007f5b40e51b1d in PropagatorWithMaterial::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #7 0x00007f5a9f514e7c in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #8 0x00007f5a9f515a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #9 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #14 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f5e23f00, waiter=..., this=0x7f5b99014180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f5a52ffe700 (LWP 1772193) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a97c0f16a in Basic2DGenericPFlowClusterizer::growPFClusters(reco::PFCluster const&, std::vector > const&, unsigned int, unsigned int, double, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #5 0x00007f5a97c0fc1e in Basic2DGenericPFlowClusterizer::buildClusters(std::vector > const&, std::vector > const&, std::vector >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #6 0x00007f5a97c33689 in PFClusterProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #7 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a4dbd1400, waiter=..., this=0x7f5b99014100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f5a539ff700 (LWP 1772192) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b413d8db9 in edmNew::DetSetVector::TSFastFiller::~TSFastFiller() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so #5 0x00007f5b413d04c7 in StMeasurementDetSet::getDetSet(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so #6 0x00007f5b413d89ab in TkStripMeasurementDet::simpleRecHits(TrajectoryStateOnSurface const&, MeasurementTrackerEvent const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so #7 0x00007f5b413d33a0 in TkGluedMeasurementDet::measurements(TrajectoryStateOnSurface const&, MeasurementEstimator const&, MeasurementTrackerEvent const&, tracking::TempMeasurements&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so #8 0x00007f5b4133d492 in LayerMeasurements::measurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMeasurementDet.so #9 0x00007f5aa0c286e5 in MuonCkfTrajectoryBuilder::collectMeasurement(DetLayer const*, std::vector > const&, TrajectoryStateOnSurface const&, std::vector >&, int&, Propagator const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonL3TrackFinder.so #10 0x00007f5aa0c2a2ae in MuonCkfTrajectoryBuilder::findCompatibleMeasurements(TrajectorySeed const&, TempTrajectory const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonL3TrackFinder.so #11 0x00007f5aa0c7fadb in CkfTrajectoryBuilder::limitedCandidates(std::shared_ptr const&, std::vector >&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #12 0x00007f5aa0c810d7 in CkfTrajectoryBuilder::limitedCandidates(TrajectorySeed const&, TempTrajectory&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #13 0x00007f5aa0c812cf in CkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector >&, unsigned int&, TrajectoryFilter const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #14 0x00007f5aa0c75263 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #15 0x00007f5aa0c76ceb in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #16 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #21 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a4634c100, waiter=..., this=0x7f5b99014080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #28 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f5a55ffe700 (LWP 1772191) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b4be3747b in edmNew::DetSetVector::FastFiller::FastFiller(edmNew::DetSetVector&, unsigned int, bool) [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #5 0x00007f5b4be2a94d in SiPixelDigisClustersFromSoAT::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const::{lambda(unsigned int)#1}::operator()(unsigned int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #6 0x00007f5b4be2b9cd in SiPixelDigisClustersFromSoAT::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #7 0x00007f5ba0b0d9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0b075d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f5e38600, waiter=..., this=0x7f5b99014000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99014000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f5a54bfc700 (LWP 1772190) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b3ed8de6c in SimpleNavigableLayer::wellInside(FreeTrajectoryState const&, PropagationDirection, ForwardDetLayer const*, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkNavigationPlugins.so #5 0x00007f5b3ed8e1bc in SimpleNavigableLayer::wellInside(FreeTrajectoryState const&, PropagationDirection, std::vector > const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkNavigationPlugins.so #6 0x00007f5b3ed8e304 in SimpleBarrelNavigableLayer::nextLayers(FreeTrajectoryState const&, PropagationDirection) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkNavigationPlugins.so #7 0x00007f5aa0c2a1d1 in MuonCkfTrajectoryBuilder::findCompatibleMeasurements(TrajectorySeed const&, TempTrajectory const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonL3TrackFinder.so #8 0x00007f5aa0c7fadb in CkfTrajectoryBuilder::limitedCandidates(std::shared_ptr const&, std::vector >&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #9 0x00007f5aa0c810d7 in CkfTrajectoryBuilder::limitedCandidates(TrajectorySeed const&, TempTrajectory&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #10 0x00007f5aa0c812cf in CkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector >&, unsigned int&, TrajectoryFilter const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #11 0x00007f5aa0c75263 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #12 0x00007f5aa0c76ceb in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #13 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #18 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a4cdb0f00, waiter=..., this=0x7f5b99013f80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #19 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013f80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #20 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #21 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #22 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #24 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #25 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f5a555fd700 (LWP 1772189) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5ab2c207d8 in VirtualJetProducer::output(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #5 0x00007f5ab2c17a82 in VirtualJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #6 0x00007f5ab2bd42dd in FastjetJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #7 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f6652d00, waiter=..., this=0x7f5b99013f00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013f00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f5a569ff700 (LWP 1772188) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a9790726c in KDTreeLinkerTrackHcal::insertFieldClusterElt(reco::PFBlockElement*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFProducerPlugins_kdtrees.so #5 0x00007f5a979a59ca in PFBlockAlgo::buildElements(edm::Event const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/libRecoParticleFlowPFProducer.so #6 0x00007f5a97a2e9a8 in PFBlockProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFProducerPlugins.so #7 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f5c1ad00, waiter=..., this=0x7f5b99013e80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013e80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f5a8fc51700 (LWP 1772170) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b92f7768e in evf::FastMonitoringService::snapshotRunner() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #3 0x00007f5b9ea3e9b4 in std::execute_native_thread_routine (__p=0x7f5a589e8d40) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f5aa03ff700 (LWP 1772054) "cmsRun"): #0 0x00007f5b9e3b83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5b9ea38e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f5b32019acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f5b3201a0ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f5b32016c68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f5b2ee404c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f5aa14d2700 (LWP 1772046) "cmsRun"): #0 0x00007f5b9e3b83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5b9ea38e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f5b32019acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f5b3201a0ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f5b32016c68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f5b2ee404c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f5aa1cd3700 (LWP 1772042) "cmsRun"): #0 0x00007f5b9e3b83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5b9ea38e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f5b32019acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f5b3201a0ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f5b32016c68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f5b2ee404c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f5ac71fe700 (LWP 1772023) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 je_je_sdallocx_noflags (ptr=0x7f5a2b62cef0, size=16) at src/jemalloc.c:3950 #5 0x00007f5b9f6e633e in sizedDeleteImpl (size=, ptr=) at src/jemalloc_cpp.cpp:195 #6 operator delete (ptr=, size=) at src/jemalloc_cpp.cpp:200 #7 0x00007f5b3e88ba2e in KalmanVertexUpdator<5u>::update(CachingVertex<5u> const&, ReferenceCountingPointer >, float, int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoVertexKalmanVertexFit.so #8 0x00007f5b3e88bb3b in KalmanVertexUpdator<5u>::add(CachingVertex<5u> const&, ReferenceCountingPointer >) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoVertexKalmanVertexFit.so #9 0x00007f5b3e8af5af in AdaptiveVertexFitter::fit(std::vector >, std::allocator > > > const&, VertexState const&, bool) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoVertexAdaptiveVertexFit.so #10 0x00007f5b3e8b1791 in AdaptiveVertexFitter::vertex(std::vector > const&, Point3DBase const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoVertexAdaptiveVertexFit.so #11 0x00007f5a8ebe859b in TemplatedInclusiveVertexFinder >, reco::Vertex>::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoVertexAdaptiveVertexFinderPlugins.so #12 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #17 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a4bdcfe00, waiter=..., this=0x7f5b99013e00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013e00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f5ac7bff700 (LWP 1772022) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b42514ef7 in HcalDetId::HcalDetId(DetId const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryHcalTowerAlgo.so #5 0x00007f5b4251541d in HcalGeometry::getGeometry(DetId const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryHcalTowerAlgo.so #6 0x00007f5b43a9d36d in CaloGeometry::getGeometry(DetId const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryCaloGeometry.so #7 0x00007f5ab2b19aee in EgammaHcalIsolation::goodHitEnergy(float, float, HBHERecHit const&, int, int, int, int, double (*)(double const&)) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaIsolationAlgos.so #8 0x00007f5ab2b19fbd in EgammaHcalIsolation::getHcalSum(Point3DBase const&, int, int, int, int, double (*)(double const&)) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaIsolationAlgos.so #9 0x00007f5a97758a4b in EgammaHLTHcalVarProducerFromRecHit::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoEgammaEgammaHLTProducersPlugins.so #10 0x00007f5ba0b0d9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0b075d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a4f562400, waiter=..., this=0x7f5b99013d00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013d00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fef980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fef980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f5ac8bfc700 (LWP 1772014) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b60a68e30 in reco::CompositePtrCandidate::~CompositePtrCandidate() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsCandidate.so #5 0x00007f5b4496a73e in void std::_Destroy_aux::__destroy(reco::RecoTauPiZero*, reco::RecoTauPiZero*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsTauReco.so #6 0x00007f5b44964a0a in std::vector >::~vector() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsTauReco.so #7 0x00007f5b44964a21 in edm::AtomicPtrCache > >::~AtomicPtrCache() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsTauReco.so #8 0x00007f5b4495bdc4 in reco::PFTau::~PFTau() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsTauReco.so #9 0x00007f5b4495bf08 in reco::PFTau::~PFTau() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsTauReco.so #10 0x00007f5a8fe6d73e in RecoTauProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTauTagRecoTauPlugins.so #11 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #16 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a49d38a00, waiter=..., this=0x7f5b99013c80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013c80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefd00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefd00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f5ac95fd700 (LWP 1772009) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b80d3da15 in ?? () from /lib64/libcuda.so.1 #5 0x00007f5b80e04acb in ?? () from /lib64/libcuda.so.1 #6 0x00007f5b80e0b513 in ?? () from /lib64/libcuda.so.1 #7 0x00007f5b80d50bdf in ?? () from /lib64/libcuda.so.1 #8 0x00007f5b80d53b97 in ?? () from /lib64/libcuda.so.1 #9 0x00007f5b80f96f2c in ?? () from /lib64/libcuda.so.1 #10 0x00007f5b80decf20 in ?? () from /lib64/libcuda.so.1 #11 0x00007f5b839a0b3e in ?? () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #12 0x00007f5b8397749b in ?? () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #13 0x00007f5b839c5151 in cudaMemsetAsync () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #14 0x00007f5b4bdcbbd0 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #15 0x00007f5b4be3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #16 0x00007f5b4be2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #17 0x00007f5ba0b1dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f5ba0b21b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f5ba0afe6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f5ba0afe846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f5ba0a959ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f5ba0a027cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #23 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a51574c00, waiter=..., this=0x7f5b99013b80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #24 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013b80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #25 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #26 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #27 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #28 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #29 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #30 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f5ac9ffe700 (LWP 1772003) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b41353fac in TrajectoryStateOnSurface (anonymous namespace)::lupdate<2u>(TrajectoryStateOnSurface const&, TrackingRecHit const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #5 0x00007f5b4135cfae in KFUpdator::update(TrajectoryStateOnSurface const&, TrackingRecHit const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #6 0x00007f5a9f515337 in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #7 0x00007f5a9f515a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #8 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f5edb400, waiter=..., this=0x7f5b99013a80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013a80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefc00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefc00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f5acb9fe700 (LWP 1772001) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b424c863e in EcalBarrelGeometry::getGeometryRawPtr(unsigned int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryEcalAlgo.so #5 0x00007f5b43a96c28 in CaloSubdetectorGeometry::cellGeomPtr(unsigned int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryCaloGeometry.so #6 0x00007f5b43a99fd2 in CaloSubdetectorGeometry::getGeometry(DetId const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryCaloGeometry.so #7 0x00007f5a97c3bb49 in PFEcalBarrelRecHitCreator::importRecHits(std::unique_ptr >, std::default_delete > > >&, std::unique_ptr >, std::default_delete > > >&, edm::Event const&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #8 0x00007f5a97c35165 in PFRecHitProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #9 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a027cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f6471e00, waiter=..., this=0x7f5b99013980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefc80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefc80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f5aca9ff700 (LWP 1772000) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b9e62ab96 in atanf32 () from /lib64/libm.so.6 #5 0x00007f5b9e626119 in __atan2f_finite () from /lib64/libm.so.6 #6 0x00007f5a972817bc in TrajSeedMatcher::processSeed(TrajectorySeed const&, Point3DBase const&, float, TrajectoryStateOnSurface const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaElectronAlgos.so #7 0x00007f5a97282dc1 in TrajSeedMatcher::operator()(Point3DBase const&, float) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaElectronAlgos.so #8 0x00007f5a9730b49a in ElectronNHitSeedProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoEgammaEgammaElectronProducersPlugins.so #9 0x00007f5ba0b0d9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0b075d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #14 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f64ea000, waiter=..., this=0x7f5b99013900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f5acc3ff700 (LWP 1771983) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b413d658d in TkPixelMeasurementDet::compHits(TrajectoryStateOnSurface const&, MeasurementTrackerEvent const&, float, float) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so #5 0x00007f5b413d6be7 in TkPixelMeasurementDet::measurements(TrajectoryStateOnSurface const&, MeasurementEstimator const&, MeasurementTrackerEvent const&, tracking::TempMeasurements&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so #6 0x00007f5b4133d492 in LayerMeasurements::measurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMeasurementDet.so #7 0x00007f5aa0c286e5 in MuonCkfTrajectoryBuilder::collectMeasurement(DetLayer const*, std::vector > const&, TrajectoryStateOnSurface const&, std::vector >&, int&, Propagator const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonL3TrackFinder.so #8 0x00007f5aa0c2a2ae in MuonCkfTrajectoryBuilder::findCompatibleMeasurements(TrajectorySeed const&, TempTrajectory const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonL3TrackFinder.so #9 0x00007f5aa0c7fadb in CkfTrajectoryBuilder::limitedCandidates(std::shared_ptr const&, std::vector >&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #10 0x00007f5aa0c810d7 in CkfTrajectoryBuilder::limitedCandidates(TrajectorySeed const&, TempTrajectory&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #11 0x00007f5aa0c812cf in CkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector >&, unsigned int&, TrajectoryFilter const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #12 0x00007f5aa0c75263 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #13 0x00007f5aa0c76ceb in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #14 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #19 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f5ceb900, waiter=..., this=0x7f5b99013880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #20 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #21 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #22 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #23 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #24 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #25 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #26 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f5acd1fe700 (LWP 1771955) "cmsRun"): #0 0x00007f5b9e3b83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5b9ea38e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f5b92f892d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f5b9ea3e9b4 in std::execute_native_thread_routine (__p=0x7f5b90038620) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f5acdbff700 (LWP 1771953) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b3dff6bd2 in std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, std::vector > >, std::_Select1st, std::allocator > const, std::vector > > >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::vector > > > >::find(std::__cxx11::basic_string, std::allocator > const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libL1TriggerL1TGlobal.so #5 0x00007f5b3dffb54b in l1t::GlobalScales::getLUT_Pt(std::__cxx11::basic_string, std::allocator > const&, int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libL1TriggerL1TGlobal.so #6 0x00007f5b3dfdea73 in l1t::CorrCondition::evaluateCondition(int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libL1TriggerL1TGlobal.so #7 0x00007f5b3dff3837 in l1t::GlobalBoard::runGTL(edm::Event const&, edm::EventSetup const&, TriggerMenu const*, bool, int, std::unique_ptr >&, unsigned int, int, int, int, int, int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libL1TriggerL1TGlobal.so #8 0x00007f5b3e0f17ce in L1TGlobalProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginL1TriggerL1TGlobalPlugins.so #9 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #14 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a49dc2400, waiter=..., this=0x7f5b99013800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f5ace9fe700 (LWP 1771944) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5ba0afefa3 in edm::Worker::esPrefetchAsync(edm::WaitingTaskHolder, edm::EventSetupImpl const&, edm::Transition, edm::ServiceToken const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5ba0b3646a in void edm::Worker::prefetchAsync >(edm::WaitingTaskHolder, edm::ServiceToken const&, edm::ParentContext const&, edm::OccurrenceTraits::TransitionInfoType const&, edm::Transition) [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5ba0a96eff in void edm::Worker::doWorkAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5ba0a975d7 in edm::Path::runNextWorkerAsync(unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0a97a03 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a97b26 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #11 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a46326f00, waiter=..., this=0x7f5b99013d80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #12 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013d80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #13 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #14 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #15 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefd80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefd80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #17 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #18 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f5acf5ff700 (LWP 1771943) "cmsRun"): #0 0x00007f5b9e3b83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5b9ea38e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f5b92f892d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f5b9ea3e9b4 in std::execute_native_thread_routine (__p=0x7f5b900384d0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f5aecbfc700 (LWP 1771936) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b44693833 in reco::LeafCandidate::pdgId() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsParticleFlowCandidate.so #5 0x00007f5ab2ac693b in reco::makeSpecific(std::vector, std::allocator > > const&, reco::PFJet::Specific*, edm::ValueMap const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoJetsJetProducers.so #6 0x00007f5ab2ac71de in reco::writeSpecific(reco::PFJet&, ROOT::Math::LorentzVector > const&, ROOT::Math::PositionVector3D, ROOT::Math::DefaultCoordinateSystemTag> const&, std::vector, std::allocator > > const&, edm::ValueMap const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoJetsJetProducers.so #7 0x00007f5ab2c1f9fc in VirtualJetProducer::output(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #8 0x00007f5ab2c17a82 in VirtualJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #9 0x00007f5ab2bd42dd in FastjetJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #10 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f6626200, waiter=..., this=0x7f5b99013b00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013b00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98ff0000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98ff0000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f5ad01fb700 (LWP 1771933) "cmsRun"): #0 0x00007f5b9e3b83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5b9ea38e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f5b92f892d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f5b9ea3e9b4 in std::execute_native_thread_routine (__p=0x7f5b90038410) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f5b055fd700 (LWP 1771919) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 cache_bin_alloc_impl (adjust_low_water=false, success=, bin=) at include/jemalloc/internal/cache_bin.h:363 #5 cache_bin_alloc_easy (success=, bin=) at include/jemalloc/internal/cache_bin.h:399 #6 imalloc_fastpath (fallback_alloc=0x7f5b9f6e5eb0 (std::size_t)>, size=48) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:326 #7 newImpl (size=48) at src/jemalloc_cpp.cpp:109 #8 operator new (size=48) at src/jemalloc_cpp.cpp:114 #9 0x00007f5b40e4f15e in CombinedMaterialEffectsUpdator::clone() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #10 0x00007f5b40e50e5e in PropagatorWithMaterial::clone() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #11 0x00007f5b448f1062 in KFTrajectorySmoother::clone() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrackFitters.so #12 0x00007f5b3ffdce47 in (anonymous namespace)::KFFittingSmoother::clone() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginTrackingToolsTrackFittersPlugins.so #13 0x00007f5a9fb94927 in TrackProducerAlgorithm::runWithCandidate(TrackingGeometry const*, MagneticField const*, std::vector > const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector::AlgoProduct, std::allocator::AlgoProduct> >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTrackProducerPlugins.so #14 0x00007f5a9fb932c2 in TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTrackProducerPlugins.so #15 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #20 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a5154e300, waiter=..., this=0x7f5b99013c00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013c00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #24 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98feff00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98feff00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #26 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #27 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f5b05ffe700 (LWP 1771918) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5ba07ec34c in edm::WaitingTaskList::add(edm::WaitingTaskHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #5 0x00007f5ba0a392d1 in void edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5ba0a39b35 in void edm::Path::runAllModulesAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5ba0a39dce in edm::StreamSchedule::processOneStreamAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0a45ce9 in tbb::detail::d1::enqueue_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f5b02000, waiter=..., this=0x7f5b99013a00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #10 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013a00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #11 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #12 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #13 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98feff80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98feff80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #15 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #16 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f5b069ff700 (LWP 1771914) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 fastjet::PseudoJet::py (this=0x7f5a36940d58) at ./../include/fastjet/PseudoJet.hh:119 #5 fastjet::PseudoJet::_finish_init (this=this@entry=0x7f5a36940d58) at PseudoJet.cc:122 #6 0x00007f5acde579cb in fastjet::PseudoJet::PseudoJet (this=0x7f5a36940d58) at ./../include/fastjet/PseudoJet.hh:82 #7 std::_Construct (__p=0x7f5a36940d58) at ./../include/fastjet/PseudoJet.hh:82 #8 std::__uninitialized_default_n_1::__uninit_default_n (__n=19, __first=0x7f5a36940d00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_0_pre3-el8_amd64_gcc11/build/CMSSW_13_0_0_pre3-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/stl_uninitialized.h:579 #9 std::__uninitialized_default_n (__n=, __first=0x7f5a36940d00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_0_pre3-el8_amd64_gcc11/build/CMSSW_13_0_0_pre3-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/stl_uninitialized.h:640 #10 std::__uninitialized_default_n_a (__n=, __first=0x7f5a36940d00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_0_pre3-el8_amd64_gcc11/build/CMSSW_13_0_0_pre3-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/stl_uninitialized.h:704 #11 std::vector >::_M_default_initialize (__n=, this=0x7f5b069f8400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_0_pre3-el8_amd64_gcc11/build/CMSSW_13_0_0_pre3-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/stl_vector.h:1606 #12 std::vector >::vector (__a=..., __n=, this=0x7f5b069f8400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_0_pre3-el8_amd64_gcc11/build/CMSSW_13_0_0_pre3-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/stl_vector.h:512 #13 fastjet::objects_sorted_by_values (objects=..., values=...) at ./../include/fastjet/PseudoJet.hh:999 #14 0x00007f5acde56b6d in fastjet::sorted_by_pt (jets=...) at PseudoJet.cc:885 #15 0x00007f5ab2c1f98b in VirtualJetProducer::output(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #16 0x00007f5ab2c17a82 in VirtualJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #17 0x00007f5ab2bd42dd in FastjetJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #18 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #23 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f65ae500, waiter=..., this=0x7f5b99013780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #24 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #25 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #26 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #27 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98fefe80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #28 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98fefe80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #29 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #30 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f5b2bd7f700 (LWP 1771881) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5ba0aabfed in edm::PuttableProductResolver::prefetchAsync_(edm::WaitingTaskHolder, edm::Principal const&, bool, edm::ServiceToken const&, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5ba0a9a8e1 in edm::Principal::prefetchAsync(edm::WaitingTaskHolder, unsigned int, bool, edm::ServiceToken const&, edm::ModuleCallingContext const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5ba0afebb0 in edm::Worker::edPrefetchAsync(edm::WaitingTaskHolder, edm::ServiceToken const&, edm::Principal const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5ba0b364f2 in void edm::Worker::prefetchAsync >(edm::WaitingTaskHolder, edm::ServiceToken const&, edm::ParentContext const&, edm::OccurrenceTraits::TransitionInfoType const&, edm::Transition) [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0a96eff in void edm::Worker::doWorkAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a975d7 in edm::Path::runNextWorkerAsync(unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a8ead1 in edm::Path::processOneOccurrenceAsync(edm::WaitingTaskHolder, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0ae860c in edm::StreamSchedule::processOneEventAsync(edm::WaitingTaskHolder, edm::EventTransitionInfo&, edm::ServiceToken const&, std::vector >, std::allocator > > >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0acc332 in edm::Schedule::processOneEventAsync(edm::WaitingTaskHolder, unsigned int, edm::EventTransitionInfo&, edm::ServiceToken const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a2ee1d in edm::waiting_task::detail::WaitingTaskChain >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder) [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba0a2c5de in edm::EventProcessor::processEventAsyncImpl(edm::WaitingTaskHolder, unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5ba0a4fc2c in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f59f5f61e00, waiter=..., this=0x7f5b99013700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98ff0100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98ff0100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f5b2c780700 (LWP 1771875) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b3ea93403 in PixelCPEBase::setTheClu(PixelCPEBase::DetParam const&, PixelCPEBase::ClusterParam&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #5 0x00007f5b3ea93f38 in PixelClusterParameterEstimator::getParameters(SiPixelCluster const&, GeomDet const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #6 0x00007f5b4470f3ce in TkClonerImpl::operator()(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTransientTrackingRecHit.so #7 0x00007f5b3eb4bc88 in SiPixelRecHit::clone_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelRecHitsPlugins.so #8 0x00007f5a9f514ec6 in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #9 0x00007f5a9f515a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #10 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f5b9f266304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a4cd40400, waiter=..., this=0x7f5b99013680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5b99013680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f5b9f2684c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5b98ff0080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5b98ff0080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f5b613ff700 (LWP 1771859) "cuda-EvtHandlr"): #0 0x00007f5b9e0d4a71 in poll () from /lib64/libc.so.6 #1 0x00007f5b80d7dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f5b80e31d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f5b80d80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f5b621bb700 (LWP 1771855) "cuda-EvtHandlr"): #0 0x00007f5b9e0d4a71 in poll () from /lib64/libc.so.6 #1 0x00007f5b80d7dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f5b80e31d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f5b80d80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f5b68cbd700 (LWP 1771781) "cmsRun"): #0 0x00007f5b9e3bc5a2 in waitpid () from /lib64/libpthread.so.0 #1 0x00007f5b96fa1157 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f5b96fa3a9a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f5b9ea3e9b4 in std::execute_native_thread_routine (__p=0x7f5b988d68d0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f5b9e3b217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5b9e0dfdf3 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f5b9d541640 (LWP 1770918) "cmsRun"): #0 0x00007f5b9e0abd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5b9e0abc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5b96fa0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5b44696c89 in reco::PFCandidate::~PFCandidate() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsParticleFlowCandidate.so #5 0x00007f5a9798b5af in PFAlgo::reconstructParticles(edm::Handle > > const&, PFEGammaFilters const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/libRecoParticleFlowPFProducer.so #6 0x00007f5a97a66c7c in PFProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFProducerPlugins.so #7 0x00007f5ba0b2195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5ba0b08072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5ba0a946da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5ba0a94b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5ba07e9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5b9f26d9cd in tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=0x7f59f5d88600, this=0x7f5b99013600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=, this=0x7f5b99013600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168 #15 0x00007f5ba0a1710d in edm::FinalWaitingTask::wait() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f5ba0a24b91 in edm::EventProcessor::processRuns() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f5ba0a250e6 in edm::EventProcessor::runToCompletion() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x000000000040c17d in tbb::detail::d1::task_arena_function::operator()() const () #19 0x00007f5b9f25b847 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:694 #20 0x00000000004101d0 in main::{lambda()#1}::operator()() const () #21 0x0000000000407d6c in main () Current Modules: Module: HitPairEDProducer:hltDisplacedhltIter4PFlowPixelLessHitDoubletsForTau (crashed) Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: InclusiveVertexFinder:hltInclusiveVertexFinder Module: RecoTauProducer:hltHpsCombinatoricRecoTausDispl Module: FastjetJetProducer:hltAK4PFJets Module: CkfTrajectoryMaker:hltL3NoFiltersTrackCandidateFromL2IOHitNoVtx Module: none Module: PFRecHitProducer:hltParticleFlowRecHitECALUnseeded Module: TrackSelectorByRegion:hltPixelTracksInRegionIter0L3MuonNoVtx Module: PFClusterProducer:hltParticleFlowClusterHBHE Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoubletsUnseeded Module: FastjetJetProducer:hltAK4PFJetsForTaus Module: none Module: TrackProducer:hltDisplacedhltIter4PFlowCtfWithMaterialTracksForTau Module: SiPixelRawToClusterCUDA:hltSiPixelClustersGPU Module: SiPixelDigisClustersFromSoAPhase1:hltSiPixelClustersFromSoA Module: none Module: PFBlockProducer:hltParticleFlowBlockCPUOnly Module: CkfTrackCandidateMaker:hltIter0PFlowCkfTrackCandidates Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: ElectronNHitSeedProducer:hltEgammaElectronPixelSeeds Module: CkfTrackCandidateMaker:hltIter0PFlowCkfTrackCandidates Module: FastjetJetProducer:hltAK4PFJetsCPUOnly Module: PFBlockProducer:hltParticleFlowBlockForDisplTaus Module: CkfTrajectoryMaker:hltL3NoFiltersTrackCandidateFromL2OIHitNoVtx Module: none Module: CkfTrackCandidateMaker:hltIterL3OITrackCandidatesNoVtx Module: L1TGlobalProducer:hltGtStage2ObjectMap Module: EgammaHLTHcalVarProducerFromRecHit:hltEgammaHoverE Module: PFProducer:hltParticleFlowForTaus Module: CkfTrackCandidateMaker:hltIter0PFlowCkfTrackCandidates Module: none A fatal system signal has occurred: segmentation violation msgtime:2023-05-07 13:29:23 doc_type:cmsswlog date:2023-05-07T11:29:23.686Z run:367112 host:fu-c2b04-21-01 pid:1281402 doctype:stacktrace severity:FATAL severityVal:4 instance:global lexicalId:549852445 message:A fatal system signal has occurred: segmentation violation The following is the call stack containing the origin of the signal. Sun May 7 13:28:58 CEST 2023 Thread 44 (Thread 0x7efed727f700 (LWP 1282769) "cmsRun"): #0 0x00007f0092d47cd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0 #1 0x00007f0092d47dc8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007f007556b0a2 in ?? () from /lib64/libcuda.so.1 #3 0x00007f0075580768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7eff3b9ff700 (LWP 1282750) "cmsRun"): #0 0x00007f0092d4574a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0087917061 in FedRawDataInputSource::readSupervisor() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #2 0x00007f00933cb9b4 in std::execute_native_thread_routine (__p=0x7eff4b995800) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #4 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7eff3effe700 (LWP 1282688) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f007ab14ef8 in ProxyPixelTopology::positionCorrection(Topology::LocalTrackPred const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryCommonTopologies.so #5 0x00007f007ab14fb5 in ProxyPixelTopology::localPosition(Point2DBase const&, Topology::LocalTrackPred const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryCommonTopologies.so #6 0x00007f0033439919 in PixelCPEGeneric::localPosition(PixelCPEBase::DetParam const&, PixelCPEBase::ClusterParam&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #7 0x00007f003341af62 in PixelClusterParameterEstimator::getParameters(SiPixelCluster const&, GeomDet const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #8 0x00007f00390ad5be in TkClonerImpl::makeShared(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTransientTrackingRecHit.so #9 0x00007f00390ab854 in SiPixelRecHit::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTransientTrackingRecHit.so #10 0x00007f0039297b57 in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector, std::allocator > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrackFitters.so #11 0x00007f003496643d in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector, std::allocator > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginTrackingToolsTrackFittersPlugins.so #12 0x00007eff927ddcd2 in TrackProducerAlgorithm::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector::AlgoProduct, std::allocator::AlgoProduct> >&, std::vector, std::allocator > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase, int, signed char) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTrackProducer.so #13 0x00007eff92d0d9dd in TrackProducerAlgorithm::runWithCandidate(TrackingGeometry const*, MagneticField const*, std::vector > const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector::AlgoProduct, std::allocator::AlgoProduct> >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTrackProducerPlugins.so #14 0x00007eff92d0c2c2 in TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTrackProducerPlugins.so #15 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #20 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb659900, waiter=..., this=0x7f008d99f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #24 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #26 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #27 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7eff40bfe700 (LWP 1282687) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f00356af2cf in SteppingHelixPropagator::propagate(SteppingHelixStateInfo const&, Plane const&, SteppingHelixStateInfo&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackPropagationSteppingHelixPropagator.so #5 0x00007f00356b21ba in SteppingHelixPropagator::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackPropagationSteppingHelixPropagator.so #6 0x00007f00356a4d95 in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackPropagationSteppingHelixPropagator.so #7 0x00007f00810b577d in GeomDetCompatibilityChecker::isCompatible(GeomDet const*, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsDetLayers.so #8 0x00007f00810b5acc in ForwardDetRingOneZ::add(int, std::vector, std::allocator > >&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsDetLayers.so #9 0x00007f0038e8ba62 in MuDetRing::compatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonDetLayers.so #10 0x00007f0038e89be0 in MuRingForwardLayer::compatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonDetLayers.so #11 0x00007f0038e8a4cc in MuRingForwardDoubleLayer::groupedCompatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonDetLayers.so #12 0x00007eff9e11fe18 in MuonDetLayerMeasurements::groupedMeasurements(DetLayer const*, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, edm::Event const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonMeasurementDet.so #13 0x00007eff9e120b39 in MuonDetLayerMeasurements::groupedMeasurements(DetLayer const*, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonMeasurementDet.so #14 0x00007eff9e07caf6 in StandAloneMuonFilter::findBestMeasurements(DetLayer const*, TrajectoryStateOnSurface const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonStandAloneTrackFinder.so #15 0x00007eff9e0801f6 in StandAloneMuonFilter::refit(TrajectoryStateOnSurface const&, DetLayer const*, Trajectory&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonStandAloneTrackFinder.so #16 0x00007eff9e08241d in StandAloneMuonTrajectoryBuilder::trajectories(TrajectorySeed const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonStandAloneTrackFinder.so #17 0x00007eff9e07aeba in ExhaustiveMuonTrajectoryBuilder::trajectories(TrajectorySeed const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonStandAloneTrackFinder.so #18 0x00007f003dc79beb in MuonTrackFinder::reconstruct(edm::Handle > const&, edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonTrackingTools.so #19 0x00007eff9e0a47b4 in L2MuonProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoMuonL2MuonProducer.so #20 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #23 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #24 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #25 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb86f500, waiter=..., this=0x7f008d99f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #26 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #27 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #28 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #29 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #30 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #31 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #32 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7eff3fbff700 (LWP 1282686) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f007e9dcde3 in BasicTrajectoryState::~BasicTrajectoryState() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrajectoryState.so #5 0x00007eff9269229a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #6 0x00007eff926b7f7a in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #7 0x00007eff926c07bd in SeedGeneratorFromProtoTracksEDProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #8 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f009538f7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb379300, waiter=..., this=0x7f008d99f480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7eff419ff700 (LWP 1282685) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0092fb7ba6 in atanf32 () from /lib64/libm.so.6 #5 0x00007f0092fb3119 in __atan2f_finite () from /lib64/libm.so.6 #6 0x00007eff8a413aab in TrajSeedMatcher::processSeed(TrajectorySeed const&, Point3DBase const&, float, TrajectoryStateOnSurface const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaElectronAlgos.so #7 0x00007eff8a414dc1 in TrajSeedMatcher::operator()(Point3DBase const&, float) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaElectronAlgos.so #8 0x00007eff8a49d49a in ElectronNHitSeedProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoEgammaEgammaElectronProducersPlugins.so #9 0x00007f009549a9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954945d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #14 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efed01d4800, waiter=..., this=0x7f008d99f400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7eff43bfd700 (LWP 1282684) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f007ab14f02 in ProxyPixelTopology::positionCorrection(Topology::LocalTrackPred const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryCommonTopologies.so #5 0x00007f007ab14fb5 in ProxyPixelTopology::localPosition(Point2DBase const&, Topology::LocalTrackPred const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryCommonTopologies.so #6 0x00007f0033439948 in PixelCPEGeneric::localPosition(PixelCPEBase::DetParam const&, PixelCPEBase::ClusterParam&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #7 0x00007f003341af62 in PixelClusterParameterEstimator::getParameters(SiPixelCluster const&, GeomDet const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #8 0x00007f00390ad3ce in TkClonerImpl::operator()(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTransientTrackingRecHit.so #9 0x00007f00334d2c88 in SiPixelRecHit::clone_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelRecHitsPlugins.so #10 0x00007eff926b7ec6 in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #11 0x00007eff926b8a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #12 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #17 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efed6082a00, waiter=..., this=0x7f008d99f380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7eff455fe700 (LWP 1282683) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0094005dce in arena_slab_reg_alloc_batch (ptrs=, cnt=12, bin_info=, slab=0x7eff3d046b80) at src/arena.c:280 #5 je_arena_cache_bin_fill_small (tsdn=, arena=, cache_bin=, cache_bin_info=, binind=, nfill=) at src/arena.c:982 #6 0x00007f009406bdf9 in je_tcache_alloc_small_hard (tsdn=tsdn@entry=0x7eff455f87f8, arena=arena@entry=0x7eff3d001040, tcache=tcache@entry=0x7eff455f8b50, cache_bin=cache_bin@entry=0x7eff455f8c00, binind=binind@entry=7, tcache_success=tcache_success@entry=0x7eff455f7730) at src/tcache.c:238 #7 0x00007f0093ffabf2 in tcache_alloc_small (slow_path=, zero=false, binind=7, size=, tcache=, arena=0x7eff3d001040, tsd=0x7eff455f87f8) at include/jemalloc/internal/tcache_inlines.h:68 #8 arena_malloc (slow_path=, tcache=, zero=false, ind=7, size=, arena=0x0, tsdn=0x7eff455f87f8) at include/jemalloc/internal/arena_inlines_b.h:151 #9 iallocztm (slow_path=, arena=0x0, is_internal=false, tcache=, zero=false, ind=7, size=, tsdn=0x7eff455f87f8) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:55 #10 imalloc_no_sample (ind=7, usize=112, size=, tsd=0x7eff455f87f8, dopts=, sopts=) at src/jemalloc.c:2398 #11 imalloc_body (tsd=0x7eff455f87f8, dopts=, sopts=) at src/jemalloc.c:2573 #12 imalloc (dopts=, sopts=) at src/jemalloc.c:2687 #13 je_malloc_default (size=) at src/jemalloc.c:2722 #14 0x00007f0094072eb9 in fallback_impl (size=112) at src/jemalloc_cpp.cpp:98 #15 0x00007f0038ef5531 in reco::PFBlockElementCluster::clone() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsParticleFlowReco.so #16 0x00007eff8abc0dca in PFBlockProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFProducerPlugins.so #17 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #22 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efed600a900, waiter=..., this=0x7f008d99f300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #23 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #24 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #25 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #26 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #27 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #28 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #29 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7eff445fe700 (LWP 1282682) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0092ad07a6 in __memmove_avx_unaligned_erms () from /lib64/libc.so.6 #5 0x00007f00954bf209 in void std::__cxx11::basic_string, std::allocator >::_M_construct(char*, char*, std::forward_iterator_tag) [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f009539fd9f in edm::Event::commit_aux(std::vector > >, std::allocator > > > >&, edm::Hash<5>*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f00953a009a in edm::Event::commit_(std::vector > const&, edm::Hash<5>*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f009549a9e7 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f00954945d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f00954217cd in std::__exception_ptr::exception_ptr edm::Worker::runModuleDirectly >(edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095421906 in edm::Path::finished(std::__exception_ptr::exception_ptr, edm::StreamContext const*, edm::EventTransitionInfo const&, edm::StreamID const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095424916 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0095424b26 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #16 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb139c00, waiter=..., this=0x7f008d99f280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7eff463fe700 (LWP 1282681) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0075445604 in ?? () from /lib64/libcuda.so.1 #5 0x00007f00756167a6 in ?? () from /lib64/libcuda.so.1 #6 0x00007f0075616b66 in ?? () from /lib64/libcuda.so.1 #7 0x00007f007567f45e in ?? () from /lib64/libcuda.so.1 #8 0x00007f007568902d in ?? () from /lib64/libcuda.so.1 #9 0x00007f00755a1599 in ?? () from /lib64/libcuda.so.1 #10 0x00007f007819f3bd in ?? () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #11 0x00007f00781731ba in ?? () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #12 0x00007f00781ac6f0 in cudaFreeHost () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #13 0x00007f0034c4ccb5 in edm::Wrapper > >::~Wrapper() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalCaloEcalRecProducersPlugins.so #14 0x00007f009539111a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0095436ee6 in edm::DataManagingProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f00954271b8 in edm::Principal::clearPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f00953a06cd in edm::EventPrincipal::clearEventPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f00953dcbb2 in edm::FunctorWaitingTask, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f009538f7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efed01ecb00, waiter=..., this=0x7f008d99f200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #24 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #26 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #27 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7eff46fff700 (LWP 1282680) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f003588bd8f in FastHelix::compute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkSeedGenerator.so #5 0x00007eff926b285c in SeedFromConsecutiveHitsCreator::initialKinematic(GlobalTrajectoryParameters&, SeedingHitSet const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #6 0x00007eff926b7c1c in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #7 0x00007eff926b8a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #8 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7effc63cc600, waiter=..., this=0x7f008d99f180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7eff48db2700 (LWP 1282679) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f00357eed75 in MSLayersAtAngle::sum2RmRn(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, float, SimpleLineRZ const&) const [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkMSParametrization.so #5 0x00007f00357f0f32 in MultipleScatteringParametrisation::operator()(float, PixelRecoPointRZ const&, PixelRecoPointRZ const&, int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkMSParametrization.so #6 0x00007f003583bfd8 in RectangularEtaPhiTrackingRegion::checkRZOld(DetLayer const*, BaseTrackerRecHit const* const&, DetLayer const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkTrackingRegions.so #7 0x00007f003583c04a in RectangularEtaPhiTrackingRegion::checkRZ(DetLayer const*, BaseTrackerRecHit const* const&, DetLayer const*, float, float, float, float) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkTrackingRegions.so #8 0x00007f0035862767 in HitPairGeneratorFromLayerPair::doublets(TrackingRegion const&, DetLayer const&, DetLayer const&, RecHitsSortedInPhi const&, RecHitsSortedInPhi const&, MagneticField const&, MultipleScatteringParametrisationMaker const&, unsigned int, HitDoublets&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #9 0x00007f0035866490 in HitPairGeneratorFromLayerPair::doublets(TrackingRegion const&, edm::Event const&, edm::EventSetup const&, SeedingLayerSetsHits::SeedingLayer const&, SeedingLayerSetsHits::SeedingLayer const&, LayerHitMapCache&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #10 0x00007eff8a34e4f7 in (anonymous namespace)::Impl<(anonymous namespace)::ImplSeedingHitSets, (anonymous namespace)::ImplIntermediateHitDoublets, (anonymous namespace)::RegionsLayersSeparate>::produce(bool, edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkHitPairsPlugins.so #11 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #16 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb14d200, waiter=..., this=0x7f008d99f100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7eff483b1700 (LWP 1282678) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007eff9e32aa21 in CSCSegAlgoRU::buildSegments(CSCChamber const*, std::vector > const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalMuonCSCSegment.so #5 0x00007eff9e32c1da in CSCSegAlgoRU::run(CSCChamber const*, std::vector > const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalMuonCSCSegment.so #6 0x00007eff9e34951f in CSCSegmentBuilder::build(edm::RangeMap >, edm::ClonePolicy > const*, edm::RangeMap >, edm::ClonePolicy >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalMuonCSCSegment.so #7 0x00007eff9e34a0ff in CSCSegmentProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalMuonCSCSegment.so #8 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efb16bb9200, waiter=..., this=0x7f008d99f080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7eff497b3700 (LWP 1282677) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007eff82fbb0ea in boost::iterators::filter_iterator > >, __gnu_cxx::__normal_iterator*, std::vector, std::allocator > > > >::satisfy_predicate() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTauTagRecoTauPlugins.so #5 0x00007eff82fc3021 in reco::tau::RecoTauBuilderCombinatoricPlugin::operator()(edm::RefToBase const&, std::vector > const&, std::vector > const&, std::vector, std::allocator > > const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTauTagRecoTauPlugins.so #6 0x00007eff82fff5e7 in RecoTauProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTauTagRecoTauPlugins.so #7 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb1ece00, waiter=..., this=0x7f008d99f000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99f000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7eff4a5b3700 (LWP 1282676) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0092d443f5 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #5 0x00007f00757a7a58 in ?? () from /lib64/libcuda.so.1 #6 0x00007f00755f3afa in ?? () from /lib64/libcuda.so.1 #7 0x00007f0078170bfa in ?? () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #8 0x00007f00781c6281 in cudaStreamAddCallback () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #9 0x00007f00352bf8a3 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #10 0x00007f00352bf911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #11 0x00007f00334dc0b5 in SiPixelRecHitFromCUDAT::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelRecHitsPlugins.so #12 0x00007f00954aaad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f00954aeb82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f009548b6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f009548b846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f00954229ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #18 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7eff401dfa00, waiter=..., this=0x7f008d99ef80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #19 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ef80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #20 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #21 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #22 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #24 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #25 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7eff82ddc700 (LWP 1282657) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008790268e in evf::FastMonitoringService::snapshotRunner() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #3 0x00007f00933cb9b4 in std::execute_native_thread_routine (__p=0x7eff4bbf3060) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7eff935ff700 (LWP 1282564) "cmsRun"): #0 0x00007f0092d453fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f00933c5e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f00269a0acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f00269a10ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f002699dc68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f00237c74c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7eff94655700 (LWP 1282552) "cmsRun"): #0 0x00007f0092d453fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f00933c5e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f00269a0acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f00269a10ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f002699dc68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f00237c74c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7eff94e56700 (LWP 1282551) "cmsRun"): #0 0x00007f0092d453fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f00933c5e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f00269a0acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f00269a10ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f002699dc68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f00237c74c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7effb95fc700 (LWP 1282518) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f00233efdf4 in tensorflow::TensorShapeBase::dim_size(int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #5 0x00007eff82f555b6 in tensorflow::TTypes::Tensor tensorflow::Tensor::tensor() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTauTagRecoTauPlugins.so #6 0x00007eff82f59fee in void DeepTauId::createConvFeatures(reco::PFTau const&, unsigned long, edm::RefToBase, reco::Vertex const&, double, std::vector > const*, std::vector > const*, edm::View const&, (anonymous namespace)::CellGrid const&, (anonymous namespace)::TauFunc, bool) [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTauTagRecoTauPlugins.so #7 0x00007eff82f4c1d1 in void DeepTauId::getPredictionsV2(reco::BaseTau const&, unsigned long, edm::RefToBase, std::vector > const*, std::vector > const*, edm::View const&, reco::Vertex const&, double, unsigned long long const&, std::vector >&, (anonymous namespace)::TauFunc) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTauTagRecoTauPlugins.so #8 0x00007eff82f4ffc7 in DeepTauId::getPredictions(edm::Event&, edm::Handle >) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTauTagRecoTauPlugins.so #9 0x00007eff82e4e87b in deep_tau::DeepTauBase::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTauTagRecoTau.so #10 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7eff40180600, waiter=..., this=0x7f008d99ef00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ef00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7effba9fe700 (LWP 1282515) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0092ac9f24 in __memcmp_avx2_movbe () from /lib64/libc.so.6 #5 0x00007f008fbfaf60 in std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, edm::messagelogger::ELseverityLevel>, std::_Select1st, std::allocator > const, edm::messagelogger::ELseverityLevel> >, std::less, std::allocator > >, std::allocator, std::allocator > const, edm::messagelogger::ELseverityLevel> > >::find(std::__cxx11::basic_string, std::allocator > const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreMessageServicePlugins.so #6 0x00007f008fbff20c in edm::service::MessageLogger::establishModule(unsigned int, edm::ModuleCallingContext const&, char const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreMessageServicePlugins.so #7 0x00007f009549a95f in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f00954945d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954217cd in std::__exception_ptr::exception_ptr edm::Worker::runModuleDirectly >(edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095421906 in edm::Path::finished(std::__exception_ptr::exception_ptr, edm::StreamContext const*, edm::EventTransitionInfo const&, edm::StreamID const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095424916 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095424b26 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeebaf0d00, waiter=..., this=0x7f008d99ee00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ee00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7effb9ffd700 (LWP 1282514) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0081084004 in JacobianCurvilinearToLocal::JacobianCurvilinearToLocal(Surface const&, LocalTrajectoryParameters const&, GlobalTrajectoryParameters const&, MagneticField const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsAnalyticalJacobians.so #5 0x00007f007e9de860 in BasicTrajectoryState::createLocalErrorFromCurvilinearError() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrajectoryState.so #6 0x00007f0035cd824d in Chi2MeasurementEstimator::estimate(TrajectoryStateOnSurface const&, TrackingRecHit const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #7 0x00007f0035d5fc92 in bool TkStripMeasurementDet::filteredRecHits, SiStripCluster, edmNew::DetSetVector::FindForDetSetVector> >(edm::Ref, SiStripCluster, edmNew::DetSetVector::FindForDetSetVector> const&, StripCPE::AlgoParam const&, TrajectoryStateOnSurface const&, MeasurementEstimator const&, std::vector > const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so #8 0x00007f0035d68fa2 in TkStripMeasurementDet::simpleRecHits(TrajectoryStateOnSurface const&, MeasurementEstimator const&, MeasurementTrackerEvent const&, std::vector >&) const [clone .isra.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so #9 0x00007f0035d5a15d in TkGluedMeasurementDet::measurements(TrajectoryStateOnSurface const&, MeasurementEstimator const&, MeasurementTrackerEvent const&, tracking::TempMeasurements&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerMeasurementDetPlugins.so #10 0x00007f0035cc4492 in LayerMeasurements::measurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMeasurementDet.so #11 0x00007eff92de113a in CkfTrajectoryBuilder::findCompatibleMeasurements(TrajectorySeed const&, TempTrajectory const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #12 0x00007eff92dd9adb in CkfTrajectoryBuilder::limitedCandidates(std::shared_ptr const&, std::vector >&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #13 0x00007eff92ddb0d7 in CkfTrajectoryBuilder::limitedCandidates(TrajectorySeed const&, TempTrajectory&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #14 0x00007eff92ddb2cf in CkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector >&, unsigned int&, TrajectoryFilter const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #15 0x00007eff92dcf263 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #16 0x00007eff92dd0ceb in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #17 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #22 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efed02ae100, waiter=..., this=0x7f008d99ed80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #23 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ed80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #24 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #25 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #26 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97ab00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #27 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97ab00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #28 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #29 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7effbb3ff700 (LWP 1282512) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007ffcd772fbba in clock_gettime () #5 0x00007f0092a3397a in clock_gettime@GLIBC_2.2.5 () from /lib64/libc.so.6 #6 0x00007f008b9f3121 in boost::chrono::thread_clock::now() () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libboost_chrono.so.1.80.0 #7 0x00007f0087726c03 in FastTimerService::Measurement::measure_and_store(FastTimerService::Resources&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginHLTriggerTimerPlugins.so #8 0x00007f0087734790 in FastTimerService::postModuleEvent(edm::StreamContext const&, edm::ModuleCallingContext const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginHLTriggerTimerPlugins.so #9 0x00007f009549aa2b in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954945d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f00954217cd in std::__exception_ptr::exception_ptr edm::Worker::runModuleDirectly >(edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095421906 in edm::Path::finished(std::__exception_ptr::exception_ptr, edm::StreamContext const*, edm::EventTransitionInfo const&, edm::StreamID const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0095424916 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0095424b26 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #17 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7eff3e3cce00, waiter=..., this=0x7f008d99ec80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ec80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97aa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97aa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7effbc1fe700 (LWP 1282504) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0092ad0c4b in __memset_avx2_unaligned_erms () from /lib64/libc.so.6 #5 0x00007effabe187e3 in CAHitNtupletGeneratorKernelsCPU::buildDoublets(TrackingRecHitSoA::TrackingRecHitSoALayout<128ul, false>::ConstViewTemplateFreeParams<128ul, false, true, false> const&, int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so #6 0x00007effabe2899b in CAHitNtupletGeneratorOnGPU::makeTuples(TrackingRecHitSoAHost const&, float) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so #7 0x00007effabe155bc in CAHitNtupletCUDAT::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so #8 0x00007f009549a9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f00954945d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb164d00, waiter=..., this=0x7f008d99eb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99eb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97a980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97a980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7effbcdff700 (LWP 1282503) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007eff8aab0825 in BlockElementLinkerBase::linkPrefilter(reco::PFBlockElement const*, reco::PFBlockElement const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFProducerPlugins_linkers.so #5 0x00007eff8ab3622f in PFBlockAlgo::findBlocks() () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/libRecoParticleFlowPFProducer.so #6 0x00007eff8abc09b5 in PFBlockProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFProducerPlugins.so #7 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7eff3e37c300, waiter=..., this=0x7f008d99eb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99eb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97aa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97aa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7effbe9fe700 (LWP 1282480) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0092d443f5 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #5 0x00007f00757a7a58 in ?? () from /lib64/libcuda.so.1 #6 0x00007f00755f3afa in ?? () from /lib64/libcuda.so.1 #7 0x00007f0078170bfa in ?? () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #8 0x00007f00781c6281 in cudaStreamAddCallback () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #9 0x00007f00352bf8a3 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #10 0x00007f00352bf911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #11 0x00007f00351f77ed in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #12 0x00007f00954aaad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f00954aeb82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f009548b6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f009548b846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f00954229ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f009538f7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeba2c300, waiter=..., this=0x7f008d99ea80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #19 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ea80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #20 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #21 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #22 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97ae00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97ae00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #24 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #25 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7effbf3ff700 (LWP 1282479) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f00954248ba in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f0095424b26 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #7 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7eff40169800, waiter=..., this=0x7f008d99e980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #8 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99e980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #9 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #10 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #11 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97ac00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97ac00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #13 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #14 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7effc1ffd700 (LWP 1282469) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f007e9df47b in BasicTrajectoryState::createLocalErrorFromCurvilinearError() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrajectoryState.so #5 0x00007f00357d6d28 in MaterialEffectsUpdator::updateStateInPlace(TrajectoryStateOnSurface&, PropagationDirection) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #6 0x00007f00357d8b1d in PropagatorWithMaterial::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #7 0x00007eff926b7e7c in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #8 0x00007eff926b8a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #9 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #14 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7eff3e374100, waiter=..., this=0x7f008d99e900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99e900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97ab80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97ab80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7effc0bfb700 (LWP 1282464) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f00955aad20 in update_get_addr () from /lib64/ld-linux-x86-64.so.2 #5 0x00007f00955b0cdc in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2 #6 0x00007f00810941b4 in AnalyticalPropagator::propagatedStateWithPath(FreeTrajectoryState const&, Surface const&, GlobalTrajectoryParameters const&, double const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #7 0x00007f008109e7fe in AnalyticalPropagator::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #8 0x00007f0081093365 in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #9 0x00007f00357d8a72 in PropagatorWithMaterial::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #10 0x00007eff926b7e7c in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #11 0x00007eff926b8a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #12 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #17 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb370a00, waiter=..., this=0x7f008d99e880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99e880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97ad80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97ad80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7effbfffa700 (LWP 1282463) "cmsRun"): #0 0x00007f0092d453fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f00933c5e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f00879142d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f00933cb9b4 in std::execute_native_thread_routine (__p=0x7f0084fa7650) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7effc15fc700 (LWP 1282460) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0035cdb5ba in TrajectoryStateOnSurface (anonymous namespace)::lupdate<2u>(TrajectoryStateOnSurface const&, TrackingRecHit const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #5 0x00007f0035ce3fae in KFUpdator::update(TrajectoryStateOnSurface const&, TrackingRecHit const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #6 0x00007eff926b8337 in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #7 0x00007eff926b8a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #8 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7eff44b41600, waiter=..., this=0x7f008d99ee80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ee80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97ad00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97ad00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7effc27fe700 (LWP 1282452) "cmsRun"): #0 0x00007f0092d453fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f00933c5e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f00879142d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f00933cb9b4 in std::execute_native_thread_routine (__p=0x7f0084fa7500) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7effc31ff700 (LWP 1282449) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0035cdbb03 in TrajectoryStateOnSurface (anonymous namespace)::lupdate<2u>(TrajectoryStateOnSurface const&, TrackingRecHit const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #5 0x00007f0035ce3fae in KFUpdator::update(TrajectoryStateOnSurface const&, TrackingRecHit const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #6 0x00007eff93e448b5 in TrajectorySegmentBuilder::updateTrajectory(TempTrajectory&, TrajectoryMeasurement) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so #7 0x00007eff93e464cf in TrajectorySegmentBuilder::updateCandidates(TempTrajectory const&, std::vector > const&, std::vector >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so #8 0x00007eff93e469bf in TrajectorySegmentBuilder::addGroup(TempTrajectory const&, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so #9 0x00007eff93e481f0 in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector >&, std::vector >&) const [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so #10 0x00007eff93e3b38d in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so #11 0x00007eff93e3e846 in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector >&, unsigned int&, TrajectoryFilter const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerCkfPatternPlugins.so #12 0x00007eff92dcf263 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #13 0x00007eff92dd0ceb in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #14 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #19 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeebc3bf00, waiter=..., this=0x7f008d99ec00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #20 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ec00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #21 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #22 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #23 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97ac80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #24 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97ac80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #25 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #26 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7effc47ff700 (LWP 1282442) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0036e9bea4 in HcalDetId::HcalDetId(DetId const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryHcalTowerAlgo.so #5 0x00007f0036e9c41d in HcalGeometry::getGeometry(DetId const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryHcalTowerAlgo.so #6 0x00007f003842936d in CaloGeometry::getGeometry(DetId const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryCaloGeometry.so #7 0x00007eff9fca4aee in EgammaHcalIsolation::goodHitEnergy(float, float, HBHERecHit const&, int, int, int, int, double (*)(double const&)) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaIsolationAlgos.so #8 0x00007eff9fca4fbd in EgammaHcalIsolation::getHcalSum(Point3DBase const&, int, int, int, int, double (*)(double const&)) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaIsolationAlgos.so #9 0x00007eff8a8eaa4b in EgammaHLTHcalVarProducerFromRecHit::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoEgammaEgammaHLTProducersPlugins.so #10 0x00007f009549a9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f00954945d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f009538f7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb312e00, waiter=..., this=0x7f008d99ed00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ed00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97af00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97af00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7effc55fe700 (LWP 1282420) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0092d43fc2 in pthread_rwlock_rdlock () from /lib64/libpthread.so.0 #5 0x00007f0075410228 in ?? () from /lib64/libcuda.so.1 #6 0x00007f00754126aa in ?? () from /lib64/libcuda.so.1 #7 0x00007f00755ef97e in ?? () from /lib64/libcuda.so.1 #8 0x00007f007817202c in ?? () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #9 0x00007f00781c6886 in cudaLaunchKernel () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #10 0x00007f0034cfc25f in __device_stub__ZN4ecal8multifit29kernel_time_compute_nullhypotEPKfS2_PKbPfS5_S5_i(float const*, float const*, bool const*, float*, float*, float*, int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalCaloEcalRecProducersPlugins.so #11 0x00007f0034cfb5cd in ecal::multifit::entryPoint(ecal::multifit::EventInputDataGPU const&, ecal::multifit::EventOutputDataGPU&, ecal::multifit::EventDataForScratchGPU&, ecal::multifit::ConditionsProducts const&, ecal::multifit::ConfigurationParameters const&, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalCaloEcalRecProducersPlugins.so #12 0x00007f0034cd2073 in EcalUncalibRecHitProducerGPU::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalCaloEcalRecProducersPlugins.so #13 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #18 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7eff40112e00, waiter=..., this=0x7f008d99e800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #19 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99e800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #20 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #21 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #22 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97ae80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97ae80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #24 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #25 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7effc5fff700 (LWP 1282414) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007eff8ad9f092 in reco::PFCluster::PFCluster(reco::PFCluster const&) [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #5 0x00007eff8ad9f14b in reco::PFCluster* std::__uninitialized_copy::__uninit_copy(reco::PFCluster const*, reco::PFCluster const*, reco::PFCluster*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #6 0x00007eff8ad9f22e in void std::vector >::_M_realloc_insert(__gnu_cxx::__normal_iterator > >, reco::PFCluster&&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #7 0x00007eff8ada17a8 in std::vector >::_M_insert_rval(__gnu_cxx::__normal_iterator > >, reco::PFCluster&&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #8 0x00007eff8ada1c67 in Basic2DGenericPFlowClusterizer::buildClusters(std::vector > const&, std::vector > const&, std::vector >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #9 0x00007eff8adc5689 in PFClusterProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #10 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efed61eb600, waiter=..., this=0x7f008d99e780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99e780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97b000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97b000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7effc6bfe700 (LWP 1282409) "cmsRun"): #0 0x00007f0092d453fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f00933c5e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f00879142d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f00933cb9b4 in std::execute_native_thread_routine (__p=0x7f0084fa7440) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f000f7ff700 (LWP 1282399) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f003588bd8f in FastHelix::compute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkSeedGenerator.so #5 0x00007eff926b285c in SeedFromConsecutiveHitsCreator::initialKinematic(GlobalTrajectoryParameters&, SeedingHitSet const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #6 0x00007eff926b7c1c in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #7 0x00007eff926b8a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #8 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeba4cf00, waiter=..., this=0x7f008d99ea00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99ea00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97af80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97af80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f0020706700 (LWP 1282382) "cmsRun"): #0 0x00007f0092a61a71 in poll () from /lib64/libc.so.6 #1 0x00007f008b96346f in full_read.constprop () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f008b92eb6c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f008b92f33b in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #4 #5 0x00007f003585f777 in void std::__introsort_loop<__gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #6 0x00007f0035861bf5 in RecHitsSortedInPhi::RecHitsSortedInPhi(std::vector > const&, Point3DBase const&, DetLayer const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #7 0x00007f0035865d11 in LayerHitMapCache::operator()(SeedingLayerSetsHits::SeedingLayer const&, TrackingRegion const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #8 0x00007f003586607a in HitPairGeneratorFromLayerPair::doublets(TrackingRegion const&, edm::Event const&, edm::EventSetup const&, SeedingLayerSetsHits::SeedingLayer const&, SeedingLayerSetsHits::SeedingLayer const&, LayerHitMapCache&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #9 0x00007eff8a34bd97 in (anonymous namespace)::Impl<(anonymous namespace)::DoNothing, (anonymous namespace)::ImplIntermediateHitDoublets, (anonymous namespace)::RegionsLayersSeparate>::produce(bool, edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkHitPairsPlugins.so #10 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7efeeb99c300, waiter=..., this=0x7f008d99e680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99e680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97b100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97b100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f0021107700 (LWP 1282373) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007eff8aec2a68 in DTTimingExtractor::fillTiming(TimeMeasurementSequence&, std::vector > const&, edm::Ref >, reco::Track, edm::refhelper::FindUsingAdvance >, reco::Track> >, edm::Event const&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonMuonIdentification.so #5 0x00007eff8aec3bb8 in DTTimingExtractor::fillTiming(TimeMeasurementSequence&, edm::Ref >, reco::Track, edm::refhelper::FindUsingAdvance >, reco::Track> >, edm::Event const&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonMuonIdentification.so #6 0x00007eff8aeeaede in MuonTimingFiller::fillTiming(reco::Muon const&, reco::MuonTimeExtra&, reco::MuonTimeExtra&, reco::MuonTime&, reco::MuonTimeExtra&, edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoMuonMuonIdentification.so #7 0x00007eff8af7a271 in MuonIdProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoMuonMuonIdentificationPlugins.so #8 0x00007f00954ae95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0095495072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00954216da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0095421b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0095176f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007f0093bf3304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7eff44b71600, waiter=..., this=0x7f008d99e700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f008d99e700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f0093bf54c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f008d97b080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f008d97b080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f0055fff700 (LWP 1282350) "cuda-EvtHandlr"): #0 0x00007f0092a61a71 in poll () from /lib64/libc.so.6 #1 0x00007f007557dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f0075631d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f0075580768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f0056c26700 (LWP 1282344) "cuda-EvtHandlr"): #0 0x00007f0092a61a71 in poll () from /lib64/libc.so.6 #1 0x00007f007557dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f0075631d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f0075580768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f005d728700 (LWP 1282281) "cmsRun"): #0 0x00007f0092d495a2 in waitpid () from /lib64/libpthread.so.0 #1 0x00007f008b92c157 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f008b92ea9a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f00933cb9b4 in std::execute_native_thread_routine (__p=0x7f008c2e48b0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f0092d3f17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f0092a6cdf3 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f0091ece640 (LWP 1281402) "cmsRun"): #0 0x00007f0092a38d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f0092a38c9e in sleep () from /lib64/libc.so.6 #2 0x00007f008b92bed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0095424a81 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f009538f7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f0093bfa9cd in tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=0x7eff3b191800, this=0x7f008d99e600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #7 tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=, this=0x7f008d99e600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #8 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168 #9 0x00007f00953a410d in edm::FinalWaitingTask::wait() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f00953b1b91 in edm::EventProcessor::processRuns() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f00953b20e6 in edm::EventProcessor::runToCompletion() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x000000000040c17d in tbb::detail::d1::task_arena_function::operator()() const () #13 0x00007f0093be8847 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:694 #14 0x00000000004101d0 in main::{lambda()#1}::operator()() const () #15 0x0000000000407d6c in main () Current Modules: Module: HitPairEDProducer:hltDisplacedhltIter4PFlowPixelLessHitDoubletsForTau (crashed) Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: HitPairEDProducer:hltElePixelHitDoublets Module: PathStatusInserter:HLT_Photon60_R9Id90_CaloIdL_IsoL_DisplacedIdL_PFHT400_v1 Module: DeepTauId:hltHpsPFTauDeepTauProducerForVBFIsoTau Module: EcalRawToDigiGPU:hltEcalDigisGPU Module: CAHitNtupletCUDAPhase1:hltPixelTracksCPUOnly Module: PathStatusInserter:Dataset_EphemeralZeroBias17 Module: EcalUncalibRecHitProducerGPU:hltEcalUncalibRecHitGPU Module: PFBlockProducer:hltPixelOnlyParticleFlowBlock Module: none Module: TrackProducer:hltIter0PFlowCtfWithMaterialTracks Module: L2MuonProducer:hltL2Muons Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: CSCSegmentProducer:hltCscSegments Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: MuonIdProducer:hltIterL3MuonsNoID Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsTriplets Module: CkfTrackCandidateMaker:hltIter0PFlowCkfTrackCandidatesForBTag Module: SiPixelRecHitFromCUDAPhase1:hltSiPixelRecHitsFromGPU Module: PFBlockProducer:hltParticleFlowBlockForDisplTaus Module: RecoTauProducer:hltHpsCombinatoricRecoTausDispl Module: PathStatusInserter:HLT_PFMETNoMu140_PFMHTNoMu140_IDTight_v22 Module: none Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: PFClusterProducer:hltParticleFlowClusterPSUnseeded Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoubletsUnseeded Module: none Module: EgammaHLTHcalVarProducerFromRecHit:hltEgammaHoverEUnseeded Module: ElectronNHitSeedProducer:hltEgammaElectronPixelSeeds Module: CkfTrackCandidateMaker:hltDisplacedhltIter4PFlowCkfTrackCandidates Module: SeedGeneratorFromProtoTracksEDProducer:hltIter0PFLowPixelSeedsFromPixelTracks A fatal system signal has occurred: segmentation violation msgtime:2023-05-07 13:45:20 doc_type:cmsswlog date:2023-05-07T11:45:20.293Z run:367112 host:fu-c2b03-31-01 pid:1753372 doctype:stacktrace severity:FATAL severityVal:4 instance:global lexicalId:453908563 message:terminate called after throwing an instance of 'cms::cuda::bad_alloc' terminate called recursively A fatal system signal has occurred: abort signal The following is the call stack containing the origin of the signal. Sun May 7 13:44:58 CEST 2023 Thread 44 (Thread 0x7fbbe37ff700 (LWP 1754804) "cmsRun"): #0 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fbda0ee8b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00007fbda0c4e1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #3 0x00007fbda1111fa1 in _Unwind_Find_FDE (pc=0x7fbda15484db <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+59>, bases=bases@entry=0x7fbbe37f8ac8) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #4 0x00007fbda110e078 in uw_frame_state_for (context=0x7fbbe37f8a20, fs=0x7fbbe37f8b10) at ../../../libgcc/unwind-dw2.c:1263 #5 0x00007fbda110f9b2 in _Unwind_RaiseException (exc=0x7fbbe6287060) at ../../../libgcc/unwind.inc:104 #6 0x00007fbda15484dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7fbda3315678 , dest=0x7fbda32da000 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #7 0x00007fbd4346e409 in (anonymous namespace)::cudaScopedContextCallback(CUstream_st*, cudaError, void*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #8 0x00007fbd8636e01e in ?? () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libcudart.so.11.0 #9 0x00007fbd8376b176 in ?? () from /lib64/libcuda.so.1 #10 0x00007fbd83780768 in ?? () from /lib64/libcuda.so.1 #11 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #12 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7fbc49bff700 (LWP 1754772) "cmsRun"): #0 0x00007fbda0eec74a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbd95ac7061 in FedRawDataInputSource::readSupervisor() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #2 0x00007fbda15729b4 in std::execute_native_thread_routine (__p=0x7fbc59b388a0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #4 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7fbc4e9fb700 (LWP 1754715) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbd4504beaa in HcalDetId::HcalDetId(DetId const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryHcalTowerAlgo.so #5 0x00007fbd4504c41d in HcalGeometry::getGeometry(DetId const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryHcalTowerAlgo.so #6 0x00007fbd465d936d in CaloGeometry::getGeometry(DetId const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryCaloGeometry.so #7 0x00007fbcb4024aee in EgammaHcalIsolation::goodHitEnergy(float, float, HBHERecHit const&, int, int, int, int, double (*)(double const&)) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaIsolationAlgos.so #8 0x00007fbcb4024fbd in EgammaHcalIsolation::getHcalSum(Point3DBase const&, int, int, int, int, double (*)(double const&)) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaIsolationAlgos.so #9 0x00007fbc98a9ba4b in EgammaHLTHcalVarProducerFromRecHit::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/lib/el8_amd64_gcc11/pluginRecoEgammaEgammaHLTProducersPlugins.so #10 0x00007fbda36419cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda363b5d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf59cfa00, waiter=..., this=0x7fbd9bb4f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #22 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7fbc4d9ff700 (LWP 1754714) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda115eafd in atanf32 () from /lib64/libm.so.6 #5 0x00007fbda115a119 in __atan2f_finite () from /lib64/libm.so.6 #6 0x00007fbc985bdaab in TrajSeedMatcher::processSeed(TrajectorySeed const&, Point3DBase const&, float, TrajectoryStateOnSurface const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaElectronAlgos.so #7 0x00007fbc985bedc1 in TrajSeedMatcher::operator()(Point3DBase const&, float) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaElectronAlgos.so #8 0x00007fbc9864749a in ElectronNHitSeedProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoEgammaEgammaElectronProducersPlugins.so #9 0x00007fbda36419cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda363b5d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4cbacf00, waiter=..., this=0x7fbd9bb4f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7fbc4f7fc700 (LWP 1754713) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbd43974bef in RKOneCashKarpStep::operator()(double, ROOT::Math::SVector const&, RKDerivative const&, RKDistance const&, double) [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackPropagationRungeKutta.so #5 0x00007fbd43974ea3 in RKAdaptiveSolver::operator()(double, ROOT::Math::SVector const&, double, RKDerivative const&, RKDistance const&, float) [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackPropagationRungeKutta.so #6 0x00007fbd43978907 in RKPropagatorInS::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackPropagationRungeKutta.so #7 0x00007fbd43972c65 in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackPropagationRungeKutta.so #8 0x00007fbd43988a72 in PropagatorWithMaterial::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #9 0x00007fbd8d3d197c in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Surface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #10 0x00007fbd474308b7 in KFTrajectorySmoother::trajectory(Trajectory const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrackFitters.so #11 0x00007fbd42b1346f in (anonymous namespace)::KFFittingSmoother::smoothingStep(Trajectory&&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginTrackingToolsTrackFittersPlugins.so #12 0x00007fbd42b16459 in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector, std::allocator > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginTrackingToolsTrackFittersPlugins.so #13 0x00007fbca09ddcd2 in TrackProducerAlgorithm::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector::AlgoProduct, std::allocator::AlgoProduct> >&, std::vector, std::allocator > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase, int, signed char) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTrackProducer.so #14 0x00007fbca0ed39dd in TrackProducerAlgorithm::runWithCandidate(TrackingGeometry const*, MagneticField const*, std::vector > const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector::AlgoProduct, std::allocator::AlgoProduct> >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTrackProducerPlugins.so #15 0x00007fbca0ed22c2 in TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTrackProducerPlugins.so #16 0x00007fbda365595d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007fbda363c072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #21 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf5c61e00, waiter=..., this=0x7fbd9bb4f400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #28 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7fbc50ffd700 (LWP 1754712) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbca0fad5a1 in CkfTrajectoryBuilder::limitedCandidates(std::shared_ptr const&, std::vector >&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #5 0x00007fbca0fae0d7 in CkfTrajectoryBuilder::limitedCandidates(TrajectorySeed const&, TempTrajectory&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #6 0x00007fbca0fae2cf in CkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector >&, unsigned int&, TrajectoryFilter const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #7 0x00007fbca0fa2263 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #8 0x00007fbca0fa3ceb in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #9 0x00007fbda365595d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda363c072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4df31b00, waiter=..., this=0x7fbd9bb4f480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7fbc51bfe700 (LWP 1754711) "cmsRun"): #0 0x00007fbda0c08a71 in poll () from /lib64/libc.so.6 #1 0x00007fbd99b1346f in full_read.constprop () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007fbd99adeb6c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007fbd99adf33b in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #4 #5 0x00007fbda0b4e37f in raise () from /lib64/libc.so.6 #6 0x00007fbda0b38db5 in abort () from /lib64/libc.so.6 #7 0x00007fbda154a1aa in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:50 #8 0x00007fbda15481ea in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #9 0x00007fbda1547249 in __cxa_call_terminate (ue_header=0x7fbc3bac9da0) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #10 0x00007fbda1547977 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7fbc3bac9da0, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685 #11 0x00007fbda110f454 in _Unwind_RaiseException_Phase2 (exc=0x7fbc3bac9da0, context=0x7fbc51bf7a20, frames_p=0x7fbc51bf7b10) at ../../../libgcc/unwind.inc:64 #12 0x00007fbda110faf1 in _Unwind_RaiseException (exc=0x7fbc3bac9da0) at ../../../libgcc/unwind.inc:136 #13 0x00007fbda15484dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7fbd4372eb90 , dest=0x7fbd42df7c40 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #14 0x00007fbd42dcecaf in edm::Wrapper > >::~Wrapper() [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalCaloEcalRecProducersPlugins.so #15 0x00007fbda353811a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007fbda35ddee6 in edm::DataManagingProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007fbda35ce1b8 in edm::Principal::clearPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007fbda35476cd in edm::EventPrincipal::clearEventPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007fbda3583bb2 in edm::FunctorWaitingTask, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4df70900, waiter=..., this=0x7fbd9bb4f380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #28 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7fbc505fc700 (LWP 1754710) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e5208f in cms::cuda::allocate_device(int, unsigned long, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbd433a7362 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007fbda3651ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda3655b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda36326d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda3632846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda35c99ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf5a66600, waiter=..., this=0x7fbd9bb4f300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7fbc525ff700 (LWP 1754709) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbd4bd355a3 in edm::Wrapper >::~Wrapper() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginBeamSpotToCUDA.so #8 0x00007fbda353811a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda35ddee6 in edm::DataManagingProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda35ce1b8 in edm::Principal::clearPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35476cd in edm::EventPrincipal::clearEventPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda3583bb2 in edm::FunctorWaitingTask, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc5867f900, waiter=..., this=0x7fbd9bb4f280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fbc53bfe700 (LWP 1754708) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda1d8ebf5 in tbb::detail::d0::machine_pause (delay=6) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_machine.h:97 #5 tbb::detail::d0::atomic_backoff::bounded_pause (this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:77 #6 tbb::detail::r1::prolonged_pause_impl () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/scheduler_common.h:202 #7 tbb::detail::r1::prolonged_pause () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/scheduler_common.h:234 #8 tbb::detail::r1::stealing_loop_backoff::pause (this=0x7fbc53bf7f38) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/scheduler_common.h:263 #9 tbb::detail::r1::waiter_base::pause (this=0x7fbc53bf7f30) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/waiters.h:35 #10 tbb::detail::r1::outermost_worker_waiter::pause (this=0x7fbc53bf7f30) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/waiters.h:69 #11 tbb::detail::r1::task_dispatcher::receive_or_steal_task (this=0x7fbd9bb4f200, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=, critical_allowed=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:231 #12 0x00007fbda1d9a460 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:350 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #19 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fbc545ff700 (LWP 1754707) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbda0c4e1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007fbda1111fa1 in _Unwind_Find_FDE (pc=0x7fbda110f828 <_Unwind_RaiseException+72>, bases=bases@entry=0x7fbc545f8288) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007fbda110e078 in uw_frame_state_for (context=0x7fbc545f81e0, fs=0x7fbc545f8030) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007fbda110f2d0 in uw_init_context_1 (context=0x7fbc545f81e0, outer_cfa=0x7fbc545f8590, outer_ra=0x7fbda15484dc <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+60>) at ../../../libgcc/unwind-dw2.c:1592 #10 0x00007fbda110f829 in _Unwind_RaiseException (exc=0x7fbc3432f3e0) at ../../../libgcc/unwind.inc:93 #11 0x00007fbda15484dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7fbda169d220 , dest=0x7fbda155cf10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #12 0x00007fbd41a8ceeb in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #13 0x00007fbd41a9ac86 in cms::cuda::cudaCheck_(char const*, int, char const*, cudaError, std::basic_string_view >) [clone .isra.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #14 0x00007fbd41a951a1 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #15 0x00007fbda3651ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007fbda3655b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007fbda36326d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007fbda3632846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007fbda35c99ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #21 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf53c7500, waiter=..., this=0x7fbd9bb4f180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #28 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fbc55ffe700 (LWP 1754706) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbd8cace827 in AnalyticalCurvilinearJacobian::computeFullJacobian(GlobalTrajectoryParameters const&, Point3DBase const&, Vector3DBase const&, Vector3DBase const&, double const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsAnalyticalJacobians.so #5 0x00007fbd8cacf985 in AnalyticalCurvilinearJacobian::AnalyticalCurvilinearJacobian(GlobalTrajectoryParameters const&, Point3DBase const&, Vector3DBase const&, double const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsAnalyticalJacobians.so #6 0x00007fbd8d3cf583 in AnalyticalPropagator::propagatedStateWithPath(FreeTrajectoryState const&, Surface const&, GlobalTrajectoryParameters const&, double const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #7 0x00007fbd8d3da7fe in AnalyticalPropagator::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #8 0x00007fbd43988115 in PropagatorWithMaterial::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #9 0x00007fbca0870faa in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #10 0x00007fbca0871a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #11 0x00007fbda365595d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda363c072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #16 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4cb40400, waiter=..., this=0x7fbd9bb4f100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #23 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fbc569ff700 (LWP 1754705) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbcc778cb63 in edm::Wrapper > >::~Wrapper() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoPixelVertexingPixelVertexFindingPlugins.so #8 0x00007fbda353811a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda35ddee6 in edm::DataManagingProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda35ce1b8 in edm::Principal::clearPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35476cd in edm::EventPrincipal::clearEventPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda3583bb2 in edm::FunctorWaitingTask, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4cb97800, waiter=..., this=0x7fbd9bb4f080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fbc577fe700 (LWP 1754704) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e5208f in cms::cuda::allocate_device(int, unsigned long, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbd4be3cf2f in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #8 0x00007fbd4be2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #9 0x00007fbda3651ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda3655b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda36326d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda3632846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35c99ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4df3aa00, waiter=..., this=0x7fbd9bb4f000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4f000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #22 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fbc583ff700 (LWP 1754703) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e5208f in cms::cuda::allocate_device(int, unsigned long, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbd4be3cf2f in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #8 0x00007fbd4be2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #9 0x00007fbda3651ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda3655b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda36326d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda3632846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35c99ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf58ccf00, waiter=..., this=0x7fbd9bb4ef80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ef80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #22 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fbc90f83700 (LWP 1754661) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd95ab268e in evf::FastMonitoringService::snapshotRunner() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #3 0x00007fbda15729b4 in std::execute_native_thread_routine (__p=0x7fbc59da2100) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fbca17ff700 (LWP 1754576) "cmsRun"): #0 0x00007fbda0eec3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbda156ce90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007fbd34b50acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007fbd34b510ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007fbd34b4dc68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007fbd319774c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fbca2805700 (LWP 1754564) "cmsRun"): #0 0x00007fbda0eec3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbda156ce90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007fbd34b50acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007fbd34b510ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007fbd34b4dc68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007fbd319774c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fbca3006700 (LWP 1754557) "cmsRun"): #0 0x00007fbda0eec3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbda156ce90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007fbd34b50acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007fbd34b510ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007fbd34b4dc68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007fbd319774c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fbcc85fa700 (LWP 1754534) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0c777a6 in __memmove_avx_unaligned_erms () from /lib64/libc.so.6 #5 0x00007fbda3666209 in void std::__cxx11::basic_string, std::allocator >::_M_construct(char*, char*, std::forward_iterator_tag) [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007fbda3546d9f in edm::Event::commit_aux(std::vector > >, std::allocator > > > >&, edm::Hash<5>*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007fbda354709a in edm::Event::commit_(std::vector > const&, edm::Hash<5>*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007fbda365597d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda363c072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf572df00, waiter=..., this=0x7fbd9bb4ef00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ef00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #20 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fbcc8ffb700 (LWP 1754530) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda35c7305 in std::_Sp_counted_ptr_inplace >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::DestroyTask, std::allocator >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::DestroyTask>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007fbda353811a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007fbda35c639d in edm::FunctorWaitingTask >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda(std::__exception_ptr::exception_ptr const*)#2}>::~FunctorWaitingTask() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007fbda3630878 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #9 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4932eb00, waiter=..., this=0x7fbd9bb4ee00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #10 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ee00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #11 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #12 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #13 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #15 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #16 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fbcc9bfc700 (LWP 1754525) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda3751d0c in update_get_addr () from /lib64/ld-linux-x86-64.so.2 #5 0x00007fbda3757cdc in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2 #6 0x00007fbd43e8bb42 in TrajectoryStateOnSurface (anonymous namespace)::lupdate<2u>(TrajectoryStateOnSurface const&, TrackingRecHit const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #7 0x00007fbd43e93fae in KFUpdator::update(TrajectoryStateOnSurface const&, TrackingRecHit const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #8 0x00007fbca0871337 in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #9 0x00007fbca0871a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #10 0x00007fbda365595d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda363c072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc49328400, waiter=..., this=0x7fbd9bb4ed80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ed80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2aa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2aa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #22 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fbcca5fd700 (LWP 1754523) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e5208f in cms::cuda::allocate_device(int, unsigned long, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbd433a7362 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007fbda3651ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda3655b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda36326d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda3632846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda35c99ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf5a37600, waiter=..., this=0x7fbd9bb4ec80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ec80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2ab00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2ab00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fbccaffe700 (LWP 1754521) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda363373f in edm::Worker::skipOnPath(edm::EventPrincipal const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007fbda35cb8c2 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007fbda35cbb26 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #8 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4fb3a900, waiter=..., this=0x7fbd9bb4ec00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #9 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ec00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #10 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #11 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #12 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2a980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #13 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2a980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #14 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #15 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fbccbffe700 (LWP 1754501) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbda0c4e1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007fbda1111fa1 in _Unwind_Find_FDE (pc=0x7fbda110ff68 <_Unwind_Resume_or_Rethrow+40>, bases=bases@entry=0x7fbccbff71c8) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007fbda110e078 in uw_frame_state_for (context=0x7fbccbff7120, fs=0x7fbccbff6e70) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007fbda110f4e1 in _Unwind_RaiseException_Phase2 (exc=0x7fbc27f77760, context=0x7fbccbff7120, frames_p=0x7fbccbff7210) at ../../../libgcc/unwind.inc:50 #10 0x00007fbda110faf1 in _Unwind_RaiseException (exc=0x7fbc27f77760) at ../../../libgcc/unwind.inc:136 #11 0x00007fbda110ff69 in _Unwind_Resume_or_Rethrow (exc=0x7fbc27f77760) at ../../../libgcc/unwind.inc:264 #12 0x00007fbda1548530 in __cxxabiv1::__cxa_rethrow () at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:126 #13 0x00007fbda153c997 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:80 #14 0x00007fbda15481ea in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #15 0x00007fbda1547249 in __cxa_call_terminate (ue_header=0x7fbc27f77760) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #16 0x00007fbda1547977 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7fbc27f77760, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685 #17 0x00007fbda110f454 in _Unwind_RaiseException_Phase2 (exc=0x7fbc27f77760, context=0x7fbccbff7a20, frames_p=0x7fbccbff7b10) at ../../../libgcc/unwind.inc:64 #18 0x00007fbda110faf1 in _Unwind_RaiseException (exc=0x7fbc27f77760) at ../../../libgcc/unwind.inc:136 #19 0x00007fbda15484dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7fbd4372eb90 , dest=0x7fbd42df7c40 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #20 0x00007fbd42dcecaf in edm::Wrapper > >::~Wrapper() [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalCaloEcalRecProducersPlugins.so #21 0x00007fbda353811a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007fbda35ddee6 in edm::DataManagingProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #23 0x00007fbda35ce1b8 in edm::Principal::clearPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #24 0x00007fbda35476cd in edm::EventPrincipal::clearEventPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #25 0x00007fbda3583bb2 in edm::FunctorWaitingTask, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #26 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #27 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbe23d8400, waiter=..., this=0x7fbd9bb4eb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #28 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4eb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #29 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #30 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #31 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2aa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #32 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2aa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #33 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #34 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fbccc9ff700 (LWP 1754496) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbcc778cb63 in edm::Wrapper > >::~Wrapper() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoPixelVertexingPixelVertexFindingPlugins.so #8 0x00007fbda353811a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda35ddee6 in edm::DataManagingProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda35ce1b8 in edm::Principal::clearPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35476cd in edm::EventPrincipal::clearEventPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda3583bb2 in edm::FunctorWaitingTask, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc555dbd00, waiter=..., this=0x7fbd9bb4ea80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ea80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2ad00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2ad00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fbccd9ff700 (LWP 1754487) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbda0c4e1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007fbda1111fa1 in _Unwind_Find_FDE (pc=0x7fbda15484db <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+59>, bases=bases@entry=0x7fbccd9f84b8) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007fbda110e078 in uw_frame_state_for (context=0x7fbccd9f8410, fs=0x7fbccd9f8500) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007fbda110f9b2 in _Unwind_RaiseException (exc=0x7fbbf2a256a0) at ../../../libgcc/unwind.inc:104 #10 0x00007fbda15484dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7fbda169d220 , dest=0x7fbda155cf10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #11 0x00007fbd43464369 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsCommon.so #12 0x00007fbd434650d2 in cms::cuda::ProductBase::isAvailable() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsCommon.so #13 0x00007fbd4168beeb in SiPixelRecHitFromCUDAT::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelRecHitsPlugins.so #14 0x00007fbda3651ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007fbda3655b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007fbda36326d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007fbda3632846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007fbda35c99ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #20 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbe625f900, waiter=..., this=0x7fbd9bb4e980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4e980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #24 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2ac80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2ac80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #26 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #27 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fbcce9fd700 (LWP 1754466) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda1149dc9 in expm1f64 () from /lib64/libm.so.6 #5 0x00007fbda1146208 in __sinh_finite () from /lib64/libm.so.6 #6 0x00007fbda1128773 in sinhf64 () from /lib64/libm.so.6 #7 0x00007fbcadeec625 in FixedGridRhoProducerFastjetFromRecHit::getHitP4(DetId const&, double, CaloGeometry const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #8 0x00007fbcadef0ad6 in FixedGridRhoProducerFastjetFromRecHit::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoJetsJetProducers_plugins.so #9 0x00007fbda365595d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda363c072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4fbefd00, waiter=..., this=0x7fbd9bb4e880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4e880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2ac00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2ac00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fbccf3fe700 (LWP 1754465) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbd4bd355a3 in edm::Wrapper >::~Wrapper() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginBeamSpotToCUDA.so #8 0x00007fbda353811a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda35ddee6 in edm::DataManagingProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda35ce1b8 in edm::Principal::clearPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35476cd in edm::EventPrincipal::clearEventPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda3583bb2 in edm::FunctorWaitingTask, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc58681c00, waiter=..., this=0x7fbd9bb4e900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4e900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2ab80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2ab80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fbccffff700 (LWP 1754455) "cmsRun"): #0 0x00007fbda0eec3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbda156ce90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007fbd95ac42d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007fbda15729b4 in std::execute_native_thread_routine (__p=0x7fbd9354b650) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fbcd1ffc700 (LWP 1754440) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 edata_arena_ind_get (edata=) at include/jemalloc/internal/edata.h:262 #5 tcache_bin_flush_match (small=true, cur_binshard=, cur_arena_ind=17, edata=) at src/tcache.c:301 #6 tcache_bin_flush_impl (small=true, nflush=, ptrs=0x7fbcd1ff5bc0, binind=, cache_bin=, tcache=, tsd=0x7fbcd1ff5c00) at src/tcache.c:434 #7 tcache_bin_flush_bottom (small=true, rem=, binind=, cache_bin=, tcache=, tsd=tsd@entry=0x7fbcd1ff5c00) at src/tcache.c:519 #8 je_tcache_bin_flush_small (tsd=tsd@entry=0x7fbcd1ff67f8, tcache=, cache_bin=, binind=, rem=) at src/tcache.c:529 #9 0x00007fbda21a83b4 in tcache_dalloc_small (slow_path=false, binind=, ptr=0x7fbc46dffd40, tcache=, tsd=0x7fbcd1ff67f8) at include/jemalloc/internal/tcache_inlines.h:157 #10 arena_sdalloc (slow_path=, caller_alloc_ctx=, tcache=, size=, ptr=, tsdn=) at include/jemalloc/internal/arena_inlines_b.h:418 #11 isdalloct (slow_path=, alloc_ctx=, tcache=, size=, ptr=, tsdn=) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:133 #12 isfree (slow_path=false, tcache=, usize=, ptr=0x7fbc46dffd40, tsd=0x7fbcd1ff67f8) at src/jemalloc.c:2982 #13 je_sdallocx_default (ptr=0x7fbc46dffd40, size=, flags=) at src/jemalloc.c:3924 #14 0x00007fbda221a33e in sizedDeleteImpl (size=, ptr=) at src/jemalloc_cpp.cpp:195 #15 operator delete (ptr=, size=) at src/jemalloc_cpp.cpp:200 #16 0x00007fbd4be1c03a in edm::Wrapper >::~Wrapper() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #17 0x00007fbda353811a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007fbda35ddee6 in edm::DataManagingProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007fbda35de893 in edm::SwitchBaseProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007fbda35de8d4 in edm::SwitchProducerProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007fbda35ce1b8 in edm::Principal::clearPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007fbda35476cd in edm::EventPrincipal::clearEventPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #23 0x00007fbda3583bb2 in edm::FunctorWaitingTask, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #24 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #25 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fb80c3b6b00, waiter=..., this=0x7fbd9bb4ee80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #26 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ee80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #27 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #28 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #29 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2af00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #30 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2af00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #31 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #32 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fbcd0bfa700 (LWP 1754439) "cmsRun"): #0 0x00007fbda0eec3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbda156ce90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007fbd95ac42d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007fbda15729b4 in std::execute_native_thread_routine (__p=0x7fbd9354b500) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fbcd15fb700 (LWP 1754435) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbc98f74e5a in PFMultiDepthClusterizer::link(std::vector > const&, std::vector > const&, std::vector > const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #5 0x00007fbc98f77229 in PFMultiDepthClusterizer::buildClusters(std::vector > const&, std::vector > const&, std::vector >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #6 0x00007fbc98f728e6 in PFMultiDepthClusterProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #7 0x00007fbda365595d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007fbda363c072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf5916100, waiter=..., this=0x7fbd9bb4ed00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ed00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2ae00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2ae00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #19 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fbcd4bfe700 (LWP 1754429) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbcaca2900a in void std::__introsort_loop<__gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter > >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter >) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginJetMETCorrectionsModulesPlugins.so #5 0x00007fbcaca292e2 in void std::__introsort_loop<__gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter > >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter >) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginJetMETCorrectionsModulesPlugins.so #6 0x00007fbcaca21a7c in reco::CorrectedJetProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginJetMETCorrectionsModulesPlugins.so #7 0x00007fbda36419cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007fbda363b5d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbc4dfbda00, waiter=..., this=0x7fbd9bb4eb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4eb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2b000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2b000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #19 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fbcd29fd700 (LWP 1754428) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0c4e2a5 in dl_iterate_phdr () from /lib64/libc.so.6 #5 0x00007fbda1111fa1 in _Unwind_Find_FDE (pc=0x7fbd4bd37af0 , bases=bases@entry=0x7fbcd29f6478) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #6 0x00007fbda110e078 in uw_frame_state_for (context=0x7fbcd29f63d0, fs=0x7fbcd29f64c0) at ../../../libgcc/unwind-dw2.c:1263 #7 0x00007fbda110f9b2 in _Unwind_RaiseException (exc=0x7fbc33d79920) at ../../../libgcc/unwind.inc:104 #8 0x00007fbda15484dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7fbda169d220 , dest=0x7fbda155cf10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #9 0x00007fbd93e4b91e in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #10 0x00007fbd93e4f476 in cms::cuda::EventCache::get() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #11 0x00007fbd4bd37af1 in BeamSpotToCUDA::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginBeamSpotToCUDA.so #12 0x00007fbda36419cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda363b5d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf5c54300, waiter=..., this=0x7fbd9bb4e800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4e800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2ad80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2ad80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #24 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fbcd35fe700 (LWP 1754427) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e5208f in cms::cuda::allocate_device(int, unsigned long, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbd433a7362 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007fbda3651ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda3655b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda36326d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda3632846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda35c99ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbe623e400, waiter=..., this=0x7fbd9bb4ea00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4ea00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2ae80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2ae80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fbcd3dff700 (LWP 1754426) "cmsRun"): #0 0x00007fbda0eec3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbda156ce90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007fbd95ac42d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007fbda15729b4 in std::execute_native_thread_routine (__p=0x7fbd9354b2f0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fbd1d7ff700 (LWP 1754410) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbd93e5208f in cms::cuda::allocate_device(int, unsigned long, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007fbd4be3cf2f in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #8 0x00007fbd4be2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #9 0x00007fbda3651ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda3655b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda36326d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda3632846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda35c99ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbf55d9300, waiter=..., this=0x7fbd9bb4e700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4e700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2af80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2af80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #22 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fbd2e8b6700 (LWP 1754397) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbd8d3660a9 in BasicTrajectoryState::BasicTrajectoryState(LocalTrajectoryParameters const&, LocalTrajectoryError const&, Surface const&, MagneticField const*, SurfaceSideDefinition::SurfaceSide) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrajectoryState.so #5 0x00007fbd43e8bb92 in TrajectoryStateOnSurface (anonymous namespace)::lupdate<2u>(TrajectoryStateOnSurface const&, TrackingRecHit const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #6 0x00007fbd43e93fae in KFUpdator::update(TrajectoryStateOnSurface const&, TrackingRecHit const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsKalmanUpdators.so #7 0x00007fbca0871337 in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #8 0x00007fbca0871a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #9 0x00007fbda365595d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda363c072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #14 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbe6167900, waiter=..., this=0x7fbd9bb4e780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4e780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2b100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2b100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #21 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fbd2f2b7700 (LWP 1754393) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda0eef75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007fbda0ee8b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007fbda0c4e1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007fbda1111fa1 in _Unwind_Find_FDE (pc=0x7fbd93e4b91d >) [clone .constprop.0] [clone .cold]+40>, bases=bases@entry=0x7fbd2f2b07f8) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007fbda110e078 in uw_frame_state_for (context=0x7fbd2f2b0750, fs=0x7fbd2f2b0840) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007fbda110f9b2 in _Unwind_RaiseException (exc=0x7fbcc5b26180) at ../../../libgcc/unwind.inc:104 #10 0x00007fbda15484dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7fbda169d220 , dest=0x7fbda155cf10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #11 0x00007fbd93e4b91e in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #12 0x00007fbd93e53229 in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #13 0x00007fbd4bd355a3 in edm::Wrapper >::~Wrapper() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginBeamSpotToCUDA.so #14 0x00007fbda353811a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007fbda35ddee6 in edm::DataManagingProductResolver::resetProductData_(bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007fbda35ce1b8 in edm::Principal::clearPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007fbda35476cd in edm::EventPrincipal::clearEventPrincipal() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007fbda3583bb2 in edm::FunctorWaitingTask, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::Conditional >, edm::waiting_task::detail::AutoExceptionHandler >::runLast(edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007fbda35367cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007fbda1d9a304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7fbbe60e6600, waiter=..., this=0x7fbd9bb4e680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7fbd9bb4e680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #24 0x00007fbda1d9c4c6 in tbb::detail::r1::rml::private_worker::run (this=0x7fbd9bb2b080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7fbd9bb2b080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #26 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #27 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fbd63fff700 (LWP 1754369) "cuda-EvtHandlr"): #0 0x00007fbda0c08a71 in poll () from /lib64/libc.so.6 #1 0x00007fbd8377dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007fbd83831d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007fbd83780768 in ?? () from /lib64/libcuda.so.1 #4 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fbd64d51700 (LWP 1754367) "cuda-EvtHandlr"): #0 0x00007fbda0c08a71 in poll () from /lib64/libc.so.6 #1 0x00007fbd8377dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007fbd83831d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007fbd83780768 in ?? () from /lib64/libcuda.so.1 #4 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fbd6b853700 (LWP 1754299) "cmsRun"): #0 0x00007fbda0ef05a2 in waitpid () from /lib64/libpthread.so.0 #1 0x00007fbd99adc157 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007fbd99adea9a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007fbda15729b4 in std::execute_native_thread_routine (__p=0x7fbd9b3f38d0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007fbda0ee617a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fbda0c13df3 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fbda0075640 (LWP 1753372) "cmsRun"): #0 0x00007fbda0bdfd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007fbda0bdfc9e in sleep () from /lib64/libc.so.6 #2 0x00007fbd99adbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007fbda3867e06 in edm::RefCore::RefCore(edm::RefCore const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsCommon.so #5 0x00007fbcaca1f6d3 in reco::Jet::Jet(reco::Jet const&) [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginJetMETCorrectionsModulesPlugins.so #6 0x00007fbcaca29f6c in void std::__unguarded_linear_insert<__gnu_cxx::__normal_iterator > >, __gnu_cxx::__ops::_Val_comp_iter > >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__ops::_Val_comp_iter >) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginJetMETCorrectionsModulesPlugins.so #7 0x00007fbcaca21aa8 in reco::CorrectedJetProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginJetMETCorrectionsModulesPlugins.so #8 0x00007fbda36419cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007fbda363b5d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007fbda35c86da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007fbda35c8b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007fbda331df79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007fbda1da19cd in tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=0x7fbbf5564600, this=0x7fbd9bb4e600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=, this=0x7fbd9bb4e600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168 #16 0x00007fbda354b10d in edm::FinalWaitingTask::wait() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007fbda3558b91 in edm::EventProcessor::processRuns() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007fbda35590e6 in edm::EventProcessor::runToCompletion() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x000000000040c17d in tbb::detail::d1::task_arena_function::operator()() const () #20 0x00007fbda1d8f847 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:694 #21 0x00000000004101d0 in main::{lambda()#1}::operator()() const () #22 0x0000000000407d6c in main () Current Modules: Module: none (crashed) Module: EcalRawToDigiGPU:hltEcalDigisGPU Module: none Module: TrackProducer:hltIter0PFlowCtfWithMaterialTracks Module: CkfTrackCandidateMaker:hltIter0PFlowCkfTrackCandidates Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoubletsUnseeded Module: EcalRawToDigiGPU:hltEcalDigisGPU Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: none Module: CorrectedPFJetProducer:hltAK4PixelOnlyPFJetsLooseIDCorrected Module: none Module: SiPixelRawToClusterCUDA:hltSiPixelClustersGPU Module: none Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: none Module: BeamSpotToCUDA:hltOnlineBeamSpotToGPU Module: FixedGridRhoProducerFastjetFromRecHit:hltFixedGridRhoFastjetAllCaloForMuons Module: EcalRawToDigiGPU:hltEcalDigisGPU Module: CkfTrajectoryMaker:hltL3TrackCandidateFromL2IOHit Module: EgammaHLTHcalVarProducerFromRecHit:hltEgammaHoverEUnseeded Module: PFMultiDepthClusterProducer:hltParticleFlowClusterHCAL Module: none Module: none Module: none Module: none Module: ElectronNHitSeedProducer:hltEgammaElectronPixelSeedsUnseeded Module: HcalDigisProducerGPU:hltHcalDigisGPU Module: SiPixelRawToClusterCUDA:hltSiPixelClustersGPU Module: none Module: SiPixelRawToClusterCUDA:hltSiPixelClustersGPU Module: SiPixelRecHitFromCUDAPhase1:hltSiPixelRecHitsFromGPU Module: CorrectedPFJetProducer:hltAK4PixelOnlyPFJetsTightIDCorrected A fatal system signal has occurred: abort signal msgtime:2023-05-07 13:45:22 doc_type:cmsswlog date:2023-05-07T11:45:22.989Z run:367112 host:fu-c2b03-31-01 pid:1781992 doctype:stacktrace severity:FATAL severityVal:4 instance:global lexicalId:4050327622 message:terminate called after throwing an instance of 'std::runtime_error' terminate called recursively terminate called recursively A fatal system signal has occurred: abort signal The following is the call stack containing the origin of the signal. Sun May 7 13:44:58 CEST 2023 Thread 44 (Thread 0x7f03d37ff700 (LWP 1782244) "cmsRun"): #0 0x00007f0592706cd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0 #1 0x00007f0592706dc8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007f0574f6b0a2 in ?? () from /lib64/libcuda.so.1 #3 0x00007f0574f80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f043b1ff700 (LWP 1782241) "cmsRun"): #0 0x00007f059270474a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f05872c7061 in FedRawDataInputSource::readSupervisor() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #2 0x00007f0592d8a9b4 in std::execute_native_thread_routine (__p=0x7f0449b49ee0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #4 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f043e3fe700 (LWP 1782240) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f0585653eb4 in cms::cuda::free_host(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b824 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03b8196200, waiter=..., this=0x7f058d34f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f043edff700 (LWP 1782239) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03e5ad1f00, waiter=..., this=0x7f058d34f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f04407ff700 (LWP 1782238) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05924661e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f0592929fa1 in _Unwind_Find_FDE (pc=0x7f0592d604db <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+59>, bases=bases@entry=0x7f04407f7b68) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f0592926078 in uw_frame_state_for (context=0x7f04407f7ac0, fs=0x7f04407f7bb0) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f05929279b2 in _Unwind_RaiseException (exc=0x7f03e18653c0) at ../../../libgcc/unwind.inc:104 #10 0x00007f0592d604dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f0592eb5220 , dest=0x7f0592d74f10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #11 0x00007f058564b91e in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #12 0x00007f0585654377 in cms::cuda::free_host(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #13 0x00007f053ddcada1 in std::unique_ptr, cms::cuda::host::impl::HostDeleter>::~unique_ptr() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #14 0x00007f053ddca3e4 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #15 0x00007f053de3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #16 0x00007f053de2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #17 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #23 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03e5b7b300, waiter=..., this=0x7f058d34f480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #24 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #25 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #26 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #27 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #28 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #29 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #30 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f04415ff700 (LWP 1782237) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f043fb7d000, waiter=..., this=0x7f058d34f400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f04429ff700 (LWP 1782236) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03dc106400, waiter=..., this=0x7f058d34f380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f04439fe700 (LWP 1782235) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f0585653eb4 in cms::cuda::free_host(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b824 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03e5baec00, waiter=..., this=0x7f058d34f300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f04447ff700 (LWP 1782234) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f043fba9300, waiter=..., this=0x7f058d34f280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f04455ff700 (LWP 1782233) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f0441fb6c00, waiter=..., this=0x7f058d34f200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f04469ff700 (LWP 1782232) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05924661e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f0592929fa1 in _Unwind_Find_FDE (pc=0x7f0594aec63b , bases=bases@entry=0x7f04469f8858) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f0592926078 in uw_frame_state_for (context=0x7f04469f87b0, fs=0x7f04469f88a0) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f05929279b2 in _Unwind_RaiseException (exc=0x7f03d26915e0) at ../../../libgcc/unwind.inc:104 #10 0x00007f0592d604dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f0594b2d660 , dest=0x7f0594af21f0 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #11 0x00007f0594aec63c in edm::convertException::stdToEDM(std::exception const&) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreUtilities.so #12 0x00007f0594d389b7 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594de0b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03b80dbf00, waiter=..., this=0x7f058d34f180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f0447d70700 (LWP 1782231) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0594f61ce4 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2 #5 0x00007f0594f66024 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2 #6 0x00007f0594f6d51e in _dl_runtime_resolve_xsavec () from /lib64/ld-linux-x86-64.so.2 #7 0x00007f0534c7e760 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #8 0x00007f0534c7f8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #9 0x00007f0534c7f911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #10 0x00007f053329a3c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #11 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0594b35f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #17 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03b80a4d00, waiter=..., this=0x7f058d34f100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f0448771700 (LWP 1782230) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05924661e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f0592929fa1 in _Unwind_Find_FDE (pc=0x7f0592d6052f <__cxxabiv1::__cxa_rethrow()+63>, bases=bases@entry=0x7f04487695f8) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f0592926078 in uw_frame_state_for (context=0x7f0448769550, fs=0x7f04487692a0) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f05929274e1 in _Unwind_RaiseException_Phase2 (exc=0x7f03dbcffba0, context=0x7f0448769550, frames_p=0x7f0448769640) at ../../../libgcc/unwind.inc:50 #10 0x00007f0592927af1 in _Unwind_RaiseException (exc=0x7f03dbcffba0) at ../../../libgcc/unwind.inc:136 #11 0x00007f0592927f69 in _Unwind_Resume_or_Rethrow (exc=0x7f03dbcffba0) at ../../../libgcc/unwind.inc:264 #12 0x00007f0592d60530 in __cxxabiv1::__cxa_rethrow () at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:126 #13 0x00007f0592d54997 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:80 #14 0x00007f0592d601ea in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #15 0x00007f0592d5f249 in __cxa_call_terminate (ue_header=0x7f03dbcffba0) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #16 0x00007f0592d5f977 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7f03dbcffba0, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685 #17 0x00007f0592927454 in _Unwind_RaiseException_Phase2 (exc=0x7f03dbcffba0, context=0x7f0448769e60, frames_p=0x7f0448769d68) at ../../../libgcc/unwind.inc:64 #18 0x00007f0592927e8e in _Unwind_Resume (exc=0x7f03dbcffba0) at ../../../libgcc/unwind.inc:241 #19 0x00007f058564c1d1 in cms::cuda::free_host(void*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #20 0x00007f053ddcada1 in std::unique_ptr, cms::cuda::host::impl::HostDeleter>::~unique_ptr() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #21 0x00007f053ddca3e4 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #22 0x00007f053de3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #23 0x00007f053de2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #24 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #25 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #26 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #27 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #28 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #29 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #30 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f043fb5e500, waiter=..., this=0x7f058d34f080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #31 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #32 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #33 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #34 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #35 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #36 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #37 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f0480f99700 (LWP 1782229) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f05872b268e in evf::FastMonitoringService::snapshotRunner() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #3 0x00007f0592d8a9b4 in std::execute_native_thread_routine (__p=0x7f0449d3b740) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f04917ff700 (LWP 1782182) "cmsRun"): #0 0x00007f05927043fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0592d84e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f0526360acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f05263610ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f052635dc68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f05231874c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f049280e700 (LWP 1782181) "cmsRun"): #0 0x00007f05927043fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0592d84e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f0526360acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f05263610ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f052635dc68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f05231874c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f049300f700 (LWP 1782180) "cmsRun"): #0 0x00007f05927043fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0592d84e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f0526360acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f05263610ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f052635dc68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f05231874c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f04b73fb700 (LWP 1782178) "cmsRun"): #0 0x00007f05927043fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0592d84e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f05872c42d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f0592d8a9b4 in std::execute_native_thread_routine (__p=0x7f05811e1650) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f04b87fd700 (LWP 1782177) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05924661e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f0592929fa1 in _Unwind_Find_FDE (pc=0x7f0592d604db <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+59>, bases=bases@entry=0x7f04b87f6858) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f0592926078 in uw_frame_state_for (context=0x7f04b87f67b0, fs=0x7f04b87f6500) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f05929274e1 in _Unwind_RaiseException_Phase2 (exc=0x7f03f00b46e0, context=0x7f04b87f67b0, frames_p=0x7f04b87f68a0) at ../../../libgcc/unwind.inc:50 #10 0x00007f0592927af1 in _Unwind_RaiseException (exc=0x7f03f00b46e0) at ../../../libgcc/unwind.inc:136 #11 0x00007f0592d604dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f0594b2d660 , dest=0x7f0594af21f0 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #12 0x00007f0594aec63c in edm::convertException::stdToEDM(std::exception const&) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreUtilities.so #13 0x00007f0594d389b7 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0594de0b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f0447293b00, waiter=..., this=0x7f058d34ef80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ef80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f04b7dfc700 (LWP 1782176) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f0442f1f900, waiter=..., this=0x7f058d34f000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f04b8ffe700 (LWP 1782175) "cmsRun"): #0 0x00007f05927043fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0592d84e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f05872c42d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f0592d8a9b4 in std::execute_native_thread_routine (__p=0x7f05811e1500) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f04b99ff700 (LWP 1782174) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f0445f28700, waiter=..., this=0x7f058d34ee00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ee00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f04ba9fe700 (LWP 1782173) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f053416cb51 in bool Eigen::internal::ldlt_inplace<1>::unblocked, Eigen::Transpositions<6, 6, int>, Eigen::Matrix >(Eigen::Matrix&, Eigen::Transpositions<6, 6, int>&, Eigen::Matrix&, Eigen::internal::SignMatrix&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalCaloHcalRecAlgos.so #5 0x00007f0534164e30 in MahiFit::solveSubmatrix(Eigen::Matrix&, Eigen::Matrix&, Eigen::Matrix&, unsigned int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalCaloHcalRecAlgos.so #6 0x00007f0534165c80 in MahiFit::nnls() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalCaloHcalRecAlgos.so #7 0x00007f0534166575 in MahiFit::minimize() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalCaloHcalRecAlgos.so #8 0x00007f05341673a5 in MahiFit::doFit(std::array&, int) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalCaloHcalRecAlgos.so #9 0x00007f0534167a6f in MahiFit::phase1Apply(HBHEChannelInfo const&, float&, float&, float&, bool&, float&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalCaloHcalRecAlgos.so #10 0x00007f053417813f in SimpleHBHEPhase1Algo::reconstruct(HBHEChannelInfo const&, HcalRecoParam const*, HcalCalibrations const&, bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalCaloHcalRecAlgos.so #11 0x00007f053381da09 in HBHEPhase1Reconstructor::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalCaloHcalRecProducers.so #12 0x00007f0594e6d95d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594e54072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0594de06da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0594de0b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0594b35f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #17 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f0441f7da00, waiter=..., this=0x7f058d34ed80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ed80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f04bb3ff700 (LWP 1782172) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f0441f0c600, waiter=..., this=0x7f058d34ed00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ed00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32aa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32aa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f04bbffe700 (LWP 1782171) "cmsRun"): #0 0x00007f05927043fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0592d84e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f05872c42d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f0592d8a9b4 in std::execute_native_thread_routine (__p=0x7f05811e1470) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f04bc9ff700 (LWP 1782170) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f0592929026 in _Unwind_IteratePhdrCallback (info=, size=64, ptr=0x7f04bc9f83e0) at ../../../libgcc/unwind-dw2-fde-dip.c:280 #5 0x00007f0592466327 in dl_iterate_phdr () from /lib64/libc.so.6 #6 0x00007f0592929fa1 in _Unwind_Find_FDE (pc=0x7f053dd37af0 , bases=bases@entry=0x7f04bc9f8818) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #7 0x00007f0592926078 in uw_frame_state_for (context=0x7f04bc9f8770, fs=0x7f04bc9f84b0) at ../../../libgcc/unwind-dw2.c:1263 #8 0x00007f05929274e1 in _Unwind_RaiseException_Phase2 (exc=0x7f042c32d380, context=0x7f04bc9f8770, frames_p=0x7f04bc9f8678) at ../../../libgcc/unwind.inc:50 #9 0x00007f0592927e8e in _Unwind_Resume (exc=0x7f042c32d380) at ../../../libgcc/unwind.inc:241 #10 0x00007f058564bacb in cms::cuda::EventCache::get() [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #11 0x00007f053dd37af1 in BeamSpotToCUDA::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginBeamSpotToCUDA.so #12 0x00007f0594e599cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594e535d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0594de06da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0594de0b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03e5b7a400, waiter=..., this=0x7f058d34ec00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ec00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f04f25ff700 (LWP 1782169) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05924661e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f0592929fa1 in _Unwind_Find_FDE (pc=0x7f0592d604db <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+59>, bases=bases@entry=0x7f04f25f8368) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f0592926078 in uw_frame_state_for (context=0x7f04f25f82c0, fs=0x7f04f25f8010) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f05929274e1 in _Unwind_RaiseException_Phase2 (exc=0x7f0427b279a0, context=0x7f04f25f82c0, frames_p=0x7f04f25f83b0) at ../../../libgcc/unwind.inc:50 #10 0x00007f0592927af1 in _Unwind_RaiseException (exc=0x7f0427b279a0) at ../../../libgcc/unwind.inc:136 #11 0x00007f0592d604dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f0592eb5220 , dest=0x7f0592d74f10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #12 0x00007f0534c7e740 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #13 0x00007f0534c7f8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #14 0x00007f0534c7f911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #15 0x00007f053329a3c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #16 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f0594b35f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #22 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f044728bd00, waiter=..., this=0x7f058d34eb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #23 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34eb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #24 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #25 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #26 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32ab00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #27 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32ab00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #28 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #29 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f05073fe700 (LWP 1782168) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05924661e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f0592929fa1 in _Unwind_Find_FDE (pc=0x7f058564c132 , bases=bases@entry=0x7f05073f7688) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f0592926078 in uw_frame_state_for (context=0x7f05073f75e0, fs=0x7f05073f7320) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f05929274e1 in _Unwind_RaiseException_Phase2 (exc=0x7f04391264c0, context=0x7f05073f75e0, frames_p=0x7f05073f74e8) at ../../../libgcc/unwind.inc:50 #10 0x00007f0592927e8e in _Unwind_Resume (exc=0x7f04391264c0) at ../../../libgcc/unwind.inc:241 #11 0x00007f058564c133 in cms::cuda::allocate_host(unsigned long, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #12 0x00007f0534bb7402 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #13 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03dc102800, waiter=..., this=0x7f058d34ea80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #20 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ea80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #21 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #22 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #23 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32aa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #24 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32aa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #25 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #26 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f0517dff700 (LWP 1782167) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f0585653eb4 in cms::cuda::free_host(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f053ddcada1 in std::unique_ptr, cms::cuda::host::impl::HostDeleter>::~unique_ptr() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #8 0x00007f053ddca3e4 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #9 0x00007f053de3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #10 0x00007f053de2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #11 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f0440beba00, waiter=..., this=0x7f058d34ea00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ea00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32ac00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32ac00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f05195fe700 (LWP 1782166) "cmsRun"): #0 0x00007f0592420a71 in poll () from /lib64/libc.so.6 #1 0x00007f058b31346f in full_read.constprop () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f058b2deb6c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f058b2df33b in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #4 #5 0x00007f059236637f in raise () from /lib64/libc.so.6 #6 0x00007f0592350db5 in abort () from /lib64/libc.so.6 #7 0x00007f0592d621aa in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:50 #8 0x00007f0592d601ea in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #9 0x00007f0592d5f249 in __cxa_call_terminate (ue_header=0x7f03891cfbe0) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #10 0x00007f0592d5f977 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7f03891cfbe0, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685 #11 0x00007f0592927454 in _Unwind_RaiseException_Phase2 (exc=0x7f03891cfbe0, context=0x7f05195f7600, frames_p=0x7f05195f7508) at ../../../libgcc/unwind.inc:64 #12 0x00007f0592927e8e in _Unwind_Resume (exc=0x7f03891cfbe0) at ../../../libgcc/unwind.inc:241 #13 0x00007f058564c05c in cms::cuda::free_device(int, void*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #14 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #15 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03dbfc1b00, waiter=..., this=0x7f058d34e900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34e900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32ab80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32ab80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #28 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f0518bfd700 (LWP 1782165) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03e5daf200, waiter=..., this=0x7f058d34e880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34e880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32ad00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32ad00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f051a1ff700 (LWP 1782164) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f05927059d5 in pthread_kill () from /lib64/libpthread.so.0 #5 0x00007f058b2df9b8 in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #6 #7 0x00007f059236637f in raise () from /lib64/libc.so.6 #8 0x00007f0592350db5 in abort () from /lib64/libc.so.6 #9 0x00007f0592d621aa in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:50 #10 0x00007f0592d601ea in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #11 0x00007f0592d5f249 in __cxa_call_terminate (ue_header=0x7f03f24ea600) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #12 0x00007f0592d5f977 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7f03f24ea600, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685 #13 0x00007f0592927454 in _Unwind_RaiseException_Phase2 (exc=0x7f03f24ea600, context=0x7f051a1f7e60, frames_p=0x7f051a1f7d68) at ../../../libgcc/unwind.inc:64 #14 0x00007f0592927e8e in _Unwind_Resume (exc=0x7f03f24ea600) at ../../../libgcc/unwind.inc:241 #15 0x00007f058564c1d1 in cms::cuda::free_host(void*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #16 0x00007f053ddcada1 in std::unique_ptr, cms::cuda::host::impl::HostDeleter>::~unique_ptr() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #17 0x00007f053ddca3e4 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #18 0x00007f053de3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #19 0x00007f053de2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #20 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #23 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #24 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #25 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #26 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03b80da700, waiter=..., this=0x7f058d34ee80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #27 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ee80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #28 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #29 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #30 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32ae00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #31 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32ae00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #32 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #33 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f051affe700 (LWP 1782163) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05924661e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f0592929fa1 in _Unwind_Find_FDE (pc=0x7f0592d604db <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+59>, bases=bases@entry=0x7f051aff7308) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f0592926078 in uw_frame_state_for (context=0x7f051aff7260, fs=0x7f051aff7350) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f05929279b2 in _Unwind_RaiseException (exc=0x7f03cf43b880) at ../../../libgcc/unwind.inc:104 #10 0x00007f0592d604dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f0592eb5220 , dest=0x7f0592d74f10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #11 0x00007f058564b91e in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #12 0x00007f0585653229 in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #13 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #14 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f0440b30500, waiter=..., this=0x7f058d34ef00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ef00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #24 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32ac80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32ac80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #26 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #27 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f051bdff700 (LWP 1782162) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f053ddca3f9 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #8 0x00007f053de3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #9 0x00007f053de2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #10 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03b8133300, waiter=..., this=0x7f058d34ec80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34ec80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32ad80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32ad80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f051cdfc700 (LWP 1782161) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f05927077fa in __lll_unlock_wake () from /lib64/libpthread.so.0 #5 0x00007f0592702286 in __pthread_mutex_unlock_usercnt () from /lib64/libpthread.so.0 #6 0x00007f0592466343 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f0592929fa1 in _Unwind_Find_FDE (pc=0x7f0534c7f8e6 , bases=bases@entry=0x7f051cdf5368) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f0592926078 in uw_frame_state_for (context=0x7f051cdf52c0, fs=0x7f051cdf53b0) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f05929279b2 in _Unwind_RaiseException (exc=0x7f0429214960) at ../../../libgcc/unwind.inc:104 #10 0x00007f0592d604dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f0592eb5220 , dest=0x7f0592d74f10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #11 0x00007f0534c7e740 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #12 0x00007f0534c7f8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #13 0x00007f0534c7f911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #14 0x00007f053329a3c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #15 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f0594b35f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #21 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03dbfbd500, waiter=..., this=0x7f058d34eb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34eb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32af00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32af00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #28 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f051e3fe700 (LWP 1782160) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f0442f11500, waiter=..., this=0x7f058d34e980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34e980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32ae80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32ae80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f051d9fd700 (LWP 1782159) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 uw_init_context_1 (context=0x7f051d9f6870, outer_cfa=0x7f051d9f6c10, outer_ra=0x7f0592d601d2 ) at ../../../libgcc/unwind-dw2.c:1587 #5 0x00007f0592927829 in _Unwind_RaiseException (exc=0x7f03bb3f79e0) at ../../../libgcc/unwind.inc:93 #6 0x00007f0592d601d2 in std::rethrow_exception (ep=...) at ../../../../libstdc++-v3/libsupc++/eh_ptr.cc:212 #7 0x00007f0594e4a62b in edm::Worker::shouldRethrowException(std::__exception_ptr::exception_ptr, edm::ParentContext const&, bool) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f0594d38870 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594de0b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f043a951100, waiter=..., this=0x7f058d34e800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #12 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34e800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #13 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #14 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #15 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32b000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32b000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #17 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #18 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f051efff700 (LWP 1782158) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03b8108d00, waiter=..., this=0x7f058d34e780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34e780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32af80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32af80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f05200c6700 (LWP 1782157) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03d22a6100, waiter=..., this=0x7f058d34e700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34e700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32b100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32b100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f0520ac7700 (LWP 1782156) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f043fb32200, waiter=..., this=0x7f058d34e680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34e680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32b080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32b080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f05557ff700 (LWP 1782122) "cuda-EvtHandlr"): #0 0x00007f0592420a71 in poll () from /lib64/libc.so.6 #1 0x00007f0574f7dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f0575031d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f0574f80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f055651c700 (LWP 1782121) "cuda-EvtHandlr"): #0 0x00007f0592420a71 in poll () from /lib64/libc.so.6 #1 0x00007f0574f7dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f0575031d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f0574f80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f055d01e700 (LWP 1782113) "cmsRun"): #0 0x00007f05927085a2 in waitpid () from /lib64/libpthread.so.0 #1 0x00007f058b2dc157 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f058b2dea9a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f0592d8a9b4 in std::execute_native_thread_routine (__p=0x7f058cc0c8c0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f059188d640 (LWP 1781992) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b99cd in tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=0x7f03e57f4200, this=0x7f058d34e600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=, this=0x7f058d34e600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168 #17 0x00007f0594d6310d in edm::FinalWaitingTask::wait() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f0594d70b91 in edm::EventProcessor::processRuns() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f0594d710e6 in edm::EventProcessor::runToCompletion() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x000000000040c17d in tbb::detail::d1::task_arena_function::operator()() const () #21 0x00007f05935a7847 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:694 #22 0x00000000004101d0 in main::{lambda()#1}::operator()() const () #23 0x0000000000407d6c in main () Current Modules: A fatal system signal has occurred: abort signal The following is the call stack containing the origin of the signal. Sun May 7 13:45:07 CEST 2023 Thread 44 (Thread 0x7f03d37ff700 (LWP 1782244) "cmsRun"): #0 0x00007f0592706cd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0 #1 0x00007f0592706dc8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007f0574f6b0a2 in ?? () from /lib64/libcuda.so.1 #3 0x00007f0574f80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f043b1ff700 (LWP 1782241) "cmsRun"): #0 0x00007f059270474a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f05872c7061 in FedRawDataInputSource::readSupervisor() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #2 0x00007f0592d8a9b4 in std::execute_native_thread_routine (__p=0x7f0449b49ee0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #4 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f043e3fe700 (LWP 1782240) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f0585653eb4 in cms::cuda::free_host(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b824 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03b8196200, waiter=..., this=0x7f058d34f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f043edff700 (LWP 1782239) "cmsRun"): #0 0x00007f05923f7d98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f05923f7c9e in sleep () from /lib64/libc.so.6 #2 0x00007f058b2dbed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f059270775d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f0592700a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f05856529cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f0534b8b7b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f0594e69ad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f0594e6db82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f0594e4a6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f0594e4a846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f0594de19ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f0594d4e7cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f05935b2304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f03e5ad1f00, waiter=..., this=0x7f058d34f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f058d34f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f05935b44c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f058d32a400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f058d32a400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f05926fe17a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f059242bdf3 in clone () from /lib64/libc.so.6 msgtime:2023-05-07 13:45:42 doc_type:cmsswlog date:2023-05-07T11:45:42.269Z run:367112 host:fu-c2b03-31-01 pid:1753378 doctype:stacktrace severity:FATAL severityVal:4 instance:global lexicalId:4050327622 message:terminate called after throwing an instance of 'std::runtime_error' terminate called recursively terminate called recursively terminate called recursively what(): /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_5-el8_amd64_gcc11/build/CMSSW_13_0_5-build/tmp/BUILDROOT/5dd16b30069b277b991de4e3be8a9236/opt/cmssw/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/src/HeterogeneousCore/CUDACore/src/ScopedContext.cc, line 86: cudaCheck(cudaStreamAddCallback(stream, cudaScopedContextCallback, new CallbackData{waitingTaskHolder_, device}, 0)); cudaErrorLaunchFailure: unspecified launch failure A fatal system signal has occurred: abort signal The following is the call stack containing the origin of the signal. Sun May 7 13:45:08 CEST 2023 Thread 44 (Thread 0x7f7cbb3ff700 (LWP 1754802) "cmsRun"): #0 0x00007f7e6d80acd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0 #1 0x00007f7e6d80adc8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007f7e5016b0a2 in ?? () from /lib64/libcuda.so.1 #3 0x00007f7e50180768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f7d15fff700 (LWP 1754773) "cmsRun"): #0 0x00007f7e6d80874a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7e623cc061 in FedRawDataInputSource::readSupervisor() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #2 0x00007f7e6de8e9b4 in std::execute_native_thread_routine (__p=0x7f7d24857620) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #4 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f7d199ff700 (LWP 1754674) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e6da2b828 <_Unwind_RaiseException+72>, bases=bases@entry=0x7f7d199f8768) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d199f86c0, fs=0x7f7d199f8510) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b2d0 in uw_init_context_1 (context=0x7f7d199f86c0, outer_cfa=0x7f7d199f8a70, outer_ra=0x7f7e6de644dc <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+60>) at ../../../libgcc/unwind-dw2.c:1592 #10 0x00007f7e6da2b829 in _Unwind_RaiseException (exc=0x7f7c293d1be0) at ../../../libgcc/unwind.inc:93 #11 0x00007f7e6de644dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f7e6fc31660 , dest=0x7f7e6fbf61f0 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #12 0x00007f7e6fbf063c in edm::convertException::stdToEDM(std::exception const&) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreUtilities.so #13 0x00007f7e6fe3c9b7 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6fee4b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cbe0a2400, waiter=..., this=0x7f7e68454580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f7d1c5fd700 (LWP 1754673) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e663e4270 in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #5 #6 0x00007f7e6d46a37f in raise () from /lib64/libc.so.6 #7 0x00007f7e6d454db5 in abort () from /lib64/libc.so.6 #8 0x00007f7e6de661aa in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:50 #9 0x00007f7e6de641ea in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #10 0x00007f7e6de63249 in __cxa_call_terminate (ue_header=0x7f7d10d300a0) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #11 0x00007f7e6de63977 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7f7d10d300a0, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685 #12 0x00007f7e6da2b454 in _Unwind_RaiseException_Phase2 (exc=0x7f7d10d300a0, context=0x7f7d1c5f6600, frames_p=0x7f7d1c5f6508) at ../../../libgcc/unwind.inc:64 #13 0x00007f7e6da2be8e in _Unwind_Resume (exc=0x7f7d10d300a0) at ../../../libgcc/unwind.inc:241 #14 0x00007f7e6075105c in cms::cuda::free_device(int, void*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #15 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #16 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cca4a3900, waiter=..., this=0x7f7e68454500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #23 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #24 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #25 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #26 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #27 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #28 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #29 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f7d1a9ff700 (LWP 1754672) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e6da2bd9a <_Unwind_Resume+58>, bases=bases@entry=0x7f7d1a9f8908) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d1a9f8860, fs=0x7f7d1a9f86a0) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b2d0 in uw_init_context_1 (context=0x7f7d1a9f8860, outer_cfa=0x7f7d1a9f8a90, outer_ra=0x7f7e6fbf0659 , std::allocator >&) [clone .cold]>) at ../../../libgcc/unwind-dw2.c:1592 #10 0x00007f7e6da2bd9b in _Unwind_Resume (exc=0x7f7c968e69e0) at ../../../libgcc/unwind.inc:235 #11 0x00007f7e6fbf0659 in edm::convertException::stdToEDM(std::exception const&) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreUtilities.so #12 0x00007f7e6fe3c9b7 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fee4b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7dbad14500, waiter=..., this=0x7f7e68454480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f7d1cffe700 (LWP 1754671) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d9d572000, waiter=..., this=0x7f7e68454400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f7d1bbfc700 (LWP 1754670) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d9b186200, waiter=..., this=0x7f7e68454380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f7d1dbff700 (LWP 1754669) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e60758eb4 in cms::cuda::free_host(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e19dcada1 in std::unique_ptr, cms::cuda::host::impl::HostDeleter>::~unique_ptr() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #8 0x00007f7e19dca3e4 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #9 0x00007f7e19e3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #10 0x00007f7e19e2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #11 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d96104500, waiter=..., this=0x7f7e68454300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f7d1e9ff700 (LWP 1754668) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d8099d5 in pthread_kill () from /lib64/libpthread.so.0 #5 0x00007f7e663e49b8 in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #6 #7 0x00007f7e6d46a37f in raise () from /lib64/libc.so.6 #8 0x00007f7e6d454db5 in abort () from /lib64/libc.so.6 #9 0x00007f7e6de5898a in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95 #10 0x00007f7e6de641ea in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #11 0x00007f7e6de63249 in __cxa_call_terminate (ue_header=0x7f7cfbd23600) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #12 0x00007f7e6de63977 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7f7cfbd23600, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685 #13 0x00007f7e6da2b454 in _Unwind_RaiseException_Phase2 (exc=0x7f7cfbd23600, context=0x7f7d1e9f8460, frames_p=0x7f7d1e9f8368) at ../../../libgcc/unwind.inc:64 #14 0x00007f7e6da2be8e in _Unwind_Resume (exc=0x7f7cfbd23600) at ../../../libgcc/unwind.inc:241 #15 0x00007f7e0fd8b768 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #16 0x00007f7e0fd8c8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #17 0x00007f7e0fd8c911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #18 0x00007f7e0e3a73c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #19 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #23 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #24 0x00007f7e6fc39f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #25 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cca61eb00, waiter=..., this=0x7f7e68454280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #26 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #27 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #28 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #29 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #30 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #31 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #32 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f7d221fe700 (LWP 1754667) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e0fc9485f >) [clone .cold]+80>, bases=bases@entry=0x7f7d221f73b8) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d221f7310, fs=0x7f7d221f7050) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b4e1 in _Unwind_RaiseException_Phase2 (exc=0x7f7cd22e4c80, context=0x7f7d221f7310, frames_p=0x7f7d221f7218) at ../../../libgcc/unwind.inc:50 #10 0x00007f7e6da2be8e in _Unwind_Resume (exc=0x7f7cd22e4c80) at ../../../libgcc/unwind.inc:241 #11 0x00007f7e0fc94860 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #12 0x00007f7e0fce1307 in ecal::raw::entryPoint(ecal::raw::InputDataCPU const&, ecal::raw::InputDataGPU&, ecal::raw::OutputDataGPU&, ecal::raw::ScratchDataGPU&, ecal::raw::OutputDataCPU&, ecal::raw::ConditionsProducts const&, CUstream_st*, unsigned int, unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #13 0x00007f7e0fcc470c in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #14 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d21706800, waiter=..., this=0x7f7e68454200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #24 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #26 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #27 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f7d1fbff700 (LWP 1754666) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cca679800, waiter=..., this=0x7f7e68454180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f7d20fff700 (LWP 1754665) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cca415c00, waiter=..., this=0x7f7e68454100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f7d22bff700 (LWP 1754664) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6da2d026 in _Unwind_IteratePhdrCallback (info=, size=64, ptr=0x7f7d22bf7910) at ../../../libgcc/unwind-dw2-fde-dip.c:280 #5 0x00007f7e6d56a327 in dl_iterate_phdr () from /lib64/libc.so.6 #6 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e19dcada0 , cms::cuda::host::impl::HostDeleter>::~unique_ptr()+22>, bases=bases@entry=0x7f7d22bf7b68) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #7 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d22bf7ac0, fs=0x7f7d22bf7bb0) at ../../../libgcc/unwind-dw2.c:1263 #8 0x00007f7e6da2b9b2 in _Unwind_RaiseException (exc=0x7f7cb516db20) at ../../../libgcc/unwind.inc:104 #9 0x00007f7e6de644dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f7e6dfb9220 , dest=0x7f7e6de78f10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #10 0x00007f7e6075091e in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #11 0x00007f7e60759377 in cms::cuda::free_host(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #12 0x00007f7e19dcada1 in std::unique_ptr, cms::cuda::host::impl::HostDeleter>::~unique_ptr() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #13 0x00007f7e19dca3e4 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #14 0x00007f7e19e3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #15 0x00007f7e19e2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #16 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d1b17e200, waiter=..., this=0x7f7e68454080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #23 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #24 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #25 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #26 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #27 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #28 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #29 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f7d5bc9d700 (LWP 1754659) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e623b768e in evf::FastMonitoringService::snapshotRunner() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #3 0x00007f7e6de8e9b4 in std::execute_native_thread_routine (__p=0x7f7d24ab9e80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f7d6c3ff700 (LWP 1754574) "cmsRun"): #0 0x00007f7e6d8083fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7e6de88e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f7e0146dacb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f7e0146e0ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f7e0146ac68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f7dfe2944c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f7d6d51d700 (LWP 1754562) "cmsRun"): #0 0x00007f7e6d8083fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7e6de88e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f7e0146dacb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f7e0146e0ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f7e0146ac68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f7dfe2944c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f7d6dd1e700 (LWP 1754558) "cmsRun"): #0 0x00007f7e6d8083fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7e6de88e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f7e0146dacb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f7e0146e0ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f7e0146ac68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f7dfe2944c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f7d92ffd700 (LWP 1754556) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cbab97400, waiter=..., this=0x7f7e68454000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68454000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f7d939fe700 (LWP 1754555) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e60758eb4 in cms::cuda::free_host(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e19dcada1 in std::unique_ptr, cms::cuda::host::impl::HostDeleter>::~unique_ptr() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #8 0x00007f7e19dca3e4 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #9 0x00007f7e19e3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #10 0x00007f7e19e2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #11 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d1df5f400, waiter=..., this=0x7f7e68453f00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453f00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f7d945ff700 (LWP 1754550) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d1ed83400, waiter=..., this=0x7f7e68453e80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453e80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f7d953fe700 (LWP 1754545) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e0fd8b73f >) [clone .constprop.0] [clone .cold]+40>, bases=bases@entry=0x7f7d953f7368) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d953f72c0, fs=0x7f7d953f7010) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b4e1 in _Unwind_RaiseException_Phase2 (exc=0x7f7cfc88e420, context=0x7f7d953f72c0, frames_p=0x7f7d953f73b0) at ../../../libgcc/unwind.inc:50 #10 0x00007f7e6da2baf1 in _Unwind_RaiseException (exc=0x7f7cfc88e420) at ../../../libgcc/unwind.inc:136 #11 0x00007f7e6de644dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f7e6dfb9220 , dest=0x7f7e6de78f10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #12 0x00007f7e0fd8b740 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #13 0x00007f7e0fd8c8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #14 0x00007f7e0fd8c911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #15 0x00007f7e0e3a73c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #16 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f7e6fc39f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #22 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cca608100, waiter=..., this=0x7f7e68453d80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #23 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453d80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #24 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #25 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #26 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #27 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #28 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #29 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f7d95dff700 (LWP 1754543) "cmsRun"): #0 0x00007f7e6d524a71 in poll () from /lib64/libc.so.6 #1 0x00007f7e6641846f in full_read.constprop () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f7e663e3b6c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f7e663e433b in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #4 #5 0x00007f7e6d46a37f in raise () from /lib64/libc.so.6 #6 0x00007f7e6d454db5 in abort () from /lib64/libc.so.6 #7 0x00007f7e6de661aa in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:50 #8 0x00007f7e6de641ea in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #9 0x00007f7e6de63249 in __cxa_call_terminate (ue_header=0x7f7d130acb40) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #10 0x00007f7e6de63977 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7f7d130acb40, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685 #11 0x00007f7e6da2b454 in _Unwind_RaiseException_Phase2 (exc=0x7f7d130acb40, context=0x7f7d95df8460, frames_p=0x7f7d95df8368) at ../../../libgcc/unwind.inc:64 #12 0x00007f7e6da2be8e in _Unwind_Resume (exc=0x7f7d130acb40) at ../../../libgcc/unwind.inc:241 #13 0x00007f7e0fd8b768 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #14 0x00007f7e0fd8c8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #15 0x00007f7e0fd8c911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #16 0x00007f7e0e3a73c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #17 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f7e6fc39f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #23 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cbe095800, waiter=..., this=0x7f7e68453d00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #24 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453d00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #25 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #26 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #27 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #28 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #29 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #30 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f7d975fe700 (LWP 1754520) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d18d73800, waiter=..., this=0x7f7e68453c00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453c00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842f980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842f980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f7d969fd700 (LWP 1754517) "cmsRun"): #0 0x00007f7e6d8083fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7e6de88e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f7e623c92d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f7e6de8e9b4 in std::execute_native_thread_routine (__p=0x7f7e5fe12650) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f7d99ffe700 (LWP 1754505) "cmsRun"): #0 0x00007f7e6d8083fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7e6de88e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f7e623c92d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f7e6de8e9b4 in std::execute_native_thread_routine (__p=0x7f7e5fe12500) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f7d97fff700 (LWP 1754503) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d18d56100, waiter=..., this=0x7f7e68453b80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453b80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f7d98dfc700 (LWP 1754491) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e6da2bd9a <_Unwind_Resume+58>, bases=bases@entry=0x7f7d98df5418) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d98df5370, fs=0x7f7d98df51b0) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b2d0 in uw_init_context_1 (context=0x7f7d98df5370, outer_cfa=0x7f7d98df55a0, outer_ra=0x7f7e0fd8b768 ) at ../../../libgcc/unwind-dw2.c:1592 #10 0x00007f7e6da2bd9b in _Unwind_Resume (exc=0x7f7d080be2c0) at ../../../libgcc/unwind.inc:235 #11 0x00007f7e0fd8b768 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #12 0x00007f7e0fd8c8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #13 0x00007f7e0fd8c911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #14 0x00007f7e0e3a73c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #15 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6fc39f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #21 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cca9b1600, waiter=..., this=0x7f7e68453a00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453a00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fd00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fd00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #28 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f7d9bbfd700 (LWP 1754489) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d1ed0cb00, waiter=..., this=0x7f7e68453a80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453a80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f7d997fd700 (LWP 1754486) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e60758228 , bases=bases@entry=0x7f7d997f6308) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d997f6260, fs=0x7f7d997f6350) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b9b2 in _Unwind_RaiseException (exc=0x7f7d12e2e680) at ../../../libgcc/unwind.inc:104 #10 0x00007f7e6de644dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f7e6dfb9220 , dest=0x7f7e6de78f10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #11 0x00007f7e6075091e in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #12 0x00007f7e60758229 in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #13 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #14 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cbab34e00, waiter=..., this=0x7f7e68453880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #21 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #22 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #23 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #24 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #26 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #27 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f7d9abff700 (LWP 1754475) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e0fd8b73f >) [clone .constprop.0] [clone .cold]+40>, bases=bases@entry=0x7f7d9abf8368) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d9abf82c0, fs=0x7f7d9abf83b0) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b9b2 in _Unwind_RaiseException (exc=0x7f7cb1e36780) at ../../../libgcc/unwind.inc:104 #10 0x00007f7e6de644dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f7e6dfb9220 , dest=0x7f7e6de78f10 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #11 0x00007f7e0fd8b740 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #12 0x00007f7e0fd8c8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #13 0x00007f7e0fd8c911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #14 0x00007f7e0e3a73c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #15 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6fc39f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #21 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cca557500, waiter=..., this=0x7f7e68453800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fc80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fc80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #28 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f7d9c5fe700 (LWP 1754461) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e0fd8c910 , bases=bases@entry=0x7f7d9c5f7508) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d9c5f7460, fs=0x7f7d9c5f71a0) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b4e1 in _Unwind_RaiseException_Phase2 (exc=0x7f7d07f6dc00, context=0x7f7d9c5f7460, frames_p=0x7f7d9c5f7368) at ../../../libgcc/unwind.inc:50 #10 0x00007f7e6da2be8e in _Unwind_Resume (exc=0x7f7d07f6dc00) at ../../../libgcc/unwind.inc:241 #11 0x00007f7e0fd8b768 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #12 0x00007f7e0fd8c8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #13 0x00007f7e0fd8c911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #14 0x00007f7e0e3a73c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #15 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6fc39f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #21 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7de8925a00, waiter=..., this=0x7f7e68453f80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453f80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fc00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fc00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #28 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f7d9d1ff700 (LWP 1754458) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d9b1bf400, waiter=..., this=0x7f7e68453e00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453e00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f7dbb7fe700 (LWP 1754449) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e0fd8b767 >) [clone .constprop.0] [clone .cold]+80>, bases=bases@entry=0x7f7dbb7f7508) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7dbb7f7460, fs=0x7f7dbb7f71a0) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b4e1 in _Unwind_RaiseException_Phase2 (exc=0x7f7cf56c3980, context=0x7f7dbb7f7460, frames_p=0x7f7dbb7f7368) at ../../../libgcc/unwind.inc:50 #10 0x00007f7e6da2be8e in _Unwind_Resume (exc=0x7f7cf56c3980) at ../../../libgcc/unwind.inc:241 #11 0x00007f7e0fd8b768 in cms::cuda::abortOnCudaError(char const*, int, char const*, char const*, char const*, std::basic_string_view >) [clone .constprop.0] [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #12 0x00007f7e0fd8c8e7 in cms::cuda::impl::ScopedContextHolderHelper::enqueueCallback(int, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #13 0x00007f7e0fd8c911 in cms::cuda::ScopedContextAcquire::~ScopedContextAcquire() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDACore.so #14 0x00007f7e0e3a73c0 in HcalDigisProducerGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterHcalRawToDigiGPUPlugins.so #15 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f7e6fc39f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #21 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7ccaa4ac00, waiter=..., this=0x7f7e68453c80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #22 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453c80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #23 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #24 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #25 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fd80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fd80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #27 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #28 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f7d9ddfe700 (LWP 1754448) "cmsRun"): #0 0x00007f7e6d80b9e4 in read () from /lib64/libpthread.so.0 #1 0x00007f7e623c93e0 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #2 0x00007f7e6de8e9b4 in std::execute_native_thread_routine (__p=0x7f7e5fe122f0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #4 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f7d9e7ff700 (LWP 1754447) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d205c4600, waiter=..., this=0x7f7e68453b00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453b00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842ff00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842ff00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f7dbc1ff700 (LWP 1754423) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cca955000, waiter=..., this=0x7f7e68453780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842fe80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842fe80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f7de93ff700 (LWP 1754417) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e6da2b828 <_Unwind_RaiseException+72>, bases=bases@entry=0x7f7de93f8768) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7de93f86c0, fs=0x7f7de93f8510) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b2d0 in uw_init_context_1 (context=0x7f7de93f86c0, outer_cfa=0x7f7de93f8a70, outer_ra=0x7f7e6de644dc <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+60>) at ../../../libgcc/unwind-dw2.c:1592 #10 0x00007f7e6da2b829 in _Unwind_RaiseException (exc=0x7f7cf9c118e0) at ../../../libgcc/unwind.inc:93 #11 0x00007f7e6de644dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f7e6fc31660 , dest=0x7f7e6fbf61f0 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #12 0x00007f7e6fbf063c in edm::convertException::stdToEDM(std::exception const&) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreUtilities.so #13 0x00007f7e6fe3c9b7 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6fee4b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d2058b600, waiter=..., this=0x7f7e68453980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e68430000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e68430000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f7dfa1ff700 (LWP 1754409) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cbe156400, waiter=..., this=0x7f7e68453900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e6842ff80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e6842ff80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f7dfb1d3700 (LWP 1754398) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d8099d5 in pthread_kill () from /lib64/libpthread.so.0 #5 0x00007f7e663e49b8 in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #6 #7 0x00007f7e6d46a37f in raise () from /lib64/libc.so.6 #8 0x00007f7e6d454db5 in abort () from /lib64/libc.so.6 #9 0x00007f7e6de661aa in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:50 #10 0x00007f7e6de641ea in __cxxabiv1::__terminate (handler=) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 #11 0x00007f7e6de63249 in __cxa_call_terminate (ue_header=0x7f7c6b8e6740) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54 #12 0x00007f7e6de63977 in __cxxabiv1::__gxx_personality_v0 (version=, actions=6, exception_class=5138137972254386944, ue_header=0x7f7c6b8e6740, context=) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:685 #13 0x00007f7e6da2b454 in _Unwind_RaiseException_Phase2 (exc=0x7f7c6b8e6740, context=0x7f7dfb1cbe60, frames_p=0x7f7dfb1cbd68) at ../../../libgcc/unwind.inc:64 #14 0x00007f7e6da2be8e in _Unwind_Resume (exc=0x7f7c6b8e6740) at ../../../libgcc/unwind.inc:241 #15 0x00007f7e607511d1 in cms::cuda::free_host(void*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #16 0x00007f7e19dcada1 in std::unique_ptr, cms::cuda::host::impl::HostDeleter>::~unique_ptr() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #17 0x00007f7e19dca3e4 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #18 0x00007f7e19e3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #19 0x00007f7e19e2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #20 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #23 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #24 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #25 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #26 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7cca9f5400, waiter=..., this=0x7f7e68453700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #27 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #28 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #29 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #30 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e68430100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #31 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e68430100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #32 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #33 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f7dfbbd4700 (LWP 1754394) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e60758eb4 in cms::cuda::free_host(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e19dcada1 in std::unique_ptr, cms::cuda::host::impl::HostDeleter>::~unique_ptr() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #8 0x00007f7e19dca3e4 in SiPixelDigiErrorsCUDA::SiPixelDigiErrorsCUDA(unsigned long, std::map >, std::less, std::allocator > > > >, CUstream_st*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libCUDADataFormatsSiPixelDigi.so #9 0x00007f7e19e3d152 in pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, SiPixelClusterThresholds, SiPixelROCsStatusAndMapping const*, unsigned char const*, SiPixelGainForHLTonGPU const*, pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map >, std::less, std::allocator > > > >&&, unsigned int, unsigned int, bool, bool, bool, CUstream_st*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #10 0x00007f7e19e2f3df in SiPixelRawToClusterCUDA::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so #11 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #17 0x00007f7e6e6b6304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f7d2177f100, waiter=..., this=0x7f7e68453680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f7e68453680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f7e6e6b84c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f7e68430080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f7e68430080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f7e1ffff700 (LWP 1754385) "cuda-EvtHandlr"): #0 0x00007f7e6d524a71 in poll () from /lib64/libc.so.6 #1 0x00007f7e5017dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f7e50231d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f7e50180768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f7e315bb700 (LWP 1754377) "cuda-EvtHandlr"): #0 0x00007f7e6d524a71 in poll () from /lib64/libc.so.6 #1 0x00007f7e5017dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f7e50231d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f7e50180768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f7e380bd700 (LWP 1754302) "cmsRun"): #0 0x00007f7e6d80c5a2 in waitpid () from /lib64/libpthread.so.0 #1 0x00007f7e663e1157 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f7e663e3a9a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f7e6de8e9b4 in std::execute_native_thread_routine (__p=0x7f7e66c658b0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f7e6c991640 (LWP 1753378) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804a79 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e607579cb in cms::cuda::free_device(int, void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libHeterogeneousCoreCUDAUtilities.so #7 0x00007f7e0fc987b3 in EcalRawToDigiGPU::acquire(edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginEventFilterEcalRawToDigiPlugins.so #8 0x00007f7e6ff6dad8 in edm::stream::doAcquireIfNeeded(edm::stream::impl::ExternalWork*, edm::Event const&, edm::EventSetup const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f7e6ff71b82 in edm::stream::EDProducerAdaptorBase::doAcquire(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f7e6ff4e6d1 in edm::Worker::runAcquire(edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f7e6ff4e846 in edm::Worker::runAcquireAfterAsyncPrefetch(std::__exception_ptr::exception_ptr, edm::EventTransitionInfo const&, edm::ParentContext const&, edm::WaitingTaskWithArenaHolder) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f7e6fee59ff in edm::Worker::AcquireTask, void>::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6e6bd9cd in tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=0x7f7d98356c00, this=0x7f7e68453600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=, this=0x7f7e68453600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168 #17 0x00007f7e6fe6710d in edm::FinalWaitingTask::wait() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #18 0x00007f7e6fe74b91 in edm::EventProcessor::processRuns() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #19 0x00007f7e6fe750e6 in edm::EventProcessor::runToCompletion() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x000000000040c17d in tbb::detail::d1::task_arena_function::operator()() const () #21 0x00007f7e6e6ab847 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:694 #22 0x00000000004101d0 in main::{lambda()#1}::operator()() const () #23 0x0000000000407d6c in main () Current Modules: Module: HcalDigisProducerGPU:hltHcalDigisGPU (crashed) A fatal system signal has occurred: abort signal The following is the call stack containing the origin of the signal. Sun May 7 13:45:14 CEST 2023 Thread 44 (Thread 0x7f7cbb3ff700 (LWP 1754802) "cmsRun"): #0 0x00007f7e6d80acd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0 #1 0x00007f7e6d80adc8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007f7e5016b0a2 in ?? () from /lib64/libcuda.so.1 #3 0x00007f7e50180768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f7d15fff700 (LWP 1754773) "cmsRun"): #0 0x00007f7e6d80874a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7e623cc061 in FedRawDataInputSource::readSupervisor() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #2 0x00007f7e6de8e9b4 in std::execute_native_thread_routine (__p=0x7f7d24857620) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007f7e6d80217a in start_thread () from /lib64/libpthread.so.0 #4 0x00007f7e6d52fdf3 in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f7d199ff700 (LWP 1754674) "cmsRun"): #0 0x00007f7e6d4fbd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f7e6d4fbc9e in sleep () from /lib64/libc.so.6 #2 0x00007f7e663e0ed0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f7e6d80b75d in __lll_lock_wait () from /lib64/libpthread.so.0 #5 0x00007f7e6d804b44 in pthread_mutex_lock () from /lib64/libpthread.so.0 #6 0x00007f7e6d56a1e3 in dl_iterate_phdr () from /lib64/libc.so.6 #7 0x00007f7e6da2dfa1 in _Unwind_Find_FDE (pc=0x7f7e6da2b828 <_Unwind_RaiseException+72>, bases=bases@entry=0x7f7d199f8768) at ../../../libgcc/unwind-dw2-fde-dip.c:470 #8 0x00007f7e6da2a078 in uw_frame_state_for (context=0x7f7d199f86c0, fs=0x7f7d199f8510) at ../../../libgcc/unwind-dw2.c:1263 #9 0x00007f7e6da2b2d0 in uw_init_context_1 (context=0x7f7d199f86c0, outer_cfa=0x7f7d199f8a70, outer_ra=0x7f7e6de644dc <__cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*))+60>) at ../../../libgcc/unwind-dw2.c:1592 #10 0x00007f7e6da2b829 in _Unwind_RaiseException (exc=0x7f7c293d1be0) at ../../../libgcc/unwind.inc:93 #11 0x00007f7e6de644dc in __cxxabiv1::__cxa_throw (obj=, tinfo=0x7f7e6fc31660 , dest=0x7f7e6fbf61f0 ) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90 #12 0x00007f7e6fbf063c in edm::convertException::stdToEDM(std::exception const&) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreUtilities.so #13 0x00007f7e6fe3c9b7 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) [clone .cold] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f7e6fee4b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f7e6fe527cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so msgtime:2023-05-07 14:39:14 doc_type:cmsswlog date:2023-05-07T12:39:14.718Z run:367112 host:fu-c2b02-19-01 pid:2545194 doctype:stacktrace severity:FATAL severityVal:4 instance:global lexicalId:549852445 message:A fatal system signal has occurred: segmentation violation The following is the call stack containing the origin of the signal. Sun May 7 14:38:53 CEST 2023 Thread 44 (Thread 0x7f58729ff700 (LWP 2546730) "cmsRun"): #0 0x00007f5a2c4aacd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0 #1 0x00007f5a2c4aadc8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 #2 0x00007f5a0ed6b0a2 in ?? () from /lib64/libcuda.so.1 #3 0x00007f5a0ed80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f58d53ff700 (LWP 2546719) "cmsRun"): #0 0x00007f5a2c4a874a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5a21088061 in FedRawDataInputSource::readSupervisor() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #2 0x00007f5a2cb2e9b4 in std::execute_native_thread_routine (__p=0x7f58e38afb20) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #4 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f58d9bfe700 (LWP 2546680) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2eb06ec0 in edm::ServiceWeakToken::lock() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5a2eb281db in edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}::operator()() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5a2eb3043a in tbb::detail::d1::function_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58816bbc00, waiter=..., this=0x7f5a27110580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #8 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #9 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #10 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #11 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #12 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #13 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #14 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f58db7ff700 (LWP 2546679) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2c1b66bb in sched_yield () from /lib64/libc.so.6 #5 0x00007f5a2d347423 in __gthread_yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693 #6 std::this_thread::yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/std_thread.h:323 #7 tbb::detail::d0::atomic_backoff::pause (this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:71 #8 tbb::detail::r1::arena_slot::lock_task_pool (this=0x7f5a2710e200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.h:338 #9 tbb::detail::r1::arena_slot::steal_task (this=this@entry=0x7f5a2710e200, a=..., isolation=isolation@entry=0, slot_index=slot_index@entry=12) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.cpp:149 #10 0x00007f5a2d34ac87 in tbb::detail::r1::arena::steal_task (isolation=0, ed=..., frnd=..., arena_index=30, this=0x7f5a2710d400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.h:574 #11 tbb::detail::r1::task_dispatcher::steal_or_get_critical (critical_allowed=true, isolation=0, random=..., arena_index=30, a=..., ed=..., this=0x7f5a27110500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:161 #12 tbb::detail::r1::task_dispatcher::receive_or_steal_task (this=0x7f5a27110500, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=, critical_allowed=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:217 #13 0x00007f5a2d356460 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:350 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f58da9fe700 (LWP 2546678) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2d34abf5 in tbb::detail::d0::machine_pause (delay=11) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_machine.h:97 #5 tbb::detail::d0::atomic_backoff::bounded_pause (this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:77 #6 tbb::detail::r1::prolonged_pause_impl () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/scheduler_common.h:202 #7 tbb::detail::r1::prolonged_pause () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/scheduler_common.h:234 #8 tbb::detail::r1::stealing_loop_backoff::pause (this=0x7f58da9f7f38) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/scheduler_common.h:263 #9 tbb::detail::r1::waiter_base::pause (this=0x7f58da9f7f30) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/waiters.h:35 #10 tbb::detail::r1::outermost_worker_waiter::pause (this=0x7f58da9f7f30) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/waiters.h:69 #11 tbb::detail::r1::task_dispatcher::receive_or_steal_task (this=0x7f5a27110480, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=, critical_allowed=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:231 #12 0x00007f5a2d356460 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:350 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f58dd1ff700 (LWP 2546677) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2ebef73f in edm::Worker::skipOnPath(edm::EventPrincipal const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5a2eb878c2 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5a2eb87b26 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #8 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58d9fc2b00, waiter=..., this=0x7f5a27110400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #9 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #10 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #11 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #12 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #13 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #14 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #15 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f58dc7fe700 (LWP 2546676) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5922ca52b6 in Basic2DGenericPFlowClusterizer::growPFClusters(reco::PFCluster const&, std::vector > const&, unsigned int, unsigned int, double, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #5 0x00007f5922ca4df7 in Basic2DGenericPFlowClusterizer::growPFClusters(reco::PFCluster const&, std::vector > const&, unsigned int, unsigned int, double, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #6 0x00007f5922ca4df7 in Basic2DGenericPFlowClusterizer::growPFClusters(reco::PFCluster const&, std::vector > const&, unsigned int, unsigned int, double, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #7 0x00007f5922ca5c1e in Basic2DGenericPFlowClusterizer::buildClusters(std::vector > const&, std::vector > const&, std::vector >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #8 0x00007f5922cc9689 in PFClusterProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #9 0x00007f5a2ec1195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2ebf8072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #14 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5874363d00, waiter=..., this=0x7f5a27110380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #15 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110380) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #16 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #17 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #18 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #19 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #20 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #21 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f58de3ff700 (LWP 2546675) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2c1c219b in unlink () from /lib64/libc.so.6 #5 0x00007f5a2c14150d in remove () from /lib64/libc.so.6 #6 0x00007f5a2cbc0405 in std::filesystem::remove (p=..., ec=...) at ../../../../../libstdc++-v3/src/c++17/fs_ops.cc:1270 #7 0x00007f5a2cbc047c in std::filesystem::remove (p=...) at ../../../../../libstdc++-v3/src/c++17/fs_ops.cc:1245 #8 0x00007f5a21085dfc in InputFile::~InputFile() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #9 0x00007f5a2107ea9e in FedRawDataInputSource::read(edm::EventPrincipal&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #10 0x00007f5a2eb6e5a7 in void edm::callWithTryCatchAndPrint(std::function, char const*, bool) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2eb6ea0b in edm::InputSource::readEvent(edm::EventPrincipal&, edm::StreamContext&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5a2eb197e6 in edm::EventProcessor::readEvent(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5a2eb19c3b in edm::EventProcessor::readNextEventForStream(edm::WaitingTaskHolder const&, unsigned int, edm::LuminosityBlockProcessingStatus&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5a2eb1bb16 in edm::EventProcessor::handleNextEventForStreamAsync(edm::WaitingTaskHolder, unsigned int)::{lambda()#1}::operator()() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5a2eb3c548 in edm::SerialTaskQueue::QueuedTask(tbb::detail::d1::task_group&, edm::EventProcessor::handleNextEventForStreamAsync(edm::WaitingTaskHolder, unsigned int)::{lambda()#1}&&)::{lambda()#1}>::execute() [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #16 0x00007f5a2e8db099 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #17 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58819f3a00, waiter=..., this=0x7f5a27110300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #18 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110300) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #19 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #20 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #21 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #23 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #24 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f58e01fc700 (LWP 2546674) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 __cxxabiv1::__dynamic_cast (src_ptr=0x7f586a4b4e00, src_type=0x7f59deb76b80 , dst_type=0x7f59d296d4b0 , src2dst=0) at ../../../../libstdc++-v3/libsupc++/dyncast.cc:49 #5 0x00007f593678029b in (anonymous namespace)::ChainedJetCorrectorImpl::correction(reco::Jet const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginJetMETCorrectionsModulesPlugins.so #6 0x00007f593678eaeb in reco::CorrectedJetProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginJetMETCorrectionsModulesPlugins.so #7 0x00007f5a2ebfd9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5a2ebf75d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5886b2eb00, waiter=..., this=0x7f5a27110200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb500) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f58df7fb700 (LWP 2546673) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f592232b5d8 in TrajSeedMatcher::clearCache() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaElectronAlgos.so #5 0x00007f592232bd07 in TrajSeedMatcher::operator()(Point3DBase const&, float) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoEgammaEgammaElectronAlgos.so #6 0x00007f59223b449a in ElectronNHitSeedProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoEgammaEgammaElectronProducersPlugins.so #7 0x00007f5a2ebfd9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5a2ebf75d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f588156e500, waiter=..., this=0x7f5a27110280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110280) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f58e0dfd700 (LWP 2546672) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2eb04f09 in edm::Event::~Event() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5a2ec119d4 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5a2ebf8072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2eaf27cf in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58640d2700, waiter=..., this=0x7f5a27110180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #11 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110180) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #12 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #13 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #14 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #15 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb580) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #16 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #17 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f58e17fe700 (LWP 2546671) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2c1b66bb in sched_yield () from /lib64/libc.so.6 #5 0x00007f5a2d347423 in __gthread_yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693 #6 std::this_thread::yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/std_thread.h:323 #7 tbb::detail::d0::atomic_backoff::pause (this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:71 #8 tbb::detail::r1::arena_slot::lock_task_pool (this=0x7f5a2710e200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.h:338 #9 tbb::detail::r1::arena_slot::steal_task (this=this@entry=0x7f5a2710e200, a=..., isolation=isolation@entry=0, slot_index=slot_index@entry=12) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.cpp:149 #10 0x00007f5a2d34ac87 in tbb::detail::r1::arena::steal_task (isolation=0, ed=..., frnd=..., arena_index=22, this=0x7f5a2710d400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.h:574 #11 tbb::detail::r1::task_dispatcher::steal_or_get_critical (critical_allowed=true, isolation=0, random=..., arena_index=22, a=..., ed=..., this=0x7f5a27110100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:161 #12 tbb::detail::r1::task_dispatcher::receive_or_steal_task (this=0x7f5a27110100, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=, critical_allowed=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:217 #13 0x00007f5a2d356460 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:350 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb480) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f58e21ff700 (LWP 2546670) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2c1b66bb in sched_yield () from /lib64/libc.so.6 #5 0x00007f5a2d347423 in __gthread_yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693 #6 std::this_thread::yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/std_thread.h:323 #7 tbb::detail::d0::atomic_backoff::pause (this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:71 #8 tbb::detail::r1::arena_slot::lock_task_pool (this=0x7f5a2710e200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.h:338 #9 tbb::detail::r1::arena_slot::steal_task (this=this@entry=0x7f5a2710e200, a=..., isolation=isolation@entry=0, slot_index=slot_index@entry=12) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.cpp:149 #10 0x00007f5a2d34ac87 in tbb::detail::r1::arena::steal_task (isolation=0, ed=..., frnd=..., arena_index=21, this=0x7f5a2710d400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.h:574 #11 tbb::detail::r1::task_dispatcher::steal_or_get_critical (critical_allowed=true, isolation=0, random=..., arena_index=21, a=..., ed=..., this=0x7f5a27110080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:161 #12 tbb::detail::r1::task_dispatcher::receive_or_steal_task (this=0x7f5a27110080, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=, critical_allowed=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:217 #13 0x00007f5a2d356460 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:350 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f591acfd700 (LWP 2546657) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2107368e in evf::FastMonitoringService::snapshotRunner() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #3 0x00007f5a2cb2e9b4 in std::execute_native_thread_routine (__p=0x7f58e3b30380) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f592b3ff700 (LWP 2546552) "cmsRun"): #0 0x00007f5a2c4a83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5a2cb28e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f59c00d1acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f59c00d20ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f59c00cec68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f59bcef84c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f592c578700 (LWP 2546543) "cmsRun"): #0 0x00007f5a2c4a83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5a2cb28e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f59c00d1acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f59c00d20ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f59c00cec68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f59bcef84c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f592cd79700 (LWP 2546540) "cmsRun"): #0 0x00007f5a2c4a83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5a2cb28e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f59c00d1acb in Eigen::ThreadPoolTempl::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #5 0x00007f59c00d20ea in Eigen::ThreadPoolTempl::WorkerLoop(int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #6 0x00007f59c00cec68 in std::_Function_handler)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_cc.so.2 #7 0x00007f59bcef84c0 in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libtensorflow_framework.so.2 #8 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #9 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f59521fe700 (LWP 2546479) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5922ca4ef1 in Basic2DGenericPFlowClusterizer::growPFClusters(reco::PFCluster const&, std::vector > const&, unsigned int, unsigned int, double, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #5 0x00007f5922ca5c1e in Basic2DGenericPFlowClusterizer::buildClusters(std::vector > const&, std::vector > const&, std::vector >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #6 0x00007f5922cc9689 in PFClusterProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #7 0x00007f5a2ec1195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5a2ebf8072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5874377800, waiter=..., this=0x7f5a27110000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a27110000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f5952bff700 (LWP 2546474) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a189d02d5 in AnalyticalPropagator::propagatedStateWithPath(FreeTrajectoryState const&, Surface const&, GlobalTrajectoryParameters const&, double const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #5 0x00007f5a189da7fe in AnalyticalPropagator::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #6 0x00007f5a189cf365 in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsGeomPropagators.so #7 0x00007f59cef09a72 in PropagatorWithMaterial::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #8 0x00007f5a189f177d in GeomDetCompatibilityChecker::isCompatible(GeomDet const*, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsDetLayers.so #9 0x00007f59d26b387a in CompatibleDetToGroupAdder::add(GeomDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkDetLayers.so #10 0x00007f59d26bf6ab in TOBRod::searchNeighbors(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, SubLayerCrossing const&, float, std::vector >&, bool) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkDetLayers.so #11 0x00007f59d26bfb08 in TOBRod::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkDetLayers.so #12 0x00007f59d26b46cb in CompatibleDetToGroupAdder::add(GeometricSearchDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkDetLayers.so #13 0x00007f59d26bd4fe in TBLayer::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkDetLayers.so #14 0x00007f5a189ef2fb in GeometricSearchDet::compatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector, std::allocator > >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsDetLayers.so #15 0x00007f5a189ee6d4 in GeometricSearchDet::compatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsDetLayers.so #16 0x00007f59cf3f53d7 in LayerMeasurements::measurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMeasurementDet.so #17 0x00007f592bd2d13a in CkfTrajectoryBuilder::findCompatibleMeasurements(TrajectorySeed const&, TempTrajectory const&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #18 0x00007f592bd25adb in CkfTrajectoryBuilder::limitedCandidates(std::shared_ptr const&, std::vector >&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #19 0x00007f592bd270d7 in CkfTrajectoryBuilder::limitedCandidates(TrajectorySeed const&, TempTrajectory&, std::vector >&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #20 0x00007f592bd272cf in CkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector >&, unsigned int&, TrajectoryFilter const*) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #21 0x00007f592bd1b263 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #22 0x00007f592bd1cceb in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerCkfPattern.so #23 0x00007f5a2ec1195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #24 0x00007f5a2ebf8072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #25 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #26 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #27 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #28 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5886a72000, waiter=..., this=0x7f5a2710ff00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #29 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710ff00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #30 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #31 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #32 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #33 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #34 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #35 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f59543fc700 (LWP 2546455) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2ebebb9d in edm::Worker::reset() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5a2ebece74 in edm::WorkerManager::resetAll() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5a2eb29cd0 in edm::StreamSchedule::processOneStreamAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5a2eb35ce9 in tbb::detail::d1::enqueue_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58d4ba7700, waiter=..., this=0x7f5a2710fe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #9 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #10 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #11 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #12 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #13 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #14 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #15 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f59539fb700 (LWP 2546454) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007ffc593a5bba in clock_gettime () #5 0x00007f5a2c19697a in clock_gettime@GLIBC_2.2.5 () from /lib64/libc.so.6 #6 0x00007f5a26e62121 in boost::chrono::thread_clock::now() () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libboost_chrono.so.1.80.0 #7 0x00007f5a20e97d03 in FastTimerService::Measurement::measure_and_accumulate(FastTimerService::AtomicResources&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginHLTriggerTimerPlugins.so #8 0x00007f5a2eb282d0 in edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}::operator()() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2eb3043a in tbb::detail::d1::function_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f586419d800, waiter=..., this=0x7f5a2710fe80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #11 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fe80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #12 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #13 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #14 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eba00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #15 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eba00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #16 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #17 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f5954ffd700 (LWP 2546449) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2eb282a5 in edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}::operator()() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5a2eb3043a in tbb::detail::d1::function_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5886bd1200, waiter=..., this=0x7f5a2710fc00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #7 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fc00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #8 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #9 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #10 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #12 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #13 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f59559fe700 (LWP 2546447) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2eb1e4e5 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_weak_add_ref() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5a2eb2932c in void edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5a2eb29b35 in void edm::Path::runAllModulesAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5a2eb29d36 in edm::StreamSchedule::processOneStreamAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5a2eb35ce9 in tbb::detail::d1::enqueue_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f57fa335600, waiter=..., this=0x7f5a2710fc80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #10 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fc80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #11 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #12 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #13 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #15 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #16 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f5957dfd700 (LWP 2546445) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007ffc593a5bba in clock_gettime () #5 0x00007f5a2c19697a in clock_gettime@GLIBC_2.2.5 () from /lib64/libc.so.6 #6 0x00007f5a26e62121 in boost::chrono::thread_clock::now() () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libboost_chrono.so.1.80.0 #7 0x00007f5a20e97d03 in FastTimerService::Measurement::measure_and_accumulate(FastTimerService::AtomicResources&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginHLTriggerTimerPlugins.so #8 0x00007f5a2eb282d0 in edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}::operator()() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2eb3043a in tbb::detail::d1::function_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5886b37b00, waiter=..., this=0x7f5a2710fb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #11 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fb00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #12 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #13 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #14 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eba80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #15 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eba80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #16 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #17 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f59563ff700 (LWP 2546442) "cmsRun"): #0 0x00007f5a2c1c4a71 in poll () from /lib64/libc.so.6 #1 0x00007f5a250d446f in full_read.constprop () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f5a2509fb6c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f5a250a033b in sig_dostack_then_abort () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #4 #5 0x00007f59cef90777 in void std::__introsort_loop<__gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, long, __gnu_cxx::__ops::_Iter_comp_iter) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #6 0x00007f59cef92bf5 in RecHitsSortedInPhi::RecHitsSortedInPhi(std::vector > const&, Point3DBase const&, DetLayer const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #7 0x00007f59cef96d11 in LayerHitMapCache::operator()(SeedingLayerSetsHits::SeedingLayer const&, TrackingRegion const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #8 0x00007f59cef9707a in HitPairGeneratorFromLayerPair::doublets(TrackingRegion const&, edm::Event const&, edm::EventSetup const&, SeedingLayerSetsHits::SeedingLayer const&, SeedingLayerSetsHits::SeedingLayer const&, LayerHitMapCache&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTkHitPairs.so #9 0x00007f5922262d97 in (anonymous namespace)::Impl<(anonymous namespace)::DoNothing, (anonymous namespace)::ImplIntermediateHitDoublets, (anonymous namespace)::RegionsLayersSeparate>::produce(bool, edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkHitPairsPlugins.so #10 0x00007f5a2ec1195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2ebf8072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58817ac700, waiter=..., this=0x7f5a2710fa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fa80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270eb880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270eb880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f59573fc700 (LWP 2546416) "cmsRun"): #0 0x00007f5a2c4a83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5a2cb28e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f5a210852d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f5a2cb2e9b4 in std::execute_native_thread_routine (__p=0x7f5a1ac81620) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f59593ff700 (LWP 2546410) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2c1b66bb in sched_yield () from /lib64/libc.so.6 #5 0x00007f5a2d347423 in __gthread_yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693 #6 std::this_thread::yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/std_thread.h:323 #7 tbb::detail::d0::atomic_backoff::pause (this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:71 #8 tbb::detail::r1::arena_slot::lock_task_pool (this=0x7f5a2710e200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.h:338 #9 tbb::detail::r1::arena_slot::steal_task (this=this@entry=0x7f5a2710e200, a=..., isolation=isolation@entry=0, slot_index=slot_index@entry=12) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.cpp:149 #10 0x00007f5a2d34ac87 in tbb::detail::r1::arena::steal_task (isolation=0, ed=..., frnd=..., arena_index=8, this=0x7f5a2710d400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.h:574 #11 tbb::detail::r1::task_dispatcher::steal_or_get_critical (critical_allowed=true, isolation=0, random=..., arena_index=8, a=..., ed=..., this=0x7f5a2710fa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:161 #12 tbb::detail::r1::task_dispatcher::receive_or_steal_task (this=0x7f5a2710fa00, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=, critical_allowed=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:217 #13 0x00007f5a2d356460 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:350 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fa00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebd00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebd00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f59589fe700 (LWP 2546405) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2eaf3b99 in tbb::detail::d1::wait_context::add_reference(long) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5a2eb298d9 in void edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5a2eb29b35 in void edm::Path::runAllModulesAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5a2eb29dce in edm::StreamSchedule::processOneStreamAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5a2eb35ce9 in tbb::detail::d1::enqueue_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58813c2100, waiter=..., this=0x7f5a2710f900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #10 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710f900) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #11 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #12 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #13 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebc00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebc00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #15 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #16 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f5959ffd700 (LWP 2546398) "cmsRun"): #0 0x00007f5a2c4a83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5a2cb28e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f5a210852d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f5a2cb2e9b4 in std::execute_native_thread_routine (__p=0x7f5a1ac814d0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f595abfe700 (LWP 2546397) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5922ce90db in PFRecHitNavigatorBase::associateNeighbour(DetId const&, reco::PFRecHit&, std::unique_ptr >, std::default_delete > > >&, edm::RefProd > >&, short, short, short) [clone .constprop.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #5 0x00007f5922cbb8b9 in PFRecHitCaloNavigator::associateNeighbours(reco::PFRecHit&, std::unique_ptr >, std::default_delete > > >&, edm::RefProd > >&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #6 0x00007f5922ccb42e in PFRecHitProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoParticleFlowPFClusterProducerPlugins.so #7 0x00007f5a2ec1195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5a2ebf8072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58814ae100, waiter=..., this=0x7f5a2710f800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710f800) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f595c5fe700 (LWP 2546396) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2d347366 in tbb::detail::r1::arena_slot::steal_task (this=this@entry=0x7f5a2710e200, a=..., isolation=isolation@entry=0, slot_index=slot_index@entry=12) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.cpp:173 #5 0x00007f5a2d34ac87 in tbb::detail::r1::arena::steal_task (isolation=0, ed=..., frnd=..., arena_index=19, this=0x7f5a2710d400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.h:574 #6 tbb::detail::r1::task_dispatcher::steal_or_get_critical (critical_allowed=true, isolation=0, random=..., arena_index=19, a=..., ed=..., this=0x7f5a2710ff80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:161 #7 tbb::detail::r1::task_dispatcher::receive_or_steal_task (this=0x7f5a2710ff80, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=, critical_allowed=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:217 #8 0x00007f5a2d356460 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710ff80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:350 #9 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710ff80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #10 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #11 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #12 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebc80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #13 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebc80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #14 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #15 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f595b7ff700 (LWP 2546390) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2eb29dab in edm::StreamSchedule::processOneStreamAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#2}::operator()() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5a2eb35ce9 in tbb::detail::d1::enqueue_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType&, edm::ServiceToken const&, bool)::{lambda()#4}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58d4b93700, waiter=..., this=0x7f5a2710fb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #7 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fb80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #8 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #9 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #10 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #11 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebe00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #12 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #13 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f595d1ff700 (LWP 2546374) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2c1b66bb in sched_yield () from /lib64/libc.so.6 #5 0x00007f5a2d347423 in __gthread_yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/x86_64-redhat-linux-gnu/bits/gthr-default.h:693 #6 std::this_thread::yield () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/bits/std_thread.h:323 #7 tbb::detail::d0::atomic_backoff::pause (this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_utils.h:71 #8 tbb::detail::r1::arena_slot::lock_task_pool (this=0x7f5a2710e200) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.h:338 #9 tbb::detail::r1::arena_slot::steal_task (this=this@entry=0x7f5a2710e200, a=..., isolation=isolation@entry=0, slot_index=slot_index@entry=12) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena_slot.cpp:149 #10 0x00007f5a2d34ac87 in tbb::detail::r1::arena::steal_task (isolation=0, ed=..., frnd=..., arena_index=15, this=0x7f5a2710d400) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.h:574 #11 tbb::detail::r1::task_dispatcher::steal_or_get_critical (critical_allowed=true, isolation=0, random=..., arena_index=15, a=..., ed=..., this=0x7f5a2710fd80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:161 #12 tbb::detail::r1::task_dispatcher::receive_or_steal_task (this=0x7f5a2710fd80, tls=..., ed=..., waiter=..., isolation=0, fifo_allowed=, critical_allowed=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:217 #13 0x00007f5a2d356460 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fd80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:350 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fd80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebd80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebd80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f595e9ff700 (LWP 2546365) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a1f45f66a in JacobianCurvilinearToLocal::compute(TkRotation const&, Vector3DBase const&, Vector3DBase const&, double) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsAnalyticalJacobians.so #5 0x00007f5a1f461087 in JacobianCurvilinearToLocal::JacobianCurvilinearToLocal(Surface const&, LocalTrajectoryParameters const&, GlobalTrajectoryParameters const&, MagneticField const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsAnalyticalJacobians.so #6 0x00007f5a18927860 in BasicTrajectoryState::createLocalErrorFromCurvilinearError() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrajectoryState.so #7 0x00007f59cef07d28 in MaterialEffectsUpdator::updateStateInPlace(TrajectoryStateOnSurface&, PropagationDirection) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #8 0x00007f59cef0916d in PropagatorWithMaterial::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsMaterialEffects.so #9 0x00007f592a5b8faa in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #10 0x00007f592a5b9a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #11 0x00007f5a2ec1195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5a2ebf8072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #16 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f58deda1100, waiter=..., this=0x7f5a2710fd00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710fd00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #19 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #20 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ec000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ec000) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #22 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #23 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f595dffe700 (LWP 2546362) "cmsRun"): #0 0x00007f5a2c4a83fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5a2cb28e90 in __gthread_cond_wait (__mutex=, __cond=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865 #2 std::__condvar::wait (__m=..., this=) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #3 std::condition_variable::wait (this=, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #4 0x00007f5a210852d5 in FedRawDataInputSource::readWorker(unsigned int) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libEventFilterUtilities.so #5 0x00007f5a2cb2e9b4 in std::execute_native_thread_routine (__p=0x7f5a1ac812c0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #6 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #7 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f59b77fd700 (LWP 2546355) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f59f003fd14 in reco::Track::~Track() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDataFormatsTrackReco.so #5 0x00007f59ccab8668 in void storeTracks > >, std::allocator > > > > >(edm::Event&, std::vector > >, std::allocator > > > > const&, TrackerTopology const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoPixelVertexingPixelTrackFittingPlugins.so #6 0x00007f59ccaba2a6 in PixelTrackProducerFromSoAT::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoPixelVertexingPixelTrackFittingPlugins.so #7 0x00007f5a2ebfd9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #8 0x00007f5a2ebf75d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #12 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5886aec200, waiter=..., this=0x7f5a2710f880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #13 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710f880) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #14 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #15 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #16 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebf80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebf80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #18 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f59b81fe700 (LWP 2546353) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a14226216 in TH1F::AddBinContent(int, double) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libHist.so #5 0x00007f5a1421e0f6 in TH1::Fill(double, double) () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libHist.so #6 0x00007f5a263ef264 in dqm::impl::MonitorElement::doFill(long) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libDQMServicesCore.so #7 0x00007f58faf0e289 in TriggerRatesMonitor::dqmAnalyze(edm::Event const&, edm::EventSetup const&, (anonymous namespace)::RunBasedHistograms const&) const [clone .lto_priv.0] () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginDQMHLTEvFPlugins.so #8 0x00007f5a2ebfd9cb in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2ebf75d2 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #13 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5886a7dc00, waiter=..., this=0x7f5a2710f980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #14 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710f980) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #15 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #16 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #17 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebf00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebf00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #19 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #20 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f59b8dff700 (LWP 2546351) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f59ccb6ac24 in PixelCPEGeneric::localPosition(PixelCPEBase::DetParam const&, PixelCPEBase::ClusterParam&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #5 0x00007f59ccb4bf62 in PixelClusterParameterEstimator::getParameters(SiPixelCluster const&, GeomDet const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoLocalTrackerSiPixelRecHits.so #6 0x00007f59d27be3ce in TkClonerImpl::operator()(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libRecoTrackerTransientTrackingRecHit.so #7 0x00007f59ccc03c88 in SiPixelRecHit::clone_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoLocalTrackerSiPixelRecHitsPlugins.so #8 0x00007f592a5b8ec6 in SeedFromConsecutiveHitsCreator::makeSeed(std::vector >&, SeedingHitSet const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #9 0x00007f592a5b9a38 in SeedCreatorFromRegionHitsEDProducerT::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoTrackerTkSeedGeneratorPlugins.so #10 0x00007f5a2ec1195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #11 0x00007f5a2ebf8072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #15 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5a2a137200, waiter=..., this=0x7f5a2710f780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #16 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710f780) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #17 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #18 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #19 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ebe80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ebe80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #21 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #22 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f59b9e37700 (LWP 2546335) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f5a2ebef74e in edm::Worker::skipOnPath(edm::EventPrincipal const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #5 0x00007f5a2eb878c2 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventTransitionInfo const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*, tbb::detail::d1::task_group&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #6 0x00007f5a2eb87b26 in edm::FunctorWaitingTask::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #7 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #8 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5886b3c800, waiter=..., this=0x7f5a2710f680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #9 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710f680) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #10 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #11 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #12 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ec100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #13 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ec100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #14 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #15 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f59ba838700 (LWP 2546332) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007ffc593a5bba in clock_gettime () #5 0x00007f5a2c19697a in clock_gettime@GLIBC_2.2.5 () from /lib64/libc.so.6 #6 0x00007f5a26e62121 in boost::chrono::thread_clock::now() () from /opt/offline/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_0_5_patch1/external/el8_amd64_gcc11/lib/libboost_chrono.so.1.80.0 #7 0x00007f5a20e97d03 in FastTimerService::Measurement::measure_and_accumulate(FastTimerService::AtomicResources&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginHLTriggerTimerPlugins.so #8 0x00007f5a2eb2833f in edm::Worker::doWorkNoPrefetchingAsync >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}::operator()() const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #9 0x00007f5a2eb3043a in tbb::detail::d1::function_task >(edm::WaitingTaskHolder, edm::OccurrenceTraits::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #10 0x00007f5a2d356304 in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x7f5886b60e00, waiter=..., this=0x7f5a2710f700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #11 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f5a2710f700) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #12 tbb::detail::r1::arena::process (tls=..., this=) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:137 #13 tbb::detail::r1::market::process (this=, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/market.cpp:599 #14 0x00007f5a2d3584c6 in tbb::detail::r1::rml::private_worker::run (this=0x7f5a270ec080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:271 #15 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f5a270ec080) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/private_server.cpp:221 #16 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #17 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f59df3ff700 (LWP 2546314) "cuda-EvtHandlr"): #0 0x00007f5a2c1c4a71 in poll () from /lib64/libc.so.6 #1 0x00007f5a0ed7dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f5a0ee31d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f5a0ed80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f59dffff700 (LWP 2546308) "cuda-EvtHandlr"): #0 0x00007f5a2c1c4a71 in poll () from /lib64/libc.so.6 #1 0x00007f5a0ed7dde9 in ?? () from /lib64/libcuda.so.1 #2 0x00007f5a0ee31d44 in ?? () from /lib64/libcuda.so.1 #3 0x00007f5a0ed80768 in ?? () from /lib64/libcuda.so.1 #4 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f59f6ddd700 (LWP 2546274) "cmsRun"): #0 0x00007f5a2c4ac5a2 in waitpid () from /lib64/libpthread.so.0 #1 0x00007f5a2509d157 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #2 0x00007f5a2509fa9a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 0x00007f5a2cb2e9b4 in std::execute_native_thread_routine (__p=0x7f5a269c88e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f5a2c4a217a in start_thread () from /lib64/libpthread.so.0 #5 0x00007f5a2c1cfdf3 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f5a2b631640 (LWP 2545194) "cmsRun"): #0 0x00007f5a2c19bd98 in nanosleep () from /lib64/libc.so.6 #1 0x00007f5a2c19bc9e in sleep () from /lib64/libc.so.6 #2 0x00007f5a2509ced0 in sig_pause_for_stacktrace () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so #3 #4 0x00007f59d1f5702c in DTGeometry::idToDet(DetId) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libGeometryDTGeometry.so #5 0x00007f59cee48f48 in MuonDetIdAssociator::getGeomDet(DetId const&) const () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginTrackingToolsTrackAssociatorPlugins.so #6 0x00007f59cee16a79 in TrackDetectorAssociator::fillMuon(edm::Event const&, TrackDetMatchInfo&, TrackAssociatorParameters const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrackAssociator.so #7 0x00007f59cee1baf6 in TrackDetectorAssociator::associate(edm::Event const&, edm::EventSetup const&, TrackAssociatorParameters const&, FreeTrajectoryState const*, FreeTrajectoryState const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrackAssociator.so #8 0x00007f59cee1c379 in TrackDetectorAssociator::associate(edm::Event const&, edm::EventSetup const&, reco::Track const&, TrackAssociatorParameters const&, TrackDetectorAssociator::Direction) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libTrackingToolsTrackAssociator.so #9 0x00007f5922e7b2be in MuonIdProducer::fillMuonId(edm::Event&, edm::EventSetup const&, reco::Muon&, TrackDetectorAssociator::Direction) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoMuonMuonIdentificationPlugins.so #10 0x00007f5922e7dc89 in MuonIdProducer::produce(edm::Event&, edm::EventSetup const&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/pluginRecoMuonMuonIdentificationPlugins.so #11 0x00007f5a2ec1195d in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #12 0x00007f5a2ebf8072 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #13 0x00007f5a2eb846da in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #14 0x00007f5a2eb84b88 in edm::Worker::RunModuleTask >::execute() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #15 0x00007f5a2e8d9f79 in tbb::detail::d1::function_task::execute(tbb::detail::d1::execution_data&) () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreConcurrency.so #16 0x00007f5a2d35d9cd in tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=0x7f58dbd42800, this=0x7f5a2710f600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322 #17 tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=, this=0x7f5a2710f600) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458 #18 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168 #19 0x00007f5a2eb0710d in edm::FinalWaitingTask::wait() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #20 0x00007f5a2eb14b91 in edm::EventProcessor::processRuns() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #21 0x00007f5a2eb150e6 in edm::EventProcessor::runToCompletion() () from /opt/offline/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_5/lib/el8_amd64_gcc11/libFWCoreFramework.so #22 0x000000000040c17d in tbb::detail::d1::task_arena_function::operator()() const () #23 0x00007f5a2d34b847 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_0_2-el8_amd64_gcc11/build/CMSSW_13_0_2-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-bb5e0283c68ca6d69bd8419f6c08f7b1/tbb-v2021.8.0/src/tbb/arena.cpp:694 #24 0x00000000004101d0 in main::{lambda()#1}::operator()() const () #25 0x0000000000407d6c in main () Current Modules: Module: HitPairEDProducer:hltDisplacedhltIter4PFlowPixelLessHitDoubletsForTau (crashed) Module: HLTPrescaler:hltPreMu23TrkIsoVVLEle12CaloIdLTrackIdLIsoVL Module: CorrectedCaloJetProducer:hltAK4CaloJetsCorrected Module: none Module: TriggerRatesMonitor:hltTriggerRatesMonitor Module: none Module: none Module: none Module: PFClusterProducer:hltParticleFlowClusterPSUnseeded Module: none Module: PFRecHitProducer:hltParticleFlowRecHitPSUnseeded Module: none Module: DTRecSegment4DProducer:hltDt4DSegments Module: PFClusterProducer:hltParticleFlowClusterHBHE Module: PixelTrackProducerFromSoAPhase1:hltPixelTracks Module: HLTPrescaler:hltPreDoubleEle9p5eta1p22mMax6 Module: PFRecHitProducer:hltParticleFlowRecHitHBHE Module: HLTHtMhtProducer:hltHtMhtPFJetsFive30er3p0 Module: none Module: none Module: none Module: none Module: ElectronNHitSeedProducer:hltEgammaElectronPixelSeeds Module: none Module: none Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: none Module: none Module: CkfTrackCandidateMaker:hltDisplacedhltIter4PFlowCkfTrackCandidatesForTau Module: SeedCreatorFromRegionConsecutiveHitsEDProducer:hltElePixelSeedsDoublets Module: none Module: MuonIdProducer:hltMuons A fatal system signal has occurred: segmentation violation