Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible memory corruption in 10023.0 #19638

Closed
dan131riley opened this issue Jul 10, 2017 · 5 comments
Closed

Possible memory corruption in 10023.0 #19638

dan131riley opened this issue Jul 10, 2017 · 5 comments
Labels
pending-assignment

Comments

@dan131riley
Copy link

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc700/CMSSW_9_3_X_2017-07-07-2300/pyRelValMatrixLogs/run/10023.0_MinBias_13+MinBias_13TeV_pythia8_TuneCUETP8M1_2017_GenSimFullINPUT+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017/step3_MinBias_13+MinBias_13TeV_pythia8_TuneCUETP8M1_2017_GenSimFullINPUT+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017.log

segmentation fault in jemalloc, only seen once so far:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f1615cbc700 (LWP 8624)):
#0  0x00000039a20acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000039a20acac0 in sleep () from /lib64/libc.so.6
#2  0x00007f16880cf621 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1680232df0 in TH1::GetBinCenter(int) const () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/external/slc6_amd64_gcc700/lib/libHist.so
#5  0x00007f163b8e6a47 in JetMETHLTOfflineSource::fillMEforMonTriggerSummary(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#6  0x00007f163b902b08 in JetMETHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#7  0x00007f168e62ffc2 in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#8  0x00007f168e567312 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#9  0x00007f168e511577 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#10 0x00007f168e51172d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f168e51300b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f168e5134d5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f168ceed8ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1614fb3e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#14 0x00007f168cee6c18 in tbb::internal::arena::process (this=0x7f168b95f780, s=...) at ../../src/tbb/arena.cpp:159
#15 0x00007f168cee56d3 in tbb::internal::market::process (this=0x7f168b97be00, j=...) at ../../src/tbb/market.cpp:697
#16 0x00007f168cee1925 in tbb::internal::rml::private_worker::run (this=0x7f168b973980) at ../../src/tbb/private_server.cpp:271
#17 0x00007f168cee1b69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#18 0x00000039a2407aa1 in start_thread () from /lib64/libpthread.so.0
#19 0x00000039a20e8bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f1616ebd700 (LWP 8623)):
#0  0x00000039a20acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000039a20acac0 in sleep () from /lib64/libc.so.6
#2  0x00007f16880cf621 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f167f5b5c07 in MonitorElement::Fill(double, double) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libDQMServicesCore.so
#5  0x00007f163a551231 in CaloTowersAnalyzer::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginDQMOfflineHcal.so
#6  0x00007f168e62ffc2 in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#7  0x00007f168e567312 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#8  0x00007f168e511577 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#9  0x00007f168e51172d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#10 0x00007f168e51300b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f168e5134d5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f168ceed8ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f16432ebe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#13 0x00007f168cee6c18 in tbb::internal::arena::process (this=0x7f168b95f780, s=...) at ../../src/tbb/arena.cpp:159
#14 0x00007f168cee56d3 in tbb::internal::market::process (this=0x7f168b97be00, j=...) at ../../src/tbb/market.cpp:697
#15 0x00007f168cee1925 in tbb::internal::rml::private_worker::run (this=0x7f168b973a80) at ../../src/tbb/private_server.cpp:271
#16 0x00007f168cee1b69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#17 0x00000039a2407aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x00000039a20e8bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f16178be700 (LWP 8622)):
#0  0x00000039a20acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000039a20acac0 in sleep () from /lib64/libc.so.6
#2  0x00007f16880cf621 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f168e3a2c90 in unsigned int edm::ParameterSet::getUntrackedParameter<unsigned int>(char const*, unsigned int const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreParameterSet.so
#5  0x00007f163b73e13b in EmDQM::SetVarsFromPSet(__gnu_cxx::__normal_iterator<edm::ParameterSet*, std::vector<edm::ParameterSet, std::allocator<edm::ParameterSet> > >) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginHLTriggerOfflineEgamma.so
#6  0x00007f163b742ee1 in EmDQM::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginHLTriggerOfflineEgamma.so
#7  0x00007f168e62ffc2 in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#8  0x00007f168e567312 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#9  0x00007f168e511577 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#10 0x00007f168e51172d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f168e51300b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f168e5134d5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f168ceed8ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f16152b3e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#14 0x00007f168cee6c18 in tbb::internal::arena::process (this=0x7f168b95f780, s=...) at ../../src/tbb/arena.cpp:159
#15 0x00007f168cee56d3 in tbb::internal::market::process (this=0x7f168b97be00, j=...) at ../../src/tbb/market.cpp:697
#16 0x00007f168cee1925 in tbb::internal::rml::private_worker::run (this=0x7f168b973a00) at ../../src/tbb/private_server.cpp:271
#17 0x00007f168cee1b69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#18 0x00000039a2407aa1 in start_thread () from /lib64/libpthread.so.0
#19 0x00000039a20e8bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f166e079700 (LWP 8542)):
#0  0x00000039a240f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f16880cf8a7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f16880cfeda in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f168c7dbc2f in std::execute_native_thread_routine (__p=0x7f1687a9fb00) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00000039a2407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000039a20e8bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f168c492360 (LWP 8355)):
#0  0x00000039a20df383 in poll () from /lib64/libc.so.6
#1  0x00007f16880cfd67 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f16880d033c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f16880d11fc in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f168d13218b in imalloc_body (sopts=0x7ffc23923a60, dopts=0x7ffc23920030, tsd=0x7f168c48b7c8) at src/jemalloc.c:1871
#6  0x00007f168d1323ce in imalloc (sopts=0x7ffc23923a60, dopts=0x7ffc23923a30) at src/jemalloc.c:1949
#7  0x00007f168d132436 in malloc (size=97) at src/jemalloc.c:1977
#8  0x00007f168d1a1a6d in newImpl<false> (size=97) at src/jemalloc_cpp.cpp:45
#9  0x00007f168d1a18c5 in operator new (size=97) at src/jemalloc_cpp.cpp:78
#10 0x000000000040c7a2 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) [clone .isra.124] ()
#11 0x000000000040c7e8 in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long) [clone .isra.125] ()
#12 0x0000000000413788 in std::string::reserve(unsigned long) ()
#13 0x0000000000413927 in std::string::append(std::string const&) ()
#14 0x00007f168e1eaad3 in edm::InputTag::encode() const () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreUtilities.so
#15 0x00007f163b92e4d0 in void egHLT::trigTools::setFiltersObjPasses<egHLT::OffEle>(std::vector<egHLT::OffEle, std::allocator<egHLT::OffEle> >&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, std::bitset<128ul> const&, egHLT::TrigCodes const&, trigger::TriggerEvent const*, std::string const&) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#16 0x00007f163b9297eb in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#17 0x00007f163b92d8b9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#18 0x00007f163b90dc58 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#19 0x00007f168e62ffc2 in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#20 0x00007f168e567312 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#21 0x00007f168e511577 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#22 0x00007f168e51172d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#23 0x00007f168e51300b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#24 0x00007f168e5134d5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#25 0x00007f168ceed8ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f168b97a400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#26 0x00007f168e5b755a in edm::EventProcessor::readAndProcessEvents() () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#27 0x00007f168e5c1ca1 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02479/slc6_amd64_gcc700/cms/cmssw/CMSSW_9_3_X_2017-07-07-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#28 0x000000000040da1a in main::{lambda()#1}::operator()() const ()
#29 0x000000000040c2ca in main ()

Current Modules:

Module: EgHLTOfflineSource:egHLTOffDQMSource (crashed)
Module: CaloTowersAnalyzer:AllCaloTowersDQMOffline
Module: JetMETHLTOfflineSource:jetMETHLTOfflineSourceAK4
Module: EmDQM:emdqm

A fatal system signal has occurred: segmentation violation
@cmsbuild
Copy link
Contributor

A new Issue was created by @dan131riley Dan Riley.

@davidlange6, @Dr15Jones, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@dan131riley
Copy link
Author

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc700/CMSSW_9_3_X_2017-07-21-1100/pyRelValMatrixLogs/run/136.78_RunHLTPhy2017B+RunHLTPhy2017B+HLTDR2_2017+RECODR2_2017reHLT_Prompt+HARVEST2017/step3_RunHLTPhy2017B+RunHLTPhy2017B+HLTDR2_2017+RECODR2_2017reHLT_Prompt+HARVEST2017.log

another one, with

Module: EgHLTOfflineSource:egHLTOffDQMSource (crashed)
Module: JetMETHLTOfflineSource:jetMETHLTOfflineSourceAK4

in common with the previous one...time to run some valgrind

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f4338534700 (LWP 5375)):
#0  0x00000031970acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000031970acac0 in sleep () from /lib64/libc.so.6
#2  0x00007f43a277d601 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003197083b06 in __memset_sse2 () from /lib64/libc.so.6
#5  0x00007f43aa87b7d5 in je_arena_alloc_junk_small (ptr=0x7f41faad5fe0, bin_info=0x7f43aa8de4f8 <je_arena_bin_info+120>, zero=false) at src/arena.c:1479
#6  0x00007f43aa8644c5 in tcache_alloc_small (tsd=0x7f433852db68, arena=0x0, tcache=0x7f433852dd20, size=48, binind=3, zero=false, slow_path=true) at include/jemalloc/internal/tcache_inlines.h:106
#7  0x00007f43aa8651ab in arena_malloc (tsdn=0x7f433852db68, arena=0x0, size=48, ind=3, zero=false, tcache=0x7f433852dd20, slow_path=true) at include/jemalloc/internal/arena_inlines_b.h:101
#8  0x00007f43aa86669c in iallocztm (tsdn=0x7f433852db68, size=48, ind=3, zero=false, tcache=0x7f433852dd20, is_internal=false, arena=0x0, slow_path=true) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:33
#9  0x00007f43aa86b8e8 in imalloc_no_sample (sopts=0x7f433852ca10, dopts=0x7f433852c9e0, tsd=0x7f433852db68, size=48, usize=48, ind=3) at src/jemalloc.c:1650
#10 0x00007f43aa86c052 in imalloc_body (sopts=0x7f433852ca10, dopts=0x7f433852c9e0, tsd=0x7f433852db68) at src/jemalloc.c:1846
#11 0x00007f43aa86c3ce in imalloc (sopts=0x7f433852ca10, dopts=0x7f433852c9e0) at src/jemalloc.c:1949
#12 0x00007f43aa86c436 in malloc (size=48) at src/jemalloc.c:1977
#13 0x00007f43aa8dba6d in newImpl<false> (size=48) at src/jemalloc_cpp.cpp:45
#14 0x00007f43aa8db8c5 in operator new (size=48) at src/jemalloc_cpp.cpp:78
#15 0x00007f43863115aa in std::_Rb_tree_node<std::pair<std::string const, unsigned int> >* std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_M_copy<std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node>(std::_Rb_tree_node<std::pair<std::string const, unsigned int> > const*, std::_Rb_tree_node_base*, std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libL1TriggerL1TGlobal.so
#16 0x00007f4386311590 in std::_Rb_tree_node<std::pair<std::string const, unsigned int> >* std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_M_copy<std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node>(std::_Rb_tree_node<std::pair<std::string const, unsigned int> > const*, std::_Rb_tree_node_base*, std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libL1TriggerL1TGlobal.so
#17 0x00007f4386311590 in std::_Rb_tree_node<std::pair<std::string const, unsigned int> >* std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_M_copy<std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node>(std::_Rb_tree_node<std::pair<std::string const, unsigned int> > const*, std::_Rb_tree_node_base*, std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libL1TriggerL1TGlobal.so
#18 0x00007f4386311450 in std::_Rb_tree_node<std::pair<std::string const, unsigned int> >* std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_M_copy<std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node>(std::_Rb_tree_node<std::pair<std::string const, unsigned int> > const*, std::_Rb_tree_node_base*, std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libL1TriggerL1TGlobal.so
#19 0x00007f4386311450 in std::_Rb_tree_node<std::pair<std::string const, unsigned int> >* std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_M_copy<std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node>(std::_Rb_tree_node<std::pair<std::string const, unsigned int> > const*, std::_Rb_tree_node_base*, std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::_Reuse_or_alloc_node&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libL1TriggerL1TGlobal.so
#20 0x00007f438631195f in std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > >::operator=(std::_Rb_tree<std::string, std::pair<std::string const, unsigned int>, std::_Select1st<std::pair<std::string const, unsigned int> >, std::less<std::string>, std::allocator<std::pair<std::string const, unsigned int> > > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libL1TriggerL1TGlobal.so
#21 0x00007f4354e0b475 in JetMETHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#22 0x00007f43abd64162 in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#23 0x00007f43abca2d72 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#24 0x00007f43abc2d827 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#25 0x00007f43abc2d9dd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#26 0x00007f43abc2dd6b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#27 0x00007f43abc2e215 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#28 0x00007f43aa6278ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f4338617e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#29 0x00007f43aa620c18 in tbb::internal::arena::process (this=0x7f43a9097780, s=...) at ../../src/tbb/arena.cpp:159
#30 0x00007f43aa61f6d3 in tbb::internal::market::process (this=0x7f43a90b3e00, j=...) at ../../src/tbb/market.cpp:697
#31 0x00007f43aa61b925 in tbb::internal::rml::private_worker::run (this=0x7f43a90ab980) at ../../src/tbb/private_server.cpp:271
#32 0x00007f43aa61bb69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#33 0x0000003197407aa1 in start_thread () from /lib64/libpthread.so.0
#34 0x00000031970e8bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4339635700 (LWP 5374)):
#0  0x00000031970acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000031970acac0 in sleep () from /lib64/libc.so.6
#2  0x00007f43a277d601 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f4356b6d746 in GenericTriggerEventFlag::acceptHlt(edm::Event const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libCommonToolsTriggerUtils.so
#5  0x00007f4354a58956 in TopMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginDQMOfflineTriggerPlugins.so
#6  0x00007f43abd64162 in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#7  0x00007f43abca2d72 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#8  0x00007f43abc2d827 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#9  0x00007f43abc2d9dd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#10 0x00007f43abc2dd6b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f43abc2e215 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f43aa6278ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f4358b5fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#13 0x00007f43aa620c18 in tbb::internal::arena::process (this=0x7f43a9097780, s=...) at ../../src/tbb/arena.cpp:159
#14 0x00007f43aa61f6d3 in tbb::internal::market::process (this=0x7f43a90b3e00, j=...) at ../../src/tbb/market.cpp:697
#15 0x00007f43aa61b925 in tbb::internal::rml::private_worker::run (this=0x7f43a90aba80) at ../../src/tbb/private_server.cpp:271
#16 0x00007f43aa61bb69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#17 0x0000003197407aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031970e8bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f433a036700 (LWP 5373)):
#0  0x00000031970df383 in poll () from /lib64/libc.so.6
#1  0x00007f43a277dd47 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f43a277e31c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f43a277f1dc in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f43aa86c18b in imalloc_body (sopts=0x7f433a02ea00, dopts=0x7f430002e9d0, tsd=0x7f433a02fb68) at src/jemalloc.c:1871
#6  0x00007f43aa86c3ce in imalloc (sopts=0x7f433a02ea00, dopts=0x7f433a02e9d0) at src/jemalloc.c:1949
#7  0x00007f43aa86c436 in malloc (size=97) at src/jemalloc.c:1977
#8  0x00007f43aa8dba6d in newImpl<false> (size=97) at src/jemalloc_cpp.cpp:45
#9  0x00007f43aa8db8c5 in operator new (size=97) at src/jemalloc_cpp.cpp:78
#10 0x000000000040c7a2 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) [clone .isra.124] ()
#11 0x000000000040c7e8 in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long) [clone .isra.125] ()
#12 0x0000000000413788 in std::string::reserve(unsigned long) ()
#13 0x0000000000413927 in std::string::append(std::string const&) ()
#14 0x00007f43ab9b3d43 in edm::InputTag::encode() const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreUtilities.so
#15 0x00007f4354e3c1f0 in void egHLT::trigTools::setFiltersObjPasses<egHLT::OffEle>(std::vector<egHLT::OffEle, std::allocator<egHLT::OffEle> >&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, std::bitset<128ul> const&, egHLT::TrigCodes const&, trigger::TriggerEvent const*, std::string const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#16 0x00007f4354e3750b in egHLT::OffHelper::setTrigInfo(edm::Event const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#17 0x00007f4354e3b5d9 in egHLT::OffHelper::makeOffEvt(edm::Event const&, edm::EventSetup const&, egHLT::OffEvt&, egHLT::TrigCodes const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#18 0x00007f4354f52f68 in EgHLTOfflineSource::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#19 0x00007f43abd64162 in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#20 0x00007f43abca2d72 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#21 0x00007f43abc2d827 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#22 0x00007f43abc2d9dd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#23 0x00007f43abc2dd6b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#24 0x00007f43abc2e215 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#25 0x00007f43aa6278ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f4338627e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#26 0x00007f43aa620c18 in tbb::internal::arena::process (this=0x7f43a9097780, s=...) at ../../src/tbb/arena.cpp:159
#27 0x00007f43aa61f6d3 in tbb::internal::market::process (this=0x7f43a90b3e00, j=...) at ../../src/tbb/market.cpp:697
#28 0x00007f43aa61b925 in tbb::internal::rml::private_worker::run (this=0x7f43a90aba00) at ../../src/tbb/private_server.cpp:271
#29 0x00007f43aa61bb69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#30 0x0000003197407aa1 in start_thread () from /lib64/libpthread.so.0
#31 0x00000031970e8bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f438bf1b700 (LWP 5244)):
#0  0x000000319740f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f43a277d887 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f43a277deba in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f43a9f14c2f in std::execute_native_thread_routine (__p=0x7f43a85c9880) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x0000003197407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000031970e8bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f43a9bca360 (LWP 5219)):
#0  0x00000031970acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000031970acac0 in sleep () from /lib64/libc.so.6
#2  0x00007f43a277d601 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f439a238718 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#5  0x00007f439a25f6c6 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libCommonToolsUtils.so
#6  0x00007f4354f310a8 in HLTMuonMatchAndPlot::selectedMuons(std::vector<reco::Muon, std::allocator<reco::Muon> > const&, reco::BeamSpot const&, bool, StringCutObjectSelector<reco::Muon, false> const&, double, double) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#7  0x00007f4354f3139d in HLTMuonMatchAndPlot::analyze(edm::Handle<std::vector<reco::Muon, std::allocator<reco::Muon> > >&, edm::Handle<reco::BeamSpot>&, edm::Handle<std::vector<reco::Vertex, std::allocator<reco::Vertex> > >&, edm::Handle<trigger::TriggerEvent>&, edm::Handle<edm::TriggerResults>&, edm::TriggerNames const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#8  0x00007f4354e9e055 in HLTMuonMatchAndPlotContainer::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/pluginDQMOfflineTrigger.so
#9  0x00007f43abd64162 in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#10 0x00007f43abca2d72 in edm::WorkerT<edm::stream::EDAnalyzerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f43abc2d827 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f43abc2d9dd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f43abc2dd6b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f43abc2e215 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f43aa6278ae in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f43a90b2400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#16 0x00007f43abce4c2a in edm::EventProcessor::readAndProcessEvents() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#17 0x00007f43abcef249 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02481/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_9_3_X_2017-07-21-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#18 0x000000000040da1a in main::{lambda()#1}::operator()() const ()
#19 0x000000000040c2ca in main ()

Current Modules:

Module: EgHLTOfflineSource:egHLTOffDQMSource (crashed)
Module: JetMETHLTOfflineSource:jetMETHLTOfflineSourceAK4
Module: TopMonitor:topDiElectronHLTMonitor_Dz
Module: HLTMuonOfflineAnalyzer:globalAnalyzerTnP

A fatal system signal has occurred: segmentation violation

@dan131riley
Copy link
Author

Looking at the stack traces I realized that this is the return of #14897 from a year ago...

@dan131riley
Copy link
Author

Poking further, both of these are for the slc6_amd64_gcc700 architecture, which is using jemalloc 5.0.1. Per jemalloc/jemalloc#369 jemalloc 5 does not support valgrind. Attempting valgrind memcheck or helgrind gives the error message

<jemalloc>: src/jemalloc.c:2837: Failed assertion: "malloc_initialized() || IS_INITIALIZER"

valgrind on an architecture with jemalloc 4.5.0 doesn't find any relevant problems (it does find lots else to complain about, though). It's possible this is actually a concurrency problem with jemalloc, or it could be a return of #14897 (which we never did understand, it just empirically went away when we reduced the InputTag churn), but our best tool for finding out doesn't work. I guess I'll try valgrind on slc6_amd64_gcc700 cmsRunGlibC.

@smuzaffar
Copy link
Contributor

look like this has been fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
pending-assignment
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dan131riley @smuzaffar @cmsbuild