Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid reads in LHAPDF::LogBicubicInterpolator::_interpolateXQ2() #44371

Open
makortel opened this issue Mar 11, 2024 · 10 comments
Open

Invalid reads in LHAPDF::LogBicubicInterpolator::_interpolateXQ2() #44371

makortel opened this issue Mar 11, 2024 · 10 comments
Labels
generators-pending pending-signatures

Comments

@makortel
Copy link
Contributor

This issue spins off the discovery in #44271 (comment) that running step1 of workflow 9.0 in CMSSW_14_1_X_2024-03-04-1100 with valgrind shows many

==4984== Invalid read of size 8
==4984==    at 0x426BC33C: LHAPDF::LogBicubicInterpolator::_interpolateXQ2(LHAPDF::KnotArray const&, double, unsigned long, double, unsigned long, int) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x426A82E9: LHAPDF::PDF::xfxQ2(int, double, double) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x4272B0EB: structm_ (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x404581CE: pypdfu_ (pypdfu.f:356)
==4984==    by 0x404A053B: pysigh_ (pysigh.f:420)
==4984==    by 0x40439FF7: pymign_ (pymign.f:80)
==4984==    by 0x404267E3: pyinit_ (pyinit.f:287)
==4984==    by 0x4025A94E: gen::Pythia6Hadronizer::initializeForInternalPartons() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4025B92E: edm::GeneratorFilter<gen::Pythia6Hadronizer, gen::ExternalDecayDriver>::beginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4C62378: edm::one::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4C4C8CF: edm::WorkerT<edm::one::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B9F95E: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4BA36ED: edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x42131CF: tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-0000/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:322)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:458)
==4984==    by 0x63F9290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==4984==    by 0x4B61CBA: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6B629: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6BB80: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/bin/slc7_amd64_gcc12/cmsRun)
==4984==    by 0x63E59BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==4984==  Address 0x7b6e53f8 is 8 bytes before a block of size 400 alloc'd
==4984==    at 0x402EEE1: operator new(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/valgrind/3.22.0-e989c977f8949fc48f1bc1206f1f87c8/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4984==    by 0x5285FFE: std::vector<double, std::allocator<double> >::_M_default_append(unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/lcg/root/6.30.05-09dd1c56883e4f434d3cfa5dffc18d6e/lib/libMathCore.so)
==4984==    by 0x426BA5C2: LHAPDF::KnotArray::fillLogKnots() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x426B68EC: LHAPDF::GridPDF::_loadData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x426CACDE: LHAPDF::mkPDF(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x427287EB: (anonymous namespace)::PDFSetHandler::loadMember(int) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x4272E0F4: pdfset_ (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/external/lhapdf/6.4.0-5969784ee06af968580d5197ca83d374/lib/libLHAPDF.so)
==4984==    by 0x404264A5: pyinit_ (pyinit.f:115)
==4984==    by 0x4025A94E: gen::Pythia6Hadronizer::initializeForInternalPartons() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4025B92E: edm::GeneratorFilter<gen::Pythia6Hadronizer, gen::ExternalDecayDriver>::beginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4C62378: edm::one::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4C4C8CF: edm::WorkerT<edm::one::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B9F95E: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4BA36ED: edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x42131CF: tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-0000/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:322)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:458)
==4984==    by 0x63F9290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==4984==    by 0x4B61CBA: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6B629: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6BB80: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{

and

==4984== Conditional jump or move depends on uninitialised value(s)
==4984==    at 0x404C0CED: pystrf_ (pystrf.f:145)
==4984==    by 0x4041C539: pyexec_ (pyexec.f:134)
==4984==    by 0x40419BB3: pyevnw_ (pyevnw.f:431)
==4984==    by 0x40259FB9: gen::Pythia6Hadronizer::generatePartonsAndHadronize() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4025D950: edm::GeneratorFilter<gen::Pythia6Hadronizer, gen::ExternalDecayDriver>::filter(edm::Event&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so)
==4984==    by 0x4C612FD: edm::one::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4C4C63D: edm::WorkerT<edm::one::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4BDE3D8: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4BDF4AE: edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x42131CF: tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-0000/lib/slc7_amd64_gcc12/libFWCoreConcurrency.so)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:322)
==4984==    by 0x63F9290: UnknownInlinedFun (task_dispatcher.h:458)
==4984==    by 0x63F9290: tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) (task_dispatcher.cpp:168)
==4984==    by 0x4B61CBA: edm::FinalWaitingTask::wait() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6B629: edm::EventProcessor::processRuns() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4B6BB80: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/lib/slc7_amd64_gcc12/libFWCoreFramework.so)
==4984==    by 0x4074EE: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/bin/slc7_amd64_gcc12/cmsRun)
==4984==    by 0x63E59BC: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:688)
==4984==    by 0x408ED1: main::{lambda()#1}::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/bin/slc7_amd64_gcc12/cmsRun)
==4984==    by 0x40518B: main (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02827/slc7_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-03-04-1100/bin/slc7_amd64_gcc12/cmsRun)
@makortel
Copy link
Contributor Author

assign generators

@cmsbuild
Copy link
Contributor

New categories assigned: generators

@alberto-sanchez,@bbilin,@GurpreetSinghChahal,@mkirsano,@menglu21,@SiewYan you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

cms-bot internal usage

@cmsbuild
Copy link
Contributor

A new Issue was created by @makortel.

@antoniovilela, @rappoccio, @sextonkennedy, @makortel, @Dr15Jones, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel makortel mentioned this issue Mar 11, 2024
Merged
@makortel makortel changed the title Invalid read in LHAPDF::LogBicubicInterpolator::_interpolateXQ2() Invalid reads in LHAPDF::LogBicubicInterpolator::_interpolateXQ2() Mar 11, 2024
@iarspider
Copy link
Contributor

Possibly related crash in CMSSW_14_1_DEVEL_X_2024-08-06-2300:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Aug  7 03:57:09 CEST 2024
Thread 5 (Thread 0x146fe05ff700 (LWP 2186482) "cmsRun"):
#0  0x0000147016faf098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147016faef9e in sleep () from /lib64/libc.so.6
#2  0x00001470126af1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147016ee041d in syscall () from /lib64/libc.so.6
#5  0x0000147018149fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x1470130a2024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x1470130a2024) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x1470130a2020) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x1470130a2000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x1470130a2000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001470172851ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000147016ee08d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x146fe15a8700 (LWP 2186480) "cmsRun"):
#0  0x0000147016faf098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147016faef9e in sleep () from /lib64/libc.so.6
#2  0x00001470126af1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147016ee041d in syscall () from /lib64/libc.so.6
#5  0x0000147018149fd2 in tbb::detail::r1::futex_wait (comparand=2, futex=0x1470130a2124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x1470130a2124) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x1470130a2120) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x1470130a2100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x1470130a2100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x00001470172851ca in start_thread () from /lib64/libpthread.so.0
#11 0x0000147016ee08d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x146fe1fa9700 (LWP 2186479) "cmsRun"):
#0  0x0000147016fd9ac1 in poll () from /lib64/libc.so.6
#1  0x00001470126b2857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001470126b2a54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000146fddc7533c in LHAPDF::LogBicubicInterpolator::_interpolateXQ2(LHAPDF::KnotArray const&, double, unsigned long, double, unsigned long, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#5  0x0000146fddc612ea in LHAPDF::PDF::xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#6  0x0000146fb8d39b26 in Pythia8::LHAPDF6::xfUpdate(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8lhapdf6.so
#7  0x0000146fde34d5b6 in Pythia8::PDF::xf(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#8  0x0000146fde30b884 in Pythia8::MultipartonInteractions::upperEnvelope() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#9  0x0000146fde31c807 in Pythia8::MultipartonInteractions::init(bool, int, Pythia8::BeamParticle*, Pythia8::BeamParticle*, std::shared_ptr<Pythia8::PartonVertex>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#10 0x0000146fde363e5e in Pythia8::PartonLevel::init(std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::SpaceShower>, Pythia8::RHadrons*, std::shared_ptr<Pythia8::MergingHooks>, std::shared_ptr<Pythia8::PartonVertex>, std::shared_ptr<Pythia8::StringInteractions>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#11 0x0000146fde3f98e2 in Pythia8::Pythia::init() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_DEVEL_X_2024-08-06-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#12 0x0000146fe0ab3eef in Pythia8Hadronizer::initializeForExternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia8Filters.so
#13 0x0000146fe0ad66ba in edm::ConcurrentHadronizerFilter<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>::initLumi(edm::gen::StreamCache<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>*, edm::LuminosityBlock const&, edm::EventSetup const&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia8Filters.so
#14 0x0000146fe0ac4dd8 in edm::ConcurrentHadronizerFilter<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>::globalBeginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia8Filters.so
#15 0x0000147019a5e2fb in edm::global::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x0000147019a55aa0 in edm::WorkerT<edm::global::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x00001470199961ca in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x0000147019985e8b in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x00001470199445de in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x0000147018147b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x147015720b00, waiter=..., this=0x1470157c9400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#21 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x1470157c9400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#22 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#23 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#24 0x0000147018149cee in tbb::detail::r1::rml::private_worker::run (this=0x1470130a2080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x1470130a2080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#26 0x00001470172851ca in start_thread () from /lib64/libpthread.so.0
#27 0x0000147016ee08d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x146ff03ff700 (LWP 2186458) "cmsRun"):
#0  0x000014701728f6a2 in waitpid () from /lib64/libpthread.so.0
#1  0x00001470126af327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001470126b263a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000147017919a73 in std::execute_native_thread_routine (__p=0x146ff19636b0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00001470172851ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000147016ee08d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1470163fd680 (LWP 2186334) "cmsRun"):
#0  0x0000147016faf098 in nanosleep () from /lib64/libc.so.6
#1  0x0000147016faef9e in sleep () from /lib64/libc.so.6
#2  0x00001470126af1d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000147016ee041d in syscall () from /lib64/libc.so.6
#5  0x000014701814eca4 in tbb::detail::r1::futex_wait (comparand=2, futex=0x7fffb8ddbb50) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x7fffb8ddbb50) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>::wait (this=0x7fffb8ddbb20) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:170
#8  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (this=<optimized out>, node=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:232
#9  tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::commit_wait (node=..., this=0x1470157cb598) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:228
#10 tbb::detail::r1::concurrent_monitor_base<tbb::detail::r1::market_context>::wait<tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&>(tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}&, tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>&&) (node=..., pred=<synthetic pointer>..., this=0x1470157cb598) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/concurrent_monitor.h:262
#11 tbb::detail::r1::sleep_waiter::sleep<tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}>(unsigned long, tbb::detail::r1::external_waiter::pause(tbb::detail::r1::arena_slot&)::{lambda()#1}) (this=<optimized out>, wakeup_condition=..., uniq_tag=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:118
#12 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:144
#13 tbb::detail::r1::external_waiter::pause (this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/waiters.h:137
#14 tbb::detail::r1::task_dispatcher::receive_or_steal_task<true, tbb::detail::r1::external_waiter> (this=<optimized out>, tls=..., ed=..., waiter=..., isolation=<optimized out>, fifo_allowed=<optimized out>, critical_allowed=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:231
#15 0x00001470181504e2 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x0, this=0x1470157c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:350
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x1470157c9380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#18 0x000014701995a5fb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x00001470199680ef in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x00001470199685a1 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02849/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_DEVEL_X_2024-08-05-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x000014701813c9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#23 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#24 0x0000000000405100 in main ()

Current Modules:

Module: Pythia8ConcurrentHadronizerFilter:generator (crashed)
Module: none
Module: none
Module: none

@iarspider
Copy link
Contributor

New occurance: link

Thread 3 (Thread 0x150df77ad700 (LWP 2456340) "cmsRun"):
#0  0x0000150e2e1baac1 in poll () from /lib64/libc.so.6
#1  0x0000150e2a5dacd7 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-08-18-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000150e2a5e07f3 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-08-18-0000/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000150df436633c in LHAPDF::LogBicubicInterpolator::_interpolateXQ2(LHAPDF::KnotArray const&, double, unsigned long, double, unsigned long, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#5  0x0000150df43522ea in LHAPDF::PDF::xfxQ2(int, double, double) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libLHAPDF.so
#6  0x0000150dcd14db26 in Pythia8::LHAPDF6::xfUpdate(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8lhapdf6.so
#7  0x0000150df323e5b6 in Pythia8::PDF::xf(int, double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#8  0x0000150df31fc884 in Pythia8::MultipartonInteractions::upperEnvelope() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#9  0x0000150df320d807 in Pythia8::MultipartonInteractions::init(bool, int, Pythia8::BeamParticle*, Pythia8::BeamParticle*, std::shared_ptr<Pythia8::PartonVertex>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#10 0x0000150df3254e5e in Pythia8::PartonLevel::init(std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::TimeShower>, std::shared_ptr<Pythia8::SpaceShower>, Pythia8::RHadrons*, std::shared_ptr<Pythia8::MergingHooks>, std::shared_ptr<Pythia8::PartonVertex>, std::shared_ptr<Pythia8::StringInteractions>, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#11 0x0000150df32ea8e2 in Pythia8::Pythia::init() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_NONLTO_X_2024-08-19-2300/external/el8_amd64_gcc12/lib/libpythia8.so
#12 0x0000150df6268abc in Pythia8Hadronizer::initializeForExternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_NONLTO_X_2024-08-18-0000/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia8Filters.so

@lviliani
Copy link
Contributor

lviliani commented Sep 6, 2024

Is this still relevant?

@makortel
Copy link
Contributor Author

makortel commented Sep 6, 2024

Last recorded occurrence was two weeks ago, so yes, looks still relevant.

@lviliani
Copy link
Contributor

lviliani commented Sep 6, 2024

Sorry, didn't realize the last messages are so recent.
Anyway I don't seem to be able to open the links above.

@makortel
Copy link
Contributor Author

makortel commented Sep 6, 2024

The IB logs are cleaned up after about two weeks. In order to reproduce I'd suggest to run one of the reported jobs through valgrind (which is how the problem first showed up), i.e. one of

  • workflow 9.0 step 1
  • workflow 515.0 step 2
  • workflow 518.0 step 2

We have also other LHAPDF issues that may be related or not: #35082, #35251, #43526. Actually #43526 (comment) seems to show these same symptoms (in workflow 561.0 step 1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
generators-pending pending-signatures
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@makortel @iarspider @cmsbuild @lviliani