Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TCling assertion failure in ROOT master #32734

Closed
makortel opened this issue Jan 25, 2021 · 14 comments
Closed

TCling assertion failure in ROOT master #32734

makortel opened this issue Jan 25, 2021 · 14 comments
Labels
core-approved fully-signed

Comments

@makortel
Copy link
Contributor

makortel commented Jan 25, 2021 •
edited
Loading

Workflow 1102 step 3 failed in ROOT master IB CMSSW_11_3_ROOT6_X_2021-01-24-2300 with

cmsRun: /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/lcg/root/6.23.01-8e7a9c4406e195d049d6d9526c8e2ae6/root-6.23.01/core/metacling/src/TCling.cxx:6147: virtual Int_t TCling::AutoLoad(const char*, Bool_t): Assertion `IsClassAutoLoadingEnabled() && "Calling when AutoLoading is off!"' failed.

Thread 5 (Thread 0x2b9008200700 (LWP 28725)):
#0  0x00002b8fb515b85d in nanosleep () from /lib64/libc.so.6
#1  0x00002b8fb515b6f4 in sleep () from /lib64/libc.so.6
#2  0x00002b8fc045dde0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b8fcf63cbbb in clang::serialization::MultiOnDiskHashTable<clang::serialization::reader::ASTDeclContextNameLookupTrait>::find(clang::DeclarationName const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#5  0x00002b8fcf63cfae in clang::ASTReader::FindExternalVisibleDeclsByName(clang::DeclContext const*, clang::DeclarationName) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#6  0x00002b8fcf758534 in clang::MultiplexExternalSemaSource::FindExternalVisibleDeclsByName(clang::DeclContext const*, clang::DeclarationName) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#7  0x00002b8fd105c4ee in clang::DeclContext::lookup(clang::DeclarationName) const () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#8  0x00002b8fcfb18f77 in LookupDirect(clang::Sema&, clang::LookupResult&, clang::DeclContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#9  0x00002b8fcfb19a5a in CppNamespaceLookup(clang::Sema&, clang::LookupResult&, clang::ASTContext&, clang::DeclContext*, (anonymous namespace)::UnqualUsingDirectiveSet&) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#10 0x00002b8fcfb1c4f5 in clang::Sema::CppLookupName(clang::LookupResult&, clang::Scope*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#11 0x00002b8fcfb1ca1d in clang::Sema::LookupName(clang::LookupResult&, clang::Scope*, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#12 0x00002b8fcf87cf29 in clang::Sema::HandleDeclarator(clang::Scope*, clang::Declarator&, llvm::MutableArrayRef<clang::TemplateParameterList*>) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#13 0x00002b8fcf87fc2c in clang::Sema::ActOnStartOfFunctionDef(clang::Scope*, clang::Declarator&, llvm::MutableArrayRef<clang::TemplateParameterList*>, clang::Sema::SkipBodyInfo*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#14 0x00002b8fcf5e98cb in clang::Parser::ParseFunctionDefinition(clang::ParsingDeclarator&, clang::Parser::ParsedTemplateInfo const&, clang::Parser::LateParsedAttrList*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#15 0x00002b8fcf566f2e in clang::Parser::ParseDeclGroup(clang::ParsingDeclSpec&, unsigned int, clang::SourceLocation*, clang::Parser::ForRangeInit*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#16 0x00002b8fcf5e5fa1 in clang::Parser::ParseDeclOrFunctionDefInternal(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec&, clang::AccessSpecifier) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#17 0x00002b8fcf5ec950 in clang::Parser::ParseExternalDeclaration(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#18 0x00002b8fcf57b004 in clang::Parser::ParseLinkage(clang::ParsingDeclSpec&, unsigned int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#19 0x00002b8fcf5e611e in clang::Parser::ParseDeclOrFunctionDefInternal(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec&, clang::AccessSpecifier) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#20 0x00002b8fcf5e64b1 in clang::Parser::ParseDeclarationOrFunctionDefinition(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec*, clang::AccessSpecifier) [clone .part.0] () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#21 0x00002b8fcf5ecf42 in clang::Parser::ParseExternalDeclaration(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#22 0x00002b8fcf5ed399 in clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#23 0x00002b8fcef7c0e6 in cling::IncrementalParser::ParseInternal(llvm::StringRef) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#24 0x00002b8fcef7dfed in cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#25 0x00002b8fceee6852 in cling::Interpreter::DeclareInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions const&, cling::Transaction**) const () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#26 0x00002b8fceee69fa in cling::Interpreter::declare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::Transaction**) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#27 0x00002b8fceee79e1 in cling::Interpreter::DeclareCFunction(llvm::StringRef, llvm::StringRef, bool, cling::Transaction*&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#28 0x00002b8fceeeaa2d in cling::Interpreter::compileFunction(llvm::StringRef, llvm::StringRef, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libCling.so
#29 0x00002b8fcee7556b in TClingCallFunc::compile_wrapper (this=0x2b913aff4c00, wrapper_name=..., wrapper=..., withAccessControl=true) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/lcg/root/6.23.01-8e7a9c4406e195d049d6d9526c8e2ae6/root-6.23.01/core/metacling/src/TClingCallFunc.cxx:267
#30 0x00002b8fcee78390 in TClingCallFunc::make_wrapper (this=0x2b913aff4c00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/lcg/root/6.23.01-8e7a9c4406e195d049d6d9526c8e2ae6/root-6.23.01/core/metacling/src/TClingCallFunc.cxx:1117
#31 0x00002b8fcee7d5a8 in TClingCallFunc::IFacePtr (this=0x2b913aff4c00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/lcg/root/6.23.01-8e7a9c4406e195d049d6d9526c8e2ae6/root-6.23.01/core/metacling/src/TClingCallFunc.cxx:2301
#32 0x00002b8fced52d75 in TCling::CallFunc_IFacePtr (this=0x2b8fb708cf80, func=0x2b913aff4c00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/lcg/root/6.23.01-8e7a9c4406e195d049d6d9526c8e2ae6/root-6.23.01/core/metacling/src/TCling.cxx:7882
#33 0x00002b8fb20bc427 in edm::FunctionWithDict::FunctionWithDict(TMethod*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreReflection.so
#34 0x00002b8fb20c619a in edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreReflection.so
#35 0x00002b8ffa1dee7e in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#36 0x00002b8ffa1c1cc9 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#37 0x00002b8ffa1bd24d in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#38 0x00002b8ffa1be2d1 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#39 0x00002b8ffa1c08c2 in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#40 0x00002b8ffa1a6041 in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#41 0x00002b8ffa1cc9c9 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#42 0x00002b8ffa1bca26 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#43 0x00002b902741ab1a in ObjectSelector<SingleElementCollectionSelector<edm::View<reco::Candidate>, StringCutObjectSelector<reco::Candidate, true>, edm::PtrVector<reco::Candidate>, edm::PtrVector<reco::Candidate>, helper::SelectionPtrViewAdder<reco::Candidate> >, edm::PtrVector<reco::Candidate>, NonNullNumberSelector, helper::NullPostProcessor<edm::PtrVector<reco::Candidate>, edm::stream::EDFilter<> >, helper::CollectionStoreManager<edm::PtrVector<reco::Candidate>, helper::IteratorToObjectConverter<edm::PtrVector<reco::Candidate> > >, helper::ObjectSelectorBase<edm::PtrVector<reco::Candidate>, edm::stream::EDFilter<> >, reco::modules::SingleElementCollectionSelectorEventSetupInit<SingleElementCollectionSelector<edm::View<reco::Candidate>, StringCutObjectSelector<reco::Candidate, true>, edm::PtrVector<reco::Candidate>, edm::PtrVector<reco::Candidate>, helper::SelectionPtrViewAdder<reco::Candidate> > > >::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginCommonToolsCandAlgos_plugins.so
#44 0x00002b8fb23f6c34 in edm::stream::EDFilterAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#45 0x00002b8fb23d4c8d in edm::WorkerT<edm::stream::EDFilterAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#46 0x00002b8fb23353c5 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#47 0x00002b8fb233557d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#48 0x00002b8fb2335886 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#49 0x00002b8fb2336fb2 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so

Thread 4 (Thread 0x2b90072d3700 (LWP 28724)):
#0  0x00002b8fb515b85d in nanosleep () from /lib64/libc.so.6
#1  0x00002b8fb515b6f4 in sleep () from /lib64/libc.so.6
#2  0x00002b8fc045dde0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b8fb4b8c859 in atanf () from /lib64/libm.so.6
#5  0x00002b8fb4b880a4 in __atan2f_finite () from /lib64/libm.so.6
#6  0x00002b9026e557a0 in TrackingParticleSelector::operator()(TrackingParticle const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libValidationRecoTrack.so
#7  0x00002b9026e39f38 in MTVHistoProducerAlgoForTracker::fill_recoAssociated_simTrack_histos(MTVHistoProducerAlgoForTrackerHistograms const&, int, TrackingParticle const&, ROOT::Math::DisplacementVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag> const&, ROOT::Math::PositionVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag> const&, double, double, double, double, int, int, int, int, reco::Track const*, int, double, double, ROOT::Math::PositionVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag> const*, ROOT::Math::LorentzVector<ROOT::Math::PxPyPzE4D<double> > const*, ROOT::Math::PositionVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag> const&, std::vector<float, std::allocator<float> > const&, unsigned int, unsigned int) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libValidationRecoTrack.so
#8  0x00002b9026cb46df in MultiTrackValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, MultiTrackValidatorHistograms const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginValidationRecoTrackPlugins.so
#9  0x00002b8fb23df3cf in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#10 0x00002b8fb23d464d in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#11 0x00002b8fb23353c5 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002b8fb233557d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#13 0x00002b8fb2335886 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002b8fb2336fb2 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so

Thread 3 (Thread 0x2b90068d2700 (LWP 28723)):
#0  0x00002b8fb5189c3d in poll () from /lib64/libc.so.6
#1  0x00002b8fc045e397 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#2  0x00002b8fc045eaac in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  0x00002b8fc045f7d4 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b8fb50cc387 in raise () from /lib64/libc.so.6
#6  0x00002b8fb50cda78 in abort () from /lib64/libc.so.6
#7  0x00002b8fb50c51a6 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00002b8fb50c5252 in __assert_fail () from /lib64/libc.so.6
#9  0x00002b8fced4b768 in TCling::AutoLoad (this=0x2b8fb708cf80, cls=0x2b90068c9c80 "size_t", knowDictNotLoaded=false) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/lcg/root/6.23.01-8e7a9c4406e195d049d6d9526c8e2ae6/root-6.23.01/core/metacling/src/TCling.cxx:6147
#10 0x00002b8fb38b5e09 in TEnum::GetEnum (enumName=0x2b90068c9c80 "size_t", sa=TEnum::kAutoload) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/lcg/root/6.23.01-8e7a9c4406e195d049d6d9526c8e2ae6/root-6.23.01/core/meta/src/TEnum.cxx:274
#11 0x00002b8fb20c4ca8 in edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreReflection.so
#12 0x00002b8fb20c519d in edm::TypeWithDict::TypeWithDict(TMethodArg*, long) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreReflection.so
#13 0x00002b8ffa1de7dd in reco::checkMethod(edm::FunctionWithDict const&, edm::TypeWithDict const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#14 0x00002b8ffa1deeab in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#15 0x00002b8ffa1c1cc9 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#16 0x00002b8ffa1bd24d in reco::parser::SingleInvoker::SingleInvoker(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#17 0x00002b8ffa1be2d1 in reco::parser::LazyInvoker::invoker(edm::TypeWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#18 0x00002b8ffa1c08c2 in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#19 0x00002b8ffa1a6041 in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#20 0x00002b8ffa1cc9c9 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCommonToolsUtils.so
#21 0x00002b902741ab1a in ObjectSelector<SingleElementCollectionSelector<edm::View<reco::Candidate>, StringCutObjectSelector<reco::Candidate, true>, edm::PtrVector<reco::Candidate>, edm::PtrVector<reco::Candidate>, helper::SelectionPtrViewAdder<reco::Candidate> >, edm::PtrVector<reco::Candidate>, NonNullNumberSelector, helper::NullPostProcessor<edm::PtrVector<reco::Candidate>, edm::stream::EDFilter<> >, helper::CollectionStoreManager<edm::PtrVector<reco::Candidate>, helper::IteratorToObjectConverter<edm::PtrVector<reco::Candidate> > >, helper::ObjectSelectorBase<edm::PtrVector<reco::Candidate>, edm::stream::EDFilter<> >, reco::modules::SingleElementCollectionSelectorEventSetupInit<SingleElementCollectionSelector<edm::View<reco::Candidate>, StringCutObjectSelector<reco::Candidate, true>, edm::PtrVector<reco::Candidate>, edm::PtrVector<reco::Candidate>, helper::SelectionPtrViewAdder<reco::Candidate> > > >::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginCommonToolsCandAlgos_plugins.so
#22 0x00002b8fb23f6c34 in edm::stream::EDFilterAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#23 0x00002b8fb23d4c8d in edm::WorkerT<edm::stream::EDFilterAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#24 0x00002b8fb23353c5 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#25 0x00002b8fb233557d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#26 0x00002b8fb2335886 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#27 0x00002b8fb2336fb2 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so

Thread 1 (Thread 0x2b8fb6dafc80 (LWP 28384)):
#0  0x00002b8fb515b85d in nanosleep () from /lib64/libc.so.6
#1  0x00002b8fb515b6f4 in sleep () from /lib64/libc.so.6
#2  0x00002b8fc045dde0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b8fb5189c3d in poll () from /lib64/libc.so.6
#5  0x00002b8ffef9150a in frontier_read () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#6  0x00002b8ffef8e7a9 in http_read () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#7  0x00002b8ffef8e7f3 in read_char () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#8  0x00002b8ffef8e8d1 in read_line () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#9  0x00002b8ffef8f1e1 in read_connection () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#10 0x00002b8ffef8f62f in frontierHttpClnt_post () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#11 0x00002b8ffef8f53d in frontierHttpClnt_get () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#12 0x00002b8ffef9ba44 in get_data () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#13 0x00002b8ffef9c26e in frontier_postRawData () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#14 0x00002b8ffef9bedc in frontier_getRawData () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#15 0x00002b8ffef93a15 in frontier::Session::getData(std::vector<frontier::Request const*, std::allocator<frontier::Request const*> > const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/libfrontier_client.so.2
#16 0x00002b8ffef2eca4 in coral::FrontierAccess::Statement::execute(coral::AttributeList const&, int) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/liblcg_FrontierAccess.so
#17 0x00002b8ffef214b9 in coral::FrontierAccess::Query::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_ROOT6_X_2021-01-24-2300/external/slc7_amd64_gcc900/lib/liblcg_FrontierAccess.so
#18 0x00002b8ff38aa5f5 in cond::persistency::PAYLOAD::Table::select(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, cond::Binary&, cond::Binary&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libCondCoreCondDB.so
#19 0x00002b90009f8929 in std::unique_ptr<PhysicsTGraphPayload, std::default_delete<PhysicsTGraphPayload> > cond::persistency::Session::fetchPayload<PhysicsTGraphPayload>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginCondCorePhysicsToolsPlugins.so
#20 0x00002b90009f8d2b in cond::persistency::PayloadProxy<PhysicsTGraphPayload>::loadPayload() () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginCondCorePhysicsToolsPlugins.so
#21 0x00002b90009ed13f in DataProxy<PhysicsTGraphPayloadRcd, PhysicsTGraphPayload, cond::DefaultInitializer<PhysicsTGraphPayload> >::prefetch(edm::eventsetup::DataKey const&, edm::EventSetupRecordDetails) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginCondCorePhysicsToolsPlugins.so
#22 0x00002b8fb229b89f in edm::SerialTaskQueue::QueuedTask<edm::eventsetup::ESSourceDataProxyBase::prefetchAsyncImpl(edm::WaitingTaskHolder, edm::eventsetup::EventSetupRecordImpl const&, edm::eventsetup::DataKey const&, edm::EventSetupImpl const*, edm::ServiceToken const&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#23 0x00002b8fb4078045 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x2b8fb7a7a600, context_guard=..., t=0x2b913dd29140, isolation=isolation@entry=0) at ../../src/tbb/scheduler.h:630
#24 0x00002b8fb40782fb in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x2b8fb7a7a600, parent=..., child=<optimized out>) at ../../include/tbb/task.h:1003
#25 0x00002b8fb2287dc0 in tbb::interface7::internal::delegated_function<edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecordImpl const&, edm::eventsetup::DataKey const&, bool, edm::ActivityRegistry const*, edm::EventSetupImpl const*) const::{lambda()#1} const, void>::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#26 0x00002b8fb4072b92 in tbb::interface7::internal::task_arena_base::internal_execute (this=0x2b8fb24bafa0 <edm::esTaskArena()::s_arena>, d=...) at ../../src/tbb/arena.cpp:1105
#27 0x00002b8fb2287f95 in edm::eventsetup::DataProxy::get(edm::eventsetup::EventSetupRecordImpl const&, edm::eventsetup::DataKey const&, bool, edm::ActivityRegistry const*, edm::EventSetupImpl const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#28 0x00002b8fb22fdc94 in edm::eventsetup::EventSetupRecordImpl::getFromProxy(edm::eventsetup::DataKey const&, edm::eventsetup::ComponentDescription const*&, bool, edm::EventSetupImpl const*) const () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#29 0x00002b901312715a in (anonymous namespace)::loadTGraphFromDB(edm::EventSetup const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int const&) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginRecoTauTagRecoTauPlugins.so
#30 0x00002b9013130b93 in TauDiscriminantCutMultiplexerT<reco::PFTau, edm::Ref<std::vector<reco::PFTau, std::allocator<reco::PFTau> >, reco::PFTau, edm::refhelper::FindUsingAdvance<std::vector<reco::PFTau, std::allocator<reco::PFTau> >, reco::PFTau> >, TauDiscriminationProducerBase<reco::PFTau, edm::ValueMap<reco::SingleTauDiscriminatorContainer>, reco::SingleTauDiscriminatorContainer, reco::PFTauDiscriminator> >::beginEvent(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/pluginRecoTauTagRecoTauPlugins.so
#31 0x00002b9013295e9c in TauDiscriminationProducerBase<reco::PFTau, edm::ValueMap<reco::SingleTauDiscriminatorContainer>, reco::SingleTauDiscriminatorContainer, reco::PFTauDiscriminator>::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libRecoTauTagRecoTau.so
#32 0x00002b8fb23f8924 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#33 0x00002b8fb23d4b4d in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#34 0x00002b8fb23353c5 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#35 0x00002b8fb233557d in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#36 0x00002b8fb2335886 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#37 0x00002b8fb2336fb2 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#38 0x00002b8fb4078045 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x2b8fb7a7a600, context_guard=..., t=0x2b913df60640, isolation=isolation@entry=0) at ../../src/tbb/scheduler.h:630
#39 0x00002b8fb40782fb in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x2b8fb7a7a600, parent=..., child=<optimized out>) at ../../include/tbb/task.h:1003
#40 0x00002b8fb22aa625 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms-ib.cern.ch/nweek-02665/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT6_X_2021-01-24-0000/lib/slc7_amd64_gcc900/libFWCoreFramework.so

Current Modules:
Module: CandPtrSelector:CHSCands (crashed)
Module: CandPtrSelector:TrkCands
Module: RecoTauDiscriminantCutMultiplexer:hpsPFTauDiscriminationByIsolationMVArun2v1DBdR03oldDMwLTBoosted
Module: MultiTrackValidator:trackValidatorBuilding

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_11_3_ROOT6_X_2021-01-24-2300/pyRelValMatrixLogs/run/1102.0_RR+TTbarINPUT+DIGI+RECO+RECOFROMRECO+COPYPASTE/step3_RR+TTbarINPUT+DIGI+RECO+RECOFROMRECO+COPYPASTE.log#/
@cmsbuild
Copy link
Contributor

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

assign core

@cmsbuild
Copy link
Contributor

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor Author

@pcanal Could you take a look?

Given that we have not seen any workflow failure sin ROOT6 IBs for at least a week, I'm wondering if this could be a threading issue?

@pcanal
Copy link
Contributor

pcanal commented Jan 25, 2021

It does indeed looks like a race condition. (On the state/value of TCling::IsClassAutoLoadingEnabled). I will be looking at the stack to see if I can find it with just that information. Additional tracing the value change for TCling::IsClassAutoLoadingEnabled in the debugger might give more clue (i.e. I would need a reproducer that does not need a proxy certificate).

@makortel
Copy link
Contributor Author

@pcanal The workflow is MC (without pileup) so you should be able to run yourself first runTheMatrix.py -l 1102.0 -t 4 and repeat running the step3_RAW2DIGI_L1Reco_RECO_RECOSIM_EI_PAT_VALIDATION_DQM.py

@pcanal pcanal mentioned this issue Jan 25, 2021
Closed
@pcanal
Copy link
Contributor

pcanal commented Jan 25, 2021

@makortel Can you try root-project/root#7094 which is likely to fix the problem?

@makortel
Copy link
Contributor Author

I'm a bit skeptical additional testing by me being useful given that the race condition occurred in 1 / (> 7x3397) = 1 / ( > ~23k) jobs.

@pcanal
Copy link
Contributor

pcanal commented Jan 25, 2021

:)

Alright, then can you incorporate the patch into the CMSSW builds once I merge them?

@pcanal
Copy link
Contributor

pcanal commented Feb 25, 2021

The patch has been merge into the master (for v6.24) and v6.22 patch branch. See root-project/root#7095

@makortel
Copy link
Contributor Author

@smuzaffar @mrodozov Please take note for the next ROOT update(s).

This was referenced Feb 25, 2021
Merged
Merged
@makortel
Copy link
Contributor Author

makortel commented Jun 3, 2021

We have not seen this a while, so I suppose we can close it.

@makortel
Copy link
Contributor Author

makortel commented Jun 3, 2021

+1

@makortel makortel closed this as completed Jun 3, 2021
@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 3, 2021

This issue is fully signed and ready to be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core-approved fully-signed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@makortel @cmsbuild @pcanal