Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault in HitPairEDProducer #36804

Closed
makortel opened this issue Jan 26, 2022 · 21 comments
Closed

Segfault in HitPairEDProducer #36804

makortel opened this issue Jan 26, 2022 · 21 comments
Labels
fully-signed reconstruction-approved

Comments

@makortel
Copy link
Contributor

Workflow 1030.0 crashed in CMSSW_12_3_X_2022-01-25-1700

Thread 10 (Thread 0x2b6770c00700 (LWP 1084) "cmsRun"):
#2  0x00002b6728fc2ca0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b67433d7e95 in BasicTrajectoryState::createLocalErrorFromCurvilinearError() const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libTrackingToolsTrajectoryState.so
#5  0x00002b6751b20dd8 in MaterialEffectsUpdator::updateStateInPlace(TrajectoryStateOnSurface&, PropagationDirection) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libTrackingToolsMaterialEffects.so
#6  0x00002b6751b227bd in PropagatorWithMaterial::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libTrackingToolsMaterialEffects.so
#7  0x00002b672dd9b87c in GeomDetCompatibilityChecker::isCompatible(GeomDet const*, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libTrackingToolsDetLayers.so
#8  0x00002b675120aaf8 in CompatibleDetToGroupAdder::add(GeomDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoTrackerTkDetLayers.so
#9  0x00002b675120d5c9 in Phase1PixelBlade::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoTrackerTkDetLayers.so
#10 0x00002b675120a32f in CompatibleDetToGroupAdder::add(GeometricSearchDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoTrackerTkDetLayers.so
#11 0x00002b675121975b in PixelForwardLayerPhase1::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoTrackerTkDetLayers.so
#12 0x00002b672dd9b981 in GeometricSearchDet::groupedCompatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libTrackingToolsDetLayers.so
#13 0x00002b67511a73de in LayerMeasurements::groupedMeasurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libTrackingToolsMeasurementDet.so
#14 0x00002b677e28824e in TrajectorySegmentBuilder::segments(TrajectoryStateOnSurface) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerCkfPatternPlugins.so
#15 0x00002b677e2703d5 in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerCkfPatternPlugins.so
#16 0x00002b677e27229c in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerCkfPatternPlugins.so
#17 0x00002b677e2724fd in GroupedCkfTrajectoryBuilder::rebuildSeedingRegion(TrajectorySeed const&, std::vector<TrackingRecHit const*, std::allocator<TrackingRecHit const*> > const&, TempTrajectory&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerCkfPatternPlugins.so
#18 0x00002b677e2737d1 in GroupedCkfTrajectoryBuilder::rebuildSeedingRegion(TrajectorySeed const&, TempTrajectory const&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerCkfPatternPlugins.so
#19 0x00002b677e273c34 in GroupedCkfTrajectoryBuilder::rebuildTrajectories(TempTrajectory const&, TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerCkfPatternPlugins.so
#20 0x00002b677e2d4383 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoTrackerCkfPattern.so
#21 0x00002b6720de64c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#22 0x00002b6720dcf7df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#23 0x00002b6720d2ac15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#24 0x00002b6720d2af0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#25 0x00002b6720d2d4f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#26 0x00002b6720ac17d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so

#34 0x00002b6723450b0d in clone () from /lib64/libc.so.6
#3  0x00002b6728fc63eb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b677e700958 in (anonymous namespace)::ImplSeedingHitSets::fill(RegionsSeedingHitSets::RegionFiller&, HitDoublets const&) [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerTkHitPairsPlugins.so
#6  0x00002b677e701f5d in (anonymous namespace)::Impl<(anonymous namespace)::ImplSeedingHitSets, (anonymous namespace)::DoNothing, (anonymous namespace)::RegionsLayersSeparate>::produce(bool, edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerTkHitPairsPlugins.so
#7  0x00002b6720de64c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b6720dcf7df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b6720d2ac15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b6720d2af0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b6720d2d4f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#12 0x00002b6720ac17d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so

Thread 8 (Thread 0x2b676ee00700 (LWP 1082) "cmsRun"):
#2  0x00002b6728fc2ca0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b678083baad in std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> >::~vector() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoVertexPrimaryVertexProducerPlugins.so
#5  0x00002b6780834ddd in PrimaryVertexProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoVertexPrimaryVertexProducerPlugins.so
#6  0x00002b6720de64c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#7  0x00002b6720dcf7df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b6720d2ac15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b6720d2af0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b6720d2d4f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b6720ac17d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so

Thread 1 (Thread 0x2b67254d4b00 (LWP 851) "cmsRun"):
#2  0x00002b6728fc2ca0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b6722e1e2b0 in *ABS*+0x146e0@plt () from /lib64/libm.so.6
#5  0x00002b67214b64ad in ROOT::Math::Cephes::igam(double, double) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/external/slc7_amd64_gcc10/lib/libMathCore.so
#6  0x00002b676664df68 in SiPixelTemplateReco::PixelTempReco1D(int, float, float, float, float, SiPixelTemplateReco::ClusMatrix&, SiPixelTemplate&, float&, float&, float&, float&, float&, float&, int&, int, bool, std::vector<std::pair<int, int>, std::allocator<std::pair<int, int> > >&, float&, int&, int&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoLocalTrackerSiPixelRecHits.so
#7  0x00002b6766635bbb in PixelCPEClusterRepair::callTempReco1D(PixelCPEBase::DetParam const&, PixelCPEClusterRepair::ClusterParamTemplate&, SiPixelTemplateReco::ClusMatrix&, int, Point3DBase<float, LocalTag>&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoLocalTrackerSiPixelRecHits.so
#8  0x00002b6766636b03 in PixelCPEClusterRepair::localPosition(PixelCPEBase::DetParam const&, PixelCPEBase::ClusterParam&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoLocalTrackerSiPixelRecHits.so
#9  0x00002b6766632e92 in PixelClusterParameterEstimator::getParameters(SiPixelCluster const&, GeomDet const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoLocalTrackerSiPixelRecHits.so
#10 0x00002b67511970b9 in TkClonerImpl::makeShared(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libRecoTrackerTransientTrackingRecHit.so
#11 0x00002b674a1b4a10 in SiPixelRecHit::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libDataFormatsTrackerRecHit2D.so
#12 0x00002b675309e2ca in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libTrackingToolsTrackFitters.so
#13 0x00002b6772023730 in TrackTransformer::transform(reco::TransientTrack const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libTrackingToolsTrackRefitter.so
#14 0x00002b6772023c19 in TrackTransformer::transform(reco::Track const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libTrackingToolsTrackRefitter.so
#15 0x00002b6771e30966 in TrackingRecoMaterialAnalyser::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginDQMTrackingMonitor.so
#16 0x00002b6720de64c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#17 0x00002b6720dcf7df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#18 0x00002b6720d2ac15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#19 0x00002b6720d2af0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#20 0x00002b6720d2d4f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#21 0x00002b6720ac17d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so

Current Modules:

Module: HitPairEDProducer:pixelPairStepHitDoublets (crashed)
Module: TrackingRecoMaterialAnalyser:materialDumperAnalyzer
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: PrimaryVertexProducer:unsortedOfflinePrimaryVertices

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc10/CMSSW_12_3_X_2022-01-25-1700/pyRelValMatrixLogs/run/1030.0_RunHLTPhy2017B+RunHLTPhy2017B+TIER0EXPHPBS+ALCASPLITHPBS+ALCAHARVDHPBS+ALCAHARVDHPBSLOWPU/step2_RunHLTPhy2017B+RunHLTPhy2017B+TIER0EXPHPBS+ALCASPLITHPBS+ALCAHARVDHPBS+ALCAHARVDHPBSLOWPU.log#/
@makortel
Copy link
Contributor Author

assign reconstruction

@cmsbuild
Copy link
Contributor

New categories assigned: reconstruction

@slava77,@jpata,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

Possible culprit could be #36184

@makortel
Copy link
Contributor Author

Crash was also seen in #36801 (comment) workflow 1040.0 step 2

Thread 1 (Thread 0x2b8ec3e0cb00 (LWP 31897) "cmsRun"):
#3  0x00002b8ec79c03eb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b8f362ec958 in (anonymous namespace)::ImplSeedingHitSets::fill(RegionsSeedingHitSets::RegionFiller&, HitDoublets const&) [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerTkHitPairsPlugins.so
#6  0x00002b8f362edf5d in (anonymous namespace)::Impl<(anonymous namespace)::ImplSeedingHitSets, (anonymous namespace)::DoNothing, (anonymous namespace)::RegionsLayersSeparate>::produce(bool, edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/pluginRecoTrackerTkHitPairsPlugins.so
#7  0x00002b8ebf71c4c3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b8ebf7057df in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b8ebf660c15 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b8ebf660f0b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b8ebf6634f5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#12 0x00002b8ebf3f97d5 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_3_X_2022-01-25-1700/lib/slc7_amd64_gcc10/libFWCoreConcurrency.so

Current Modules:

Module: HitPairEDProducer:pixelPairStepHitDoublets (crashed)

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-87be02/21993/runTheMatrixINPUT-results/1040.0_RunZeroBias2017F+RunZeroBias2017F+TIER0RAWSIPIXELCAL+ALCASPLITSIPIXELCAL+ALCAHARVDSIPIXELCAL/step2_RunZeroBias2017F+RunZeroBias2017F+TIER0RAWSIPIXELCAL+ALCASPLITSIPIXELCAL+ALCAHARVDSIPIXELCAL.log

@makortel makortel mentioned this issue Jan 26, 2022
Merged
@slava77
Copy link
Contributor

slava77 commented Jan 26, 2022

@AdrianoDee
please check

@qliphy qliphy mentioned this issue Jan 26, 2022
Merged
@makortel makortel mentioned this issue Jan 26, 2022
Closed
@makortel
Copy link
Contributor Author

Also CAHitQuadrupletEDProducer is showing crashes (#36808).

@AdrianoDee
Copy link
Contributor

AdrianoDee commented Jan 26, 2022 •
edited
Loading

I'm not being able to reproduce this (both for 1030.0 and 1040.0) for neither CMSSW_12_3_X_2022-01-25-1700, nor CMSSW_12_3_X_2022-01-25-2300 and neither CMSSW_12_3_X_2022-01-26-1100. All on slc7_amd64_gcc10 and on different machines:

  • Intel(R) Xeon(R) Silver 4110 at Bari T2;
  • Intel(R) Xeon(R) Gold 6130 on GPU machines@P5;
  • Intel Core Processor (Broadwell, IBRS) on lxplus.

with four threads and streams. Everything is running smoothly.
Any suggestion on how to reproduce it? A sample setup?

@makortel
Copy link
Contributor Author

Could valgrind reveal anything? (#36801 (comment) showed a crash on first event in 1040.0 step 2 on one thread)

@Dr15Jones
Copy link
Contributor

In the traceback, the problem comes from this call

cmssw/RecoTracker/TkHitPairs/plugins/HitPairEDProducer.cc

Line 123 in ec53ab6

seedingHitSetsProducer.fill(std::get<1>(hitCachePtr_filler_shs), doublets);

and is somewhere in this function

cmssw/RecoTracker/TkHitPairs/plugins/HitPairEDProducer.cc

Lines 172 to 176 in ec53ab6

void fill(RegionsSeedingHitSets::RegionFiller& filler, const HitDoublets& doublets) {
for (size_t i = 0, size = doublets.size(); i < size; ++i) {
filler.emplace_back(doublets.hit(i, HitDoublets::inner), doublets.hit(i, HitDoublets::outer));
}
}

@Dr15Jones
Copy link
Contributor

Looking at the code, I think the most likely problem is the doublets, especially since the instance of the class does not appear to own the memory for the RecHitsSortedInPhi.

@makortel
Copy link
Contributor Author

The call chain from the filler.emplace_back() is to

cmssw/RecoTracker/TkHitPairs/interface/RegionsSeedingHitSets.h

Lines 36 to 39 in ec53ab6

template <typename... Args>
void emplace_back(Args&&... args) {
obj_->hitSets_.emplace_back(std::forward<Args>(args)...);
}

cmssw/RecoTracker/TkSeedingLayers/interface/SeedingHitSet.h

Line 20 in a8490c8

SeedingHitSet(ConstRecHitPointer one, ConstRecHitPointer two) : theRecHits({one, two}) { setSize(); }

cmssw/RecoTracker/TkSeedingLayers/interface/SeedingHitSet.h

Lines 45 to 50 in a8490c8

void setSize() {
theSize = 0;
while (theRecHits[++theSize] and theSize < theRecHits.size())
;
theSize = theSize > 1 ? theSize : 0;
}

The last loop does (with theRecHits having two elements)

  • increase theSize to 1
  • check theRecHits[1] != nullptr (evaluates to true)
  • check 1 < 2
  • increase theSize to 2
  • check theRecHits[2] != nullptr, read goes out of bounds and UB hits

@AdrianoDee
Copy link
Contributor

Should move it to theRecHits[theSize++]

@makortel
Copy link
Contributor Author

How about filtering out the argument pointers after first nullptr in the constructors and relying on vector::size() for the size?

@AdrianoDee
Copy link
Contributor

AdrianoDee commented Jan 26, 2022 •
edited
Loading

Indeed, this method was there while the container had fixed size. Now doesn't make too much sense to stick to it. Let me prepare a PR.

@dan131riley
Copy link

The latest ASAN builds have a large number of heap-buffer-overflow errors that I suspect are related (I don't see any other obvious candidates). It seems to be highly reproducible in the ASAN build (803 failures):

==6181==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x603004d55ad8 at pc 0x2b3d87fbeed1 bp 0x2b3d1badc800 sp 0x2b3d1badc7f8
READ of size 8 at 0x603004d55ad8 thread T4
    #0 0x2b3d87fbeed0 in PixelTripletHLTGenerator::hitTriplets(TrackingRegion const&, OrderedHitTriplets&, edm::EventSetup const&, HitDoublets const&, RecHitsSortedInPhi const**, std::vector<DetLayer const*, std::allocator<DetLayer const*> > const&, int, std::vector<int, std::allocator<int> >*) (/cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-01-26-2300/lib/slc7_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so+0x1b1ed0)
    #1 0x2b3d87fc18c4 in PixelTripletHLTGenerator::hitTriplets(TrackingRegion const&, OrderedHitTriplets&, edm::Event const&, edm::EventSetup const&, HitDoublets const&, std::vector<SeedingLayerSetsHits::SeedingLayer, std::allocator<SeedingLayerSetsHits::SeedingLayer> > const&, std::vector<int, std::allocator<int> >*, LayerHitMapCache&) (/cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-01-26-2300/lib/slc7_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so+0x1b48c4)
    #2 0x2b3d881a9801 in hitTripletEDProducerT::Impl<PixelTripletHLTGenerator, hitTripletEDProducerT::ImplSeedingHitSets, hitTripletEDProducerT::DoNothing>::produce(IntermediateHitDoublets const&, edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-01-26-2300/lib/slc7_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so+0x39c801)
    #3 0x2b3d881be09d in HitTripletEDProducerT<PixelTripletHLTGenerator>::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-01-26-2300/lib/slc7_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so+0x3b109d)

0x603004d55ad8 is located 0 bytes to the right of 24-byte region [0x603004d55ac0,0x603004d55ad8)
allocated by thread T4 here:
    #0 0x2b3cc15bcd07 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2b3d87fbad4c in PixelTripletHLTGenerator::hitTriplets(TrackingRegion const&, OrderedHitTriplets&, edm::EventSetup const&, HitDoublets const&, RecHitsSortedInPhi const**, std::vector<DetLayer const*, std::allocator<DetLayer const*> > const&, int, std::vector<int, std::allocator<int> >*) (/cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-01-26-2300/lib/slc7_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so+0x1add4c)
    #2 0x2b3d87fc18c4 in PixelTripletHLTGenerator::hitTriplets(TrackingRegion const&, OrderedHitTriplets&, edm::Event const&, edm::EventSetup const&, HitDoublets const&, std::vector<SeedingLayerSetsHits::SeedingLayer, std::allocator<SeedingLayerSetsHits::SeedingLayer> > const&, std::vector<int, std::allocator<int> >*, LayerHitMapCache&) (/cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-01-26-2300/lib/slc7_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so+0x1b48c4)
    #3 0x2b3d881a9801 in hitTripletEDProducerT::Impl<PixelTripletHLTGenerator, hitTripletEDProducerT::ImplSeedingHitSets, hitTripletEDProducerT::DoNothing>::produce(IntermediateHitDoublets const&, edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-01-26-2300/lib/slc7_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so+0x39c801)
    #4 0x2b3d881be09d in HitTripletEDProducerT<PixelTripletHLTGenerator>::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02717/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-01-26-2300/lib/slc7_amd64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so+0x3b109d)

@AdrianoDee AdrianoDee mentioned this issue Jan 27, 2022
Merged
@AdrianoDee
Copy link
Contributor

@dan131riley, thanks. Indeed it was reproducible with the ASAN build (e.g. CMSSW_12_3_ASAN_X_2022-01-26-2300). #36819 should fix this.

@makortel makortel mentioned this issue Feb 1, 2022
Merged
@mmusich
Copy link
Contributor

mmusich commented Feb 15, 2022 •
edited
Loading

this is solved by #36819 right?
@cms-sw/reconstruction-l2 please sign and close.

@smuzaffar
Copy link
Contributor

@cms-sw/reconstruction-l2 , I think this has been resolved, can you please confirm?

@clacaputo
Copy link
Contributor

+reconstruction

@cmsbuild
Copy link
Contributor

This issue is fully signed and ready to be closed.

@qliphy qliphy closed this as completed Feb 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fully-signed reconstruction-approved
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@makortel @dan131riley @Dr15Jones @smuzaffar @cmsbuild @slava77 @mmusich @qliphy @clacaputo @AdrianoDee