Random comparison difference in HLT/SiStrip/ControlView #41200

makortel · 2023-03-27T17:48:35Z

The HLT/SiStrip/ControlView/{ClusterStoNCorr_OnTrack_FECCratevsFECSlot,ClusterStoNCorr_OnTrack_FECSlotVsFECRing_TECP} histograms showed differences in workflow 11634.911 in PR tests of #41186 (comment) . The PR itself is very unlikely to be the cause of the differences. The differences have also not been visible in other recent PR tests, so these differences have likely random origin. The purpose of this issue is to nevertheless document them, in case they are visible in other tests later on.

The 11634.911 is the DD4Hep workflow that, IIUC, reads the geometry from the XML file instead from the CondDB. These differences may be evidence of some rare non-reproducibility in DD4Hep code path (that we have observed, but not really solved, before).

The text was updated successfully, but these errors were encountered:

makortel · 2023-03-27T17:48:43Z

assign geometry

cmsbuild · 2023-03-27T17:48:52Z

New categories assigned: geometry

@mdhildreth,@Dr15Jones,@makortel,@bsunanda,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild · 2023-03-27T17:48:55Z

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

VinInn · 2023-03-28T13:17:53Z

is architecture dependence excluded (aka INTEL vs AMD)?

makortel · 2023-03-28T13:58:02Z

Good point. I checked the PR test and baseline runTheMatrix output of #41186 (comment), and both were run on Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz.

VinInn · 2023-03-28T14:34:15Z

I'm running valgrind on step1 and see a bunch of these

==1882951== Invalid read of size 8
==1882951==    at 0x40F3AA2E: vecgeom::cxx::CommonUnplacedVolumeImplHelper<vecgeom::cxx::PolyhedronImplementation<(EInnerRadii)0, (EPhiCutout)0>, vecgeom::cxx::VUnplacedVolume>::SafetyToIn(vecgeom::cxx::Vector3D<double> const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x40EF79F6: G4UAdapter<vecgeom::cxx::UnplacedPolyhedron>::DistanceToIn(CLHEP::Hep3Vector const&) const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x40FC659C: G4VoxelNavigation::ComputeStep(CLHEP::Hep3Vector const&, CLHEP::Hep3Vector const&, double, double&, G4NavigationHistory&, bool&, CLHEP::Hep3Vector&, bool&, bool&, G4VPhysicalVolume**, int&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x40CDF04A: G4Navigator::ComputeStep(CLHEP::Hep3Vector const&, CLHEP::Hep3Vector const&, double, double&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x40E52FA6: G4Transportation::AlongStepGetPhysicalInteractionLength(G4Track const&, double, double, double&, G4GPILSelection*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x40E4B87B: G4TrackingManager::ProcessOneTrack(G4Track*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x40C38D19: G4EventManager::DoProcessing(G4Event*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x40989BB9: RunManagerMTWorker::produce(edm::Event const&, edm::EventSetup const&, RunManagerMT&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x40995831: omt::ThreadHandoff::Functor<OscarMTProducer::produce(edm::Event&, edm::EventSetup const&)::{lambda()#1}>::execute() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x4097A919: omt::ThreadHandoff::threadLoop(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/biglib/el8_amd64_gcc11/pluginSimulation.so)
==1882951==    by 0x70861C9: start_thread (in /usr/lib64/libpthread-2.28.so)
==1882951==    by 0x72D7E72: clone (in /usr/lib64/libc-2.28.so)
==1882951==  Address 0x54664188 is 24 bytes before an unallocated block of size 0 in arena "client"
==1882951==

need to be understood if related specifically to DD4HEP

VinInn · 2023-03-28T14:36:19Z

my valgrind command

valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --tool=memcheck \
--suppressions=$ROOTSYS/etc/valgrind-root.supp \
--suppressions=$CMSSW_RELEASE_BASE/src/Utilities/ReleaseScripts/data/cms-valgrind-memcheck.supp cmsRun $1

VinInn · 2023-03-28T14:45:57Z

not sure if the valgrind report is actually still related to this
https://sft.its.cern.ch/jira/projects/VECGEOM/issues/VECGEOM-600?filter=allopenissues

civanch · 2023-03-28T16:45:17Z

@VinInn , this issue was understood as a compiler bug when -O3 optimisation is used. The solution was to use -O2 optimisation for VecGeom. However, I am not sure if the problem Matti are reporting here is the same.

VinInn · 2023-03-28T16:48:54Z

I got the report above in latest 13_1_X nighty. Maybe understood, not solved apparently.
VecGeom not vectorized is a bit incongruous...

VinInn · 2023-03-28T17:33:35Z

one more valgrind message in step2 (most probably for a different issue)

==1891808== Invalid free() / delete / delete[] / realloc()
==1891808==    at 0x403BF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1891808==    by 0x48F90DDB: edm::Wrapper<ZVertexSoAHeterogeneousHost<131072> >::~Wrapper() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/lib/el8_amd64_gcc11/libCUDADataFormatsVe
rtex.so)
==1891808==    by 0x48F90DF3: edm::Wrapper<ZVertexSoAHeterogeneousHost<131072> >::~Wrapper() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/lib/el8_amd64_gcc11/libCUDADataFormatsVe
rtex.so)
==1891808==    by 0x4DD56FA: edm::productholderindexhelper::getContainedTypeFromWrapper(edm::TypeID const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-0277
8/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==1891808==    by 0x4DDB31F: edm::ProductRegistry::initializeLookupTables(std::set<edm::TypeID, std::less<edm::TypeID>, std::allocator<edm::TypeID> > const*, std::set<edm::TypeID, std::less<edm::TypeID>, std::allocator<edm::TypeID> > const*,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/lib/el8_amd64_gcc11/libDataFormatsProvenance.
so)
==1891808==    by 0x4DD15EF: edm::ProductRegistry::setFrozen(std::set<edm::TypeID, std::less<edm::TypeID>, std::allocator<edm::TypeID> > const&, std::set<edm::TypeID, std::less<edm::TypeID>, std::allocator<edm::TypeID> > const&, std::__cxx11:
:basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==1891808==    by 0x4C161B6: edm::Schedule::finishSetup(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ProcessBlockHelperBase&, edm::ThinnedAssociationsHelper&, edm::SubProc
essParentageHelper const*, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64
_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==1891808==    by 0x4C2676D: edm::ScheduleItems::finishSchedule(edm::ScheduleItems::MadeModules, edm::ParameterSet&, edm::service::TriggerNamesService const&, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::Proc
essBlockHelperBase&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023-03-28-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==1891808==    by 0x4B68977: edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13
_1_X_2023-03-28-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==1891808==    by 0x4B6BAD0: edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch
/CMSSW_13_1_X_2023-03-28-1100/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==1891808==    by 0x40C0AC: tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_1_X_2023
-03-28-1100/bin/el8_amd64_gcc11/cmsRun)
==1891808==    by 0x63D3846: tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) (arena.cpp:694)
==1891808==  Address 0xaa644380 is in a rw- anonymous segment
==1891808==

makortel · 2023-04-04T20:05:32Z

Another occurrence in #41274 (comment), this time in workflow 12434.0

makortel · 2023-04-04T20:05:47Z

assign dqm

cmsbuild · 2023-04-04T20:05:51Z

New categories assigned: dqm

@micsucmed,@rvenditti,@emanueleusai,@syuvivida,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel · 2023-04-04T20:05:56Z

FYI @cms-sw/trk-dpg-l2

makortel · 2023-04-20T15:59:52Z

Another occurrence in #41328 (comment), this time in workflow 12434.0

makortel · 2023-05-02T14:36:11Z

Another occurrence in #41460 (comment) in workflow 12434.0

makortel · 2023-06-06T08:37:20Z

Another occurrence in #41876 (comment) in workflow 12434.0

makortel · 2023-06-20T13:45:57Z

Another occurrence in cms-sw/cmsdist#8545 (comment) in workflow 12434.0 (although there because of an update of the compiler minor(?) differences in generated code can not be excluded)

makortel · 2023-06-26T12:48:57Z

Another occurrence in #42075 (comment) in workflow 12434.0.

mmusich · 2023-07-31T07:13:03Z

type trk

cmsbuild added geometry-pending pending-signatures labels Mar 27, 2023

makortel mentioned this issue Mar 27, 2023

(13_0_X) streamer format padding backport #41186

Merged

makortel changed the title ~~Random comparison difference in HLT/SiStrip/ControlView in 11634.911~~ Random comparison difference in HLT/SiStrip/ControlView Apr 4, 2023

cmsbuild added the dqm-pending label Apr 4, 2023

makortel mentioned this issue Apr 4, 2023

Remove FWCore TestHelper utility #41274

Merged

mmusich mentioned this issue Apr 10, 2023

redefine IT digitizer ToF window + add customize fcn for activating IT signal shape with RelVals #41273

Merged

makortel mentioned this issue Apr 20, 2023

Implement return value for ESProducer acquire function #41328

Merged

makortel mentioned this issue May 2, 2023

GetterOfProducts for output modules, use in DQMRootOutputModule #41460

Merged

perrotta mentioned this issue May 5, 2023

[DB-RECONSTRUCTION] [LLVM16]Fix mismatched bound warnings #41504

Merged

makortel mentioned this issue Jun 6, 2023

Adding automated thread controls for fallback triton servers #41876

Merged

makortel mentioned this issue Jun 20, 2023

GCC 11.4; CUDA 11.8; cleanup for cmsplatf/cmsos use cms-sw/cmsdist#8545

Merged

makortel mentioned this issue Jun 26, 2023

Rename DataProxy to ESProductResolver #42075

Merged

This was referenced Jun 29, 2023

Make TritonCpuShmResource class compile with C++20 #42144

Merged

Revert "Update TBB to version 2021.9.0" cms-sw/cmsdist#8574

Merged

mmusich mentioned this issue Jul 31, 2023

[13_0_X] Update 2023 MC GTs with the fixed L1T tag and re-snapshot the data GTs #42372

Merged

cmsbuild added the trk label Jul 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Random comparison difference in HLT/SiStrip/ControlView #41200

Random comparison difference in HLT/SiStrip/ControlView #41200

makortel commented Mar 27, 2023

makortel commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

VinInn commented Mar 28, 2023

makortel commented Mar 28, 2023

VinInn commented Mar 28, 2023

VinInn commented Mar 28, 2023

VinInn commented Mar 28, 2023

civanch commented Mar 28, 2023

VinInn commented Mar 28, 2023

VinInn commented Mar 28, 2023

makortel commented Apr 4, 2023

makortel commented Apr 4, 2023

cmsbuild commented Apr 4, 2023

makortel commented Apr 4, 2023

makortel commented Apr 20, 2023

makortel commented May 2, 2023

makortel commented Jun 6, 2023

makortel commented Jun 20, 2023

makortel commented Jun 26, 2023

mmusich commented Jul 31, 2023

Random comparison difference in HLT/SiStrip/ControlView #41200

Random comparison difference in HLT/SiStrip/ControlView #41200

Comments

makortel commented Mar 27, 2023

makortel commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

cmsbuild commented Mar 27, 2023

VinInn commented Mar 28, 2023

makortel commented Mar 28, 2023

VinInn commented Mar 28, 2023

VinInn commented Mar 28, 2023

VinInn commented Mar 28, 2023

civanch commented Mar 28, 2023

VinInn commented Mar 28, 2023

VinInn commented Mar 28, 2023

makortel commented Apr 4, 2023

makortel commented Apr 4, 2023

cmsbuild commented Apr 4, 2023

makortel commented Apr 4, 2023

makortel commented Apr 20, 2023

makortel commented May 2, 2023

makortel commented Jun 6, 2023

makortel commented Jun 20, 2023

makortel commented Jun 26, 2023

mmusich commented Jul 31, 2023