Recurring error: module 'std.codecvt' is incompatible with feature 'header_existence' #39735

rappoccio · 2022-10-14T15:21:22Z

There is currently a recurring error (see here for instance) in some validation WFs that seems to be related to validate.C (see link here).

The text was updated successfully, but these errors were encountered:

cmsbuild · 2022-10-14T15:21:55Z

A new Issue was created by @rappoccio .

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

rappoccio · 2022-10-14T15:21:56Z

Affecting at least #39711

rappoccio · 2022-10-14T15:22:17Z

assign reconstruction

cmsbuild · 2022-10-14T15:22:33Z

New categories assigned: reconstruction

@mandrenguyen,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel · 2022-10-14T15:30:41Z

The error message is

WARNING: AutoLibraryloader::enable() and AutoLibraryLoader.h are deprecated.
Use FWLiteEnabler::enable() and FWLiteEnabler.h instead
Info in <TUnixSystem::ACLiC>: creating shared library /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/all_OldVSNew_TTbar14TeV2026D88wf20834p0/./validate_C.so
Start making plots for Events with 10 events and refEvents with 10 events ==> check  10
plotting variable: [email protected]()
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.count
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.module.size()
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.category.size()
plotting variable: HBHEDataFramesSorted_simHcalUnsuppressedDigis__RECO.obj.obj@.size()
/cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/etc//cling/std.modulemap:368:10: error: module 'std.codecvt' is incompatible with feature 'header_existence'
  module "codecvt" {
         ^
/cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/external/gcc/10.3.0-84898dea653199466402e67d73657f10/bin/../lib/gcc/x86_64-redhat-linux-gnu/10.3.0/../../../../include/c++/10.3.0/bits/fs_path.h:40:10: note: submodule of top-level module 'std' implicitly imported here
#include <codecvt>
         ^

 *** Break *** segmentation violation

#0  0x00002af647d053cb in waitpid () from /lib64/libc.so.6
#1  0x00002af647c66bff in do_system () from /lib64/libc.so.6
#2  0x00002af6474acd1b in TUnixSystem::StackTrace() () from /cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/lib/libCore.so
#3  0x00002af6474aa2b5 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/lib/libCore.so
#4  <signal handler called>
#5  0x00002af64b565b6c in clang::CXXRecordDecl::getLambdaCallOperator() const () from /cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/lib/libCling.so
#6  0x00002af6490e3c38 in clang::RecursiveASTVisitor<cling::(anonymous namespace)::StaticVarCollector>::TraverseLambdaExpr(clang::LambdaExpr*, llvm::SmallVectorImpl<llvm::PointerIntPair<clang::Stmt*, 1u, bool, llvm::PointerLikeTypeTraits<clang::Stmt*>, llvm::PointerIntPairInfo<clang::Stmt*, 1u, llvm::PointerLikeTypeTraits<clang::Stmt*> > > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/lib/libCling.so

Could the appearance be related to cms-sw/cms-bot#1856?

Is this pointing to some error in our ROOT setup? @vgvassilev @pcanal

vgvassilev · 2022-10-14T16:21:54Z

Which version of ROOT are you using. Do you have this commit: root-project/root@f8bcafc ?

vgvassilev · 2022-10-14T16:26:04Z

@makortel, @smuzaffar can you confirm that root-project/root#11571 fixes the issue for you?

makortel · 2022-10-14T16:36:53Z

The ROOT version is 6.24, more specifically https://github.com/cms-sw/root/commits/cms/v6-24-00-patches/b5aa8fd i.e. root-project/root@b5aa8fd + cms-sw/root@556667e. It does not contain root-project/root@f8bcafc.

smuzaffar · 2022-10-15T08:43:26Z

I have started the tests for root-project/root#11571 here cms-sw/root#176

smuzaffar · 2022-10-15T08:57:40Z

Could the appearance be related to cms-sw/cms-bot#1856?

@makortel , no this is not related to cms-sw/cms-bot#1856. log file has message like

WARNING: AutoLibraryloader::enable() and AutoLibraryLoader.h are deprecated.
Use FWLiteEnabler::enable() and FWLiteEnabler.h instead
Info in <TUnixSystem::ACLiC>: creating shared library /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/all_OldVSNew_TTbar14TeV2026D88wf20834p0/./validate_C.so

which means the tests were still using old cms-bot as it generated the validate_C.so. With latest cms-bot validateJR.sh change one should get something like this where we generate validate_C.so once

WARNING: AutoLibraryloader::enable() and AutoLibraryLoader.h are deprecated.
Use FWLiteEnabler::enable() and FWLiteEnabler.h instead
Start making plots for Events with 10 events and refEvents with 10 events ==> check  10

smuzaffar · 2022-10-17T09:03:59Z

@vgvassilev , root-project/root#11571 worked for CMS. Comparison results here do not show any crashes

smuzaffar · 2022-10-17T20:32:54Z

@vgvassilev any idea why we suddenly hit this issue? Note that we have not changed root in CMSSW 12.6.X since May 2022. IBs and PR tests always run under singularity container which was re-built and is in use since 4th OCT 2022. Both CMSSW_12_5_X and CMSSW_12_6_X IBs use the same ROOT and same singularity container. Tests in CMSSW_12_5_X do not fail e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_5_X_2022-10-17-1100+296a35/53469/validateJR.html is all clean. Is it some thing we have integrated in CMSSW_12_6_X which exposed this bug?

smuzaffar · 2022-10-17T20:34:25Z

@makortel , @cms-sw/orp-l2 how should we proceed? should we cherry-pick just this in 12.6.X ?

vgvassilev · 2022-10-17T20:35:52Z

@vgvassilev any idea why we suddenly hit this issue? Note that we have not changed root in CMSSW 12.6.X since May 2022. IBs and PR tests always run under singularity container which was re-built and is in use since 4th OCT 2022. Both CMSSW_12_5_X and CMSSW_12_6_X IBs use the same ROOT and same singularity container. Tests in CMSSW_12_5_X do not fail e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_5_X_2022-10-17-1100+296a35/53469/validateJR.html is all clean. Is it some thing we have integrated in CMSSW_12_6_X which exposed this bug?

Did you switch recently to newer version of gcc, libstdc++ or bumped the language version? If you have then likely that is the cause.

smuzaffar · 2022-10-17T20:39:19Z

not really @vgvassilev . As I wrote tests do not fail for CMSSW_12_5_X which uses exact same root and same singularity image.

smuzaffar · 2022-10-17T20:42:04Z

ah could it be that rebuilt of root in new singularity container causes this. We do have rebuilt root for 12.6.X IBs ( due to dependency update). Let me rebuilt root for 12.5.X too and see if that fails too

vgvassilev · 2022-10-17T20:42:18Z

WARNING: AutoLibraryloader::enable() and AutoLibraryLoader.h are deprecated.
Use FWLiteEnabler::enable() and FWLiteEnabler.h instead
Info in <TUnixSystem::ACLiC>: creating shared library /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/all_OldVSNew_TTbar14TeV2026D88wf20834p0/./validate_C.so
Start making plots for Events with 10 events and refEvents with 10 events ==> check  10
plotting variable: [email protected]()
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.count
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.module.size()
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.category.size()
plotting variable: HBHEDataFramesSorted_simHcalUnsuppressedDigis__RECO.obj.obj@.size()
/cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/etc//cling/std.modulemap:368:10: error: module 'std.codecvt' is incompatible with feature 'header_existence'
  module "codecvt" {
         ^
/cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/external/gcc/10.3.0-84898dea653199466402e67d73657f10/bin/../lib/gcc/x86_64-redhat-linux-gnu/10.3.0/../../../../include/c++/10.3.0/bits/fs_path.h:40:10: note: submodule of top-level module 'std' implicitly imported here
#include <codecvt>
         ^

 *** Break *** segmentation violation

Our test started using bits/fs_path.h which seems to include codecvt and then we got the failure. So yes, I think you are right to say we integrated something that triggered this issue...

smuzaffar · 2022-10-18T05:47:17Z

CMSSW_12_5_X PR tests with root rebuilt in new container are all good . So the failure we see in 12.6.X is due to some new code in cmssw 12.6.X itself. PR tests based on CMSSW_12_6_X_2022-10-13-1100 IB were good and they start failing for CMSSW_12_6_X_2022-10-13-2300 and above. Looking at the PRs merged for CMSSW_12_6_X_2022-10-13-2300, it looks like #38341 introduced some change which exposed this issue.

smuzaffar · 2022-10-27T07:11:05Z

@makortel , @perrotta @rappoccio any objections on including cms-sw/root#176 ( suggested by @vgvassilev here )? This should fix these DQM comparison failures

perrotta · 2022-10-27T08:00:13Z

@makortel , @perrotta @rappoccio any objections on including cms-sw/root#176 ( suggested by @vgvassilev here )? This should fix these DQM comparison failures

@smuzaffar we have already started building pre4.
Are you suggesting to stop the build, merge that one, and restart the build afterwards?
Of course, after pre4 that fix is welcome in master

smuzaffar · 2022-10-27T08:12:28Z

no, do not stop pre4, we can include it after pre4.

Fixes cms-sw/cmssw#39735

rappoccio mentioned this issue Oct 14, 2022

Fix SiPixelRawToClusterGPUKernel for spurious ROCs #39711

Merged

cmsbuild added the pending-assignment label Oct 14, 2022

cmsbuild added pending-signatures reconstruction-pending and removed pending-assignment labels Oct 14, 2022

vgvassilev mentioned this issue Oct 14, 2022

[backport] Revert "[cxxmodules] Allow submodules to contain headers which may be… root-project/root#11571

Merged

smuzaffar mentioned this issue Oct 17, 2022

[Do not Merge][12.5.X] testing root rebuild with PR comparison cms-sw/cmsdist#8132

Closed

smuzaffar mentioned this issue Oct 26, 2022

mkfit pixelLess-specific DNN training #39854

Merged

smuzaffar mentioned this issue Oct 27, 2022

Revert "[cxxmodules] Allow submodules to contain headers which may be missing cms-sw/root#176

Merged

smuzaffar added a commit to cms-sw/cmsdist that referenced this issue Oct 27, 2022

[ROOT] Allow submodules to contain headers missing headers

c979f9e

Fixes cms-sw/cmssw#39735

smuzaffar mentioned this issue Oct 27, 2022

[ROOT] Allow submodules to contain headers missing headers cms-sw/cmsdist#8152

Merged

smuzaffar closed this as completed in cms-sw/cmsdist#8152 Oct 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recurring error: module 'std.codecvt' is incompatible with feature 'header_existence' #39735

Recurring error: module 'std.codecvt' is incompatible with feature 'header_existence' #39735

rappoccio commented Oct 14, 2022

cmsbuild commented Oct 14, 2022

rappoccio commented Oct 14, 2022

rappoccio commented Oct 14, 2022

cmsbuild commented Oct 14, 2022

makortel commented Oct 14, 2022

vgvassilev commented Oct 14, 2022

vgvassilev commented Oct 14, 2022

makortel commented Oct 14, 2022

smuzaffar commented Oct 15, 2022

smuzaffar commented Oct 15, 2022 •

edited

Loading

smuzaffar commented Oct 17, 2022

smuzaffar commented Oct 17, 2022

smuzaffar commented Oct 17, 2022

vgvassilev commented Oct 17, 2022

smuzaffar commented Oct 17, 2022

smuzaffar commented Oct 17, 2022

vgvassilev commented Oct 17, 2022

smuzaffar commented Oct 18, 2022

smuzaffar commented Oct 27, 2022

perrotta commented Oct 27, 2022

smuzaffar commented Oct 27, 2022

Recurring error: module 'std.codecvt' is incompatible with feature 'header_existence' #39735

Recurring error: module 'std.codecvt' is incompatible with feature 'header_existence' #39735

Comments

rappoccio commented Oct 14, 2022

cmsbuild commented Oct 14, 2022

rappoccio commented Oct 14, 2022

rappoccio commented Oct 14, 2022

cmsbuild commented Oct 14, 2022

makortel commented Oct 14, 2022

vgvassilev commented Oct 14, 2022

vgvassilev commented Oct 14, 2022

makortel commented Oct 14, 2022

smuzaffar commented Oct 15, 2022

smuzaffar commented Oct 15, 2022 • edited Loading

smuzaffar commented Oct 17, 2022

smuzaffar commented Oct 17, 2022

smuzaffar commented Oct 17, 2022

vgvassilev commented Oct 17, 2022

smuzaffar commented Oct 17, 2022

smuzaffar commented Oct 17, 2022

vgvassilev commented Oct 17, 2022

smuzaffar commented Oct 18, 2022

smuzaffar commented Oct 27, 2022

perrotta commented Oct 27, 2022

smuzaffar commented Oct 27, 2022

smuzaffar commented Oct 15, 2022 •

edited

Loading