ROOT 6 master: Histogram merge error

[Dr15Jones]

IN the ROOT6 IB, we are periodically seeing workflows (e.g. 137.8) failing in the DQM Harvest step from a new ROOT error message

----- Begin Fatal Exception 10-Dec-2018 08:32:43 CET-----------------------
	An exception of category 'FatalRootError' occurred while
	   [0] Calling InputSource::readRun_
	   Additional Info:
	      [a] Fatal Root Error: @SUB=TH1Merger::CheckForDuplicateLabels
	Histogram eventsPerPath_all has duplicate labels in the x axis. Bin contents will be merged in a single bin
	
----- End Fatal Exception -------------------------------------------------

[cmsbuild]

A new Issue was created by @Dr15Jones Chris Jones.

@davidlange6, @Dr15Jones, @smuzaffar, @fabiocos, @kpedro88 can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

assign dqm

[cmsbuild]

New categories assigned: dqm

@jfernan2,@andrius-k,@schneiml,@kmaeshima you have been requested to review this Pull request/Issue and eventually sign? Thanks

[andrius-k]

Hi @Dr15Jones, which IB is crashing and could you please provide a link to the Jenkins page or any reference that we could look into?

[Dr15Jones]

The failure has appeared in several CSSW_10_4_ROOT6_X builds, the latest of which was
CMSSW_10_4_X_2018-12-10-2300 which can be seen from the IB dashboard

https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/ib/CMSSW_10_4_X

The link to the most recent failed IB workflow is
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc700/CMSSW_10_4_ROOT6_X_2018-12-10-2300/pyRelValMatrixLogs/run/137.8_RunEGamma2018C+RunEGamma2018C+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Prompt_L1TEgDQM+RunEGamma2018D+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Prompt_L1TEgDQM+HARVEST2018_L1TEgDQM_MULTIRUN/step7_RunEGamma2018C+RunEGamma2018C+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Prompt_L1TEgDQM+RunEGamma2018D+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Prompt_L1TEgDQM+HARVEST2018_L1TEgDQM_MULTIRUN.log#/

[schneiml]

The 137.8 is the new multi-run harvesting Workflow. Good to see that it caught something. My crystal ball guess is something that was illegal/does not make sense even in the current production release, and was caught now, probably in some subsystem module that does not expect multi-run harvesting to happen.

However, I can't reproduce that currently; ROOT crashes on initialization on lxplus7 in the CMSSW_10_4_ROOT6_X_2018-12-10-2300 IB:

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc700/lcg/root/6.15.01/root-6.15.01/interpreter/llvm/src/tools/clang/include/clang/Serialization/Module.h:72: clang::serialization::InputFile::InputFile(const clang::FileEntry*, bool, bool): Assertion `!(isOverridden && isOutOfDate) && "an overridden cannot be out-of-date"' failed.

Any hints?

[Dr15Jones]

@pcanal any idea what this failure could be?

/build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc700/lcg/root/6.15.01/root-6.15.01/interpreter/llvm/src/tools/clang/include/clang/Serialization/Module.h:72: clang::serialization::InputFile::InputFile(const clang::FileEntry*, bool, bool): Assertion `!(isOverridden && isOutOfDate) && "an overridden cannot be out-of-date"' failed.

[Dr15Jones]

@smuzaffar what is the environment you use to run the ROOT6 IBs?

[pcanal]

@Dr15Jones sorta. This indicates that some of the headers files that are part of the ROOT pch files have been updated since ROOT was build. i.e. likely some system headers.

[smuzaffar]

@Dr15Jones , we use docker to build run ROOT6 IBs. All of 10.4.X Ibs now run under docker (as nearly all of them are slc7 based).

[Dr15Jones]

@pcanal @smuzaffar This seems extremely bad. This is implying that ROOT6 master can only run on a machine on which it was compiled. We need to determine what differences between the docker container and lxplus7 are causing the problem and make ROOT not care about them (since such diffferences are bound to happen on grid sites as well).

[davidlange6]

adding @yamaguchi1024 as we talked of similar issues a week ago in the context of root modules..

…

On Dec 14, 2018, at 8:29 AM, Chris Jones ***@***.***> wrote: @pcanal @smuzaffar This seems extremely bad. This is implying that ROOT6 master can only run on a machine on which it was compiled. We need to determine what differences between the docker container and lxplus7 are causing the problem and make ROOT not care about them (since such diffferences are bound to happen on grid sites as well). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[smuzaffar]

we have opened a JIRA ticket https://sft.its.cern.ch/jira/browse/ROOT-9843 about this.
On lxplus7 it is due to glibc version update. But our docker container we still have old glibc

[cmsbuild@050d8f6b1a80 build]$  rpm -qa | grep glibc
glibc-devel-2.17-222.el7.x86_64
glibc-common-2.17-222.el7.x86_64
glibc-2.17-222.el7.x86_64
glibc-headers-2.17-222.el7.x86_64

[vgvassilev]

Shahzad, could you bisect?

Is it possible that we build ROOT on a system with glibc version 2.17-260.el7 and the then deploy it on 2.17-222.el7 or vice versa?

If that’s the case I’d expect that the error is unclear but correct. ROOT would store a zip of the header files of glibc and then find out that some of them have changed.

[smuzaffar]

@vgvassilev , as we run under docker so this is not possible then we pick up different glic version. Anyway, yesterday we updated root master to bcd447b (commits from 18th DEC) also we use -DLLVM_BUILD_TYPE=Release but this workflow/test still fails with same error. Both root, and cmssw build was done (under docker) on the same machine where the test was run. So no chance that glic version could have changed.

[yamaguchi1024]

Hi all,

I asked Axel about this histogram merge issue with a link to this test failure, and got the following reply:

Yes, but that's fairly old on ROOT's side, this was changed months ago.
It's when having histograms with text labels, i.e.{"cat1": 12, "cat2": 13}
We can merge two of these histograms, by simply creating the super-set of the bin labels, and then adding the values for each label.
But in ROOT it's allowed to have {"cat1": 12, "cat1": 13}, i.e. repeated bin labels. And merging that will be - weird; we will be combining these labels, and likely that's not what the user expected, because they created two bins with the same label. So they need to think to do a conscious decision.
I.e. this is not a bug in ROOT; this is likely a design issue on their side, with whomever fills that histogram.

I think Lorenzo and Axel are the responsive people for Histogram, so you can discuss this issue with them. Let me know if I can help also.

[fioriNTU]

+1

Fixed by the PR above

[cmsbuild]

This issue is fully signed and ready to be closed.