Enable experimental PYROOT in master

[davidlange6]

path towards root 6.22 and dual py2 and py3 support

[davidlange6]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4163/console Started: 2020/01/08 14:09

[cmsbuild]

A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_11_1_X/rootmaster.

@cmsbuild, @smuzaffar, @mrodozov, @tulamor can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

[cmsbuild]

+1
Tested at: 2c1d331
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b99ed5/4163/summary.html
CMSSW: CMSSW_11_1_ROOT6_X_2020-01-07-2300
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b99ed5/4163/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2815513
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2815166
• DQMHistoTests: Total skipped: 346
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 147 log files, 16 edm output root files, 34 DQM output files

[smuzaffar]

+externals

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_1_X/rootmaster IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos, @silviodonato (and backports should be raised in the release meeting by the corresponding L2)

[makortel]

It seems that this change appears to have caused the edmCheckClassVersion to load unneeded shared libraries in CMSSW_11_1_ROOT6_X_2020-01-08-2300
https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc820/CMSSW_11_1_ROOT6_X_2020-01-08-2300/DataFormats/Phase2TrackerDigi

>> Checking EDM Class Version for src/DataFormats/Phase2TrackerDigi/src/classes_def.xml in libDataFormatsPhase2TrackerDigi.so
/data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/90326439580c9294f4e33092b6059872/opt/cmssw/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ROOT6_X_2020-01-08-2300/src/FWCore/Utilities/scripts/edmCheckClassVersion -l tmp/slc7_amd64_gcc820/src/DataFormats/Phase2TrackerDigi/src/DataFormatsPhase2TrackerDigi/libDataFormatsPhase2TrackerDigi.so -x src/DataFormats/Phase2TrackerDigi/src/classes_def.xml
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libSimDataFormatsGeneratorProducts.so for HepMC::GenVertex
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libSimDataFormatsGeneratorProducts.so for HepMC::GenVertex

We checked the package (DataFormats/Phase2TrackerDigi) with @Dr15Jones, and it has no explicit nor implicit dependencies beyond std and DataFormats/Common.

[smuzaffar]

@makortel , error was not due to this change. It was already there in ROOT6 IBs before this change.
I think the reason for this error message is due to the fact that scram copies PKG.rootmap file (in to lib directory) before copying the libPKG.so and root master loads all rootmaps and complains that libPKG.so not found. I am updating build rules to copy both PKG.rootmap and libPKG.so together.

[a]

[a] https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc820/CMSSW_11_1_ROOT6_X_2020-01-06-2300/DataFormats/Phase2TrackerDigi

Error in <TInterpreter::TCling::AutoLoad>: failure loading library libSimDataFormatsGeneratorProducts.so for HepMC::GenVertex
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libSimDataFormatsGeneratorProducts.so for HepMC::GenVertex

[makortel]

Ok, thanks for the clarification!

[davidlange6]

In any case it looks like my PR was a no-op - at some point the cmake variable changed from experimental_pyroot to pyroot_experimental and I found old documentation. I’m looking locally as also I believe the location of the pyroot lib and python files should have changed On Jan 9, 2020, at 4:03 PM, Malik Shahzad Muzaffar <[email protected]<mailto:[email protected]>> wrote: @makortel<https://github.com/makortel> , error was not due to this change. It was already there in ROOT6 IBs before this change. I think the reason for this error message is due to the fact that scram copies PKG.rootmap file (in to lib directory) before copying the libPKG.so and root master loads all rootmaps and complains that libPKG.so not found. I am updating build rules to copy both PKG.rootmap and libPKG.so together. [a] [a] https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc820/CMSSW_11_1_ROOT6_X_2020-01-06-2300/DataFormats/Phase2TrackerDigi Error in <TInterpreter::TCling::AutoLoad>: failure loading library libSimDataFormatsGeneratorProducts.so for HepMC::GenVertex Error in <TInterpreter::TCling::AutoLoad>: failure loading library libSimDataFormatsGeneratorProducts.so for HepMC::GenVertex — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#5455?email_source=notifications&email_token=ABGPFQ5APQGKDL4ALNWFPTTQ444CZA5CNFSM4KEICE4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIQTDYY#issuecomment-572600803>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQ2JCTNLUSRG4EHMQ7TQ444CZANCNFSM4KEICE4A>.

[makortel]

On the other hand, it is a bit weird that edmCheckClassVersion loads libSimDataFormatsGeneratorProducts.so when operating on something independent.

I played a bit between CMSSW_11_1_X_2020-01-06-2300 and CMSSW_11_1_ROOT6_X_2020-01-06-2300, and running the edmCheckClassVersion on libDataFormatsPhase2TrackerDigi.so with strace indeed shows that in ROOT6 IB the libSimDataFormatsGeneratorProducts.so is loaded, while in master IB it is not. I did not see any other "strange" CMSSW shared objects to be loaded, only libDataFormatsStdDictionaries.so, libDataFormatsProvenance.so, ibFWCoreMessageLogger.so, libFWCoreReflection.so, libFWCoreUtilities.so.

It feels like there is something special in libSimDataFormatsGeneratorProducts.so that causes it to be loaded.

[makortel]

The special one seems to be HepMC. If I remove all cases with HepMC from the selected classes of (a local copy of) SimDataFormatsGeneratorProducts_xr.rootmap, the libSimDataFormatsGeneratorProducts.so is no longer loaded, but instead I get printouts

Warning in <TClass::Init>: no dictionary for class HepMC::GenVertex is available
Warning in <TClass::Init>: no dictionary for class HepMC::GenParticle is available

This happens also already when running edmCheckClassVersion for libDataFormatsCommon.so. Looks like something is asking dictionary for HepMC::GenParticle and HepMC::GenVertex (if I add either of those back to the rootmap file, the libSimDataFormatsGeneratorProducts.so gets loaded again)

[makortel]

Inserting a breakpoint for write() in gdb gave the following stack trace

#7  0x00007ff3c331a178 in Warning (location=0x7ff3c351cdbe "TClass::Init", fmt=0x7ff3c351cd00 "no dictionary for class %s is available")
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/base/src/TError.cxx:336
#8  0x00007ff3c33f6947 in TClass::Init (this=0x37cd050, name=0x325eb30 "HepMC::GenVertex", cversion=1, typeinfo=0x0, isa=0x0, dfil=0x0, ifil=0x0, dl=-1, il=-1, givenInfo=0x0, 
    silent=false) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/meta/src/TClass.cxx:1481
#9  0x00007ff3c33f475c in TClass::TClass (this=0x37cd050, name=0x325eb30 "HepMC::GenVertex", cversion=1, silent=false)
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/meta/src/TClass.cxx:1124
#10 0x00007ff3bc8b1870 in TCling::GenerateTClass (this=0x1926af0, classname=0x325eb30 "HepMC::GenVertex", emulation=false, silent=false)
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/metacling/src/TCling.cxx:4220
#11 0x00007ff3c33fc344 in TClass::GetClass (name=0x358d3a0 "HepMC::GenVertex", load=true, silent=false)
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/meta/src/TClass.cxx:3112
#12 0x00007ff3c33f821b in TClass::AddRule (rule=0x37d95f0 "HepMC::GenVertex   m_event", ' ' <repeats 13 times>, "attributes=NotOwner")
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/meta/src/TClass.cxx:1830
#13 0x00007ff3c33f7f06 in (anonymous namespace)::ReadRulesContent (f=0x276ee90)
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/meta/src/TClass.cxx:1701
#14 0x00007ff3c33f8067 in TClass::ReadRules ()
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/meta/src/TClass.cxx:1760
#15 0x00007ff3c32dfcb3 in TROOT::InitInterpreter (this=0x7ff3c3630a60 <ROOT::Internal::GetROOT1()::alloc>)
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/base/src/TROOT.cxx:2147
#16 0x00007ff3c32d934b in ROOT::Internal::GetROOT2 ()
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/base/src/TROOT.cxx:388
#17 0x00007ff3c32d984c in ROOT::GetROOT () at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/base/src/TROOT.cxx:521
#18 0x00007ff3c32e1a83 in TROOT::Initialize ()
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/lcg/root/6.19.01-himbmo/root-6.19.01/core/base/src/TROOT.cxx:2857

The TClass::ReadRules() appears to be reading classes.rules file, that does contain HepMC stuff like

HepMC::GenVertex   m_event             attributes=NotOwner

The file itself (coming from ROOT) is equal between master and ROOT6 IBs, so maybe something has changed related to the "rule reading"?

[makortel]

I opened a ticket in ROOT JIRA (https://sft.its.cern.ch/jira/browse/ROOT-10514) about the HepMC dictionary loading.