drop high-level python2 tools #6848

davidlange6 · 2021-04-27T17:10:26Z

try again for #6846 (but with more packages removed)

davidlange6 · 2021-04-27T17:10:51Z

please test

cmsbuild · 2021-04-27T17:10:52Z

A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_12_0_X/master.

@cmsbuild, @smuzaffar, @mrodozov can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

cmsbuild · 2021-04-27T17:16:52Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14634/summary.html
COMMIT: f177579
CMSSW: CMSSW_12_0_X_2021-04-27-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6848/14634/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

cmsbuild · 2021-04-27T17:21:34Z

Pull request #6848 was updated.

davidlange6 · 2021-04-27T17:21:39Z

please test

cmsbuild · 2021-04-28T00:26:44Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14638/summary.html
COMMIT: 15680af
CMSSW: CMSSW_12_0_X_2021-04-27-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6848/14638/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test import-cjson had ERRORS
---> test testBottleneck had ERRORS
---> test testPandas had ERRORS
---> test testRootNumpy had ERRORS
and more ...

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 3034 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2877605
DQMHistoTests: Total failures: 7704
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2869879
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: found differences in 1 / 37 workflows

davidlange6 · 2021-04-28T07:46:56Z

please test with cms-sw/cmssw#33560

davidlange6 · 2021-05-03T17:52:19Z

Ok, thanks! (We have these sorts of errors in the modules IB - thats interesting…)

…

On May 3, 2021, at 7:42 PM, Malik Shahzad Muzaffar ***@***.***> wrote: please test with cms-sw/cmssw#33597 yes @davidlange6 , I think these should be fixed for today's 11h00 IB. I am starting the tests again — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

smuzaffar · 2021-05-04T05:22:00Z

please test with cms-sw/cmssw#33597

smuzaffar · 2021-05-04T14:30:49Z

Ok, thanks! (We have these sorts of errors in the modules IB - thats interesting…)

@davidlange6 , Modules IB issues are real one and I think they might be related to the cms-sw/cmssw#33186 .

I have built module IB using SCRAM V2 (perl base) and workflow 135.1 fails with the same error

2021-05-04 16:18:09.491313: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2394450000 Hz
04-May-2021 16:18:16 CEST  Closed file file:step1.root
----- Begin Fatal Exception 04-May-2021 16:18:16 CEST-----------------------
An exception of category 'DictionaryNotFound' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=PATTriggerProducer label='patTrigger'
Exception Message:
No Dictionary for class: 'edm::AssociationMap<edm::OneToManyWithQualityGeneric<edm::View<reco::Track>,vector<TrackingParticle>,double,unsigned int,edm::RefToBaseProd<reco::Track>,edm::RefProd<vector<TrackingParticle> >,edm::RefToBase<reco::Track>,edm::Ref<vector<TrackingParticle>,TrackingParticle,edm::refhelper::FindUsingAdvance<vector<TrackingParticle>,TrackingParticle> > > >'
----- End Fatal Exception -------------------------------------------------

davidlange6 · 2021-05-04T14:36:18Z

Yes, I need to look into this error.. Instead I mean some older ones affecting a few workflows (unit tests mostly) that had the same sort of missing dictionary errors as in these tests (eg, the same missing dictionaries..)

…

On May 4, 2021, at 4:31 PM, Malik Shahzad Muzaffar ***@***.***> wrote: Ok, thanks! (We have these sorts of errors in the modules IB - thats interesting…) @davidlange6 , Modules IB issues are real one and I think they might be related to the cms-sw/cmssw#33186 . I have built module IB using SCRAM V2 (perl base) and workflow 135.1 fails with the same error 2021-05-04 16:18:09.491313: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2394450000 Hz 04-May-2021 16:18:16 CEST Closed file file:step1.root ----- Begin Fatal Exception 04-May-2021 16:18:16 CEST----------------------- An exception of category 'DictionaryNotFound' occurred while [0] Constructing the EventProcessor [1] Constructing module: class=PATTriggerProducer label='patTrigger' Exception Message: No Dictionary for class: 'edm::AssociationMap<edm::OneToManyWithQualityGeneric<edm::View<reco::Track>,vector<TrackingParticle>,double,unsigned int,edm::RefToBaseProd<reco::Track>,edm::RefProd<vector<TrackingParticle> >,edm::RefToBase<reco::Track>,edm::Ref<vector<TrackingParticle>,TrackingParticle,edm::refhelper::FindUsingAdvance<vector<TrackingParticle>,TrackingParticle> > > >' ----- End Fatal Exception ------------------------------------------------- — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

cmsbuild · 2021-05-04T17:45:02Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14820/summary.html
COMMIT: a9b6f27
CMSSW: CMSSW_12_0_X_2021-05-03-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/6848/14820/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2662646
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2662617
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 36 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 155 log files, 37 edm output root files, 37 DQM output files
TriggerResults: no differences found

smuzaffar · 2021-05-04T18:24:15Z

please test for CMSSW_12_0_PY3_X

smuzaffar · 2021-05-04T18:24:42Z

please test for slc7_aarch64_gcc9

smuzaffar · 2021-05-04T18:25:02Z

please test for slc7_ppc64le_gcc9

cmsbuild · 2021-05-04T22:16:27Z

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14857/summary.html
COMMIT: a9b6f27
CMSSW: CMSSW_12_0_X_2021-05-03-2300/slc7_ppc64le_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/6848/14857/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

Igprof patch for TBB resumable tasks #6867 @gartung: Igprof patch for TBB resumable tasks
Update TBB to 2021.2.0 and use cmake to build #6792 @mrodozov: Update TBB to 2021.2.0 and use cmake to build
Update tag for GeneratorInterface-EvtGenInterface to V02-04-00 #6871 @cms-sw: Update tag for GeneratorInterface-EvtGenInterface to V02-04-00
Update tag for RecoEgamma-ElectronIdentification to V01-05-00 #6870 @cms-sw: Update tag for RecoEgamma-ElectronIdentification to V01-05-00

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14857/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14857/git-merge-result

Build

I found compilation error when building:

>> Compiling  /scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-03-2300/src/SimG4Core/Physics/src/PhysicsList.cc
>> Compiling  /scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-03-2300/src/SimG4Core/Physics/src/PhysicsListFactory.cc
In file included from /scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-03-2300/src/FWCore/PluginManager/interface/PluginFactory.h:27,
                 from /scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-03-2300/src/SimG4Core/Physics/interface/PhysicsListFactory.h:7,
                 from /scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-03-2300/src/SimG4Core/Physics/src/PhysicsListFactory.cc:1:
/scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-03-2300/src/FWCore/PluginManager/interface/PluginFactoryBase.h:64:58: error: 'zero_allocator' is not a member of 'tbb'
   64 |     typedef tbb::concurrent_vector> PMakers;
      |                                                          ^~~~~~~~~~~~~~
/scratch/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-03-2300/src/FWCore/PluginManager/interface/PluginFactoryBase.h:64:73: error: template argument 2 is invalid
   64 |     typedef tbb::concurrent_vector> PMakers;
      |                                                                         ^~~~~~~~~~~~~~~

cmsbuild · 2021-05-04T22:28:31Z

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14859/summary.html
COMMIT: a9b6f27
CMSSW: CMSSW_12_0_X_2021-05-04-1100/slc7_aarch64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/6848/14859/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

Igprof patch for TBB resumable tasks #6867 @gartung: Igprof patch for TBB resumable tasks
Update TBB to 2021.2.0 and use cmake to build #6792 @mrodozov: Update TBB to 2021.2.0 and use cmake to build
Update tag for RecoEgamma-ElectronIdentification to V01-05-00 #6870 @cms-sw: Update tag for RecoEgamma-ElectronIdentification to V01-05-00

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14859/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14859/git-merge-result

Build

I found compilation error when building:

>> Compiling  /home/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-04-1100/src/SimG4Core/Physics/src/PhysicsList.cc
>> Compiling  /home/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-04-1100/src/SimG4Core/Physics/src/PhysicsListFactory.cc
In file included from /home/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-04-1100/src/FWCore/PluginManager/interface/PluginFactory.h:27,
                 from /home/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-04-1100/src/SimG4Core/Physics/interface/PhysicsListFactory.h:7,
                 from /home/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-04-1100/src/SimG4Core/Physics/src/PhysicsListFactory.cc:1:
/home/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-04-1100/src/FWCore/PluginManager/interface/PluginFactoryBase.h:64:58: error: 'zero_allocator' is not a member of 'tbb'
   64 |     typedef tbb::concurrent_vector> PMakers;
      |                                                          ^~~~~~~~~~~~~~
/home/cmsbuild/jenkins_a/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-05-04-1100/src/FWCore/PluginManager/interface/PluginFactoryBase.h:64:73: error: template argument 2 is invalid
   64 |     typedef tbb::concurrent_vector> PMakers;
      |                                                                         ^~~~~~~~~~~~~~~

cmsbuild · 2021-05-04T23:04:12Z

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14856/summary.html
COMMIT: a9b6f27
CMSSW: CMSSW_12_0_PY3_X_2021-05-03-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/6848/14856/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

Update TBB to 2021.2.0 and use cmake to build #6792 @mrodozov: Update TBB to 2021.2.0 and use cmake to build
Igprof patch for TBB resumable tasks #6867 @gartung: Igprof patch for TBB resumable tasks
Update tag for RecoEgamma-ElectronIdentification to V01-05-00 #6870 @cms-sw: Update tag for RecoEgamma-ElectronIdentification to V01-05-00
Update tag for GeneratorInterface-EvtGenInterface to V02-04-00 #6871 @cms-sw: Update tag for GeneratorInterface-EvtGenInterface to V02-04-00

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14856/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-74c3b8/14856/git-merge-result

Build

I found compilation error when building:

>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_PY3_X_2021-05-03-2300/src/SimG4Core/Physics/src/PhysicsList.cc
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_PY3_X_2021-05-03-2300/src/SimG4Core/Physics/src/PhysicsListFactory.cc
In file included from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_PY3_X_2021-05-03-2300/src/FWCore/PluginManager/interface/PluginFactory.h:27,
                 from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_PY3_X_2021-05-03-2300/src/SimG4Core/Physics/interface/PhysicsListFactory.h:7,
                 from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_PY3_X_2021-05-03-2300/src/SimG4Core/Physics/src/PhysicsListFactory.cc:1:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_PY3_X_2021-05-03-2300/src/FWCore/PluginManager/interface/PluginFactoryBase.h:64:58: error: 'zero_allocator' is not a member of 'tbb'
   64 |     typedef tbb::concurrent_vector> PMakers;
      |                                                          ^~~~~~~~~~~~~~
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_PY3_X_2021-05-03-2300/src/FWCore/PluginManager/interface/PluginFactoryBase.h:64:73: error: template argument 2 is invalid
   64 |     typedef tbb::concurrent_vector> PMakers;
      |                                                                         ^~~~~~~~~~~~~~~

davidlange6 · 2021-05-05T07:13:24Z

hi @smuzaffar - should we put this in DEVEL for 1100?
(I don't see why any of these changes would give arch specific problems. Its removing things:) )

smuzaffar · 2021-05-05T07:23:36Z

hi @smuzaffar - should we put this in DEVEL for 1100?
(I don't see why any of these changes would give arch specific problems. Its removing things:) )

@davidlange6 , Just wanted to make sure to catch any %ifarch aarch64|ppc64le type statements in spec files.
Sure, we can put it in DEVEL IBs

smuzaffar · 2021-05-05T07:27:56Z

+externals
merging it for DVEL IBs.

cmsbuild · 2021-05-05T07:28:19Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_0_X/devel IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

smuzaffar · 2021-05-06T06:05:17Z

@davidlange6 , DEVEL IBs looks good. Lokk like it is safe to include it in master IBs

cmsbuild · 2021-05-06T06:07:22Z

backport done
Successfully backported PR #6848 as #6878 for branch IB/CMSSW_12_0_X/master

davidlange6 added 6 commits April 27, 2021 17:10

drop leaf packages from python2

1220930

a few more packages

a37705c

a few more packages removed

66bc785

removing more python2

5acc87e

matplotlib

08ac996

more migrations away from python2

f177579

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Apr 27, 2021

cmsbuild added tests-started and removed tests-pending labels Apr 27, 2021

cmsbuild added tests-rejected and removed tests-started labels Apr 27, 2021

cleanup dependencies I missed

15680af

cmsbuild added tests-pending and removed tests-rejected labels Apr 27, 2021

cmsbuild added tests-started and removed tests-pending labels Apr 27, 2021

cmsbuild added tests-rejected and removed tests-started labels Apr 28, 2021

cmsbuild added requires-external and removed tests-rejected labels Apr 28, 2021

cmsbuild added tests-approved and removed tests-started labels May 4, 2021

cmsbuild removed the requires-external label May 4, 2021

smuzaffar changed the base branch from IB/CMSSW_12_0_X/master to IB/CMSSW_12_0_X/devel May 5, 2021 07:23

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels May 5, 2021

smuzaffar merged commit cb4ec6e into cms-sw:IB/CMSSW_12_0_X/devel May 5, 2021

cmsbuild mentioned this pull request May 6, 2021

drop high-level python2 tools #6878

Merged

cmsbuild mentioned this pull request Jun 13, 2021

cleaning up unused py2 tools ; restoring tensorflow as part of python_tools.spec #7019

Merged

cmsbuild mentioned this pull request Jul 10, 2021

Update to python 3.9.6 #7111

Merged

cmsbuild mentioned this pull request Nov 21, 2022

SCRAM V2 disto via latest cmssw releases #8187

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

drop high-level python2 tools #6848

drop high-level python2 tools #6848

davidlange6 commented Apr 27, 2021

davidlange6 commented Apr 27, 2021

cmsbuild commented Apr 27, 2021

cmsbuild commented Apr 27, 2021

cmsbuild commented Apr 27, 2021

davidlange6 commented Apr 27, 2021

cmsbuild commented Apr 28, 2021

davidlange6 commented Apr 28, 2021

davidlange6 commented May 3, 2021 via email

smuzaffar commented May 4, 2021

smuzaffar commented May 4, 2021

davidlange6 commented May 4, 2021 via email

cmsbuild commented May 4, 2021

smuzaffar commented May 4, 2021

smuzaffar commented May 4, 2021

smuzaffar commented May 4, 2021

cmsbuild commented May 4, 2021

cmsbuild commented May 4, 2021

cmsbuild commented May 4, 2021

davidlange6 commented May 5, 2021

smuzaffar commented May 5, 2021

smuzaffar commented May 5, 2021

cmsbuild commented May 5, 2021

smuzaffar commented May 6, 2021

cmsbuild commented May 6, 2021

drop high-level python2 tools #6848

drop high-level python2 tools #6848

Conversation

davidlange6 commented Apr 27, 2021

davidlange6 commented Apr 27, 2021

cmsbuild commented Apr 27, 2021

cmsbuild commented Apr 27, 2021

External Build

cmsbuild commented Apr 27, 2021

davidlange6 commented Apr 27, 2021

cmsbuild commented Apr 28, 2021

Unit Tests

Comparison Summary

davidlange6 commented Apr 28, 2021

davidlange6 commented May 3, 2021 via email

smuzaffar commented May 4, 2021

smuzaffar commented May 4, 2021

davidlange6 commented May 4, 2021 via email

cmsbuild commented May 4, 2021

Comparison Summary

smuzaffar commented May 4, 2021

smuzaffar commented May 4, 2021

smuzaffar commented May 4, 2021

cmsbuild commented May 4, 2021

Build

cmsbuild commented May 4, 2021

Build

cmsbuild commented May 4, 2021

Build

davidlange6 commented May 5, 2021

smuzaffar commented May 5, 2021

smuzaffar commented May 5, 2021

cmsbuild commented May 5, 2021

smuzaffar commented May 6, 2021

cmsbuild commented May 6, 2021