Python 3.9.14 plus bulk update of many python package #8077

smuzaffar · 2022-09-10T07:41:15Z

update python3 to version 3.9.14 ( which is a security release)
- Can not update python to 3.10 series which requires OpenSSL 1.1.1. As we still support slc7 releases which contain openssl 1.0.2. Although we can maintain two branches ( one for slc7 and one for el8) but lets stick to python 3.9
update pip3 to latest version 22.2.2
update setuptools to 63.4.3 ( can not update to latest 65.x or 64.x series as numpy does not work with that)
Update many pip based packages
- Some of the packages are not yet using their latest versions due to build issues ( new meson backend build system not working correctly)
- Keep numpy <1.23.0 as keras, TensorFlow and xgboost does not work with numpy >= 1.23.0
- Keras version also need to be at 2.6.0 to match the TensorFlow version series
- Patched dxr to use Markup from markupsafe as Jinja2 3.1.2 version does not provide Markup

smuzaffar · 2022-09-10T07:41:31Z

test parameters:

full_cmssw = true

cmsbuild · 2022-09-10T07:41:36Z

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_12_6_X/master.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.
cms-bot commands are listed here

smuzaffar · 2022-09-10T07:41:41Z

please test

cmsbuild · 2022-09-10T22:21:09Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27456/summary.html
COMMIT: a128e50
CMSSW: CMSSW_12_6_X_2022-09-09-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8077/27456/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27456/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27456/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test dxr-build had ERRORS
---> test import-ipywidgets had ERRORS
---> test import-keras had ERRORS
---> test run-ipython had ERRORS
and more ...

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 12 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3618326
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3618293
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

cmsbuild · 2022-09-12T06:35:35Z

Pull request #8077 was updated.

smuzaffar · 2022-09-12T06:37:03Z

test parameters:

addpkg = PhysicsTools/PythonAnalysis, PhysicsTools/TensorFlow

smuzaffar · 2022-09-12T06:37:14Z

please test

smuzaffar · 2022-09-12T07:08:48Z

please test

cmsbuild · 2022-09-12T07:09:03Z

Pull request #8077 was updated.

cmsbuild · 2022-09-15T10:05:14Z

Pull request #8077 was updated.

smuzaffar · 2022-09-15T10:05:28Z

please test for el8_ppc64le_gcc10

smuzaffar · 2022-09-15T10:05:45Z

please test for el8_aarch64_gcc10

smuzaffar · 2022-09-15T10:06:12Z

please test

cmsbuild · 2022-09-15T14:01:05Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27561/summary.html
COMMIT: e93a4a7
CMSSW: CMSSW_12_6_X_2022-09-14-2300/el8_aarch64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8077/27561/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestFWCoreServicesDriver had ERRORS
---> test testFWCoreUtilities had ERRORS
---> test DRNTest had ERRORS

cmsbuild · 2022-09-15T14:53:02Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27560/summary.html
COMMIT: e93a4a7
CMSSW: CMSSW_12_6_X_2022-09-14-2300/el8_ppc64le_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8077/27560/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27560/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27560/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test testMPI had ERRORS
---> test DRNTest had ERRORS

cmsbuild · 2022-09-15T19:50:40Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27559/summary.html
COMMIT: e93a4a7
CMSSW: CMSSW_12_6_X_2022-09-14-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8077/27559/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27559/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27559/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3618326
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3618296
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-09-18T09:55:06Z

please test

cmsbuild · 2022-09-19T05:14:57Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00d21c/27632/summary.html
COMMIT: e93a4a7
CMSSW: CMSSW_12_6_X_2022-09-18-0000/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8077/27632/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3621956
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3621932
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-09-19T06:18:26Z

+externals

cmsbuild · 2022-09-19T06:18:46Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_6_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

smuzaffar added 4 commits September 10, 2022 09:24

update python to security release 3.9.14

42d4d29

update settup tools to 65.3.0

69b8e09

update python packages

734fc3e

added new pip package files

a128e50

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Sep 10, 2022

cmsbuild added tests-started and removed tests-pending labels Sep 10, 2022

smuzaffar changed the title ~~Bulk update of many python package~~ Python 3.9.14 plus bulk update of many python package Sep 10, 2022

cmsbuild added tests-rejected and removed tests-started labels Sep 10, 2022

smuzaffar added 2 commits September 12, 2022 08:34

update xgboost 1.6.2

e2a1951

revert numpy 1.22.4; update stack-data deps

d396cec

cmsbuild added tests-pending and removed tests-rejected labels Sep 12, 2022

cmsbuild added tests-started and removed tests-pending labels Sep 12, 2022

patch dxr to use Markup from markupsafe

d51eb86

revert keras to 2.6.0 to matchit withTF version

0a5c610

update xgboost-arm-and-ppc patch for xgboost 1.6.2

e93a4a7

cmsbuild added tests-pending and removed tests-approved labels Sep 15, 2022

cmsbuild added tests-started and removed tests-pending labels Sep 15, 2022

cmsbuild added tests-approved and removed tests-started labels Sep 15, 2022

cmsbuild added tests-started and removed tests-approved labels Sep 18, 2022

cmsbuild added tests-approved and removed tests-started labels Sep 19, 2022

smuzaffar merged commit ae67940 into IB/CMSSW_12_6_X/master Sep 19, 2022

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Sep 19, 2022

cmsbuild mentioned this pull request Sep 19, 2022

Introduction of New-All-In-One tool for Offline Validation of TrkAlignment cms-sw/cmssw#38304

Merged

iarspider added a commit to iarspider/cms-spack-repo that referenced this pull request Sep 20, 2022

Merge changes from cms-sw/cmsdist#8077

38f5082

smuzaffar deleted the python-pack-update branch September 22, 2022 06:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.9.14 plus bulk update of many python package #8077

Python 3.9.14 plus bulk update of many python package #8077

smuzaffar commented Sep 10, 2022 •

edited

Loading

smuzaffar commented Sep 10, 2022

cmsbuild commented Sep 10, 2022

smuzaffar commented Sep 10, 2022

cmsbuild commented Sep 10, 2022

cmsbuild commented Sep 12, 2022

smuzaffar commented Sep 12, 2022

smuzaffar commented Sep 12, 2022

smuzaffar commented Sep 12, 2022

cmsbuild commented Sep 12, 2022

cmsbuild commented Sep 15, 2022

smuzaffar commented Sep 15, 2022

smuzaffar commented Sep 15, 2022

smuzaffar commented Sep 15, 2022

cmsbuild commented Sep 15, 2022

cmsbuild commented Sep 15, 2022

cmsbuild commented Sep 15, 2022

smuzaffar commented Sep 18, 2022

cmsbuild commented Sep 19, 2022

smuzaffar commented Sep 19, 2022

cmsbuild commented Sep 19, 2022

Python 3.9.14 plus bulk update of many python package #8077

Python 3.9.14 plus bulk update of many python package #8077

Conversation

smuzaffar commented Sep 10, 2022 • edited Loading

smuzaffar commented Sep 10, 2022

cmsbuild commented Sep 10, 2022

smuzaffar commented Sep 10, 2022

cmsbuild commented Sep 10, 2022

Unit Tests

Comparison Summary

cmsbuild commented Sep 12, 2022

smuzaffar commented Sep 12, 2022

smuzaffar commented Sep 12, 2022

smuzaffar commented Sep 12, 2022

cmsbuild commented Sep 12, 2022

cmsbuild commented Sep 15, 2022

smuzaffar commented Sep 15, 2022

smuzaffar commented Sep 15, 2022

smuzaffar commented Sep 15, 2022

cmsbuild commented Sep 15, 2022

Unit Tests

cmsbuild commented Sep 15, 2022

Unit Tests

cmsbuild commented Sep 15, 2022

Comparison Summary

smuzaffar commented Sep 18, 2022

cmsbuild commented Sep 19, 2022

Comparison Summary

smuzaffar commented Sep 19, 2022

cmsbuild commented Sep 19, 2022

smuzaffar commented Sep 10, 2022 •

edited

Loading