uploadConditions.py is broken in recent IBs #34349

mmusich · 2021-07-06T12:18:50Z

$ uploadConditions.py SiPixelGainCalibration_2021_v0_full.db
Traceback (most recent call last):
  File "/cvmfs/cms-ib.cern.ch/nweek-02688/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-06-0300/bin/slc7_amd64_gcc900/uploadConditions.py", line 25, in <module>
    import cx_Oracle
ImportError: No module named cx_Oracle

related to py3 migration?

The text was updated successfully, but these errors were encountered:

cmsbuild · 2021-07-06T12:19:09Z

A new Issue was created by @mmusich Marco Musich.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

davidlange6 · 2021-07-06T12:24:22Z

looks like it. I gather there is no unit test coverage for this script..

mmusich · 2021-07-06T12:30:25Z

I gather there is no unit test coverage for this script..

looks like not.
OTOH, I presume one doesn't want to actually upload to DB every time the test is exercised.

davidlange6 · 2021-07-06T12:30:42Z

Could you change /usr/bin/env python to /usr/bin/env python3 and confirm that it works?

…

On Jul 6, 2021, at 2:24 PM, David Lange ***@***.***> wrote: looks like it. I gather there is no unit test coverage for this script.. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

smuzaffar · 2021-07-06T12:33:20Z

correct cx_Oracle is available for python3 only, so try using python3

smuzaffar · 2021-07-06T12:35:11Z

@ggovi also reported the same issue for conddb --db oradev -a /build/gg list runinfo_31X_mc but this was only in CMSSW_12_0_X_2021-07-01-2300 IB where condb was using #!/usr/bin/env python

davidlange6 · 2021-07-06T12:37:57Z

I’ll make a PR changing all the scripts in CondCore

…

On Jul 6, 2021, at 2:35 PM, Malik Shahzad Muzaffar ***@***.***> wrote: @ggovi also reported the same issue for conddb --db oradev -a /build/gg list runinfo_31X_mc but this was only in CMSSW_12_0_X_2021-07-01-2300 IB where condb was using #!/usr/bin/env python — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

mmusich · 2021-07-06T12:38:36Z

Could you change /usr/bin/env python to /usr/bin/env python3 and confirm that it works?

it doesn't.
With:

diff --git a/CondCore/Utilities/scripts/uploadConditions.py b/CondCore/Utilities/scripts/uploadConditions.py
index 25f57d30c27..73b0bc2ff06 100755
--- a/CondCore/Utilities/scripts/uploadConditions.py
+++ b/CondCore/Utilities/scripts/uploadConditions.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 '''Script that uploads to the new CMS conditions uploader.
 Adapted to the new infrastructure from v6 of the upload.py script for the DropBox from Miguel Ojeda.
 '''

I get

$ uploadConditions.py SiPixelGainCalibration_2021_v0_full.db                                                                                                                          Traceback (most recent call last):
  File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 46, in <module>
    import cStringIO
ModuleNotFoundError: No module named 'cStringIO'

davidlange6 · 2021-07-06T12:39:34Z

One doesn’t have to upload to a db that persists for more than the length of the test or do you? But indeed if there is no support for uploading to some file based or otherwise fake db , it would mean having a bit of infrastructure behind..

…

On Jul 6, 2021, at 2:30 PM, Marco Musich ***@***.***> wrote: I gather there is no unit test coverage for this script.. looks like not. OTOH, I presume one doesn't want to actually upload to DB every time the test is exercised. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

davidlange6 · 2021-07-06T12:42:42Z

Yea, cStringIO is gone in python3. You probably need ```from io import StringIO```

…

On Jul 6, 2021, at 2:38 PM, Marco Musich ***@***.***> wrote: Could you change /usr/bin/env python to /usr/bin/env python3 and confirm that it works? it doesn't. With: diff --git a/CondCore/Utilities/scripts/uploadConditions.py b/CondCore/Utilities/scripts/uploadConditions.py index 25f57d30c27..73b0bc2ff06 100755 --- a/CondCore/Utilities/scripts/uploadConditions.py +++ b/CondCore/Utilities/scripts/uploadConditions.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 '''Script that uploads to the new CMS conditions uploader. Adapted to the new infrastructure from v6 of the upload.py script for the DropBox from Miguel Ojeda. ''' I get $ uploadConditions.py SiPixelGainCalibration_2021_v0_full.db Traceback (most recent call last): File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 46, in <module> import cStringIO ModuleNotFoundError: No module named 'cStringIO' — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

mmusich · 2021-07-06T12:43:05Z

But indeed if there is no support for uploading to some file based or otherwise fake db , it would mean having a bit of infrastructure behind..

at the moment I think one can just either upload to the production or to the preparation db, I am not totally sure uploading to a local sqlite file would test all the functionalities. For sure one can make a simple test just running the wizard in order to spot wrong includes or something like that.
I let @ggovi to comment more.

ggovi · 2021-07-06T12:43:14Z

@davidlange6 thanks ,a PR is almost ready, i'm about to submit it...

davidlange6 · 2021-07-06T12:44:14Z

Sorry, ```from io import StringIO as cStringIO``` If you don’t want to change code below

…

On Jul 6, 2021, at 2:42 PM, David Lange ***@***.***> wrote: Yea, cStringIO is gone in python3. You probably need ```from io import StringIO``` > On Jul 6, 2021, at 2:38 PM, Marco Musich ***@***.***> wrote: > > > Could you change /usr/bin/env python to /usr/bin/env python3 and confirm that it works? > > it doesn't. > With: > > diff --git a/CondCore/Utilities/scripts/uploadConditions.py b/CondCore/Utilities/scripts/uploadConditions.py > > index 25f57d30c27..73b0bc2ff06 100755 > > --- a/CondCore/Utilities/scripts/uploadConditions.py > +++ b/CondCore/Utilities/scripts/uploadConditions.py > @@ -1,4 +1,4 @@ > -#!/usr/bin/env python > +#!/usr/bin/env python3 > > '''Script that uploads to the new CMS conditions uploader. > Adapted to the new infrastructure from v6 of the upload.py script for the DropBox from Miguel Ojeda. > ''' > > I get > > $ uploadConditions.py SiPixelGainCalibration_2021_v0_full.db Traceback (most recent call last): > File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 46, in <module> > import cStringIO > ModuleNotFoundError: No module named 'cStringIO' > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub, or unsubscribe. >

mmusich · 2021-07-06T12:56:54Z

traceback (most recent call last):
  File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 711, in uploadAllFiles
    with open(metadataFilename, 'rb') as metadataFile:
FileNotFoundError: [Errno 2] No such file or directory: 'SiPixelGainCalibration_2021_v0_full.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 992, in <module>
    sys.exit(main())
  File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 977, in main
    return upload(options, arguments)
  File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 890, in upload
    results = uploadAllFiles(options, arguments)
  File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 721, in uploadAllFiles
    if getInput('y', '\nIt looks like the metadata file %s does not exist. Do you want me to create it and help you fill it?\nAnswer [y]: ' % metadataFilename).lower() != 'y':
  File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 56, in getInput
    answer = raw_input(prompt)
NameError: name 'raw_input' is not defined

I guess there's more work to it... @ggovi

davidlange6 · 2021-07-06T13:04:08Z

Funny, I thought I fixed these things years ago.. Seems I missed these raw_input is also gone in python3. answer = input(prompt) Is likely sufficient.

…

On Jul 6, 2021, at 2:57 PM, Marco Musich ***@***.***> wrote: traceback (most recent call last): File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 711, in uploadAllFiles with open(metadataFilename, 'rb') as metadataFile: FileNotFoundError: [Errno 2] No such file or directory: 'SiPixelGainCalibration_2021_v0_full.txt' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 992, in <module> sys.exit(main()) File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 977, in main return upload(options, arguments) File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 890, in upload results = uploadAllFiles(options, arguments) File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 721, in uploadAllFiles if getInput('y', '\nIt looks like the metadata file %s does not exist. Do you want me to create it and help you fill it?\nAnswer [y]: ' % metadataFilename).lower() != 'y': File "/tmp/musich/vanilla/CMSSW_12_0_X_2021-07-06-1100/bin/slc7_amd64_gcc900/uploadConditions.py", line 56, in getInput answer = raw_input(prompt) NameError: name 'raw_input' is not defined I guess there's more work to it... @ggovi — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

davidlange6 · 2021-07-06T13:11:07Z

On Jul 6, 2021, at 3:04 PM, David Lange ***@***.***> wrote: Funny, I thought I fixed these things years ago.. Seems I missed these

Correction, I obviously didn’t actually fix cStringIO or raw_input in CMSSW. I’m rather surprised that none of the users of cStringIO are found in the PY3 unit tests

smuzaffar · 2021-07-06T13:13:49Z

@davidlange6 , I think unit tests are py3 compatible. This uploadConditions.py is part of scripts directory and is not run by any unit test

mmusich · 2021-07-06T13:21:18Z

Is likely sufficient.

It does work (sort of), for some reason the wizard doesn't recognize yet the internal list of tags present in the input sqlite file, but if it's prompted by the user it does work....

davidlange6 · 2021-07-06T13:21:47Z

Yes, bad luck it seems.

…

On Jul 6, 2021, at 3:13 PM, Malik Shahzad Muzaffar ***@***.***> wrote: @davidlange6 , I think unit tests are py3 compatible. This uploadConditions.py is part of scripts directory and is never run during IB tests — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

davidlange6 · 2021-07-06T13:30:46Z

Is that something done via the cx-oracle package or something in cmssw?

…

On Jul 6, 2021, at 3:21 PM, Marco Musich ***@***.***> wrote: Is likely sufficient. It does work (sort of), for some reason the wizard doesn't recognize yet the internal list of tags present in the input sqlite file, but if it's prompted by the user it does work.... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

mmusich · 2021-07-06T13:35:55Z

Is that something done via the cx-oracle package or something in cmssw?

I am chairing a meeting now, I'll investigate later.
I am tagging also @muhammadhasib and @Dres90 in case they more readily know

ggovi · 2021-07-06T13:41:59Z

@mmusich @davidlange6 as I said i have a fix for that issue, i'm currently working to migrate the pycurl related code...

ggovi · 2021-07-06T13:42:43Z

@mmusich hasib left CMS more than one year ago...

mmusich · 2021-07-06T13:44:46Z

@ggovi he just wrote me he has a py3 version of the script in skype chat :)

ggovi · 2021-07-07T09:31:27Z

addressed by #34373

tvami · 2021-07-08T18:38:11Z

assign alca

cmsbuild · 2021-07-08T18:38:33Z

New categories assigned: alca

@francescobrivio,@tvami,@malbouis,@pohsun,@tlampen,@yuanchao you have been requested to review this Pull request/Issue and eventually sign? Thanks

tvami · 2021-07-08T18:38:42Z

+alca

cmsbuild · 2021-07-08T18:38:54Z

This issue is fully signed and ready to be closed.

mmusich · 2021-07-08T18:40:35Z

I think it should be actually assigned db...

qliphy · 2021-07-09T10:11:44Z

#34349 (comment)
@ggovi is for db, so should be fine to close this issue.

mmusich · 2021-07-09T10:14:26Z

@qliphy there is still the question of the unit test PR at #34379 which is not signed by db...

qliphy · 2021-07-09T10:18:55Z

@qliphy there is still the question of the unit test PR at #34379 which is not signed by db...

Sorry for missing this.

qliphy · 2021-07-09T10:19:03Z

assign db

cmsbuild · 2021-07-09T10:19:23Z

New categories assigned: db

@ggovi you have been requested to review this Pull request/Issue and eventually sign? Thanks

ggovi · 2021-07-09T10:27:12Z

+1

cmsbuild · 2021-07-09T10:27:35Z

This issue is fully signed and ready to be closed.

qliphy · 2021-07-12T01:08:42Z

#34379

@cms-sw/db-l2 Would you please check #34379 ?

cmsbuild added the pending-assignment label Jul 6, 2021

mmusich mentioned this issue Jul 6, 2021

move scripts in CondCore to python3 #34350

Merged

This was referenced Jul 7, 2021

several conddb tools broken in recent 12_0_X IBs #34378

Closed

introduce uploadConditions.py unit test #34379

Merged

cmsbuild added alca-pending pending-signatures and removed pending-assignment labels Jul 8, 2021

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Jul 8, 2021

qliphy closed this as completed Jul 9, 2021

qliphy reopened this Jul 9, 2021

cmsbuild added db-pending pending-signatures and removed fully-signed labels Jul 9, 2021

cmsbuild added db-approved fully-signed and removed db-pending pending-signatures labels Jul 9, 2021

qliphy closed this as completed Jul 16, 2021

uploadConditions.py is broken in recent IBs #34349

uploadConditions.py is broken in recent IBs #34349

Comments

mmusich commented Jul 6, 2021

cmsbuild commented Jul 6, 2021

davidlange6 commented Jul 6, 2021

mmusich commented Jul 6, 2021

davidlange6 commented Jul 6, 2021 via email

smuzaffar commented Jul 6, 2021

smuzaffar commented Jul 6, 2021

davidlange6 commented Jul 6, 2021 via email

mmusich commented Jul 6, 2021

davidlange6 commented Jul 6, 2021 via email

davidlange6 commented Jul 6, 2021 via email

mmusich commented Jul 6, 2021

ggovi commented Jul 6, 2021

davidlange6 commented Jul 6, 2021 via email

mmusich commented Jul 6, 2021 • edited Loading

davidlange6 commented Jul 6, 2021 via email

davidlange6 commented Jul 6, 2021 via email

smuzaffar commented Jul 6, 2021 • edited Loading

mmusich commented Jul 6, 2021

davidlange6 commented Jul 6, 2021 via email

davidlange6 commented Jul 6, 2021 via email

mmusich commented Jul 6, 2021

ggovi commented Jul 6, 2021

ggovi commented Jul 6, 2021

mmusich commented Jul 6, 2021

ggovi commented Jul 7, 2021

tvami commented Jul 8, 2021

cmsbuild commented Jul 8, 2021

tvami commented Jul 8, 2021

cmsbuild commented Jul 8, 2021

mmusich commented Jul 8, 2021

qliphy commented Jul 9, 2021

mmusich commented Jul 9, 2021

qliphy commented Jul 9, 2021

qliphy commented Jul 9, 2021

cmsbuild commented Jul 9, 2021

ggovi commented Jul 9, 2021

cmsbuild commented Jul 9, 2021

qliphy commented Jul 12, 2021

mmusich commented Jul 6, 2021 •

edited

Loading

smuzaffar commented Jul 6, 2021 •

edited

Loading