testCondToolsSiStripBuildersReaders fails occasionally #36319

makortel · 2021-12-01T14:03:49Z

The unit test testCondToolsSiStripBuildersReaders in CondTools/SiStrip appears to fail occasionally along

connectionID=575ae316-5255-11ec-86f9-fa163e323604
Connection to service "SiStripConditionsDBFile.db" with connectionID=575ae316-5255-11ec-86f9-fa163e323604 will be disconnected Info Connection to service "SiStripConditionsDBFile.db" with connectionID=575ae316-5255-11ec-86f9-fa163e323604 will be disconnected
Deleting the ConnectionPool Info Deleting the ConnectionPool
 Done with the readers 


Using Global Tag: 122X_dataRun2_v1
SQLiteStatement::finalize 10 disk I/O error Error SQLiteStatement::finalize 10 disk I/O error
SQLiteStatement::execute 1 disk I/O error Error SQLiteStatement::execute 1 disk I/O error
----- Begin Fatal Exception 01-Dec-2021 04:18:31 CET-----------------------
An exception of category 'ConditionDatabase' occurred while
   [0] Processing  Event run: 303014 lumi: 1 event: 1 stream: 0
   [1] Running path 'p'
   [2] Calling method for module SiStripChannelGainFromDBMiscalibrator/'scaleAndSmearSiStripGains'
Exception Message:
disk I/O error ( CORAL : "SQLiteStatement::execute" from "CORAL/RelationalPlugins/sqlite" ) from PoolDBOutputService::writeOne 
----- End Fatal Exception -------------------------------------------------

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc10/CMSSW_12_2_X_2021-11-30-2300/unitTestLogs/CondTools/SiStrip#/

So far seen in e.g.

CMSSW_12_2_X_2021-11-30-2300 slc7_amd64_gcc10
CMSSW_12_2_X_2021-11-29-2300 slc7_aarch64_gcc9
CMSSW_12_2_X_2021-11-28-2300 cs8_amd64_gcc11
CMSSW_12_2_X_2021-11-26-2300 slc7_amd64_gcc10
CMSSW_12_2_ROOT6_X_2021-11-26-2300 slc7_amd64_gcc900
CMSSW_12_2_X_2021-11-25-2300 slc7_amd64_gcc10
CMSSW_12_2_DBG_X_2021-11-25-2300 slc7_amd64_gcc900

The text was updated successfully, but these errors were encountered:

cmsbuild · 2021-12-01T14:04:06Z

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel · 2021-12-01T14:06:54Z

assign db

cmsbuild · 2021-12-01T14:06:57Z

New categories assigned: db

@ggovi,@francescobrivio,@malbouis,@tvami you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel · 2021-12-01T14:07:01Z

FYI @cms-sw/trk-dpg-l2

makortel · 2021-12-01T14:07:29Z

Seems like similar concurrency issue as in #36175, #26741

mmusich · 2021-12-01T14:31:37Z

Seems like similar concurrency issue as in #36175, #26741

As far as I can tell the name of the output sqlite file of the failing test is unique in cmssw

cmssw/CondTools/SiStrip/test/SiStripChannelGainFromDBMiscalibrator_cfg.py

Line 160 in 6d2f660

    
           process.CondDB.connect = 'sqlite_file:modifiedGains_'+ process.GlobalTag.globaltag._value+'_IOV_'+str(options.runNumber)+".db"

(see https://github.com/cms-sw/cmssw/search?q=modifiedGains_)

makortel · 2021-12-01T14:39:46Z

As far as I can tell the name of the output sqlite file of the failing test is unique in cmssw

Thanks for checking, this is something different then.

mmusich · 2021-12-01T15:39:20Z

SQLiteStatement::finalize 10 disk I/O error Error SQLiteStatement::finalize 10 disk I/O error
SQLiteStatement::execute 1 disk I/O error Error SQLiteStatement::execute 1 disk I/O error
----- Begin Fatal Exception 01-Dec-2021 04:18:31 CET-----------------------
An exception of category 'ConditionDatabase' occurred while
   [0] Processing  Event run: 303014 lumi: 1 event: 1 stream: 0
   [1] Running path 'p'
   [2] Calling method for module SiStripChannelGainFromDBMiscalibrator/'scaleAndSmearSiStripGains'
Exception Message:
disk I/O error ( CORAL : "SQLiteStatement::execute" from "CORAL/RelationalPlugins/sqlite" ) from PoolDBOutputService::writeOne

@smuzaffar is it possible we're getting an issue due to full disk?

mmusich · 2021-12-03T16:07:55Z

so #36334 (changing the most obvious things could go wrong) didn't help.
There's still an error in CMSSW_12_2_X_2021-12-03-1100 unit tests.
I am open for suggestions on what to check next.

mmusich · 2021-12-03T16:20:08Z

From the manual: https://www.sqlite.org/rescode.html#ioerr

The SQLITE_IOERR result code says that the operation could not finish because the operating system reported an I/O error.
A full disk drive will normally give an SQLITE_FULL error rather than an SQLITE_IOERR error.
There are many different extended result codes for I/O errors that identify the specific I/O operation that failed.

mmusich · 2021-12-06T09:45:26Z

Dear @cms-sw/db-l2 after #36361 has been merged in CMSSW_12_2_X_2021-12-05-0000 I have monitored the IB unit tests and I didn't observe any failure related to testCondToolsSiStripBuildersReaders in:

CMSSW_12_2_X_2021-12-05-0000
CMSSW_12_2_X_2021-12-05-2300

shall we wait for the next IB for confirmation and then close this issue?

tvami · 2021-12-06T16:31:07Z

Seems to be clear for the latest IB as well
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_2_X_2021-12-06-1100/unitTestLogs/CondTools/SiStrip#/

tvami · 2021-12-06T16:31:36Z

+db

seems like resolved in Update miscalibration tools in CondTools/SiStrip #36334

cmsbuild · 2021-12-06T16:32:10Z

This issue is fully signed and ready to be closed.

makortel · 2021-12-07T14:00:32Z

This failure occurred again in CMSSW_12_3_ROOT624_X_2021-12-06-2300

Using Global Tag: 122X_dataRun2_v1
SiStripChannelGainFromDBMiscalibrator::analyze detid 369120277 	 APV 1 	 new gain: 0.565074 	 old gain: 0.845799 	
SiStripChannelGainFromDBMiscalibrator::analyze detid 369120277 	 APV 2 	 new gain: 0.497616 	 old gain: 0.83945 	
SiStripChannelGainFromDBMiscalibrator::analyze detid 369120277 	 APV 3 	 new gain: 0.568472 	 old gain: 0.874687 	
SiStripChannelGainFromDBMiscalibrator::analyze detid 369120277 	 APV 4 	 new gain: 0.54064 	 old gain: 0.817904 	
SiStripChannelGainFromDBMiscalibrator::analyze detid 369120277 	 APV 5 	 new gain: 0.555092 	 old gain: 0.880128 	
SiStripChannelGainFromDBMiscalibrator::analyze detid 369120277 	 APV 6 	 new gain: 0.548598 	 old gain: 0.892149 	
SQLiteStatement::finalize 10 disk I/O error Error SQLiteStatement::finalize 10 disk I/O error
SQLiteStatement::execute 1 disk I/O error Error SQLiteStatement::execute 1 disk I/O error
----- Begin Fatal Exception 07-Dec-2021 08:02:35 CET-----------------------
An exception of category 'ConditionDatabase' occurred while
   [0] Processing  Event run: 303014 lumi: 1 event: 1 stream: 0
   [1] Running path 'p'
   [2] Calling method for module SiStripChannelGainFromDBMiscalibrator/'scaleAndSmearSiStripGains'
Exception Message:
disk I/O error ( CORAL : "SQLiteStatement::execute" from "CORAL/RelationalPlugins/sqlite" ) from PoolDBOutputService::createNewIov 
----- End Fatal Exception -------------------------------------------------

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_3_ROOT624_X_2021-12-06-2300/unitTestLogs/CondTools/SiStrip#/

makortel · 2021-12-08T15:10:17Z

Occurred again in CMSSW_12_3_DEVEL_X_2021-12-07-2300 and CMSSW_12_3_CLANG_X_2021-12-07-2300, and in CMSSW_12_3_X_2021-12-07-2300 for cs8_amd64_gcc10.

makortel · 2021-12-08T15:12:59Z

The test appears to be failing in some PR tests too. I think we can conclude that there is a correlation with the overall load of the build+test machines. But why is it always this specific test that fails and not others that write to SQLite files?

qliphy · 2021-12-09T06:34:24Z

Occurred also in CMSSW_12_2_X_2021-12-08-1100 and CMSSW_12_2_X_2021-12-08-2300

cmsbuild added the pending-assignment label Dec 1, 2021

cmsbuild added db-pending pending-signatures and removed pending-assignment labels Dec 1, 2021

mmusich mentioned this issue Dec 2, 2021

Update miscalibration tools in CondTools/SiStrip #36334

Merged

mmusich mentioned this issue Dec 3, 2021

fix CondDB connection in CondTools/SiStrip miscalibration tools #36361

Merged

cmsbuild added db-approved fully-signed and removed db-pending pending-signatures labels Dec 6, 2021

makortel closed this as completed Dec 6, 2021

mmusich mentioned this issue Dec 9, 2021

More changes in CondTools/SiStrip unit tests #36431

Merged

makortel mentioned this issue Dec 10, 2021

Added FinalPath implementation [12_2] #36438

Merged

qliphy mentioned this issue Dec 13, 2021

Enforce EventSetup products can be obtained only with ESGetToken #35588

Merged

mmusich mentioned this issue Dec 17, 2021

Run testBuildersReaders unit test in its own directory #36533

Merged

mmusich mentioned this issue Jan 10, 2022

CondTools/SiStrip unit tests: introduce gain rescaler tool in unit tests #36508

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

testCondToolsSiStripBuildersReaders fails occasionally #36319

testCondToolsSiStripBuildersReaders fails occasionally #36319

makortel commented Dec 1, 2021

cmsbuild commented Dec 1, 2021

makortel commented Dec 1, 2021

cmsbuild commented Dec 1, 2021

makortel commented Dec 1, 2021

makortel commented Dec 1, 2021

mmusich commented Dec 1, 2021

makortel commented Dec 1, 2021

mmusich commented Dec 1, 2021

mmusich commented Dec 3, 2021

mmusich commented Dec 3, 2021 •

edited

Loading

mmusich commented Dec 6, 2021

tvami commented Dec 6, 2021

tvami commented Dec 6, 2021

cmsbuild commented Dec 6, 2021

makortel commented Dec 7, 2021

makortel commented Dec 8, 2021 •

edited

Loading

makortel commented Dec 8, 2021

qliphy commented Dec 9, 2021

testCondToolsSiStripBuildersReaders fails occasionally #36319

testCondToolsSiStripBuildersReaders fails occasionally #36319

Comments

makortel commented Dec 1, 2021

cmsbuild commented Dec 1, 2021

makortel commented Dec 1, 2021

cmsbuild commented Dec 1, 2021

makortel commented Dec 1, 2021

makortel commented Dec 1, 2021

mmusich commented Dec 1, 2021

makortel commented Dec 1, 2021

mmusich commented Dec 1, 2021

mmusich commented Dec 3, 2021

mmusich commented Dec 3, 2021 • edited Loading

mmusich commented Dec 6, 2021

tvami commented Dec 6, 2021

tvami commented Dec 6, 2021

cmsbuild commented Dec 6, 2021

makortel commented Dec 7, 2021

makortel commented Dec 8, 2021 • edited Loading

makortel commented Dec 8, 2021

qliphy commented Dec 9, 2021

mmusich commented Dec 3, 2021 •

edited

Loading

makortel commented Dec 8, 2021 •

edited

Loading