cms-bot and fake AddOn test failures

[silviodonato]

We observed some fake AddOn test failures in at least two recent PRs (#29347 two times, #29387 )
eg.

CMSSW: CMSSW_11_1_X_2020-04-06-2300
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ebf4ec/5563/summary.html

I found follow errors while testing this PR

Failed tests: AddOn

    AddOn:

I found errors in the following addon tests:

Surprising, cms-bot does not show which AddOn tests are failing.

As a reference in #28936 , on March 6, cms-bot caught correctly the AddOn test failures, showing the failing tests.
Eg.

    AddOn:

I found errors in the following addon tests:

cmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Fri Mar 6 22:29:10 2020-date Fri Mar 6 22:26:29 2020 s - exit: 16640
cmsRun /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_1_X_2020-03-06-1100/src/PhysicsTools/PatAlgos/test/IntegrationTest_cfg.py : FAILED - time: date Fri Mar 6 22:27:08 2020-date Fri Mar 6 22:26:34 2020 s - exit: 17920
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc_l1stage1 --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns : FAILED - time: date Fri Mar 6 22:31:09 2020-date Fri Mar 6 22:27:01 2020 s - exit: 16640
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_2016 : FAILED - time: date Fri Mar 6 22:31:16 2020-date Fri Mar 6 22:27:07 2020 s - exit: 16640

#28936

[silviodonato]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @silviodonato Silvio Donato.

@Dr15Jones, @silviodonato, @dpiparo, @smuzaffar, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

looks like addOn tests were timeout after 2 hours. I think there was some network issues at CERN that is why these tests were timing out . This also happened for IBs few days back. Things looks better now.

I will update cms-bot to properly report is addOnTests were timeout.

[silviodonato]

thanks @smuzaffar , feel free to close the ticket then

[smuzaffar]

I will close it after fixing cms-bot to properly report the timeouts.

[smuzaffar]

cms-bot change cms-sw/cms-bot@9969c92 should properly indicate if addOnTest was not properly finished (main reason is timeout).