2021-10-13 17:30:14,536:139884748211968:ERROR:AccountantWorker:Error while adding files to WMBS!
(cx_Oracle.IntegrityError) ORA-00001: unique constraint (CMS_T0AST_REPLAY3.WMBS_FILDETAILS_UNIQUE) violated
[SQL: INSERT INTO wmbs_file_details (id, lfn, filesize, events,
                                            first_event, merged)
             VALUES (wmbs_file_details_SEQ.nextval, :lfn, :filesize, :events,
                     :first_event, :merged)]
[parameters: [{'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/ALCARECO/Express-v2110131547/000/343/082/00000/502327d9-0b9f-422a-8513-209b40c1b7e8.root', 'filesize': 1480327, 'events': 2304, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/DQMIO/Express-v2110131547/000/343/082/00000/CB10EE3C-2C2F-11EC-8745-030011ACBEEF.root', 'filesize': 135153, 'events': 0, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/TestEnablesEcalHcal/RAW/Express-v2110131547/000/343/082/00000/23345b92-b1d9-4128-b016-cda93a4a65aa.root', 'filesize': 394395785, 'events': 2304, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/logs/prod/2021/10/13/Express_Run343082_StreamCalibration_Tier0_REPLAY_2021_v2110131547_211013_1547/Express/0000/0/Express-2afd69f4-4646-4387-a904-b650fd54f57f-0-logArchive.tar.gz', 'filesize': 0, 'events': 0, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/ALCARECO/Express-v2110131547/000/343/082/00000/bb42c957-0a58-40bc-af1f-7340888a46cc.root', 'filesize': 1500599, 'events': 2341, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/DQMIO/Express-v2110131547/000/343/082/00000/CCFF0C94-2C32-11EC-934D-2B8A8E80BEEF.root', 'filesize': 135151, 'events': 0, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/TestEnablesEcalHcal/RAW/Express-v2110131547/000/343/082/00000/0e9e50df-4c9e-4bd3-b9f3-abad4e1a1dcd.root', 'filesize': 400714348, 'events': 2341, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/logs/prod/2021/10/13/Express_Run343082_StreamCalibration_Tier0_REPLAY_2021_v2110131547_211013_1547/Express/0000/1/Express-80b096a9-d3ec-42d5-bb64-ca7ae38b73c9-1-logArchive.tar.gz', 'filesize': 0, 'events': 0, 'first_event': 0, 'merged': 0}  ... displaying 10 of 396 total bound parameter sets ...  {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/TestEnablesEcalHcal/RAW/Express-v2110131547/000/343/082/00000/107d0964-6b5e-4886-95d3-f97d1cebeb85.root', 'filesize': 395058340, 'events': 2308, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/logs/prod/2021/10/13/Express_Run343082_StreamCalibration_Tier0_REPLAY_2021_v2110131547_211013_1547/Express/0000/0/Express-b59e23ba-2adc-4eef-8573-25d075eb0711-0-logArchive.tar.gz', 'filesize': 0, 'events': 0, 'first_event': 0, 'merged': 0}]]
(Background on this error at: http://sqlalche.me/e/gkpj)
2021-10-13 17:30:14,622:139884748211968:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMComponent.JobAccountant.JobAccountantPoller.JobAccountantPoller object at 0x7f397ff12f40> <@========== WMException Start ==========@>
Exception Class: AccountantWorkerException
Message: Error while adding files to WMBS!
(cx_Oracle.IntegrityError) ORA-00001: unique constraint (CMS_T0AST_REPLAY3.WMBS_FILDETAILS_UNIQUE) violated
[SQL: INSERT INTO wmbs_file_details (id, lfn, filesize, events,
                                            first_event, merged)
             VALUES (wmbs_file_details_SEQ.nextval, :lfn, :filesize, :events,
                     :first_event, :merged)]
[parameters: [{'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/ALCARECO/Express-v2110131547/000/343/082/00000/502327d9-0b9f-422a-8513-209b40c1b7e8.root', 'filesize': 1480327, 'events': 2304, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/DQMIO/Express-v2110131547/000/343/082/00000/CB10EE3C-2C2F-11EC-8745-030011ACBEEF.root', 'filesize': 135153, 'events': 0, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/TestEnablesEcalHcal/RAW/Express-v2110131547/000/343/082/00000/23345b92-b1d9-4128-b016-cda93a4a65aa.root', 'filesize': 394395785, 'events': 2304, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/logs/prod/2021/10/13/Express_Run343082_StreamCalibration_Tier0_REPLAY_2021_v2110131547_211013_1547/Express/0000/0/Express-2afd69f4-4646-4387-a904-b650fd54f57f-0-logArchive.tar.gz', 'filesize': 0, 'events': 0, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/ALCARECO/Express-v2110131547/000/343/082/00000/bb42c957-0a58-40bc-af1f-7340888a46cc.root', 'filesize': 1500599, 'events': 2341, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/DQMIO/Express-v2110131547/000/343/082/00000/CCFF0C94-2C32-11EC-934D-2B8A8E80BEEF.root', 'filesize': 135151, 'events': 0, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/TestEnablesEcalHcal/RAW/Express-v2110131547/000/343/082/00000/0e9e50df-4c9e-4bd3-b9f3-abad4e1a1dcd.root', 'filesize': 400714348, 'events': 2341, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/logs/prod/2021/10/13/Express_Run343082_StreamCalibration_Tier0_REPLAY_2021_v2110131547_211013_1547/Express/0000/1/Express-80b096a9-d3ec-42d5-bb64-ca7ae38b73c9-1-logArchive.tar.gz', 'filesize': 0, 'events': 0, 'first_event': 0, 'merged': 0}  ... displaying 10 of 396 total bound parameter sets ...  {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/TestEnablesEcalHcal/RAW/Express-v2110131547/000/343/082/00000/107d0964-6b5e-4886-95d3-f97d1cebeb85.root', 'filesize': 395058340, 'events': 2308, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/logs/prod/2021/10/13/Express_Run343082_StreamCalibration_Tier0_REPLAY_2021_v2110131547_211013_1547/Express/0000/0/Express-b59e23ba-2adc-4eef-8573-25d075eb0711-0-logArchive.tar.gz', 'filesize': 0, 'events': 0, 'first_event': 0, 'merged': 0}]]
(Background on this error at: http://sqlalche.me/e/gkpj)
	ClassName : None
	ModuleName : WMComponent.JobAccountant.AccountantWorker
	MethodName : handleWMBSFiles
	ClassInstance : None
	FileName : /data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/AccountantWorker.py
	LineNumber : 875
	ErrorNr : 0

Traceback: 
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 846, in handleWMBSFiles
    self.addFileAction.execute(files=fileCreate,

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/WMBS/MySQL/Files/Add.py", line 40, in execute
    result = self.dbi.processData(self.sql, binds,

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/Database/DBCore.py", line 159, in processData
    result.extend(self.executemanybinds(sqlstmt[0], subBinds,

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/Database/DBCore.py", line 113, in executemanybinds
    result = connection.execute(s, b)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return self._execute_text(object_, multiparams, params)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1149, in _execute_text
    ret = self._execute_context(

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1247, in _execute_context
    self._handle_dbapi_exception(

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
    raise value.with_traceback(tb)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1223, in _execute_context
    self.dialect.do_executemany(

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/dialects/oracle/cx_oracle.py", line 1080, in do_executemany
    cursor.executemany(statement, parameters)

<@---------- WMException End ----------@>  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 183, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/Database/DBExceptionHandler.py", line 40, in wrapper
    return f(*args, **kwargs)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/Utils/Timers.py", line 26, in wrapper
    res = func(*arg, **kw)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 70, in algorithm
    self.accountantWorker(jobsSlice)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 249, in __call__
    self.handleWMBSFiles(self.wmbsFilesToBuild, self.parentageBinds)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 875, in handleWMBSFiles
    raise AccountantWorkerException(msg)

2021-10-13 17:30:14,623:139884748211968:INFO:Harness:>>>Terminating worker threads
2021-10-13 17:30:14,623:139884748211968:ERROR:BaseWorkerThread:Error in event loop (2): <WMComponent.JobAccountant.JobAccountantPoller.JobAccountantPoller object at 0x7f397ff12f40> <@========== WMException Start ==========@>
Exception Class: AccountantWorkerException
Message: Error while adding files to WMBS!
(cx_Oracle.IntegrityError) ORA-00001: unique constraint (CMS_T0AST_REPLAY3.WMBS_FILDETAILS_UNIQUE) violated
[SQL: INSERT INTO wmbs_file_details (id, lfn, filesize, events,
                                            first_event, merged)
             VALUES (wmbs_file_details_SEQ.nextval, :lfn, :filesize, :events,
                     :first_event, :merged)]
[parameters: [{'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/ALCARECO/Express-v2110131547/000/343/082/00000/502327d9-0b9f-422a-8513-209b40c1b7e8.root', 'filesize': 1480327, 'events': 2304, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/DQMIO/Express-v2110131547/000/343/082/00000/CB10EE3C-2C2F-11EC-8745-030011ACBEEF.root', 'filesize': 135153, 'events': 0, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/TestEnablesEcalHcal/RAW/Express-v2110131547/000/343/082/00000/23345b92-b1d9-4128-b016-cda93a4a65aa.root', 'filesize': 394395785, 'events': 2304, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/logs/prod/2021/10/13/Express_Run343082_StreamCalibration_Tier0_REPLAY_2021_v2110131547_211013_1547/Express/0000/0/Express-2afd69f4-4646-4387-a904-b650fd54f57f-0-logArchive.tar.gz', 'filesize': 0, 'events': 0, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/ALCARECO/Express-v2110131547/000/343/082/00000/bb42c957-0a58-40bc-af1f-7340888a46cc.root', 'filesize': 1500599, 'events': 2341, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/StreamCalibration/DQMIO/Express-v2110131547/000/343/082/00000/CCFF0C94-2C32-11EC-934D-2B8A8E80BEEF.root', 'filesize': 135151, 'events': 0, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/TestEnablesEcalHcal/RAW/Express-v2110131547/000/343/082/00000/0e9e50df-4c9e-4bd3-b9f3-abad4e1a1dcd.root', 'filesize': 400714348, 'events': 2341, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/logs/prod/2021/10/13/Express_Run343082_StreamCalibration_Tier0_REPLAY_2021_v2110131547_211013_1547/Express/0000/1/Express-80b096a9-d3ec-42d5-bb64-ca7ae38b73c9-1-logArchive.tar.gz', 'filesize': 0, 'events': 0, 'first_event': 0, 'merged': 0}  ... displaying 10 of 396 total bound parameter sets ...  {'lfn': '/store/unmerged/data/Tier0_REPLAY_2021/TestEnablesEcalHcal/RAW/Express-v2110131547/000/343/082/00000/107d0964-6b5e-4886-95d3-f97d1cebeb85.root', 'filesize': 395058340, 'events': 2308, 'first_event': 0, 'merged': 0}, {'lfn': '/store/unmerged/data/logs/prod/2021/10/13/Express_Run343082_StreamCalibration_Tier0_REPLAY_2021_v2110131547_211013_1547/Express/0000/0/Express-b59e23ba-2adc-4eef-8573-25d075eb0711-0-logArchive.tar.gz', 'filesize': 0, 'events': 0, 'first_event': 0, 'merged': 0}]]
(Background on this error at: http://sqlalche.me/e/gkpj)
	ClassName : None
	ModuleName : WMComponent.JobAccountant.AccountantWorker
	MethodName : handleWMBSFiles
	ClassInstance : None
	FileName : /data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/AccountantWorker.py
	LineNumber : 875
	ErrorNr : 0

Traceback: 
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 846, in handleWMBSFiles
    self.addFileAction.execute(files=fileCreate,

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/WMBS/MySQL/Files/Add.py", line 40, in execute
    result = self.dbi.processData(self.sql, binds,

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/Database/DBCore.py", line 159, in processData
    result.extend(self.executemanybinds(sqlstmt[0], subBinds,

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/Database/DBCore.py", line 113, in executemanybinds
    result = connection.execute(s, b)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return self._execute_text(object_, multiparams, params)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1149, in _execute_text
    ret = self._execute_context(

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1247, in _execute_context
    self._handle_dbapi_exception(

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
    raise value.with_traceback(tb)

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1223, in _execute_context
    self.dialect.do_executemany(

  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.3.3/lib/python3.8/site-packages/sqlalchemy/dialects/oracle/cx_oracle.py", line 1080, in do_executemany
    cursor.executemany(statement, parameters)

<@---------- WMException End ----------@>
Backtrace:
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 209, in __call__
    raise ex
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 183, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/Database/DBExceptionHandler.py", line 40, in wrapper
    return f(*args, **kwargs)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/Utils/Timers.py", line 26, in wrapper
    res = func(*arg, **kw)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 70, in algorithm
    self.accountantWorker(jobsSlice)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 249, in __call__
    self.handleWMBSFiles(self.wmbsFilesToBuild, self.parentageBinds)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 875, in handleWMBSFiles
    raise AccountantWorkerException(msg)

2021-10-13 17:30:15,617:139884748211968:INFO:BaseWorkerThread:Worker thread <WMComponent.JobAccountant.JobAccountantPoller.JobAccountantPoller object at 0x7f397ff12f40> terminated