Publisher - block publication fails

[mapellidario]

I deployed the latest tag v3.220826 and noticed many publication failures in Publisher_schedd/logs/log.txt, one of them look like [1].

[1]

from tasks/shin/220826_013902:shin_crab_HplusToCB_LO_M120_Summer20UL16APV_Private.log

2022-08-26 15:26:59,985:INFO: Found 3 files not already present in DBS which will be published.
2022-08-26 15:26:59,986:ERROR: Error for files: ['/store/user/shin/HplusToCB_LO_M120_Summer20UL16APV_Private/RunIISummer20UL16APV_DIGIPremix/220826_013902/0000/DIGIPremix_340.root', '/store/user/shin/HplusToCB_LO_M120_Summer20UL16APV_Private/RunIISummer20UL16APV_DIGIPremix/220826_013902/0000/DIGIPremix_435.root', '/store/user/shin/HplusToCB_LO_M120_Summer20UL16APV_Private/RunIISummer20UL16APV_DIGIPremix/220826_013902/0000/DIGIPremix_433.root']
2022-08-26 15:26:59,988:ERROR: Error when publishing (/store/temp/user/shin.487353ee37db1eab9ce4b168bd8d1154aa0ff287/HplusToCB_LO_M120_Summer20UL16APV_Private/RunIISummer20UL16APV_DIGIPremix/220826_013902/0000/DIGIPremix_340.root, /store/temp/user/shin.487353ee37db1eab9ce4b168bd8d1154aa0ff287/HplusToCB_LO_M120_Summer20UL16APV_Private/RunIISummer20UL16APV_DIGIPremix/220826_013902/0000/DIGIPremix_435.root, /store/temp/user/shin.487353ee37db1eab9ce4b168bd8d1154aa0ff287/HplusToCB_LO_M120_Summer20UL16APV_Private/RunIISummer20UL16APV_DIGIPremix/220826_013902/0000/DIGIPremix_433.root) 'list' object has no attribute 'get'Traceback (most recent call last):
  File "/data/srv/TaskManager/v3.220826-3f9ef41a7fa5dcfe2dae719b7fcfce6b/slc7_amd64_gcc630/cms/crabtaskworker/v3.220826-3f9ef41a7fa5dcfe2dae719b7fcfce6b/lib/python3.8/site-packages/Publisher/TaskPublishGo.py", line 903, in publishInDBS3
    destApi.insertBulkBlock(blockDump)
  File "/data/srv/TaskManager/v3.220826-3f9ef41a7fa5dcfe2dae719b7fcfce6b/slc7_amd64_gcc630/cms/py3-dbs3-client/4.0.9/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 610, in insertBulkBlock
    if fileLumiList.get('event_count') == None:
AttributeError: 'list' object has no attribute 'get'

2022-08-26 15:26:59,994:ERROR: FAILING BLOCK DUE TO 'list' object has no attribute 'get' SAVED AS /data/srv/Publisher_files/FailedBlocks/failed-block-at-1661520419.98896.txt

[mapellidario]

So far 20 tasks have been affected

> [15:38] crab-prod-tw01 ~
> tac /data/container/Publisher_schedd/logs/log.txt| grep "Failed block" | awk '{print $3}' | sort | uniq | wc -l
20

However, one task ( 220826_105638:ktaeun_crab_BToPiD0 ) has some blocks that failed at 14:00 and 14:37 (due to a " 502 proxy error" so i do not think it is relevant at the moment), while all the other blocks failed between 15:26 and 15:27 with the same error reported in the issue description.

[mapellidario]

I think that the problem has been introduced with the new version of the py3-dbs-client cms-sw/cmsdist#8040

[amaltaro]

@mapellidario I think you are right! This is the latest change that has been introduced before cutting 4.0.9 dbs3-client:
dmwm/DBSClient@3ba18ca#diff-9468ff64509aa700b869f2882592becf5a827e5db553de519169c759e3a3a04bR610

@vkuznet can you please investigate it and make a new pypi release with the fix?
@mapellidario once the new release is made available, can you please update that cmsdist spec? I can do that if it's too late for you

[vkuznet]

the fix in dmwm/DBSClient#73, please review it.