Can't download files when file already exists locally and message checksum is set to arbitrary

[andreleblanc11]

When a message comes in with an arbitrary checksum, and we want to download this file in a location where

1. The filename already exists, but
2. The checksum is different from that of the incoming message.

The file gets labeled as a duplicate and is rejected.

2024-12-18 16:01:21,372 [DEBUG] sarracenia.flow file_should_be_downloaded checksum in message: {'method': 'arbitrary', 'value': 'f47363343f9330bc36652458255f17ab'} vs. local: {'method': 'a
rbitrary', 'value': 'f47363343f9330bc36652458255f17ab'}
2024-12-18 16:01:21,372 [DEBUG] sarracenia.flow file_should_be_downloaded checksum in message: {'method': 'arbitrary', 'value': '3fb870ca02dd4040c0c3c6ba0237a9e8'} vs. local: {'method': 'a
rbitrary', 'value': '3fb870ca02dd4040c0c3c6ba0237a9e8'}
2024-12-18 16:01:21,372 [INFO] sarracenia.flowcb.log after_work rejected: 304 same checksum /apps/sarra/public_data/20241218/WXO-DD-STAGE/observations/swob-ml/latest/CPCI-AUTO-minute-swob.
xml
2024-12-18 16:01:21,372 [INFO] sarracenia.flowcb.log after_work rejected: 304 same checksum /apps/sarra/public_data/20241218/WXO-DD-STAGE/observations/swob-ml/latest/CWEW-AUTO-minute-swob.
xml
2024-12-18 16:01:21,372 [INFO] sarracenia.flowcb.log after_work rejected: 304 same checksum /apps/sarra/public_data/20241218/WXO-DD-STAGE/observations/swob-ml/latest/CZGH-AUTO-minute-swob.
xml
2024-12-18 16:01:21,372 [INFO] sarracenia.flowcb.log after_work rejected: 304 same checksum /apps/sarra/public_data/20241218/WXO-DD-STAGE/observations/swob-ml/latest/CESB-AUTO-minute-swob.
xml
2024-12-18 16:01:21,372 [INFO] sarracenia.flowcb.log after_work rejected: 304 same checksum /apps/sarra/public_data/20241218/WXO-DD-STAGE/observations/swob-ml/latest/CWZW-AUTO-minute-swob.
xml
2024-12-18 16:01:21,372 [INFO] sarracenia.flowcb.log after_work rejected: 304 same checksum /apps/sarra/public_data/20241218/WXO-DD-STAGE/observations/swob-ml/latest/CWOK-AUTO-minute-swob.
xml
2024-12-18 16:01:21,372 [INFO] sarracenia.flowcb.log after_work rejected: 304 same checksum /apps/sarra/public_data/20241218/WXO-DD-STAGE/observations/swob-ml/latest/CVOP-AUTO-minute-swob.
xml

For my use case, I found, that xattr wasn't installed when I did a sr3 features.
I installed it and see'd if that helped, but some files were still being labeled as duplicates.

I think something might be wrong in compute_local_checksum

sarracenia/sarracenia/flow/__init__.py

Lines 1430 to 1454 in 9b2ff58

	def compute_local_checksum(self, msg) -> None:
	if sarracenia.filemetadata.supports_extended_attributes:
	try:
	x = sarracenia.filemetadata.FileMetadata(msg['new_path'])
	s = x.get('identity')
	if s:
	metadata_cached_mtime = x.get('mtime')
	if ((metadata_cached_mtime >= msg['mtime'])):
	# file has not been modified since checksum value was stored.
	if (( 'identity' in msg ) and ( 'method' in msg['identity'] ) and \
	( msg['identity']['method'] == s['method'] )) or \
	( s['method'] == self.o.identity_method ) :
	# file
	# cache good.
	msg['local_identity'] = s
	msg['_deleteOnPost'] |= set(['local_identity'])
	b = x.get('blocks')
	msg['local_blocks'] = b
	msg['_deleteOnPost'] |= set(['local_blocks'])
	return
	except:
	pass

I've added debugging in the source and found that line 1439 returns as False.
When this happens, this line of code is executed and will always set an arbitrary checksum as a duplicate.

sarracenia/sarracenia/flow/__init__.py

Lines 1459 to 1460 in 9b2ff58

	if msg['identity']['method'] == 'arbitrary':
	local_identity.value = msg['identity']['value']

I think we might need to change the >= to a =<? If the file at the destination already exists, the mtime should be lesser than that of the file we're trying to download.

Work around

We can work around this by overriding the checksum with identity cod,sha512 however this isn't good practice, because we can't truely validate the integrity of the file that comes from the source.

[petersilva]

This problem occurs, when:

• file with same name exists on local tree.
• file checksum is arbitrary.
• calculate_local_checksum... is called... the xattr python module was missing... so it always returns the checksum from the message because there is no way to calculate local checksum without xattr.
• installing the xattr routine should fix the problem.

oh... found a bug in flow/compute_local_checksum... working on it.

[petersilva]

regression vs. v2.