OBS-463: Use custom time as modification time for GCS objects if set. #3112
Conversation
| @@ -126,7 +126,7 @@ def get_object_metadata(self, key: str) -> Optional[ObjectMetadata]: | |||
| content_encoding=blob.content_encoding, | |||
| original_content_length=original_content_length, | |||
| original_md5_sum=gcs_metadata.get("original_md5_hash"), | |||
| last_modified=blob.updated, | |||
| last_modified=blob.custom_time or blob.updated, | |||
There was a problem hiding this comment.
I'm not sure I understand this. For files that we migrated from S3 to GCS, custom_time will be set and will be the same value type as updated (e.g. seconds since epoch, Python datetime in UTC, etc). Is that right? What is that value type?
For files that were not migrated from S3, what does custom_time get set to? Does it exist as an attribute of blob object?
There was a problem hiding this comment.
migrated files will have a custom time that matches the s3 upload time and an upload time in the last week, and yes they will have the same type, though i don't know what that is.
when custom time isn't present the attribute will be None, which is false-y.
There was a problem hiding this comment.
The type of both blob.custom_time and blob.updated is datetime.datetime | None. The latter will always be set in this case, since it can only be None for objects that haven't been uploaded yet.
For objects directly uploaded to GCS, blob.updated is the time the upload finished, and blob.custom_time is None.
For objects migrated from S3, blob.updated is the time the object was migrated, and blob.custom_time is the original upload time.
In both cases, blob.custom_time or blob.updated evaluates to the original upload time, which is what we want.
I added type annotations to ObjectMetadata, and the GCS Python client also has type hints, so VS Code shows me type information for everything while typing. I also tested this by manually creating a GCS client and retrieving blobs with an without custom time metadata.
https://mozilla-hub.atlassian.net/browse/OBS-463