Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: data integrity using md5 #113

Merged
merged 1 commit into from
Dec 16, 2022
Merged

Conversation

vasco-santos
Copy link
Contributor

We have sha256 as an attribute in our bucket entries. This could enable us to use S3's GetObjectAttributesCommand to get the sha256 of this. However, we can't follow same pattern as dotStorage workers because CF Workers have an internal API for R2 that supports sha256, while S3 API is still not compatible.

Moved towards trying to integrate md5, with the assumption of ETag header having md5 https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html in it and that we can use S3 Client for R2 with   ✅ Content-MD5

We get md5 ETag in base16 (thanks docs for not mentioning this anywhere 😘) wrapped with quotation marks and S3 client for PutObjectCommand expects the base64-encoded 128-bit MD5 digest of the message (without the headers) according to RFC 1864

Closes #58

@seed-deploy seed-deploy bot temporarily deployed to pr113 December 16, 2022 11:57 Inactive
@seed-deploy
Copy link

seed-deploy bot commented Dec 16, 2022

View stack outputs

@vasco-santos vasco-santos force-pushed the feat/data-integrity-using-md5 branch from 85e00bc to eaeba70 Compare December 16, 2022 12:07
@seed-deploy seed-deploy bot temporarily deployed to pr113 December 16, 2022 12:07 Inactive
Copy link
Contributor

@olizilla olizilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's always the string encoding surprise. They should make hash digests be self describing.

@vasco-santos vasco-santos merged commit 9d9b291 into main Dec 16, 2022
@vasco-santos vasco-santos deleted the feat/data-integrity-using-md5 branch December 16, 2022 12:34
@vasco-santos vasco-santos mentioned this pull request Dec 19, 2022
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

md5 / sha256 on replicator
2 participants