Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Uploader Cloud Storage Interface #17042

Closed

Conversation

thomas-swirlds-labs
Copy link
Contributor

Description:

  • Create CloudBucketUploader interface
  • Implement MinioBucketUploader class which implements CloudBuckerUploader
  • Create RetryUtils helper class for standardized retry logic
  • Implement proper error handling and logging
  • Add comprehensive unit tests
  • Support both AWS S3 and GCP Cloud Storage
  • Implement MD5 hash validation

Related issue(s):

Fixes #16973

petreze and others added 18 commits December 10, 2024 14:53
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
…erface' into 16973-uploader-cloud-storage-interface

# Conflicts:
#	gradle/plugins/src/main/kotlin/com.hedera.gradle.jpms-modules.gradle.kts
…ion' into 16973-uploader-cloud-storage-interface
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
Signed-off-by: Thomas Moran <[email protected]>
Copy link
Collaborator

@jjohannes jjohannes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed only changes in build.gradle.kts and module-info.java files.

private String calculateMD5Hash(Path filePath) throws IOException, NoSuchAlgorithmException {
// Calculate MD5 hash
byte[] fileBytes = Files.readAllBytes(filePath); // Read the file content as bytes
MessageDigest md = MessageDigest.getInstance("MD5");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we can have the MessageDigest instance created once for a MinioBucketUploader?

.build());
return null;
},
maxRetryAttempts > 0 ? maxRetryAttempts : 3);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have the default set to 3 in BlockStreamConfig, do we need this conditional here?

BucketConfigurationManager bucketConfigurationManager,
@CommonExecutor ExecutorService executor,
ConfigProvider configProvider) {
List<CompleteBucketConfig> completeBucketConfigs = bucketConfigurationManager.getCompleteBucketConfigs();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why provide a list of bucket configs if below we always call .getFirst()?

@thomas-swirlds-labs
Copy link
Contributor Author

Closing as will not be implementing now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cloud Storage Interface Implementation
8 participants