Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug fix on the S3OutputStream wrapper #56

Merged
merged 8 commits into from
Mar 16, 2022
Merged

Bug fix on the S3OutputStream wrapper #56

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9931be7
Un-Tested S3 document storage option
millmanw Mar 1, 2022
4030a63
Cleaning up the key generation
millmanw Mar 2, 2022
415099a
Updated to use the AWS SDK V2 corrected the improperly closing InputS…
millmanw Mar 3, 2022
54eac74
Updated the getAssociatedDocument to wrap the S3 ResponseInputStream …
millmanw Mar 4, 2022
1c740b5
Merge branch 'main' of https://github.com/zuliaio/zuliasearch into S3…
millmanw Mar 4, 2022
a9d691f
Merge pull request #2 from millmanw/S3StorageOption
millmanw Mar 14, 2022
daf125a
Merge pull request #3 from zuliaio/main
millmanw Mar 14, 2022
7604edc
Bug fix to properly upload files that are less than the buffer size
millmanw Mar 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions zulia-server/src/main/java/io/zulia/server/filestorage/io/S3OutputStream.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,22 @@
package io.zulia.server.filestorage.io;

import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.AbortMultipartUploadRequest;
import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest;
import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload;
import software.amazon.awssdk.services.s3.model.CompletedPart;
import software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest;
import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.UploadPartRequest;
import software.amazon.awssdk.services.s3.model.UploadPartResponse;

import java.io.ByteArrayInputStream;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;

import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;

/**
* OutputStream which wraps S3Client, with support for streaming large files directly to S3
*
Expand Down Expand Up @@ -141,8 +148,8 @@ public void close() {
this.s3Client.completeMultipartUpload(cmur);
}
else {
PutObjectRequest req = PutObjectRequest.builder().bucket(bucket).key(key).contentLength((long) this.buf.length).build();
s3Client.putObject(req, RequestBody.fromBytes(this.buf));
PutObjectRequest req = PutObjectRequest.builder().bucket(bucket).key(key).contentLength((long) this.position).build();
s3Client.putObject(req, RequestBody.fromInputStream(new ByteArrayInputStream(buf, 0, this.position), this.position));
}
}
}
Expand Down