You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#10710 showed that we are lacking test coverage for S3 streaming upload.
We should write testcases which upload files of multiple sizes with S3 streaming enabled (default currently). Specifically
For tests which talk to real S3 endpoint on AWS we should extend AbstractTestHiveFileSystemS3 (this test is run for master and for PRs originated from trinodb/trino).
AbstractTestHiveFileSystemS3 is a low-level test in which we directly create PageSink etc. To have more blackbox coverage we can also write an integration test which uses MiniIO based QueryRunner and consumes SQL statements to trigger writing.
For that BaseTestHiveOnDataLake can be extended.
Extra guards in TrinoS3FileSystem
Additionally it would be nice to add extra guards to TrinoS3FileSystem filesystem which ensure that close() would not call finishUpload if file upload is alredy considered completed; when code path which uploads small files without multipart mechanism
is executed. Having such guard would prevent #10710 from happening.
Extra testing
#10710 showed that we are lacking test coverage for S3 streaming upload.
We should write testcases which upload files of multiple sizes with S3 streaming enabled (default currently). Specifically
For tests which talk to real S3 endpoint on AWS we should extend
AbstractTestHiveFileSystemS3
(this test is run formaster
and for PRs originated from trinodb/trino).AbstractTestHiveFileSystemS3
is a low-level test in which we directly create PageSink etc. To have more blackbox coverage we can also write an integration test which uses MiniIO based QueryRunner and consumes SQL statements to trigger writing.For that
BaseTestHiveOnDataLake
can be extended.Extra guards in TrinoS3FileSystem
Additionally it would be nice to add extra guards to
TrinoS3FileSystem
filesystem which ensure thatclose()
would not callfinishUpload
if file upload is alredy considered completed; when code path which uploads small files without multipart mechanismis executed. Having such guard would prevent #10710 from happening.
cc: @linzebing @findepi @joshthoward
The text was updated successfully, but these errors were encountered: