Skip to content

Commit

Permalink
Disable streaming upload for MinIO failure recovery tests
Browse files Browse the repository at this point in the history
Streaming upload to S3 allocates a 16MB buffer (by default) for each
output stream. Failure recovery tests create a table partitioned into
~60 partitions. Since for each partition at least one file
must be created the engine has to allocate ~1GB of buffer space. These
buffer allocations push the memory reservation beyond the maximum heap
size.
  • Loading branch information
arhimondr authored and martint committed Jan 21, 2022
1 parent 0d9a512 commit dab2fae
Showing 1 changed file with 6 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ protected QueryRunner createQueryRunner(List<TpchTable<?>> requiredTpchTables, M
.setInitialTables(requiredTpchTables)
.setExtraProperties(configProperties)
.setCoordinatorProperties(coordinatorProperties)
.setHiveProperties(ImmutableMap.<String, String>builder()
// Streaming upload allocates non trivial amount of memory for buffering (16MB per output file by default).
// When streaming upload is enabled insert into a table with high number of buckets / partitions may cause
// the tests to run out of memory as the buffer space is eagerly allocated for each output file.
.put("hive.s3.streaming.enabled", "false")
.build())
.build();
}

Expand Down

0 comments on commit dab2fae

Please sign in to comment.