Add segment partitioning #23386
Add segment partitioning #23386
Conversation
| @ConfigDescription("Number of storage partitions used") | ||
| @Config("fs.segment.partitions") | ||
| public FileSystemSpoolingConfig setPartitions(int partitions) |
There was a problem hiding this comment.
I think naming this partition is a bit misleading. It is introduced to bump number of "partitions" AWS S3 uses for spooled files, but it is not 1-1 mapping, also all of that bases on undocumented AWS S3 behavior, which we cannot validate.
Two suggestions:
-
I would make this S3 specific. As we have no data (at least I do not have) to believe that it benefits other filesystems. And even if we need similar mechanism for other providers it may require different implemenation
-
Maybe let's be explicit in what this is doing, instead pretending we know that it will bump number of S3 partitions to some exact value. Maybe better name for property would be something arund "spooling.file.prefix.entropy" ?
|
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
No description provided.