Skip to content

Commit

Permalink
Drop ParallelIterable's queue low water mark (#10979)
Browse files Browse the repository at this point in the history
As part of the change in commit
7831a8d, queue low water mark was
introduced. However, it resulted in increased number of manifests being
read when planning LIMIT queries in Trino Iceberg connector. To avoid
increased I/O, back out the change for now.
  • Loading branch information
findepi authored Aug 21, 2024
1 parent e18a2fe commit ed53c6d
Showing 1 changed file with 3 additions and 6 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -192,12 +192,9 @@ public synchronized boolean hasNext() {
// If the consumer is processing records more slowly than the producers, the producers will
// eventually fill the queue and yield, returning continuations. Continuations and new tasks
// are started by checkTasks(). The check here prevents us from restarting continuations or
// starting new tasks too early (when queue is almost full) or too late (when queue is already
// emptied). Restarting too early would lead to tasks yielding very quickly (CPU waste on
// scheduling). Restarting too late would mean the consumer may need to wait for the tasks
// to produce new items. A consumer slower than producers shouldn't need to wait.
int queueLowWaterMark = maxQueueSize / 2;
if (queue.size() > queueLowWaterMark) {
// starting new tasks before the queue is emptied. Restarting too early would lead to tasks
// yielding very quickly (CPU waste on scheduling).
if (!queue.isEmpty()) {
return true;
}

Expand Down

0 comments on commit ed53c6d

Please sign in to comment.