Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: destroy issue and use maximum workers #44

Merged
merged 7 commits into from
Jan 26, 2023
Merged

fix: destroy issue and use maximum workers #44

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
25cc93b
use maximum workers for isolate
Aslemammad Jan 13, 2023
26f58ae
WIP: cannot reproduce the halt issue
Aslemammad Jan 13, 2023
c3bccaa
fix destroy issue
Aslemammad Jan 15, 2023
ae42cb2
trace warning
Aslemammad Jan 15, 2023
0a60bdd
update
Aslemammad Jan 15, 2023
34a204d
WIP:_ensureEnouhWorkersForTaskQueue
Aslemammad Jan 15, 2023
2e304d0
update test
Aslemammad Jan 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -602,6 +602,14 @@ class ThreadPool {
this._ensureMinimumWorkers()
this.startingUp = false
}
_ensureEnoughWorkersForTaskQueue(): void {
while (
this.workers.size < this.taskQueue.size &&
this.workers.size < this.options.maxThreads
) {
this._addNewWorker()
}
}
Comment on lines +605 to +612
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@overlookmotel Do you think this solution is an optimal one?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it's not wrong to have workers for those that already have since we do not have info! And I think on the second condition, it creates a limit for the workers and can be helpful. I'd like to see your opinion.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@overlookmotel Let me know if you're there.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm really sorry but I can't help more. As I mentioned on #42, I am very tight for time. The complexity (which we're discussing here) was exactly why I didn't feel I could make a PR myself. I find piscina's code pretty hard to disentangle.

But my guess is that your solution above is not quite right, and will create new workers when none is actually needed in some case.

My suggestion would be to look at the existing code for how it's handled when a thread is terminated due to use of an AbortController. That results in same situation (worker removed from pool, do we need to add a new one?) so I imagine the code for that contains a good solution.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure! ok, thank you so much! I'll investigate it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In AbortController the ensureMin approach is used which is not proper for our case!
I think let's stick with the _ensureEnoughWorkersForTaskQueue approach for now and let's see how it goes! Thanks for yout help.


_ensureMaximumWorkers(): void {
while (this.workers.size < this.options.maxThreads) {
Expand Down Expand Up @@ -844,7 +852,7 @@ class ThreadPool {
// When `isolateWorkers` is enabled, remove the worker after task is finished
if (this.options.isolateWorkers && taskInfo.workerInfo) {
this._removeWorker(taskInfo.workerInfo)
this._ensureMaximumWorkers()
this._ensureEnoughWorkersForTaskQueue()
}
},
signal,
Expand Down