Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prioritization or somewhat controlled ordering of submitted tasks? #291

Open
josephschorr opened this issue May 30, 2021 · 4 comments · May be fixed by #292
Open

Prioritization or somewhat controlled ordering of submitted tasks? #291

josephschorr opened this issue May 30, 2021 · 4 comments · May be fixed by #292

Comments

@josephschorr
Copy link

I'd first like to thank the maintainers of the library: it has solved a number of deadlocks and other issues I encountered while attempting to run multiprocess code

I'm making use of loky as part of an engine which has jobs submitted to it at random times: these jobs are sharded to run as tasks in parallel via loky, thus making maximum use of available CPU resources whenever possible. However, currently, the order in which tasks submitted to loky runs appears to be fixed, which can result in a later job being "starved"/blocked until a sufficient set of submitted tasks for the former job(s) has completed. While I can manually mitigate this somewhat by reducing the use of available processes for a specific job, this leaves CPU cores unused. Is there a known way to indicate to loky that certain submitted tasks have a higher priority, or to perhaps submit tasks to be placed at the front of the queue, rather than the back? My brief delve into the Queue implementation within loky suggests the answer is "no" (as then it would be a stack, rather than a queue), but I wanted to confirm.
@tomMoral
Copy link
Collaborator

Hello and thanks for the kind words for loky :)

For now, loky only provides a fix ordering for the task indeed.
This part is mainly due to the use of a queue.Queue when we submit a new task.

However, I don't think this is very complex to change this behavior. Indeed, when a job is submitted, it is put in a local _work_ids queue with put. Then when there is some free space for computations, a task is fetch from this queue (get(block=False), throws queue.Empty error when empty) and send to the pool of workers (through a queue, this one cannot be changed but I don't think this matter for what you describe).
If you change the work_ids queue with another structure that implements these two methods (put/get(block=False)) I don't see any major issuers. This would change the ordering with which the task are run. As this is local, there should be no concurrence issues.
If you implement such strategy, I would be happy to see the result.

@josephschorr
Copy link
Author

@tomMoral I'll definitely take a stab at implementing that then. Will send a PR if I get it working :)

@josephschorr josephschorr linked a pull request May 31, 2021 that will close this issue
Open
@josephschorr
Copy link
Author

@tomMoral #292. Let me know if I made any bad assumptions or didn't follow a style somewhere

@josephschorr
Copy link
Author

@tomMoral Let me know if you need me to make any changes on #292 in order to get it in

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add submit_high_priority for high priority jobs josephschorr/loky
2 participants
@tomMoral @josephschorr