Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Always use the logical processor count for system max cores #385

Merged
merged 1 commit into from
Jan 24, 2021

Conversation

clintval
Copy link
Member

I have an 8-core MacOS with hyperthreading (16 virtual cores total).

I'd like to tell my tasks to use them all!

Running ExamplePipeline before this PR:

[2021/01/23 22:40:21 | TaskManager | Info] Executing with 8.0 cores and 24g system memory.

And after:

[2021/01/23 22:45:31 | TaskManager | Info] Executing with 16.0 cores and 24g system memory.

Closes #370.

@nh13
Copy link
Member

nh13 commented Jan 24, 2021

@clintval thanks for this PR, let us think about if we ant cores to be the # of physical cores, or the # of virtual cores/threads.

@tfenne
Copy link
Member

tfenne commented Jan 24, 2021

I'm 👍 . I think I was the one who wanted physical cores originally, and back in the earlier days of hyper-threading it wasn't really clear to me how useful it was. However, nowadays my habit it to, e.g. run bwa on my mac with 12 threads, not 6. So I realized I'm doing this when running things manually.

If we wanted to we could always have a config setting for dagr.core_type=[logical|physical] but I don't think that's really necessary.

@nh13 nh13 merged commit 1547cf0 into fulcrumgenomics:master Jan 24, 2021
@clintval clintval deleted the cv_logical_cores branch January 24, 2021 20:43
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Why does Dagr use the physical processor count instead of logical for Macs only?
3 participants