Allow setting number of processes/threads through DRAMATIQ_NPROCS, DRAMATIQ_NTHREADS. #186
Conversation
| if default is None: | ||
| raise | ||
| msgf = "Invalid value for %s: %r. Reverting to default." | ||
| logging.warning(msgf, varname, envstr) |
There was a problem hiding this comment.
suggestion: i wonder if we should reraise the ValueError here with the warning message. WDYT?
There was a problem hiding this comment.
While I wouldn't object to (essentially) not handle ValueError, as a user I would expect a warning and some best-effort fallback instead of a hard failure.
This is common behaviour when it comes to values set through environment variables. E.g. VIM=/does/not/exist vim script.py will have vim complain about the non-existing directory, but the editor will open. Or with TERM=wtf ls, ls will complain but still run and assume a dumb terminal.
| # Number of processes to use. Default: one per CPU. | ||
| NPROCS = getenv_int("DRAMATIQ_NPROCS", default=multiprocessing.cpu_count) | ||
|
|
||
| # Number of threads per process to use. Default: 8. | ||
| NTHREADS = getenv_int("DRAMATIQ_NTHREADS", 8) |
There was a problem hiding this comment.
suggestion: Another way to do this is to use a setting, and then allow the user to set the setting themselves, rather than django-dramatiq grabbing it from the environment.
There was a problem hiding this comment.
This is an intentional choice. I have considered adding a setting, but I concluded that an environment variable would be the appropriate solution here, since the value that should be used depends on the deployment environment.
Adding a setting would be less convenient because application setting files are typically versioned (in scm) and reused across different deployment environments. A setting which is dictated by the deployment environment would create complexities for the users.
E.g. I run the same developer setup as everyone else in my team, but I happen to have less cores on my computer. Putting e.g. DRAMATIQ_NPROCS in django settings, means that we will have to now add custom code to properly set the value for everyone. And that custom code would probably involve checking some environment variable. Having django-dramatiq handle this out of the box is
Also, in containerized environments (k8s, docker-compose etc.) the preferred way of configuring resource-related settings is through environment variables, as you don't want to update/fork your application settings file for every clone of your setup.
One could of course add settings as an additional link in the fallback chain: cli argument -> environment -> application setting -> auto-guess default value. But this can always be done at a later time, in a separate PR. At this point, IMHO, it would only add extra complexity to the codebase without offering any additional convenience.
I hope this explains this design choice.
There was a problem hiding this comment.
Thanks for the explanation.
One could of course add settings as an additional link in the fallback chain: cli argument -> environment -> application setting -> auto-guess default value.
Lets get this in a see if the users are missing it.
Thanks for the feedback @andrewgy8. Please see comments above. |
| python manage.py rundramatiq | ||
| ``` | ||
|
|
||
| By default, `rundramatiq` will adjust the number of processes/threads used |
There was a problem hiding this comment.
thought: We really should get a separate documentation page...
|
Thanks a lot for the contribution! |
Use case: I use rundramatiq to start dramatiq in a docker-compose environment. However, since all my CPUs are detected by docker, I get an excessive number of workers which are slow to restart, but otherwise mostly idling.
With this PR, it is allowed to instruct
rundramatiqto use a specific number of processes, threads per process via theDRAMATIQ_NPROCS,DRAMATIQ_NTHREADSenvironment variables. These variables can be set e.g. in your .env file, where docker-compose will pick them up and pass them torundramatiq.This would also be useful in general when using
rundramatiqin any containerized environment, as it is usually preferred to configure software through the environment, rather than modifying the actual command line used by the container.