-
Notifications
You must be signed in to change notification settings - Fork 14k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run a separate celery cluster from the webserver? #3917
Comments
Could you please explain what do you mean? celery workers are by definition separated from gunicorn workers. |
From the documentation it seems the workers are running in the same servers as gunicorn. In prod environments is it recommended to setup the workers on different nodes, similar to airflow |
Yes, as they are otherwise going to be competing for CPU and other resources. In fact, you should probably also separate your Redis or AMQP setup that you're using as your Celery broker. Also, for production, be sure to stick an Nginx in front of your Gunicorns. |
Thank you that helps! |
It's designed as two different processes so that you can choose how to set things up in your environment. I recently set up 2 * |
We currently have ours on a autoscaling cluster of t2.medium's. Do you know in general what typically utilized more - cpu, memory etc.. |
We are not having a lot of luck with request going to the celery cluster. we used this in the superset_config for both the webserver cluster and the worker (celery) cluster. Does this look correct.
|
Did you set your database as "Allow Async"? |
Thanks for the help. |
Make sure these boxes are checked before submitting your issue - thank you!
This is a question -
would it be recommend to run a cluster of Celery workers separate from the web server cluster?
Superset version
0.20.1
Expected results
Actual results
Steps to reproduce
The text was updated successfully, but these errors were encountered: