Setting up flower with docker-compose for multiple celery workers?

[dstromberg]

Hi.

I know celery can do multiple file.py's of tasks, and we're doing that, but we still want more than one celery worker for greater parallelism - more than we have cores in any given machine (VM).

And we're hoping to monitor each celery worker (two in our development environments, possibly more in production) with a single flower instance - ISTR hearing that flower is good at that? Or was?

I'm attempting to fire up flower in docker-compose with:

  flower:
    # build: ./
    image: mher/flower
    container_name: flower
    command: celery --app=service.run:celery flower
    volumes:
      # - ./examples:/data
      - ${CODE}/services/api:/data
    working_dir: /data
    ports:
      - 5555:5555
    environment:
      CELERY_BROKER_URL: amqp://username:password@rabbitmq:5672/tact-dev
      CELERY_RESULT_BACKEND: db+mysql+pymysql://username:password@pub_mysql:3306/celery_dev
    depends_on:
      - celery_worker
      - celery_worker2
      - mysql
      - rabbitmq

...but it seems to be tracebacking on a dependency that our celery workers need. I don't really know why flower would care about that, but I have next to no flower experience (and not a lot more celery experience).

Does flower have to be set up with a complete celery worker to function properly?

Can a single flower monitor multiple celery worker hosts/containers?

Is there a way to keep flower from tracebacking on dependencies I'm not sure it should need, short of actually installing those dependencies in the flower container?

Thanks! I'm eager to see flower in action.

[dstromberg]

It looks like my question is also in this not-100%-answered Stack Overflow question:
https://stackoverflow.com/questions/71503906/celery-flower-with-multiple-workers-in-different-docker-containers

Sort of.

[dstromberg]

I just needed "celery flower" instead of "celery --app whatever flower".

Here's the relevant portion of my docker-compose.yml:

  flower:
    image: mher/flower
    container_name: flower
    command: celery flower
    working_dir: /data
    ports:
      - 5555:5555
    environment:
      CELERY_BROKER_URL: amqp://${RABBIT_USER}:${RABBIT_PASSWORD}@${RABBIT_HOST}:5672/${RABBIT_VHOST}
      CELERY_RESULT_BACKEND: db+mysql+pymysql://${DB_USERNAME}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/celery_dev
    depends_on:
      - celery_worker
      - celery_worker2
      - mysql
      - rabbitmq