gregavrbancic / fastapi-celery

Minimal example utilizing fastapi and celery with RabbitMQ for task queue, Redis for celery backend and flower for monitoring the celery tasks.

Hi 👋, I'm Grega

A passionate deep learning entusiast and data scientist working as a Assistant Professor at the University of Maribor, Slovenia.

• 🔭 in free time I am working on NiaPy

• 👨‍💻 All of my projects are available at https://github.com/GregaVrbancic?tab=repositories&q=&type=source

• 📝 I sometimes write articles on https://grega.xyz/

• 💬 Ask me about deep learning and nature inspired population based metaheuristics.

• 📫 How to reach me [email protected]

🙌 You can support my work with donations 🙌

• Bitcoin: 356y5uzyTsrXBGUGJWhU4fetcQKHKHW8YA
• bitcoincash:pq4urptppsc6g7uxw2tpggqvr3482z4lts3p66xuxr
• Lumen: GA5XIGA5C7QTPTWXQHY6MCJRMTRZDOSHR6EFIBNDQTCQHG262N4GGKTM
• ZCash: zs18a47ucg2hpg8tprejvxkmtmghr288eqe67snlm6wnsm3rd3kp6lml3tleh74tx5veur2gk7c8hl

fastapi-celery's Issues

No jobs work in AWS ubuntu server 20.04

Hi.
I cloned on my local machine and tested and everything worked fine with Docker Desktop v4.12.0 in W10.
Logging into AWS and cloning and running docker-compose up everything seems to work but the jobs never get out of PENDING state.

I found this anomaly in the logs:

It does not appear locally with docker desktop.

Any ideas how to debug this?

Celery and FastApi containers not working

ERROR:

packages/kombu/utils/compat.py", line 82, in entrypoints
    for ep in importlib_metadata.entry_points().get(namespace, [])
AttributeError: 'EntryPoints' object has no attribute 'get'

Containerize the whole project

Hi, thank you for the great tutorial.

Do you have any idea how we could completely containerize the project? I'm struggling with the docker-compose.yml, trying to containerize the API as well as the worker. Doing as in the file below doesn't run neither the API nor the worker.

version: "2"

services:
  rabbitmq:
    image: "bitnami/rabbitmq:3.7"
    ports:
      - "4369:4369"
      - "5672:5672"
      - "25672:25672"
      - "15672:15672"
    volumes:
      - "rabbitmq_data:/bitnami"

  redis:
    image: "bitnami/redis:5.0.4"
    environment:
      - REDIS_PASSWORD=password123
    ports:
      - "6379:6379"
    volumes:
      - "redis_data:/bitnami/redis/data"

  celery-flower:
    image: gregsi/latest-celery-flower-docker:latest
    environment:
      - AMQP_USERNAME=user
      - AMQP_PASSWORD=bitnami
      - AMQP_ADMIN_USERNAME=user
      - AMQP_ADMIN_PASSWORD=bitnami
      - AMQP_HOST=rabbitmq
      - AMQP_PORT=5672
      - AMQP_ADMIN_HOST=rabbitmq
      - AMQP_ADMIN_PORT=15672
      - FLOWER_BASIC_AUTH=user:test
    ports:
      - "5555:5555"

  api:
    build:
      context: src/api_dir
      dockerfile: Dockerfile
    ports:
      - "8000:8000"
    depends_on:
      - redis
    restart: always
    volumes:
      - "public:/app/public"

  worker:
    build:
      context: src/api_dir
    command: celery worker -A path.to.worker -l info -Q queue -c 2
    depends_on:
      - redis
    volumes:
      - "public:/app/public"

volumes:
  rabbitmq_data:
    driver: local
  redis_data:
    driver: local
  public:
    driver: local

关于send_task(task_name)的问题

我在启动时
celery -A example worker -l info -P eventlet
我在源码中print注册的task name，他是以example.xxx形式注册
和你在send_task(task_name)的形式并不相同，请问你是如何启动celery的能以app.celery_app.woker.exmple的形式send_task

Increase the number of workers

How can I increase the number of workers in celery?

docker-compose

Hi,
thanks for sharing,
can u explain why there are 2 docker-compose files?