[QUESTION] Not understanding the web concurrency going on about uvicorn-gunicorn-docker

[NixBiks]

Thanks for the feedback. It makes sense!

[tiangolo]

So, if you have several def functions that only wait there, they will block the thread they are in. And if you will still have a limited amount of threads, and if all of them are handling a request that is blocking with a sleep, then that will end up blocking the concurrency.

If you use async and asyncio.sleep, that "sleeps", but doesn't block and lets the rest of the code continue executing, without holding the resources. Because it is not really using the CPU, just waiting for something else.

But if you are doing something that is CPU intensive, like training a model, then you should probably still use def so that it runs in threadpool and doesn't block the rest, but if you run several of those at the same time, there will be a point where all your CPUs are used in full. And then it doesn't matter which tricks you use, your machine won't be able to handle more concurrency, just because all the CPU available is used.

Now, if that's the case and you are using something that is CPU intensive, if your use case is simple enough, and you expect a small concurrency, let's say, it's an internal tool used only by a small internal team, then it might be OK to just do it all internally in your FastAPI app.

But for anything bigger than that, you will probably end up needing to have more machines dedicating to training the models (or whatever is the slow resource-intensive task). And then your API would just receive requests and report the status of the training of those tasks, but the API code wouldn't be in charge of actually training the models.

Then, the way you would coordinate those different machines (or at least separate worker processes) would be with a tool like Celery or ARQ.

If you check the project generators, those have Celery setup and included by default.

If you want to try something new, I think ARQ is probably a very good option.