based-jace / concurrency-parallelism-and-asyncio

Blog post for testdriven.io

Concurrency, Parallelism, and asyncio

Blog post for testdriven.io. If you want to follow along and create the projects yourself, you can find the post here.

Code Examples

Create and activate a new virtual environment:

Windows Powershell, assuming the Python version in your PATH is 3.9+:

PS X:> python -m venv venv
PS X:> .\venv\Scripts\Activate.ps1
(venv) PS X:>

(venv) PS X:> pip install -r requirements.txt

Mac/Linux:

$ python3.9 -m venv venv
$ source venv/bin/activate
(venv)$

(venv)$ pip install -r requirements.txt

Concurrency

(venv)$ python code_examples/concurrency/sync.py
(venv)$ python code_examples/concurrency/threads.py
(venv)$ python code_examples/concurrency/async.py

Parallelism

(venv)$ python code_examples/parallelism/sync.py
(venv)$ python code_examples/parallelism/threads.py
(venv)$ python code_examples/parallelism/multi.py

pytest async

(venv)$ python -m pytest code_examples/pytest_asyncio/test_hello_asyncio.py
(venv)$ python -m pytest code_examples/pytest_asyncio/test_hello_asyncio2.py

asyncio and multiprocessing

(venv)$ python code_examples/asyncio_and_multiprocessing/sync.py
(venv)$ python code_examples/asyncio_and_multiprocessing/multiprocessing_only.py
(venv)$ python code_examples/asyncio_and_multiprocessing/asyncio_only.py
(venv)$ python code_examples/asyncio_and_multiprocessing/asyncio_with_multiprocessing.py

concurrency-parallelism-and-asyncio's Issues

Use of for bucle in asyncio_with_multiprocessing.py

Why are you using a for bucle to iterate over the num_pages in the get_and_scrap_pages function?

[...]
    async with \
    aiohttp.ClientSession() as client, \
    aiofiles.open(output_file, "a+", encoding="utf-8") as f:

        for _ in range(num_pages):
            async with client.get("https://en.wikipedia.org/wiki/Special:Random") as `response:`
[...]

As I understand it, by doing this you are not applying concurrency to the web calls and they are not being launched in parallel. Would it be better to remove the loop and modify the function to access a single page? By doing this we could call the function n_pages times and run all the requests in parallel inside each core using asyncio.gather(), as in the asyncio_only.py code.

Is my approach correct? Would it be faster to do it this way?

Thank you!