benedikt-budig / mure

Perform multiple HTTP requests in parallel – without writing boilerplate code or worrying about async functions.

mure

This is a thin layer on top of aiohttp to perform multiple HTTP requests in parallel – without writing boilerplate code or worrying about async functions.

mure means multiple requests, but is also the German term for a form of mass wasting involving fast-moving flow of debris and dirt that has become liquified by the addition of water.

Install the latest stable version from PyPI:

pip install mure

Usage

Pass an iterable of dictionaries with at least a value for url and get a generator with the responses:

>>> import mure
>>> resources = [
...     {"url": "https://httpbin.org/get"},
...     {"url": "https://httpbin.org/get", "params": {"foo": "bar"}},
...     {"url": "invalid"},
... ]
>>> responses = mure.get(resources, batch_size=2)
>>> responses
<generator object get at 0x7fc2a9aae420>
>>> for resource, response in zip(resources, responses):
...     print(resource, "status code:", response.status)
...
{'url': 'https://httpbin.org/get'} status code: 200
{'url': 'https://httpbin.org/get', 'params': {'foo': 'bar'}} status code: 200
{'url': 'invalid'} status code: 0

The keyword argument batch_size defines the number of requests to perform in parallel (don't be too greedy).

There is also a convenience function for POST requests:

>>> resources = [
...     {"url": "https://httpbin.org/post"},
...     {"url": "https://httpbin.org/post", "json": {"foo": "bar"}},
...     {"url": "invalid"},
... ]
>>> responses = mure.post(resources)

You can even mix HTTP methods in the list of resources (but have to specify the method for each resource):

>>> resources = [
...     {"method": "GET", "url": "https://httpbin.org/get"},
...     {"method": "GET", "url": "https://httpbin.org/get", "params": {"foo": "bar"}},
...     {"method": "POST", "url": "https://httpbin.org/post"},
...     {"method": "POST", "url": "https://httpbin.org/post", "json": {"foo": "bar"}},
...     {"method": "GET", "url": "invalid"},
... ]
>>> responses = mure.request(resources)