Python asyncio bindings about foundationdb

[kurlova]

Hi, are there any updates on Python asyncio support for FoundationDB? For Python3.7 and foundationdb==6.1.8:

$ python -V
Python 3.7.3
$ python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import fdb
>>> fdb.api_version(610)
>>> fdb.open(event_model='asyncio')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/lib/python3.7/site-packages/fdb/impl.py", line 1663, in open
    init(event_model=event_model)
  File "/venv/lib/python3.7/site-packages/fdb/impl.py", line 1587, in init
    if isinstance(asyncio.futures._FUTURE_CLASSES, type):
AttributeError: module 'asyncio.futures' has no attribute '_FUTURE_CLASSES'

Would you recommend to use @amirouche's way with using found in production, or are there any other options?

[alecgrieser]

Interesting. Well, that does seem like a bug. We'd certainly be open to bug fixes that address limitations specific to 3.5 or 3.6. The Python 3 bindings should work for the most part already, I believe, so I don't think it would be that much work. The real work would be to test it well, though maybe there is some low-hanging fruit.

[alecgrieser]

You can set an event model for asynchronous tasks in the Python bindings by setting the event_model parameter of the open method on start up: https://apple.github.io/foundationdb/api-python.html#opening-a-database

We don’t document which event models are supported at the current time (probably something to fix), but “asyncio” and “gevent” are both supported.

[asgoel]

Are there any plans to upgrade support to 3.5 and/or 3.6? IIRC, the asyncio API was improved a decent amount from 3.4 -> 3.5, including support for the async/await syntax

[asgoel]

Actually, it doesn't even seem like the current asyncio bindings work with python 3.4? Let me know if I did something wrong here:

Python 3.4.6 (default, Apr 20 2018, 08:24:52) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fdb
>>> fdb.api_version(510)
>>> fdb.open(event_model="asyncio")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ashu/Documents/scratch/3.4/foundationdb-5.1.5/fdb/impl.py", line 1668, in open
    init(event_model=event_model)
  File "/home/ashu/Documents/scratch/3.4/foundationdb-5.1.5/fdb/impl.py", line 1591, in init
    if isinstance(asyncio.futures._FUTURE_CLASSES, type):
AttributeError: 'module' object has no attribute '_FUTURE_CLASSES'
>>> 

[asgoel]

Fair enough, though I guess my point was that this bug is present even in python 3.4, which is listed as supported by the binding.

[som3on3]

python3 setup.py clean
python3 setup.py build
python3 setup.py install

(worked for me)

[davidscherer]

I think the real question is, in what event_model(s) does Apple run the binding tester? Because if it's not tested, it's probably not working.

[ajbeamon]

@davidscherer The tester runs only the default one.

[amirouche]

I had a look at the code of the python bindings. At first, it seems like there support for event_model=asyncio but with PyPy 3.5 it doesn't work. Maybe it works with CPython 2.7 and trollius, but trollius is deprecated.

I would like to work on asyncio bindings, I plan to drop Python 2.7 support and Python 3 prior to 3.5. That is I plan to only support async/await syntax which was introduced in CPython 3.5 by PEP 492.

There is still something that troubles me: fdb.api_version. The issue with the current code is that it break 'autocompletion' (in emacs using elpy) because of monkey patching that happens in api_version. I think it's not possible to have autocompletion without avoiding monkeypatching because autocompletion works using static analysis and inference and that at least the elpy jedi backend is not smart enough.

My understanding is that api_version allows to upgrade the C client library and take advantage of it while using an old API. I am fine with that. That being, said, that is a problem only if the C client library is distributed along the Python client library. That is, if Python and C clients can be installed separately, one can upgrade them separately. WDYT?

Also, it seems to me that PyPy prefers cffi, do you think it's ok to build the asyncio bindings using cffi?

[alecgrieser]

Is the plan to have your own version of the python bindings that supports asyncio using async/await or to contribute a PR back that uses the new syntax. I don't believe that we are willing to drop Python 2 at this time.

The API version actually does a few different things. One of them, as suggested, is it allows the bindings to communicate their version to the C client library, and it is exactly that that allows the user to upgrade the C library before upgrading the Python code. But it also allows users of the bindings to upgrade the bindings without having to upgrade their client code with the contract being that all breaking changes will be guarded by API version changes. For example, prior to API version 500, the tuple layer did not support distinct boolean types, so running fdb.tuple.pack((True,)) returned the same result as fdb.tuple.pack((1,)). Starting at API version 500, fdb.tuple.pack((True,)) now returns a separate byte sequence that distinguishes booleans from integers. But the 5.1 version of the Python bindings supports both (depending on what API version is being used).

I think that you are right in the general case that auto-completion and monkey patching don't play well (unless you can predict statically which functions will end up being loaded, which sounds hard in the general case). This is somewhat speculative that something could be done where certain functions are loaded with "dummy" implementations that get loaded in manually. This won't work for newly added functions, but maybe the minimum set of all supported functions over all API versions supported by the bindings would get you most of the way there?

[amirouche]

Is the plan to have your own version of the python bindings that supports asyncio using async/await or to contribute a PR back that uses the new syntax.

Yes. Here is the repository

The API version actually does a few different things.

Nevermind, I will do my thing. I will figure it out later. I have created this issue to track it.

[amirouche]

FWIW, I started the bindings. If you want to help I made a small things of things to do. I am more interested in building layers so I might not implement everything this week, but I will respond to feature requests.

[amirouche]

There is an easier path that will avoid duplicating efforts to support Python asyncio. One can use the official bindings via loop.run_in_executor.

Here is an example program:

import asyncio
import random

import fdb


loop = asyncio.get_event_loop()


fdb.api_version(520)
db = fdb.open()


def random_bytes(count):
    return bytes([random.getrandbits(8) for _ in range(count)])


@fdb.transactional
def set(tr):
    key = b'\x00' + random_bytes(256)
    value = random_bytes(1024)
    tr.set(key, value)


del db[b'':b'0xff']



async def main():
    concurrency = 1000
    steps = int(10**4 / concurrency)
    for _ in range(steps):
        coroutines = [loop.run_in_executor(None, set, db) for _ in range(concurrency)]
        await asyncio.gather(*coroutines)


loop.run_until_complete(main())

[JendaPlhak]

@amirouche We are currently using the foundationDB with executors in asyncio, just as you described, but it's quite annoying and it would be good both performance and UX wise if FoundationDB had native asyncio support.

[amirouche]

@JendaPlhak It is not fully tested but I made a release at pypi:

In [1]: import asyncio

In [2]: from found import v510 as fdb

In [3]: loop = asyncio.get_event_loop()

In [4]: db = loop.run_until_complete(fdb.open())

In [5]: tr = db._create_transaction()

In [6]: loop.run_until_complete(tr.get(b'hello'))

In [7]: tr.set(b'hello', b'world')

In [8]: loop.run_until_complete(tr.get(b'hello'))
Out[8]: b'world'

Feature requests welcome! It's tested with python 3.5, 3.6 and 3.7

[JendaPlhak]

@amirouche Wow! Very cool. I wanted to check it out, but the link you provided gives me 404 :-[ What's the status on this anyway? Are you planning on creating a merge request to get it live?

[amirouche]

@JendaPlhak the correct link is https://pypi.org/project/asyncio-foundationdb/#description I updated the link. The pypi name is asyncio-foundationdb and the python package is found

[amirouche]

Are you planning on creating a merge request to get it live?

I think it requires more features and more testing.

[timephy]

Hey guys,
@amirouche Good work on found!

Does anyone know exactly why the bindings of the official python package don't work with asyncio?

Also, what is the status of found, is it going to be merged? How's the testing going?

[amirouche]

Good work on found!

Tx!

Does anyone know exactly why the bindings of the official python package don't work with asyncio?

They should work with python before 3.5, before the async and await syntax was introduced in CPython. I did not test, tho.

what is the status of found

Working so far, check out test coverage: https://codecov.io/gh/amirouche/asyncio-foundationdb/tree/master/found and TODO.

Is it going to be merged?

I don't know.

How's the testing going?

I need to check there is no memory leak, I did not have time to do it.

[amirouche]

Here is the forum thread if you want to discuss matters related to found.

[ozankabak]

FYI, we hit the above issue today.

[ja8zyjits]

Hey guys, any updates yet?
It will be really good if we could have native asyncio support in this package.

[amirouche]

Would you recommend to use @amirouche's way with using found in production,

I do not want to be rude, but I need to rehash a basic idea from open-source: through collaboration we can produce better software.

Ok, I have been busy with Scheme bindings, but I am restarting the effort with python asyncio bindings. During two years, I got zero code contributions but the code was good enough to be forked privately. That is a true, and sad story.