matrix-nio / matrix-nio

A Python Matrix client library, designed according to sans I/O (http://sans-io.readthedocs.io/) principles

nio

nio is a multilayered Matrix client library. The underlying base layer doesn't do any network IO on its own, but on top of that is a full-fledged batteries-included asyncio layer using aiohttp. File IO is only done if you enable end-to-end encryption (E2EE).

Documentation

The full API documentation for nio can be found at https://matrix-nio.readthedocs.io

Features

nio has most of the features you'd expect in a Matrix library, but it's still a work in progress.

• ✅ transparent end-to-end encryption (EE2E)
• ✅ encrypted file uploads & downloads
• ✅ space parents/children
• ✅ manual and emoji verification
• ✅ custom authentication types
• ✅ threading support
• ✅ well-integrated type system
• ✅ knocking, kick, ban and unban
• ✅ typing notifications
• ✅ message redaction
• ✅ token based login
• ✅ user registration
• ✅ read receipts
• ✅ live syncing
• ✅ m.reactions
• ✅ m.tags
• ❌ cross-signing support
• ❌ server-side key backups (room key backup, "Secure Backup")
• ❌ user deactivation (#112)
• ❌ in-room emoji verification

Installation

To install nio, simply use pip:

$ pip install matrix-nio

Note that this installs nio without end-to-end encryption support. For e2ee support, python-olm is needed which requires the libolm C library (version 3.x). On Debian and Ubuntu one can use apt-get to install package libolm-dev. On Fedora one can use dnf to install package libolm-devel. On MacOS one can use brew to install package libolm. Make sure version 3 is installed.

After libolm has been installed, the e2ee enabled version of nio can be installed using pip:

$ pip install matrix-nio[e2e]

Additionally, a docker image with the e2ee enabled version of nio is provided in the docker/ directory.

Examples

For examples of how to use nio, and how others are using it, read the docs

matrix-nio's Issues

How to write a client

Hello:

I am trying to write a client using the matrix-nio. I have the client code found at the link https://matrix-nio.readthedocs.io/ en/latest/ going. However I want to move beyond this example. Is there a more developed piece of functioning client code available that I can run and learn from?

thank you.

Key queries don't use the users_for_key_query user set.

Key queries are currently done for all users that are sharing an encrypted room with our own user.

Not only is this highly inefficient but this can lead to LocalProtocol errors when it shouldn't. The logic for making more intelligent key queries should be in place in the Olm class, namely the users_for_key_query set which also determines if a key query is necessary.

The logic needs to be double checked, possibly some test should be written and the HttpClient and AsyncClient should just get the users for a key query from the users_for_key_query attribute.

OlmTrustError doesn't have an easy way to get the unverified devices.

The unverified devices should be part of the OlmTrustError so users can do something else besides printing out a string.

Document what needs to be done so the crytpo store loads an existing account.

Documentation formatting error

See https://matrix-nio.readthedocs.io/en/latest/nio.html#nio.events.room_events.RoomCreateEvent.room_version

room_version

The version of the room. Different room versions

Type: | str

will have different event formats. Clients shouldn't worry about this
too much unless they want to perform room upgrades.

Probably due to some missing continuation indent here:
https://github.com/poljar/matrix-nio/blob/7c9b05f923b3fe8d7bdaafa4b118e988e6a50a41/nio/events/room_events.py#L565-L574

The client should check upon class instantiation if a store path without a store class is given.

logout missing

A full client needs the logout. I am just implementing it but there are some chicken/egg problems:

With logout you loose the device info and (Async)Client shouldn't exist anymore after a successful call. So the client object needs to kill itself.

The call of logout would be simple for me but with the object de-instanciation I get problems to get the whole thing into the state as before calling AsyncClient()

Add user registration and deactivation

I don't see any functions for either in the documentation.

questions about olm install

hey there! I'd like to check out this project, above all for its E2EE capabilities, but I cannot seem to be able to satisfy the libolm dependency.
are you able to provide any help?
I was expecting to be able to install using apt or, at worst, pip, yet I have failed miserably.

are you able to provide a best-case scenario for installing everything on a vanilla ubuntu install? :)

Pass access_token as Authorization Bearer token?

Query parameters tend to get logged along with the rest of the URL, so probably prudent to pass the access_token via headers instead.

async client never stores sync token

base client calls

self.store.save_sync_token(self.next_batch)

async client calls this never.
So using store_sync_tokens=True at config does not have any impact on async client

AsyncClient download, thumbnail... make timeout optional

It would be nice to have an optional timeout setting for AsyncClient download, thumbnail...
As I understand it is expected that the end user should cancel a download. If the client itself needs e.g. to download icons of images/thumbnails there should be an optional timeout setting so that such systems do not hang when e.g. offline.

Something like

timeout: Optional[int] = 0

in the function's definitions (and using it in self._send calls) should do the work.

nio.exceptions.LocalProtocolError: Not logged in.

Hi! I created a communication bot based on the Matrix-nio protocol in python, everything was working fine until yesterday, it displays the following error message:

  File "main.py", line 88, in <module>
    loop.run_until_complete(main())
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "main.py", line 54, in main
    sync_response = await async_client.sync(3000)
  File "/home/vagrant/Documents/matrix-python-nio/env_python/lib64/python3.6/site-packages/nio/client/base_client.py", line 64, in wrapper
    raise LocalProtocolError("Not logged in.")
nio.exceptions.LocalProtocolError: Not logged in.

Is there any way a recent update broke the login method?
I can't figure out what's wrong. I would be glad to provide any details if needed. Currently using python 3.6.

edit: it seems to have a link with the recent update made here https://github.com/poljar/matrix-nio/blob/master/nio/client/async_client.py

Missing room_redact method in AsyncClient

AsyncClient lacks a room_redact method that HttpClient has.

jsonschema.exceptions.ValidationError: 'presence' is a required property when querying presence of a user that has never been logged in

Hi.

I just updated to matrix-nio 0.13.0 and implemented the new AsyncClient get_presence method like this:

response = await self.client.get_presence(userid)

but when I query the presence state of a user that has never been logged in, I get the following exception:

exception=TypeError('from_dict() takes 2 positional arguments but 3 were given')>
Traceback (most recent call last):
  File "/home/diana/.envs/matrix-nio/lib/python3.8/site-packages/nio/responses.py", line 158, in wrapper
    validate_json(parsed_dict, schema)
  File "/home/diana/.envs/matrix-nio/lib/python3.8/site-packages/nio/schemas.py", line 64, in validate_json
    Validator(schema, format_checker=FormatChecker()).validate(instance)
  File "/home/diana/.envs/matrix-nio/lib/python3.8/site-packages/jsonschema/validators.py", line 353, in validate
    raise error
jsonschema.exceptions.ValidationError: 'presence' is a required property

Failed validating 'required' in schema:
    {'properties': {'currently_active': {'type': 'boolean'},
                    'last_active_ago': {'type': 'integer'},
                    'presence': {'type': 'string'},
                    'status_msg': {'type': 'string'}},
     'required': ['presence'],
     'type': 'object'}

On instance:
    {'errcode': 'M_FORBIDDEN',
     'error': 'You are not allowed to see their presence.'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/diana/proyectos/matrix/python/matrix-nio/acd/bot_commands.py", line 128, in loop_agents
    presence_response = await self.get_user_presence(agentid)
  File "/home/diana/proyectos/matrix/python/matrix-nio/acd/bot_commands.py", line 249, in get_user_presence
    response = await self.client.get_presence(userid)
  File "/home/diana/.envs/matrix-nio/lib/python3.8/site-packages/nio/client/async_client.py", line 248, in wrapper
    return await func(self, *args, **kwargs)
  File "/home/diana/.envs/matrix-nio/lib/python3.8/site-packages/nio/client/async_client.py", line 2670, in get_presence
    return await self._send(
  File "/home/diana/.envs/matrix-nio/lib/python3.8/site-packages/nio/client/async_client.py", line 692, in _send
    resp = await self.create_matrix_response(
  File "/home/diana/.envs/matrix-nio/lib/python3.8/site-packages/nio/client/async_client.py", line 485, in create_matrix_response
    resp = response_class.from_dict(parsed_dict, *data)
  File "/home/diana/.envs/matrix-nio/lib/python3.8/site-packages/nio/responses.py", line 163, in wrapper
    return error_class.from_dict(parsed_dict, *args, **kwargs)
TypeError: from_dict() takes 2 positional arguments but 3 were given

Is this the expected behaviour?

Questions about installing python-olm

I'm trying to get an environment set up for contributing to matrix-nio.

The only solution I have was to

1. Create and activate a python venv
2. Clone libolm on v3.1.5
3. Build libolm shared libraries
4. Go to olm/python directory
5. Build python bindings
6. pip install -e . to successfully install python-olm inside the virtualenv

I'm really not sure if this is the correct way to install python-olm. Is there any documentation on the correct way to get python-olm set up?

message event callbacks not firing in E2E room

I am logging into my account using username and password credentials with the AsyncClient.

I have installed libolm and am using the latest matrix-nio[e2e] installed via pip.

Then, I am subscribing for message events like:

client.add_event_callback(on_message, (nio.RoomMessageText,))

This works for messages in non-encrypted rooms, but in encrypted rooms the events never fire.
I have tried logging into the account using Riot, and the messages are indeed there and visible to the bot user. However, when accessed through nio, the messages are not read.

Support for room creation.

The ability to create a new room is completely missing from nio.

To add support for this a couple of things need to be done:

• The Api class needs to be expanded to contain the createRoom endpoint.
• A Response/ErrorResponse class pair needs to be created for CreateRoom responses.
• The clients need to get methods that make such a request.

Optionally we might want to provide some higher level room builder class that makes it easier for users to build a sensible content for this request.

The relevant spec entry can be found here

Room tags

• Handle account_data events
• Store account_data tags in MatrixRoom as info list
• implement room account_data as defined in spec 13.19.2.3 for changing tags (https://matrix.org/docs/spec/client_server/r0.6.0#put-matrix-client-r0-user-userid-rooms-roomid-account-data-type)

Error while importing store:

Its breaking while importing any of the stores.

python: 3.7
os: macosx 10.14.6
nio version: 0.7.1

Error

cannot import name 'Accounts' from 'nio.store'

from nio.store.database import (SqliteStore)

  store = SqliteStore(
        user_id=self.userID,
        device_id=self.deviceId,
        store_path=self.storePath,
        database_name="robot"
    )
    config = ClientConfig(
        store=store,
        store_name="robot",
        store_sync_tokens=True
    )

    self.client = AsyncClient(
        matrix['homeserver'],
        self.userID,
        device_id=self.deviceId,
        config=config,
        store_path=self.storePath,
    )

Use without olm

Since it doesn't seem like Olm will appear on pypi anytime soon, is there any chance of using this lib without encryption?

Room upgrades and m.room.tombstone events.

Matrix supports room upgrades. Room upgrades are essentially killing off an existing room and suggesting users to move to the new room.

To support room upgrades nio will need to parse the m.room.tombstone state events and store relevant information from the event in the MatrixRoom object of the room. Mainly the successor room of the event, and upgraded rooms should remember their predecessors as well.

To achieve this a new event schema should be specified in nio/schemas.py and a new event in nio/events/room_events.py. After that the handle_event method of the MatrixRoom class needs to be enhanced to handle the new event.

Relevant spec entry: https://matrix.org/docs/spec/client_server/latest#id205

Incorrect return type of MatrixRoom.handle_membership

The return type of MatrixRoom.handle_membership is bool.

https://github.com/poljar/matrix-nio/blob/41079d11fd16707324dbe7d82a4234542aba7e5e/nio/rooms.py#L227

Yet we return self.add_member which returns a NoneType:

https://github.com/poljar/matrix-nio/blob/41079d11fd16707324dbe7d82a4234542aba7e5e/nio/rooms.py#L241-L243

As well as self.remove_member:

https://github.com/poljar/matrix-nio/blob/41079d11fd16707324dbe7d82a4234542aba7e5e/nio/rooms.py#L258

This is not right.

The download method of the AsyncClient requires API users to parse the mxc URL.

The download API takes a server name and a media ID but media events in nio only provide the full MXC URL.

It would be nicer if the download API takes the full MXC URL so we can do this type of thing:

response = await client.download(event.url)

Client (config) option for ignoring self-sent events for callbacks

Currently in each of my callback methods I have to put:

# Ignore messages from ourselves
if event.sender == self.client.user:
    return

It'd be nice if I could specify ignore_self_events=True or something similar on the client, or its config, to remove the need for this check.

clearly enumerate current capabilities

once again, thanks again for the project!
it really seems something that I might end up using for years. 👍

unfortunately, I've experimented a bit and I've come across several problems that made me think of opening up this issue -- do close it if my questions are pointless.

I think this project lacks a clear roadmap of current and future features for the wandering user.
while trying to play with this, I am not sure whether I'm just frustrated by the lack of documentation (aspect on which I might be able to help, eventually) or whether some features are still to be coded (aspect on which I doubt I'll be helpful anytime soon).

e.g.: I have created a new Matrix account and I'd like to set up my pc to send and receive messages.
I've managed to follow the very thin examples on the main readme, but now I'd like to do it more.
since I'm interested in command line for sending and receiving messages, I thought I could have started with the appropriately named nio-cli.py, but I couldn't find it in the path of my installation.
is this distributed through the pip install matrix-cli? is it supposed to?
I have then tried and cloned the repo to execute it locally.
are the requirements from nio-cli.py larger than those of the pip-installed matrix-nio?
it seems I lack clicks, socket, and so on...
finally, I'd like to understand whether matrix-nio currently can support encryption (verifying the session, sending/receiving encrypted messages?) and attachments (images? videos? any size limit?).
thanks!

Support Olm session unwedging.

Olm sessions can sometimes get out of sync. When that happens sending encrypted Olm messages to a device will fail at the decryption step.

This fixes this by creating a new session and using the new session to send a dummy message.

The database scheme needs to be modified to store date when the session was last used and sending of dummy messages needs to be supported.

Support other databases like postgres

That would make it much easier to host the stuff built on this on a cluster of nodes. Also, it would probably scale better?

Using a file object directly in a function upload()

According to the documentation, in the upload function you can use the file object directly:

with open ("test_file", "rb") as f:
     await client.upload (f, "text / plain")

But this option does not work:

Error - '_io.BufferedReader' object is not callable

So far, the following solution has been proposed:

with open ("test_file", "rb") as f:
     await client.upload (lambda * _: f, "text / plain", filesize = file_size)

Missing Pypi Package

I would be happy to be able to install it via pip. The setup.py is already there.

Document how to properly setup a sync for encryption to work.

Document that client users need to make sure that the room state is synced for encryption to work.

There are multiple ways that this needs to be handled depending on the other sync settings (e.g. member lazy loading).

Handle whitespace in headers

As noted in: poljar/weechat-matrix#206

The h2 connection can be set not to validate headers with something similar to:

config = H2Configuration(client_side=True, validate_inbound_headers=False)
connection = H2Connection(config=config)

Is it reasonable to set that for all connections or does there need to be a parameter on HttpClient.connect and some logic elsewhere to set it?

The upload method of the async client should be able to encrypt the file before uploading

File uploads on Matrix can be encrypted or non-encrypted depending on the room where the MXC URL will be posted to.

The current upload method of the encrypted client uploads the raw data as. This should be changed, the method should take an optional argument to decide if the data should be encrypted.

The encryption should happen in an executor to not block the current thread while encryption is going on.

AsyncClient login method does not offer token login

The HttpClient login can either use token or password login, but the AsyncClient only offers password login. Is there a reason behind it? Can it be added?

Left rooms aren't handeled.

Left rooms didn't receive any love up until now, because of that it's hard to get the events for left rooms.

A new room class should be added for left rooms similarly to the invited rooms. Callbacks should be called for left rooms equally as for invited rooms.

Readme and docs do not mention python-olm / olm dependency

Through https://github.com/poljar/matrix-nio/blob/master/setup.py#L27 this package depends on https://github.com/poljar/python-olm and recursively https://git.matrix.org/git/olm/about/

To ease discoverability of these build dependencies for developers unknown to this code base, they may be added to the Readme or docs.

Nio should honor the encryption settings for a room.

An encrypted room has settings for the rotation period and rotation messages for the outbound megolm sessions.

Nio currently ignores those and just uses the default.

matrix-nio[e2e] version: image not found

olm-3.1.3 installed
matrix-nio 0.4.1 installed

pip3 install "matrix-nio[e2e]

import nio, get "image not found" error:

Python 3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nio/__init__.py", line 2, in <module>
    from .client import *
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nio/client/__init__.py", line 3, in <module>
    from .base_client import Client, ClientConfig
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nio/client/base_client.py", line 25, in <module>
    from ..crypto import ENCRYPTION_ENABLED
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nio/crypto/__init__.py", line 17, in <module>
    from .sessions import (
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nio/crypto/sessions.py", line 24, in <module>
    import olm
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/python_olm-3.1.3-py3.6-macosx-10.9-x86_64.egg/olm/__init__.py", line 24, in <module>
    from .utility import ed25519_verify, OlmVerifyError, OlmHashError, sha256
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/python_olm-3.1.3-py3.6-macosx-10.9-x86_64.egg/olm/utility.py", line 39, in <module>
    from _libolm import ffi, lib  # type: ignore
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/python_olm-3.1.3-py3.6-macosx-10.9-x86_64.egg/_libolm.abi3.so, 2): Library not loaded: build/libolm.dylib.3.1.3
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/python_olm-3.1.3-py3.6-macosx-10.9-x86_64.egg/_libolm.abi3.so
  Reason: image not found

Creating a new Olm session failed with the BAD_MESSAGE_KEY_ID error.

Two nio based clients seem to have gone into a state where unwedging fails because one of them has uploaded one-time keys that seem to be unusable.

When devices try to create a new session with those keys session creation fails with the mentioned error.

While that error might indicate that a MITM attack is going on, we should have noticed it beforehand when checking the signatures of the one-time keys.

It seems that the device has uploaded one-time keys and forgot about them.

TypeError exceptions must derive from BaseException

When testing pantalaimon 0.6.0 with matrix-nio 0.11:

mai 10 23:02:48 nlaptop xsession[1858]: fractal_gtk::app::backend_loop: sending 2e7e93c9ccfa06a6ad796052f1baf60f: retrying send
mai 10 23:02:58 nlaptop pantalaimon[31821]: Error handling request
mai 10 23:02:58 nlaptop pantalaimon[31821]: Traceback (most recent call last):
mai 10 23:02:58 nlaptop pantalaimon[31821]:   File "/nix/store/y668anqjzk8phxl312rl530x276mjh6i-python3.7-aiohttp-3.6.2/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
mai 10 23:02:58 nlaptop pantalaimon[31821]:     resp = await task
mai 10 23:02:58 nlaptop pantalaimon[31821]:   File "/nix/store/y668anqjzk8phxl312rl530x276mjh6i-python3.7-aiohttp-3.6.2/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
mai 10 23:02:58 nlaptop pantalaimon[31821]:     resp = await handler(request)
mai 10 23:02:58 nlaptop pantalaimon[31821]:   File "/nix/store/6i3m21djjdd07pcc1mf58jkgnrk70zcv-pantalaimon-0.6.0/lib/python3.7/site-packages/pantalaimon/daemon.py", line 861, in send_message
mai 10 23:02:58 nlaptop pantalaimon[31821]:     return await _send(self.conf.ignore_verification)
mai 10 23:02:58 nlaptop pantalaimon[31821]:   File "/nix/store/6i3m21djjdd07pcc1mf58jkgnrk70zcv-pantalaimon-0.6.0/lib/python3.7/site-packages/pantalaimon/daemon.py", line 830, in _send
mai 10 23:02:58 nlaptop pantalaimon[31821]:     room_id, msgtype, content, txnid, ignore_unverified
mai 10 23:02:58 nlaptop pantalaimon[31821]:   File "/nix/store/ijs66gcqygznd34idmmj4sqbk3z3y403-python3.7-matrix-nio-0.11.0/lib/python3.7/site-packages/nio/client/async_client.py", line 1370, in room_send
mai 10 23:02:58 nlaptop pantalaimon[31821]:     ignore_unverified_devices=ignore_unverified_devices,
mai 10 23:02:58 nlaptop pantalaimon[31821]:   File "/nix/store/ijs66gcqygznd34idmmj4sqbk3z3y403-python3.7-matrix-nio-0.11.0/lib/python3.7/site-packages/nio/client/async_client.py", line 1614, in share_group_session
mai 10 23:02:58 nlaptop pantalaimon[31821]:     ignore_unverified_devices=ignore_unverified_devices
mai 10 23:02:58 nlaptop pantalaimon[31821]:   File "/nix/store/ijs66gcqygznd34idmmj4sqbk3z3y403-python3.7-matrix-nio-0.11.0/lib/python3.7/site-packages/nio/crypto/olm_machine.py", line 1838, in share_group_session_parallel
mai 10 23:02:58 nlaptop pantalaimon[31821]:     f"Missing Olm session for user {user_id} and device "
mai 10 23:02:58 nlaptop pantalaimon[31821]: TypeError: exceptions must derive from BaseException

The SqliteMemoryStore doesn't work anymore.

The constructor for the SqliteMemoryStore is incompatible with the parent class and thus creating one fails once the user logs in.

Add automatic formatting/linting setup.

We should set up some way to automatically format/lint the codebase.

Black seems to be popular nowadays as an automatic formatter. Isort as it seems will become compatible with black.

Pre-commit can be used to easily introduce commit hooks with the required lint/formatting steps.

Missing methods in AsyncClient

These methods are present in HttpClient, but missing in AsyncClient:

• devices() 7f3eb1c
• update_device() #140
• lag()
• login_info() #143
• room_invite() #89
• room_kick() #123
• room_put_state() #102
• room_read_markers() #131
• room_redact() #84
• login() with token argument #106

Unable to install matrix-nio[e2e] via pip

Describe the bug
Unable to install matrix-nio[e2e] via pip

System information:
OS: Linux Debian Buster
Python Version: 3.8.2
PIP Version: 20.1

What i did
apt install python-olm
pip install "matrix-nio[e2e]

What did happen

Building wheels for collected packages: future, logbook, peewee, python-olm, pyrsistent
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491058 sha256=74276ed5c5fe8cedf02160ed29f7140383744dd01fcf00c586aee50c1239e2d6
  Stored in directory: /root/.cache/pip/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4
  Building wheel for logbook (setup.py) ... done
  Created wheel for logbook: filename=Logbook-1.5.3-cp38-cp38-linux_x86_64.whl size=66380 sha256=eb04dc6db1a04a80f5372415ab75bf19e81100745153469d561947e464926e4b
  Stored in directory: /root/.cache/pip/wheels/25/9a/c0/03bf368ac5f673e9c0bde58837146b004424b25555ca3aedfb
  Building wheel for peewee (setup.py) ... done
  Created wheel for peewee: filename=peewee-3.13.3-cp38-cp38-linux_x86_64.whl size=811182 sha256=cfc3ee7a640135f7af09bd7f1ef9019c68e211ea806c2c5d34b5ccba0a46c474
  Stored in directory: /root/.cache/pip/wheels/31/c2/e3/758b28810363cabd2ea3126fa71ee31be916528dcbc6c59eaf
  Building wheel for python-olm (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-5u5j1g0y/python-olm/setup.py'"'"'; __file__='"'"'/tmp/pip-install-5u5j1g0y/python-olm/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-s6ameo4y
       cwd: /tmp/pip-install-5u5j1g0y/python-olm/
  Complete output (29 lines):
  make: *** No rule to make target '../include/olm/olm.h', needed by 'include/olm/olm.h'.  Stop.
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/olm
  copying olm/_compat.py -> build/lib.linux-x86_64-3.8/olm
  copying olm/__init__.py -> build/lib.linux-x86_64-3.8/olm
  copying olm/group_session.py -> build/lib.linux-x86_64-3.8/olm
  copying olm/sas.py -> build/lib.linux-x86_64-3.8/olm
  copying olm/account.py -> build/lib.linux-x86_64-3.8/olm
  copying olm/__version__.py -> build/lib.linux-x86_64-3.8/olm
  copying olm/_finalize.py -> build/lib.linux-x86_64-3.8/olm
  copying olm/pk.py -> build/lib.linux-x86_64-3.8/olm
  copying olm/utility.py -> build/lib.linux-x86_64-3.8/olm
  copying olm/session.py -> build/lib.linux-x86_64-3.8/olm
  running build_ext
  generating cffi module 'build/temp.linux-x86_64-3.8/_libolm.c'
  creating build/temp.linux-x86_64-3.8
  building '_libolm' extension
  creating build/temp.linux-x86_64-3.8/build
  creating build/temp.linux-x86_64-3.8/build/temp.linux-x86_64-3.8
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.8 -c build/temp.linux-x86_64-3.8/_libolm.c -o build/temp.linux-x86_64-3.8/build/temp.linux-x86_64-3.8/_libolm.o -I../include
  build/temp.linux-x86_64-3.8/_libolm.c:541:18: fatal error: olm/olm.h: No such file or directory
           #include <olm/olm.h>
                    ^~~~~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for python-olm
  Running setup.py clean for python-olm
  Building wheel for pyrsistent (setup.py) ... done
  Created wheel for pyrsistent: filename=pyrsistent-0.16.0-cp38-cp38-linux_x86_64.whl size=126688 sha256=8f87d3c055f1da44959aa9a70dcb71173d51e1adbff30499a4d3b14b626b7554
  Stored in directory: /root/.cache/pip/wheels/17/be/0f/727fb20889ada6aaaaba861f5f0eb21663533915429ad43f28
Successfully built future logbook peewee pyrsistent
Failed to build python-olm
Installing collected packages: future, h11, hpack, hyperframe, h2, multidict, idna, yarl, async-timeout, chardet, attrs, aiohttp, aiofiles, pycryptodome, six, pyrsistent, jsonschema, unpaddedbase64, logbook, cachetools, peewee, pycparser, cffi, python-olm, atomicwrites, matrix-nio
    Running setup.py install for python-olm ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-5u5j1g0y/python-olm/setup.py'"'"'; __file__='"'"'/tmp/pip-install-5u5j1g0y/python-olm/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-xp_f8cu7/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/python-olm
         cwd: /tmp/pip-install-5u5j1g0y/python-olm/
    Complete output (29 lines):
    make: *** No rule to make target '../include/olm/olm.h', needed by 'include/olm/olm.h'.  Stop.
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/olm
    copying olm/_compat.py -> build/lib.linux-x86_64-3.8/olm
    copying olm/__init__.py -> build/lib.linux-x86_64-3.8/olm
    copying olm/group_session.py -> build/lib.linux-x86_64-3.8/olm
    copying olm/sas.py -> build/lib.linux-x86_64-3.8/olm
    copying olm/account.py -> build/lib.linux-x86_64-3.8/olm
    copying olm/__version__.py -> build/lib.linux-x86_64-3.8/olm
    copying olm/_finalize.py -> build/lib.linux-x86_64-3.8/olm
    copying olm/pk.py -> build/lib.linux-x86_64-3.8/olm
    copying olm/utility.py -> build/lib.linux-x86_64-3.8/olm
    copying olm/session.py -> build/lib.linux-x86_64-3.8/olm
    running build_ext
    generating cffi module 'build/temp.linux-x86_64-3.8/_libolm.c'
    creating build/temp.linux-x86_64-3.8
    building '_libolm' extension
    creating build/temp.linux-x86_64-3.8/build
    creating build/temp.linux-x86_64-3.8/build/temp.linux-x86_64-3.8
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.8 -c build/temp.linux-x86_64-3.8/_libolm.c -o build/temp.linux-x86_64-3.8/build/temp.linux-x86_64-3.8/_libolm.o -I../include
    build/temp.linux-x86_64-3.8/_libolm.c:541:18: fatal error: olm/olm.h: No such file or directory
             #include <olm/olm.h>
                      ^~~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-5u5j1g0y/python-olm/setup.py'"'"'; __file__='"'"'/tmp/pip-install-5u5j1g0y/python-olm/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-xp_f8cu7/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/python-olm Check the logs for full command output.

Tests fail under python 3.8.2

Describe the bug
A lot of pytests fail under python 3.8.2

System information:
OS: Linux Debian Buster
Python Version: 3.8.2
PIP Version: 20.1

What i did

pip install -r test-requirements.txt
pip install -r rtd-requirements.txt
make test

What did happen

FAILED tests/async_attachment_test.py::TestClass::test_encrypt - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_encrypt_large_bytes - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_encrypt_str - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_encrypt_path_object - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_encrypt_iterable - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_encrypt_async_iterable - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_encrypt_file_object - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_encrypt_async_file_object - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_encrypt_bad_argument_type - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_hash_verification - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_invalid_key - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_invalid_iv - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_short_key - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_short_iv - AttributeError: module 'time' has no attribute 'clock'
FAILED tests/async_attachment_test.py::TestClass::test_fake_key - AttributeError: module 'time' has no attribute 'clock'

I have found that time.clock() was removed in python 3.8
See: https://docs.python.org/3/whatsnew/3.8.html#api-and-feature-removals

If i run the same under python 3.7.7 the tests will pass but with 227 deprecation warnings

/usr/local/lib/python3.7/site-packages/Crypto/Random/_UserFriendlyRNG.py:77: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead

Implement Read Receipt Ephemeral Event

Looks like there is current an implemented event for typing. Would be nice to also have read receipts as an easy to use option.

AsyncClient upload rejected by Synapse

Content-Length header is not being set when uploading a file to a Synapse homeserver using AsyncClient.Upload(), causing Synapse to error.

Store and restore the encryption state of rooms.

No room state is currently cached in nio.

A malicious server could hide the state event that marks a room as encrypted from us and downgrade the room to a non-encrypted one after a client logs in again.

The encryption state of rooms should be stored in the database and restored afterwards again.

Missing room_context method in HttpClient

HttpClient lacks a room_context method that AsyncClient has.

UnboundLocalError: local variable 'room' referenced before assignment

I got this error once in weechat-matrix:

00:49 python: stdout/stderr (matrix): Traceback (most recent call last):
00:49 python: stdout/stderr (matrix):   File "/home/trygve/dotfiles/.weechat/python/matrix/utf.py", line 89, in wrapper
00:49 python: stdout/stderr (matrix):     return function(*args, **kwargs)
00:49 python: stdout/stderr (matrix):   File "/home/trygve/dotfiles/.weechat/python/matrix/bar_items.py", line 112, in matrix_bar_item_buffer_modes
00:49 python: stdout/stderr (matrix):     and server.client.room_contains_unverified(room.room_id)):
00:49 python: stdout/stderr (matrix):   File "/home/trygve/dev/weechat-matrix/venv/lib/python3.5/site-packages/nio/client/base_client.py", line 349, in room_contains_unverified
00:49 python: stdout/stderr (matrix):     if not room.encrypted:
00:49 python: stdout/stderr (matrix): UnboundLocalError: local variable 'room' referenced before assignment
00:49 =!= python: error in function "matrix_bar_item_buffer_modes"

Looks like it would happen if you get the exception here: https://github.com/poljar/matrix-nio/blob/master/nio/client/base_client.py#L347

Then room would not be set, but the function would continue, and try to use it.