syncrypt / client Goto Github PK

View Code? Open in Web Editor NEW

11.0 11.0 1.0 1.63 MB

Syncrypt command line client and sync daemon

Home Page: https://syncrypt.space

License: GNU General Public License v3.0

Python 99.80% Shell 0.05% Dockerfile 0.15%

backup cloud-storage command-line encryption file-sharing syncrypt syncrypt-client

client's People

Contributors

Stargazers

Watchers

Forkers

digitalsatori

client's Issues

Export vault import and export functionality as API endpoints

Add crypto information to metadata in API endpoint

Fingerprint
RSA Key size
AES key size
Algorithm
Hash Algorithm

Implement signup command in binary protocol

Expose signup command via web API so the desktop UI can initiate a user registration during initial account setup.

Add vault option to use vault key instead of file key on small files

Upload encrypted vault key after creation via API

Implement a consistent way of reporting errors in the API

The current idea is to have an exception for each error. These exception classes will each have a code to identify the error type as well as an english message. Whenever a known exception is given to the API server, it will translate it to a proper json dict and http code (4xx). Unknown exceptions will continue to serve as 500 http replies.

File path issues when sharing vaults between UNIX and Windows

Client should not crash when vault folder does not exist anymore

  File "syncrypt/client/syncrypt/models/vault.py", line 43, in config
    assert os.path.exists(self.folder)
AssertionError

Create automatic pypi deployment on new tag

Rework websockets logging code with a queue

Syncrypt daemon allows "." as vault path in global config

I've run into issues where somehow a path with value "." is stored in the global config's vault list used by the daemon.
This causes errors since "." isn't an absolute path.
We should only allow absolute paths in the global syncrypt config's vault list.

Delete/update .vault folder config in vault when deleting vault from server

When deleting a vault on the server from the desktop client or the cli, the client should delete the .vault file or config files in order to allow the folder to be added again as a vault later.
Either that, or let the client/daemon detect when it's attempting to create a new vault from a folder that has an outdated .vault config (with a vault ID that doesn't exist anymore) and let it either update that vault config (delete the vault ID) or delete it (probably cleaner but maybe there's a case where keeping all the other vault options the same makes sense?). By deleting and replacing the outdated .vault config we get the additional benefit, that we're recreating all the file keys as well. That might be the more secure option.

Add reliable API for restarting & shutting down the daemon

Related to syncrypt/desktop#11

The current api endpoints don't seem to work correctly. We need a way to reliably start, stop and restart the daemon process from the desktop client (via system tray menu)

Make sure Syncrypt is not vulnerable to CSRF attack

Locally bound HTTP RPC services can be called from the browser if no measure are taken against it.

Here is an example of Monero:

https://labs.mwrinfosecurity.com/advisories/csrf-vulnerability-allows-for-remote-compromise-of-monero-wallets/

The temporary solution of Monero is to set a pre-defined user agent, but that does not seem secure enough. We probably need some simple authentication token for API calls.

missing api.client

I just updated to 0.3.8 via pip install git+https://github.com/syncrypt/client but running into problems right now:

syncrypt push
Traceback (most recent call last):
  File "/usr/local/bin/syncrypt", line 10, in <module>
    from syncrypt.app import SyncryptApp
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/app/__init__.py", line 1, in <module>
    from .syncrypt import SyncryptApp
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py", line 21, in <module>
    from syncrypt.api import SyncryptAPI, APIClient
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/api/__init__.py", line 14, in <module>
    from .client import APIClient
ImportError: No module named 'syncrypt.api.client'

Did I miss something during the update?

Missing snappy-c crashes pip installation.

Never wrote my own package, so I am not quite sure if this is even a problem you can address: The PyPI installation crashed because I was missing the snappy-c libraries. Running sudo apt-get install libsnappy-dev solved the issue...

pip.log.txt

ignore subfolder

I want to create a vault containing a subfolder which has files that shouldn't be synchronized.
Therefore I'm wondering whether there is something like a .gitignore in syncrypt?

syncrypt clone error

Thanks for coding, it's a very promising project!
I'm working with:

syncrypt (0.1.4 via pip), debian testing, python-3.5.2

I just create a vault and pushed to the server. Then I switched to a different computer, login worked as well as the syncrypt vault to get the vault id (e.g., myvault).
But afterwards the syncrypt clone myvault command failed.
Detailed error log:

Traceback (most recent call last):
  File "/usr/local/bin/syncrypt", line 266, in <module>
    c.run()
  File "/usr/local/bin/syncrypt", line 97, in run
    self.config.directory or self.config.vault_id))
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py", line 289, in clone
    backend.get_user_vault_key(self.identity.get_fingerprint(), vault_id)
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 648, in myco
    return (yield from getattr(conn, name)(*args, **kwargs))
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 414, in get_user_vault_key
    response = yield from self.read_response()
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 85, in read_response
    decoded = yield from self.read_term(assert_ok=True)
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 78, in read_term
    raise ServerError(decoded[1:])
syncrypt.backends.binary.ServerError: ((Atom('not_found'), Atom('Elixir.Syncrypt.UserKey')),)

ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'exception': OSError(9, 'Bad file descriptor'), 'message': 'Fatal write error on socket transport', 'transport': <_SelectorSocketTransport fd=7>, 'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f6db83b64a8>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'exception': RuntimeError('Event loop is closed',), 'message': 'Fatal error on SSL transport', 'transport': <_SelectorSocketTransport closing fd=7>, 'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f6db83b64a8>}), {}

Add timestamp to daemon log file

Add "import" command for simple restore of a backup export

Respect vault name and other metadata during POST/PUT vault

Add API endpoint for stopping daemon

We should allow the desktop client to stop (and maybe also restart) the currently running daemon if necessary (e.g. when installing an update).

Expose correct user count in stats daemon api

Currently it's just always sending a user count of 1.

[alchemy branch] syncrypt CLI commands operate on all vaults instead of single vault

E.g. running syncrypt push or syncrypt log inside a vault folder will still run those commands for all vaults the daemon is configured to watch/sync.

Duplicate vault possible?

How do we want to deal with duplicate vaults? Right now, when the user wants to create a new vault with a folder that is already used for an existing & currently synchronized vault, the following happens:

The vault gets added again and shows up twice, with the same remote vault ID
Removing the newly added duplicate from sync in the admin panel removes only the duplicate vault
After restarting the daemon, both entries for the removed vault are now gone from the vault list.

It seems like there's a different state in the daemon before and after restarting it with regard to these duplicate vaults.
I think the daemon should either still show the original vault card even after restarting in this scenario, or it should cause some kind of error (in daemon & GUI) that it's not possible to use the folder of another currently synced vault for a "new" vault.

Perform client-side caching for commonly requested vault metadata & history

Currently it can take quite a long time to download & decrypt a vault's history and metadata.
Most of that data never changes or is only being appended to over time, so we should get quite large performance gains by storing this metadata locally for future reference and only request the latest diff with only the new data.

Change default port to 41337

API endpoint for user key import

Vault stays in "uninitialized" state after being cloned

When cloning a vault, the vault state stays set to "uninitialized" even after all files have been downloaded already and it looks like the daemon isn't doing anything. This is based on the states value in the response from /v1/stats/. After manually restarting the daemon, the states has the vault marked as "ready" as expected. Same problem seems to be for the status values in the response from /v1/vault/.

Newly created vault state is set to "failure"

When creating a new vault, the daemon returns state: "failure" after cloning when hitting the /v1/vault API endpoint. This happens even when the vault creation worked and there was no error.

This is only fixed by restarting the daemon. After a restart, it now returns state: "ready".

Non-existent vaults dir should put vault in error mode during daemon init

Write test case.

Fetching flying vaults while syncing vault fails

When syncing a vault with many files and while running e.g. file stat operations, the daemon returns a 500 status when trying to fetch the list of flying vaults.

Exception backtrace:

Traceback (most recent call last):
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/api/__init__.py", line 243, in error_middleware
    response = await handler(request)
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/aiohttp/web_urldispatcher.py", line 142, in handler_wrapper
    result = yield from result
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/api/resources.py", line 54, in dispatch_list
    objs = await self.get_obj_list(request)
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/api/resources.py", line 309, in get_obj_list
    my_fingerprint = self.app.identity.get_fingerprint()
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/models/identity.py", line 144, in get_fingerprint
    raise IdentityNotInitialized()
syncrypt.models.identity.IdentityNotInitialized
2018-04-08 19:30:54 [I] 127.0.0.1 [08/Apr/2018:17:30:54 +0000] "GET /v1/flying-vault/ HTTP/1.1" 500 195 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Electron/1.8.3 Safari/537.36"

Protect against malicious paths

Make sure (and create some test cases) that a malicious client can not slip in file paths that are outside of the vault folder (could be eith absolute /, or relative ../../../).

error pushing files

I'm getting an error while pushing my vault to the server.
It seems to be an encoding problem. Data is a diverse set of html and pdf files.

INFO:syncrypt.backends.binary:Uploading bundle (metadata: 512 bytes, content: 144 bytes)
Traceback (most recent call last):
  File "/usr/local/bin/syncrypt", line 266, in <module>
    c.run()
  File "/usr/local/bin/syncrypt", line 153, in run
    self.loop.run_until_complete(self.app.push())
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py", line 388, in push
    for bundle in vault.walk_disk():
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/models/vault.py", line 156, in walk_disk
    yield self.bundle_for(relpath)
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/models/vault.py", line 182, in bundle_for
    Bundle(os.path.join(self.folder, relpath), vault=self)
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/models/bundle.py", line 36, in __init__
    h.update(self.relpath.encode(self.vault.config.encoding))
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcfc' in position 32: surrogates not allowed
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<SyncryptApp._push_bundle() running at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:447> wait_for=<Future pending cb=[Task._wakeup()]> cb=[SyncryptApp.push_bundle.<locals>.cb() at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:149]>}), {}
Exception ignored in: <generator object SyncryptApp._push_bundle at 0x7fc4780584c0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py", line 447, in _push_bundle
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 634, in upload
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 227, in __exit__
  File "/usr/lib/python3.5/asyncio/locks.py", line 242, in set
  File "/usr/lib/python3.5/asyncio/futures.py", line 332, in set_result
  File "/usr/lib/python3.5/asyncio/futures.py", line 242, in _schedule_callbacks
  File "/usr/lib/python3.5/asyncio/base_events.py", line 497, in call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
RuntimeError: Event loop is closed
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<SyncryptApp._push_bundle() running at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:447> wait_for=<Future pending cb=[Task._wakeup()]> cb=[SyncryptApp.push_bundle.<locals>.cb() at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:149]>}), {}
Exception ignored in: <generator object SyncryptApp._push_bundle at 0x7fc47806f728>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py", line 447, in _push_bundle
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 634, in upload
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 227, in __exit__
  File "/usr/lib/python3.5/asyncio/locks.py", line 242, in set
  File "/usr/lib/python3.5/asyncio/futures.py", line 332, in set_result
  File "/usr/lib/python3.5/asyncio/futures.py", line 242, in _schedule_callbacks
  File "/usr/lib/python3.5/asyncio/base_events.py", line 497, in call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
RuntimeError: Event loop is closed
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<SyncryptApp._push_bundle() running at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:442> wait_for=<Future pending cb=[Task._wakeup()]> cb=[SyncryptApp.push_bundle.<locals>.cb() at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:149]>}), {}
Exception ignored in: <generator object SyncryptApp._push_bundle at 0x7fc47806f0a0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py", line 442, in _push_bundle
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 615, in stat
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 227, in __exit__
  File "/usr/lib/python3.5/asyncio/locks.py", line 242, in set
  File "/usr/lib/python3.5/asyncio/futures.py", line 332, in set_result
  File "/usr/lib/python3.5/asyncio/futures.py", line 242, in _schedule_callbacks
  File "/usr/lib/python3.5/asyncio/base_events.py", line 497, in call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
RuntimeError: Event loop is closed
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future finished result=True> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future finished result=True> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future finished result=True> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<Event.wait() running at /usr/lib/python3.5/asyncio/locks.py:264> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<SyncryptApp._push_bundle() running at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:447> wait_for=<Future finished result=b'GIF89a\x10\...xe7K\x01\x00;'> cb=[SyncryptApp.push_bundle.<locals>.cb() at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:149]>}), {}
Exception ignored in: <generator object SyncryptApp._push_bundle at 0x7fc4780665c8>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py", line 447, in _push_bundle
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 634, in upload
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/backends/binary.py", line 271, in upload
  File "/usr/lib/python3.5/asyncio/coroutines.py", line 209, in coro
  File "/usr/lib/python3.5/asyncio/coroutines.py", line 209, in coro
  File "/usr/lib/python3.5/asyncio/coroutines.py", line 209, in coro
  File "/usr/lib/python3.5/asyncio/coroutines.py", line 209, in coro
  File "/usr/local/lib/python3.5/dist-packages/syncrypt/pipes/io.py", line 40, in close
  File "/usr/local/lib/python3.5/dist-packages/aiofiles/threadpool/utils.py", line 36, in method
  File "/usr/lib/python3.5/asyncio/base_events.py", line 542, in run_in_executor
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
RuntimeError: Event loop is closed
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<SyncryptApp._push_bundle() running at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:439> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.5/asyncio/futures.py:431, Task._wakeup()]> cb=[SyncryptApp.push_bundle.<locals>.cb() at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:149]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<SyncryptApp._push_bundle() running at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:442> wait_for=<Future pending cb=[Task._wakeup()]> cb=[SyncryptApp.push_bundle.<locals>.cb() at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:149]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<SyncryptApp._push_bundle() running at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:442> wait_for=<Future pending cb=[Task._wakeup()]> cb=[SyncryptApp.push_bundle.<locals>.cb() at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:149]>}), {}
ERROR:syncrypt.app.syncrypt:(<_UnixSelectorEventLoop running=False closed=True debug=False>, {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<SyncryptApp._push_bundle() running at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:442> wait_for=<Future pending cb=[Task._wakeup()]> cb=[SyncryptApp.push_bundle.<locals>.cb() at /usr/local/lib/python3.5/dist-packages/syncrypt/app/syncrypt.py:149]>}), {}

Starting the daemon for the first time fails

When running the daemon without any existing config in ~/.config/syncrypt/ it crashes with the following backtrace:

$ syncrypt_daemon
2018-04-12 00:30:51 [I] Generating API auth token...
2018-04-12 00:30:51 [I] REST API Server started at http://127.0.0.1:28080
2018-04-12 00:30:51 [I] Refreshing vault information
2018-04-12 00:30:51 [E] Invalid login: No username/email or auth token provided
Traceback (most recent call last):
  File "/Users/bakkdoor/projects/syncrypt/client/bin/syncrypt_daemon", line 72, in <module>
    while run_syncrypt():
  File "/Users/bakkdoor/projects/syncrypt/client/bin/syncrypt_daemon", line 52, in run_syncrypt
    loop.run_until_complete(app.start())
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/base_events.py", line 337, in run_until_complete
    return future.result()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/app/daemon.py", line 72, in start
    await self.refresh_vault_info()
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/app/syncrypt.py", line 474, in refresh_vault_info
    backend = await self.open_backend()
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/app/syncrypt.py", line 362, in open_backend
    raise ValueError('Can not login, do not have auth provider')
ValueError: Can not login, do not have auth provider

This fails even on subsequent attempts as long as one is not logged in. But since the daemon is crashing the user can't login from the desktop UI login screen.

Allow cloning a vault into a previously cloned location

Currently, it aborts with an exception because the folder is not empty.

If the vault ids match, we should just treat this the same way as if the user added the folder for syncing.

Add file remove and rename command

There should be two commands (remove and rename) to allow the user to explicitly handle file changes.

Daemon stats API fails after running for long time

This happens whenever I've had the daemon running in the background for a longer time without running the GUI. It might be related to resuming from sleep mode, but it could also just be whenever it's been running idle for a while. Not sure yet.
In any case, this causes the GUI to not get the vault's user list in the VaultSettings sidebar.
Might just be happening on Mac OS. Haven't seen this on Windows yet, but that could just be because I don't use Windows very much.

2016-11-24 16:16:37 INFO [aiohttp.access] 127.0.0.1 - - [24/Nov/2016:15:16:37 +0000] "GET /v1/stats HTTP/1.1" 200 371 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.113 Electron/1.4.3 Safari/537.36"
2016-11-24 16:16:39 DEBUG [syncrypt.backends.binary] Found an available connection!
2016-11-24 16:16:39 INFO [syncrypt.backends.binary] Getting a list of vault users
2016-11-24 16:16:39 ERROR [aiohttp.web] Error handling request
Traceback (most recent call last):
  File "venv/lib/python3.5/site-packages/aiohttp/server.py", line 265, in start
  File "venv/lib/python3.5/site-packages/aiohttp/web.py", line 96, in handle_request
  File "asyncio/coroutines.py", line 209, in coro
  File "syncrypt/api/resources.py", line 50, in dispatch_list
  File "syncrypt/api/resources.py", line 333, in get_obj_list
  File "syncrypt/backends/binary.py", line 761, in myco
  File "syncrypt/backends/binary.py", line 436, in list_vault_users
  File "syncrypt/backends/binary.py", line 124, in write_term
  File "asyncio/streams.py", line 323, in drain
  File "venv/lib/python3.5/site-packages/aiohttp/server.py", line 265, in start
  File "venv/lib/python3.5/site-packages/aiohttp/web.py", line 96, in handle_request
  File "asyncio/coroutines.py", line 209, in coro
  File "syncrypt/api/resources.py", line 50, in dispatch_list
  File "syncrypt/api/resources.py", line 333, in get_obj_list
  File "syncrypt/backends/binary.py", line 761, in myco
  File "syncrypt/backends/binary.py", line 436, in list_vault_users
  File "syncrypt/backends/binary.py", line 124, in write_term
  File "asyncio/streams.py", line 323, in drain
  File "venv/lib/python3.5/site-packages/aiohttp/server.py", line 265, in start
  File "venv/lib/python3.5/site-packages/aiohttp/web.py", line 96, in handle_request
  File "asyncio/coroutines.py", line 209, in coro
  File "syncrypt/api/resources.py", line 50, in dispatch_list
  File "syncrypt/api/resources.py", line 333, in get_obj_list
  File "syncrypt/backends/binary.py", line 761, in myco
  File "syncrypt/backends/binary.py", line 436, in list_vault_users
  File "syncrypt/backends/binary.py", line 124, in write_term
  File "asyncio/streams.py", line 323, in drain
  File "venv/lib/python3.5/site-packages/aiohttp/server.py", line 265, in start
  File "venv/lib/python3.5/site-packages/aiohttp/web.py", line 96, in handle_request
  File "asyncio/coroutines.py", line 209, in coro
  File "syncrypt/api/resources.py", line 50, in dispatch_list
  File "syncrypt/api/resources.py", line 333, in get_obj_list
  File "syncrypt/backends/binary.py", line 761, in myco
  File "syncrypt/backends/binary.py", line 436, in list_vault_users
  File "syncrypt/backends/binary.py", line 124, in write_term
  File "asyncio/streams.py", line 323, in drain
  File "venv/lib/python3.5/site-packages/aiohttp/server.py", line 265, in start
  File "venv/lib/python3.5/site-packages/aiohttp/web.py", line 96, in handle_request
  File "asyncio/coroutines.py", line 209, in coro
  File "syncrypt/api/resources.py", line 50, in dispatch_list
  File "syncrypt/api/resources.py", line 333, in get_obj_list
  File "syncrypt/backends/binary.py", line 761, in myco
  File "syncrypt/backends/binary.py", line 436, in list_vault_users
  File "syncrypt/backends/binary.py", line 124, in write_term
  File "asyncio/streams.py", line 323, in drain
  File "venv/lib/python3.5/site-packages/aiohttp/server.py", line 265, in start
  File "venv/lib/python3.5/site-packages/aiohttp/web.py", line 96, in handle_request
  File "asyncio/coroutines.py", line 209, in coro
  File "syncrypt/api/resources.py", line 50, in dispatch_list
  File "syncrypt/api/resources.py", line 333, in get_obj_list
  File "syncrypt/backends/binary.py", line 761, in myco
  File "syncrypt/backends/binary.py", line 436, in list_vault_users
  File "syncrypt/backends/binary.py", line 124, in write_term
  File "asyncio/streams.py", line 323, in drain
  File "asyncio/tasks.py", line 241, in _step
  File "syncrypt/app/syncrypt.py", line 683, in pull_vault_periodically
  File "syncrypt/app/syncrypt.py", line 697, in pull_vault
  File "syncrypt/app/syncrypt.py", line 622, in retrieve_metadata
  File "syncrypt/backends/binary.py", line 761, in myco
  File "syncrypt/backends/binary.py", line 328, in vault_metadata
  File "syncrypt/backends/binary.py", line 109, in read_response
  File "syncrypt/backends/binary.py", line 81, in read_term
  File "asyncio/streams.py", line 627, in read
  File "asyncio/streams.py", line 457, in _wait_for_data
  File "asyncio/futures.py", line 361, in __iter__
  File "asyncio/tasks.py", line 296, in _wakeup
  File "asyncio/futures.py", line 274, in result
  File "asyncio/selector_events.py", line 669, in _read_ready
TimeoutError: [Errno 60] Operation timed out

Daemon error when cloning vault with latest master

I'm getting the following error when trying to clone a vault:

2018-04-23 07:10:24 [W] Continuing without verifying SSL cert
2018-04-23 07:10:24 [D] Connecting to server 127.0.0.1:1337 ssl=True...
2018-04-23 07:10:24 [I] 127.0.0.1 [23/Apr/2018:05:10:24 +0000] "GET /v1/vault/ HTTP/1.1" 200 370 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Electron/1.8.3 Safari/537.36"
2018-04-23 07:10:24 [D] Identifying to server as ('syncrypt', '0.4.3.post0+24.g4f98c66d46')
2018-04-23 07:10:24 [D] Connected (client: 0.4.3.post0+24.g4f98c66d46; server; 0.11.0)
2018-04-23 07:10:24 [D] Choosing <Connection 0x10d11eeb8 busy>
2018-04-23 07:10:24 [D] Logged in to server (version 0.11.0)
2018-04-23 07:10:24 [I] Retrieving encrypted key for vault 10cce746-8ff7-4d6d-a1c4-6c65bc0e2696 (Fingerprint: d2:cd:46:ea:6c:40:8f:2e)
2018-04-23 07:10:24 [D] Found an available connection <Connection 0x10d11eeb8 busy>
2018-04-23 07:10:24 [D] Decrypting block size: 512 bytes
2018-04-23 07:10:24 [D] RSA Decrypted 512 -> 470 bytes
2018-04-23 07:10:24 [D] RSA Decrypted 512 -> 470 bytes
2018-04-23 07:10:24 [D] RSA Decrypted 512 -> 470 bytes
2018-04-23 07:10:24 [D] RSA Decrypted 512 -> 470 bytes
2018-04-23 07:10:25 [D] RSA Decrypted 512 -> 470 bytes
2018-04-23 07:10:25 [D] RSA Decrypted 512 -> 470 bytes
2018-04-23 07:10:25 [D] RSA Decrypted 512 -> 470 bytes
2018-04-23 07:10:25 [D] RSA Decrypted 512 -> 155 bytes
2018-04-23 07:10:25 [I] Creating vault from package in /private/tmp/deeply nested
2018-04-23 07:10:25 [D] Directory exists, checking if empty
2018-04-23 07:10:25 [E]
Traceback (most recent call last):
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/api/__init__.py", line 243, in error_middleware
    response = await handler(request)
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/aiohttp/web_urldispatcher.py", line 142, in handler_wrapper
    result = yield from result
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/api/resources.py", line 78, in dispatch_post
    obj = await self.post_obj(request)
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/api/resources.py", line 249, in post_obj
    vault = await self.app.clone(request_dict['id'], request_dict['folder'])
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/app/syncrypt.py", line 375, in clone
    await self.retrieve_metadata(vault)
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/app/syncrypt.py", line 442, in retrieve_metadata
    await vault.backend.open()
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/syncrypt/backends/binary.py", line 937, in open
    raise VaultNotInitialized()
syncrypt.exceptions.VaultNotInitialized
2018-04-23 07:10:25 [I] 127.0.0.1 [23/Apr/2018:05:10:25 +0000] "POST /v1/vault/ HTTP/1.1" 500 195 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Electron/1.8.3 Safari/537.36"
2018-04-23 07:10:25 [I] 127.0.0.1 [23/Apr/2018:05:10:25 +0000] "GET /v1/stats/ HTTP/1.1" 200 504 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Electron/1.8.3 Safari/537.36"
2018-04-23 07:10:25 [I] Refreshing vault information
2018-04-23 07:10:25 [D] Found an available connection <Connection 0x10d11eeb8 busy>

I see the following vault card in the UI afterwards:

Also, when opening the vault dialog for that "empty" vault I can view the vault's history log but it doesn't show any other metadata and also doesn't sync any files.

Let me know if you want me to test anything or whatever. I've reproduced this with different vaults. Always getting the same error. This is happening with latest master on mac OS.

API endpoint for vault bundle import

Refactor storage backend manager to be a singleton

We want a global backend manager that manages all connections. Its connections have to remember the vault they are logged in to.

Update revision only when bundle is successfully pulled

syncrypt clone using vault name

I have given a name to a vault and tried to clone it from another machine but syncrypt clone vault_name does not work (contrary to syncrypt clone vault_id). Would be helpful to use the shorter vault name, wouldn't it?

On-demand key generation

`syncrypt pull` prints, that it ignored an exception?

I successfully uploaded an image, but when I try to pull in the same folder, this happens:

$ syncrypt pull
INFO:syncrypt.backends.binary:Getting a list of changes for <Vault: .> (latest to efca8e56-c7c0-49ed-8533-e55e2f728172)
Exception ignored in: $

Daemon deletes auth token from vault config

I've run into multiple cases where the daemon process ends up deleting the auth token from one or more of the vaults it's managing (it had a valid auth token before). This causes all kinds of operations to suddenly fail and since we don't yet have a way to easily re-fetch that token there's no easy way to get it working again.

I'm guessing that there's some kind of race condition or other kind of error that only gets triggered under very specific conditions (not sure what those would be but I noticed that while switching between many different vaults within a short amount of time in the desktop GUI this happened).

Here's a sample log of this error:

[ERROR   ] Error handling request
Traceback (most recent call last):
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/aiohttp/server.py", line 266, in start
    yield from self.handle_request(message, payload)
  File "/Users/bakkdoor/projects/syncrypt/client/lib/python3.5/site-packages/aiohttp/web.py", line 90, in handle_request
    resp = yield from handler(request)
  File "/Users/bakkdoor/projects/syncrypt/client/syncrypt/api/resources.py", line 47, in dispatch_list
    objs = yield from self.get_obj_list(request)
  File "/Users/bakkdoor/projects/syncrypt/client/syncrypt/api/resources.py", line 282, in get_obj_list
    yield from vault.backend.open()
  File "/Users/bakkdoor/projects/syncrypt/client/syncrypt/backends/binary.py", line 672, in open
    with (yield from self.manager.acquire_connection()) as conn:
  File "/Users/bakkdoor/projects/syncrypt/client/syncrypt/backends/binary.py", line 610, in acquire_connection
    yield from conn.connect()
  File "/Users/bakkdoor/projects/syncrypt/client/syncrypt/backends/binary.py", line 145, in connect
    raise StorageBackendInvalidAuth('No username/email or auth token provided')
syncrypt.backends.base.StorageBackendInvalidAuth: No username/email or auth token provided

Uncaught exception when attempting to initialize vault

Running
Syncrypt v0.1.4
pip 1.5.4 from /usr/lib/python3/dist-packages (python 3.4)
Python 3.4.3

Two issues:

I used the credentials I signed up with before, so the authentication shouldn't fail
Probably if authentication fails, the exception should be caught somewhere

xxx@xxx:~/syncript_test$ /usr/local/bin/syncrypt init
INFO:syncrypt.app.syncrypt:Initializing <Vault: .>
Email for storage.syncrypt.space: [email protected]
Password:
ERROR:syncrypt.backends.binary:Invalid auth
Traceback (most recent call last):
File "/usr/local/bin/syncrypt", line 266, in
c.run()
File "/usr/local/bin/syncrypt", line 168, in run
self.loop.run_until_complete(self.app.init())
File "/usr/lib/python3.4/asyncio/base_events.py", line 316, in run_until_complete
return future.result()
File "/usr/lib/python3.4/asyncio/futures.py", line 275, in result
raise self._exception
File "/usr/lib/python3.4/asyncio/tasks.py", line 238, in _step
result = next(coro)
File "/usr/local/lib/python3.4/dist-packages/syncrypt/app/syncrypt.py", line 110, in init
yield from vault.backend.upload_identity(self.identity)
File "/usr/local/lib/python3.4/dist-packages/syncrypt/backends/binary.py", line 647, in myco
with (yield from self.manager.acquire_connection()) as conn:
File "/usr/local/lib/python3.4/dist-packages/syncrypt/backends/binary.py", line 540, in acquire_connection
yield from conn.connect()
File "/usr/local/lib/python3.4/dist-packages/syncrypt/backends/binary.py", line 165, in connect
raise StorageBackendInvalidAuth(response)
syncrypt.backends.base.StorageBackendInvalidAuth: (Atom('error'), (Atom('login_failed'), b'[email protected]'))