Coder Social home page Coder Social logo

cloud_mdir_sync's People

Contributors

binary-eater avatar gobenji avatar jgunthorpe avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

timfel chenblueridge us91 saeedtx grivet gobenji stain binary-eater

cloud_mdir_sync's Issues

multiple accounts

I am running multiple local accounts and would like to access different cloud accounts using the program. However this does not seem to be easily done through configuration, other than changing the source code by setting a different port for each account. Without manual source code change, the port 8080 would conflict between different accounts.

Request: it would be great if the program can allow the local port to be configurable.

inotify is not available on macosx-13.1-arm64 on ./cloud-mdir-sync

Hi,

I clone the repository and ran ./cloud-mdir-sync, then this message stopped the process:

Collecting pyinotify>=0.9.6
  Using cached pyinotify-0.9.6.tar.gz (60 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      inotify is not available on macosx-13.1-arm64
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Thanks

cms-oauth IMAP failure

I tried the IMAP mode of cms-oauth with an Office365 account but it failed:

ben@d3:~/.mail/nvidia/cloud_mdir_sync$ ./cms-oauth --cms_sock=/var/run/user/1000/cms.sock --proto=IMAP [email protected] --output=token
Traceback (most recent call last):
  File "/home/ben/.mail/nvidia/cloud_mdir_sync/.venv/bin/cms-oauth", line 33, in <module>
    sys.exit(load_entry_point('cloud-mdir-sync', 'console_scripts', 'cms-oauth')())
  File "/home/ben/.mail/nvidia/cloud_mdir_sync/cloud_mdir_sync/cms_oauth_main.py", line 98, in main
    xoauth2_token = get_xoauth2_token(args)
  File "/home/ben/.mail/nvidia/cloud_mdir_sync/cloud_mdir_sync/cms_oauth_main.py", line 24, in get_xoauth2_token
    raise ValueError(f"Invalid CMS server response {ret!r}")
ValueError: Invalid CMS server response ''

The '--test-imap' option failed likewise:

ben@d3:~/.mail/nvidia/cloud_mdir_sync$ ./cms-oauth --cms_sock=/var/run/user/1000/cms.sock [email protected] --test-imap outlook.office365.com
Traceback (most recent call last):
  File "/home/ben/.mail/nvidia/cloud_mdir_sync/.venv/bin/cms-oauth", line 33, in <module>
    sys.exit(load_entry_point('cloud-mdir-sync', 'console_scripts', 'cms-oauth')())
  File "/home/ben/.mail/nvidia/cloud_mdir_sync/cloud_mdir_sync/cms_oauth_main.py", line 98, in main
    xoauth2_token = get_xoauth2_token(args)
  File "/home/ben/.mail/nvidia/cloud_mdir_sync/cloud_mdir_sync/cms_oauth_main.py", line 24, in get_xoauth2_token
    raise ValueError(f"Invalid CMS server response {ret!r}")
ValueError: Invalid CMS server response ''

The SMTP mode works with the same configuration:

ben@d3:~/.mail/nvidia/cloud_mdir_sync$ ./cms-oauth [email protected] --cms_sock=/var/run/user/1000/cms.sock --test-smtp=outlook.office365.com
send: 'ehlo [127.0.0.1]\r\n'

[...]

reply: b'235 2.7.0 Authentication successful\r\n'
reply: retcode (235); Msg: b'2.7.0 Authentication successful'

In the meantime, the output of cloud-mdir-sync does not hint at a problem:

03-03 14:30:25 DEBUG    Credential request 'IMAP' [] '[email protected]'
03-03 14:32:33 DEBUG    Credential request 'IMAP' [] '[email protected]'
03-03 14:34:26 DEBUG    Credential request 'SMTP' [] '[email protected]'

assertion failure for deleted message

In office365.py, When a message is deleted at local folder, and before the deletion is propagated to the cloud, the program checks to assert os.stat(os.path.join(self.msgdb.hashes_dir, ch)).st_nlink == 1. This works well by default. However, sometime, I make a copy of the message in a different folder, and the number of links is more than 1 after the message is deleted from the synced local folder. In this case, the assertion fails, and the program aborts abnormally. The desired behavior is to skip the assertion, and trust the local folder for the deletion.

KeyError: 'expires_at' in self.owa_token

When using cloud-mdir-sync only for smtp and imap with O365, I occasionally get an error that expires_at key is missing from the owa_token:

(base) stain@xena:~$ bin/cloud-mdir-sync 
07-27 08:55:31 DEBUG    Starting Loading cached state
07-27 08:55:37 INFO     Completed Loading cached state, 12311 msgs, 49126 cached ids (took 5.6297 secs)
07-27 08:55:37 INFO     Starting Azure AD Authentication for [email protected]//common
07-27 08:55:38 DEBUG    Credential request 'SMTP' [] '[email protected]'
Task exception was never retrieved
future: <Task finished name='Task-11' coro=<CredentialServer.handle_client() done, defined at /home/stain/miniconda3/envs/email/lib/python3.11/site-packages/cloud_mdir_sync/credsrv.py:37> exception=KeyError('expires_at')>
Traceback (most recent call last):
  File "/home/stain/miniconda3/envs/email/lib/python3.11/site-packages/cloud_mdir_sync/credsrv.py", line 58, in handle_client
    xoauth2 = await account.get_xoauth2_bytes(proto)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/stain/miniconda3/envs/email/lib/python3.11/site-packages/cloud_mdir_sync/office365.py", line 497, in get_xoauth2_bytes
    or self.owa_token["expires_at"] <= time.time() + 10):
       ~~~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'expires_at'
07-27 08:55:38 INFO     Completed Azure AD Authentication for [email protected]//common  (took 1.2742 secs)
07-27 08:55:44 DEBUG    Credential request 'SMTP' [] '[email protected]'

Not sure why the attribute is missing - is that a good sign or bad sign? It is not required by RFC 6749 although that specifies expires_in (seconds remaining). Here is all I found in self.owa_token:

{"refresh_token": "0.ARAAB8tSwU5...VRx"}

dos2unix problem

I am using the program in Unix. I found that I need to set the dos2unix in the function get_to_file of office365.py to True to avoid repeated downloading of long messages. The problem still persists, even after dos2unix is set to True:

async def get_to_file(self, outf, ver, path, params=None, dos2unix=True):

The problem seems to be related to how the \r is removed. Currently \r\n is replaced with \n. This does not completely solve the problem because the last \r of each chunk seems to be missed. This causes the downloaded file to be dependent on chunking, and therefore leading to non-unique content hash.

Suggestion is to make the following change in the get_to_file function:

data = data.replace(b"\r", b"")

It seems to have solved the problem - all \r gets removed, including the very last one of each chunk. If you feel this is probably not safe, you may want to do the \r\n to \n replacement before the file write, and remember to remove the possible last \r before the EOF.

content_hash saving

I could not get the program to save the content_hash files ch-.* . As a result, the messages need to be downloaded every time the program is restarted. I couldn't find where msgdb.close() or msgdb._save_content_hashes() is called.

Perhaps this has to do with how I exit the program. I couldn't find a graceful way to force the program to quit. I tried the keyboard Ctrl-C, and also sending signal TERM to the process. Neither will lead to the saving of the content hashes.

Can the content hashes be saved when the program exits, either gracefully or being forced to exit through signals?

missing subfolder

I have some subfolders in my outlook inbox. When I sync the mailbox, the subfolders are not synced.

Troubles when configuring one account and multiple folders

I'm trying to migrate from mbsync where I had an mbsyncrc with

Patterns "INBOX" "Drafts" "Sent Items" "Junk Email" "Deleted Items" "sw-cumulus"

I've created the following cms.cfg:

# vim: set ft=python:

import os.path


def mdpath(name: str) -> str:
    return os.path.join("/home/ben/.mail/nvidia/maildirs", name)


account = Office365_Account(user="[email protected]")
CredentialServer("/var/run/user/1000/cms.sock", accounts=[account])

inbox = MailDir(mdpath("INBOX"))
Office365("inbox", account)

mds = {}


def add_dir(name: str):
    mds[Office365(name, account)] = MailDir(mdpath(name))


add_dir("Deleted Items")
add_dir("Drafts")
add_dir("Junk Email")
add_dir("Sent Items")
add_dir("sw-cumulus")


def direct_message(msg):
    md = mds.get(msg.mailbox)
    if md is not None:
        return md
    else:
        return inbox


cfg.direct_message = direct_message
cfg.message_db_dir = "/home/ben/.mail/nvidia/cms_message_db"

I've been able to get cms to work but I'm getting many timeouts and errors in the logs. At least some messages do pull and flags push with this configuration but there are long delays and according to the logs many operations seem to take around 180s or some multiple of it. I had to do the initial synchronization by commenting out all but one or two folders at a time.

From a bit of experimentation, the problems start with >2 folders. At 3 the problems are not systematic. Beyond that I've never seen a run without some errors.

Here is a truncated log that shows some of the errors with the above config:

ben@d3:~$ ~/.mail/nvidia/cloud_mdir_sync/cloud-mdir-sync -c ~/.mail/nvidia/cms.cfg
03-02 07:29:50 DEBUG    Starting Loading cached state
03-02 07:29:51 INFO     Completed Loading cached state, 8170 msgs, 8170 cached ids (took 1.5429 secs)
03-02 07:29:51 INFO     Starting Azure AD Authentication for [email protected]//common
03-02 07:29:52 INFO     Completed Azure AD Authentication for [email protected]//common  (took 0.4424 secs)
03-02 07:29:52 DEBUG    Got HTTP Error 429 in <function GraphAPI.get_json at 0x7fa85ea014c0> for URL('https://graph.microsoft.com/v1.0/me/mailFolders?$filter=displayName+eq+'Junk+Email'')
03-02 07:29:52 ERROR    Graph returns 429, message='', url=URL('https://graph.microsoft.com/v1.0/me/mailFolders?$filter=displayName+eq+'Junk+Email'') Too Many Requests, None, delaying 1
03-02 07:32:05 DEBUG    Starting Updating Message List for /home/ben/.mail/nvidia/maildirs/INBOX
03-02 07:32:05 INFO     Completed Updating Message List for /home/ben/.mail/nvidia/maildirs/INBOX, 5882 msgs (took 0.0470 secs)
03-02 07:32:05 DEBUG    Starting Updating Message List for /home/ben/.mail/nvidia/maildirs/Deleted Items
03-02 07:32:05 INFO     Completed Updating Message List for /home/ben/.mail/nvidia/maildirs/Deleted Items, 2 msgs (took 0.0000 secs)
03-02 07:32:05 DEBUG    Starting Updating Message List for /home/ben/.mail/nvidia/maildirs/Drafts
03-02 07:32:05 INFO     Completed Updating Message List for /home/ben/.mail/nvidia/maildirs/Drafts, 0 msgs (took 0.0000 secs)
03-02 07:32:05 DEBUG    Starting Updating Message List for /home/ben/.mail/nvidia/maildirs/Junk Email
03-02 07:32:05 INFO     Completed Updating Message List for /home/ben/.mail/nvidia/maildirs/Junk Email, 1 msgs (took 0.0000 secs)
03-02 07:32:05 DEBUG    Starting Updating Message List for /home/ben/.mail/nvidia/maildirs/Sent Items
03-02 07:32:05 INFO     Completed Updating Message List for /home/ben/.mail/nvidia/maildirs/Sent Items, 572 msgs (took 0.0032 secs)
03-02 07:32:05 DEBUG    Starting Updating Message List for /home/ben/.mail/nvidia/maildirs/sw-cumulus
03-02 07:32:05 INFO     Completed Updating Message List for /home/ben/.mail/nvidia/maildirs/sw-cumulus, 1711 msgs (took 0.0091 secs)
03-02 07:32:05 INFO     Starting Updating Message List for [email protected]//common:inbox
03-02 07:32:05 INFO     Starting Updating Message List for [email protected]//common:Deleted Items
03-02 07:32:05 INFO     Starting Updating Message List for [email protected]//common:Drafts
03-02 07:32:05 INFO     Starting Updating Message List for [email protected]//common:Junk Email
03-02 07:32:05 INFO     Starting Updating Message List for [email protected]//common:Sent Items
03-02 07:32:05 INFO     Starting Updating Message List for [email protected]//common:sw-cumulus
03-02 07:35:05 INFO     Completed Updating Message List for [email protected]//common:Drafts, 0 msgs (took 179.9544 secs)
03-02 07:35:05 INFO     Completed Updating Message List for [email protected]//common:Deleted Items, 2 msgs (took 179.9560 secs)
03-02 07:35:05 INFO     Completed Updating Message List for [email protected]//common:Junk Email, 1 msgs (took 180.1429 secs)
03-02 07:38:05 INFO     Completed Updating Message List for [email protected]//common:Sent Items, 572 msgs (took 360.0216 secs)
03-02 07:41:06 INFO     Completed Updating Message List for [email protected]//common:sw-cumulus, 1712 msgs (took 541.6039 secs)
03-02 07:55:07 DEBUG    Got non-HTTP Error in <function GraphAPI.__get_json_paged_next at 0x7fa85ea01a60> TimeoutError()
03-02 07:56:06 INFO     Completed Updating Message List for [email protected]//common:inbox, 5883 msgs (took 1440.6920 secs)
03-02 07:56:06 INFO     Starting Applying cloud changes for /home/ben/.mail/nvidia/maildirs/INBOX
03-02 07:56:06 INFO     Completed Applying cloud changes for /home/ben/.mail/nvidia/maildirs/INBOX, 1 added, 0 removed, 5882 same (took 0.0048 secs)

[...]

03-02 08:01:06 DEBUG    Got non-HTTP Error in <function GraphAPI.get_json at 0x7fa85ea014c0> TimeoutError()
03-02 08:01:06 DEBUG    Got non-HTTP Error in <function GraphAPI.get_json at 0x7fa85ea014c0> TimeoutError()

[...]

03-02 08:13:06 DEBUG    Got non-HTTP Error in <function GraphAPI.get_json at 0x7fa85ea014c0> TimeoutError()
03-02 08:13:06 DEBUG    Got non-HTTP Error in <function GraphAPI.get_json at 0x7fa85ea014c0> TimeoutError()

[...]

03-02 08:38:06 DEBUG    Got HTTP Error 401 in <function GraphAPI.owa_subscribe at 0x7fa85ea01d30> for URL('https://outlook.office.com/api/beta/me/subscriptions')
03-02 08:38:06 INFO     Starting Azure AD Authentication for [email protected]//common
03-02 08:38:06 DEBUG    Got HTTP Error 401 in <function GraphAPI.owa_subscribe at 0x7fa85ea01d30> for URL('https://outlook.office.com/api/beta/me/subscriptions')
03-02 08:38:06 INFO     Completed Updating Message List for [email protected]//common:Sent Items, 572 msgs (took 180.1073 secs)
03-02 08:38:06 DEBUG    Got HTTP Error 401 in <function GraphAPI.owa_subscribe at 0x7fa85ea01d30> for URL('https://outlook.office.com/api/beta/me/subscriptions')
03-02 08:38:06 DEBUG    Got HTTP Error 401 in <function GraphAPI.owa_subscribe at 0x7fa85ea01d30> for URL('https://outlook.office.com/api/beta/me/subscriptions')
03-02 08:38:06 DEBUG    Got HTTP Error 401 in <function GraphAPI.owa_subscribe at 0x7fa85ea01d30> for URL('https://outlook.office.com/api/beta/me/subscriptions')
03-02 08:38:07 INFO     Completed Azure AD Authentication for [email protected]//common  (took 0.5285 secs)
03-02 08:38:07 DEBUG    Got HTTP Error 429 in <function GraphAPI.owa_subscribe at 0x7fa85ea01d30> for URL('https://outlook.office.com/api/beta/me/subscriptions')
03-02 08:38:07 ERROR    Graph returns 429, message=' ', url=URL('https://outlook.office.com/api/beta/me/subscriptions') Too Many Requests, BudgetDepleted, delaying 1

[...]

03-02 09:47:07 DEBUG    Changed event, looping
03-02 09:47:07 INFO     Starting Updating Message List for [email protected]//common:Drafts
03-02 09:47:07 INFO     Starting Updating Message List for [email protected]//common:sw-cumulus
03-02 09:50:06 DEBUG    Got HTTP Error 401 in <function GraphAPI.get_json at 0x7fa85ea014c0> for URL('https://graph.microsoft.com/v1.0/me/mailFolders/AQMkADljNDg0ZGVmLTI0MDgtNGFiYy04M2Y0LTBkYzhlYzY3MzIzNwAuAAADb7LbYCy52ka16VAD9flt5gEApCgwMaf2QUO9os4QQXbmbQAAAgEPAAAA/messages?$select=internetMessageId,isRead,Flag,receivedDateTime,singleValueExtendedProperties&$expand=SingleValueExtendedProperties($filter%3D(id+eq+'Integer+0xe17')+or+(id+eq+'Integer+0x1080'))&$top=500')
03-02 09:50:06 INFO     Starting Azure AD Authentication for [email protected]//common
03-02 09:50:06 DEBUG    Got HTTP Error 401 in <function GraphAPI.get_json at 0x7fa85ea014c0> for URL('https://graph.microsoft.com/v1.0/me/mailFolders/AAMkADljNDg0ZGVmLTI0MDgtNGFiYy04M2Y0LTBkYzhlYzY3MzIzNwAuAAAAAABvsttgLLnaRrXpUAP1_W3mAQB7DThHax87SIH3d7Ck1-mmAAAEpysWAAA=/messages?$select=internetMessageId,isRead,Flag,receivedDateTime,singleValueExtendedProperties&$expand=SingleValueExtendedProperties($filter%3D(id+eq+'Integer+0xe17')+or+(id+eq+'Integer+0x1080'))&$top=500')

Is this indicative of a problem in cms or is this kind of use case (multiple folders from one account) not supported?

Feature request

Feature Request:

I would like to request a feature that allows a user to store an extra copy of any new cloud message to a local maildir folder.

More explanation: I would like to save every new message of a cloud folder to a permanent local folder so that even if the message is deleted in the cloud folder, the local backup folder is not updated. I plan to save INBOX and "Sent Items" so that I always have a local backup copy of the cloud messages.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.