ikvk / imap_tools

Work with email by IMAP

imap_tools 📧

High level lib for work with email by IMAP:

• Basic message operations: fetch, uids, numbers
• Parsed email message attributes
• Query builder for search criteria
• Actions with emails: copy, delete, flag, move, append
• Actions with folders: list, set, get, create, exists, rename, subscribe, delete, status
• IDLE commands: start, poll, stop, wait
• Exceptions on failed IMAP operations
• No external dependencies, tested

Python version	3.5+
License	Apache-2.0
PyPI	https://pypi.python.org/pypi/imap_tools/
RFC	IMAP4.1, EMAIL, IMAP related RFCs
Repo mirror	https://gitflic.ru/project/ikvk/imap-tools

Contents

• Installation
• Guide
  • Basic
  • Email attributes
  • Search criteria
  • Actions with emails
  • Actions with folders
  • IDLE workflow
  • Exceptions
• Release notes
• Contribute
• Reasons
• Thanks
• Help the project

Installation

$ pip install imap-tools

Guide

Basic

Info about lib are at: this page, docstrings, issues, pull requests, examples, source, stackoverflow.com

from imap_tools import MailBox, AND

# Get date, subject and body len of all emails from INBOX folder
with MailBox('imap.mail.com').login('[email protected]', 'pwd') as mailbox:
    for msg in mailbox.fetch():
        print(msg.date, msg.subject, len(msg.text or msg.html))

Description of this^ example.

MailBox, MailBoxTls, MailBoxUnencrypted - for create mailbox client. TLS example.

BaseMailBox.<auth> - login, login_utf8, xoauth2, logout - authentication functions, they support context manager.

BaseMailBox.fetch - first searches email uids by criteria in current folder, then fetch and yields MailMessage, args:

• criteria = 'ALL', message search criteria, query builder
• charset = 'US-ASCII', indicates charset of the strings that appear in the search criteria. See rfc2978
• limit = None, limit on the number of read emails, useful for actions with a large number of messages, like "move". May be int or slice.
• mark_seen = True, mark emails as seen on fetch
• reverse = False, in order from the larger date to the smaller
• headers_only = False, get only email headers (without text, html, attachments)
• bulk = False, False - fetch each message separately per N commands - low memory consumption, slow; True - fetch all messages per 1 command - high memory consumption, fast; int - fetch all messages by bulks of the specified size, for 20 messages and bulk=5 -> 4 commands
• sort = None, criteria for sort messages on server, use SortCriteria constants. Charset arg is important for sort

BaseMailBox.uids - search mailbox for matching message uids in current folder, returns [str | None], None when MailMessage.from_bytes used, args:

• criteria = 'ALL', message search criteria, query builder
• charset = 'US-ASCII', indicates charset of the strings that appear in the search criteria. See rfc2978
• sort = None, criteria for sort messages on server, use SortCriteria constants. Charset arg is important for sort

BaseMailBox.<action> - copy, move, delete, flag, append

BaseMailBox.folder - folder manager

BaseMailBox.idle - idle manager

BaseMailBox.numbers - search mailbox for matching message numbers in current folder, returns [str]

BaseMailBox.client - imaplib.IMAP4/IMAP4_SSL client instance.

Email attributes

Email has 2 basic body variants: text and html. Sender can choose to include: one, other, both or neither(rare).

MailMessage and MailAttachment public attributes are cached by functools.lru_cache

for msg in mailbox.fetch():  # generator: imap_tools.MailMessage
    msg.uid          # str | None: '123'
    msg.subject      # str: 'some subject 你 привет'
    msg.from_        # str: 'Bartö[email protected]'
    msg.to           # tuple: ('[email protected]', '[email protected]', )
    msg.cc           # tuple: ('[email protected]', )
    msg.bcc          # tuple: ('[email protected]', )
    msg.reply_to     # tuple: ('[email protected]', )
    msg.date         # datetime.datetime: 1900-1-1 for unparsed, may be naive or with tzinfo
    msg.date_str     # str: original date - 'Tue, 03 Jan 2017 22:26:59 +0500'
    msg.text         # str: 'Hello 你 Привет'
    msg.html         # str: '<b>Hello 你 Привет</b>'
    msg.flags        # tuple: ('\\Seen', '\\Flagged', 'ENCRYPTED')
    msg.headers      # dict: {'received': ('from 1.m.ru', 'from 2.m.ru'), 'anti-virus': ('Clean',)}
    msg.size_rfc822  # int: 20664 bytes - size info from server (*useful with headers_only arg)
    msg.size         # int: 20377 bytes - size of received message

    for att in msg.attachments:  # list: imap_tools.MailAttachment
        att.filename             # str: 'cat.jpg'
        att.payload              # bytes: b'\xff\xd8\xff\xe0\'
        att.content_id           # str: '[email protected]'
        att.content_type         # str: 'image/jpeg'
        att.content_disposition  # str: 'inline'
        att.part                 # email.message.Message: original object
        att.size                 # int: 17361 bytes

    msg.obj              # email.message.Message: original object
    msg.from_values      # imap_tools.EmailAddress | None
    msg.to_values        # tuple: (imap_tools.EmailAddress,)
    msg.cc_values        # tuple: (imap_tools.EmailAddress,)
    msg.bcc_values       # tuple: (imap_tools.EmailAddress,)
    msg.reply_to_values  # tuple: (imap_tools.EmailAddress,)
    # EmailAddress(name='Ya', email='[email protected]')  # "full" property = 'Ya <[email protected]>'

Search criteria

The "criteria" argument is used at fetch, uids, numbers methods of MailBox. Criteria can be of three types:

from imap_tools import AND

mailbox.fetch(AND(subject='weather'))  # query, the str-like object
mailbox.fetch('TEXT "hello"')          # str
mailbox.fetch(b'TEXT "\xd1\x8f"')      # bytes

Use "charset" argument for encode criteria to the desired encoding. If criteria is bytes - encoding will be ignored.

mailbox.uids(A(subject='жёлтый'), charset='utf8')

Query builder implements all search logic described in rfc3501. It uses this classes:

Class	Alias	Description	Arguments
AND	A	Combine conditions by logical "AND"	Search keys (see table below) | str
OR	O	Combine conditions by logical "OR"	Search keys (see table below) | str
NOT	N	Invert the result of a logical expression	AND/OR instances | str
Header	H	Header value for search by header key	name: str, value: str
UidRange	U	UID range value for search by uid key	start: str, end: str

See query examples. A few examples:

from imap_tools import A, AND, OR, NOT
# AND
A(text='hello', new=True)  # '(TEXT "hello" NEW)'
# OR
OR(text='hello', date=datetime.date(2000, 3, 15))  # '(OR TEXT "hello" ON 15-Mar-2000)'
# NOT
NOT(text='hello', new=True)  # 'NOT (TEXT "hello" NEW)'
# complex
A(OR(from_='[email protected]', text='"the text"'), NOT(OR(A(answered=False), A(new=True))), to='[email protected]')
# python note: you can't do: A(text='two', NOT(subject='one'))
A(NOT(subject='one'), text='two')  # use kwargs after logic classes (args)

Server side search notes:

• For string search keys a message matches if the string is a substring of the field. The matching is case-insensitive.
• When searching by dates - email's time and timezone are disregarding.

Search key table. Key types marked with * can accepts a sequence of values like list, tuple, set or generator.

Key	Types	Results	Description
answered	bool	ANSWERED/UNANSWERED	with/without the Answered flag
seen	bool	SEEN/UNSEEN	with/without the Seen flag
flagged	bool	FLAGGED/UNFLAGGED	with/without the Flagged flag
draft	bool	DRAFT/UNDRAFT	with/without the Draft flag
deleted	bool	DELETED/UNDELETED	with/without the Deleted flag
keyword	str*	KEYWORD KEY	with the specified keyword flag
no_keyword	str*	UNKEYWORD KEY	without the specified keyword flag
from_	str*	FROM "[email protected]"	contain specified str in envelope struct's FROM field
to	str*	TO "[email protected]"	contain specified str in envelope struct's TO field
subject	str*	SUBJECT "hello"	contain specified str in envelope struct's SUBJECT field
body	str*	BODY "some_key"	contain specified str in body of the message
text	str*	TEXT "some_key"	contain specified str in header or body of the message
bcc	str*	BCC "[email protected]"	contain specified str in envelope struct's BCC field
cc	str*	CC "[email protected]"	contain specified str in envelope struct's CC field
date	datetime.date*	ON 15-Mar-2000	internal date is within specified date
date_gte	datetime.date*	SINCE 15-Mar-2000	internal date is within or later than the specified date
date_lt	datetime.date*	BEFORE 15-Mar-2000	internal date is earlier than the specified date
sent_date	datetime.date*	SENTON 15-Mar-2000	rfc2822 Date: header is within the specified date
sent_date_gte	datetime.date*	SENTSINCE 15-Mar-2000	rfc2822 Date: header is within or later than the specified date
sent_date_lt	datetime.date*	SENTBEFORE 1-Mar-2000	rfc2822 Date: header is earlier than the specified date
size_gt	int >= 0	LARGER 1024	rfc2822 size larger than specified number of octets
size_lt	int >= 0	SMALLER 512	rfc2822 size smaller than specified number of octets
new	True	NEW	have the Recent flag set but not the Seen flag
old	True	OLD	do not have the Recent flag set
recent	True	RECENT	have the Recent flag set
all	True	ALL	all, criteria by default
uid	iter(str)/str/U	UID 1,2,17	corresponding to the specified unique identifier set
header	H(str, str)*	HEADER "A-Spam" "5.8"	have a header that contains the specified str in the text
gmail_label	str*	X-GM-LABELS "label1"	have this gmail label

Actions with emails

First of all read about UID at rfc3501.

Action's uid_list arg may takes:

• str, that is comma separated uids
• Sequence, that contains str uids

To get uids, use the maibox methods: uids, fetch.

For actions with a large number of messages imap command may be too large and will cause exception at server side, use 'limit' argument for fetch in this case.

with MailBox('imap.mail.com').login('[email protected]', 'pwd', initial_folder='INBOX') as mailbox:

    # COPY messages with uid in 23,27 from current folder to folder1
    mailbox.copy('23,27', 'folder1')

    # MOVE all messages from current folder to INBOX/folder2
    mailbox.move(mailbox.uids(), 'INBOX/folder2')

    # DELETE messages with 'cat' word in its html from current folder
    mailbox.delete([msg.uid for msg in mailbox.fetch() if 'cat' in msg.html])

    # FLAG unseen messages in current folder as \Seen, \Flagged and TAG1
    flags = (imap_tools.MailMessageFlags.SEEN, imap_tools.MailMessageFlags.FLAGGED, 'TAG1')
    mailbox.flag(mailbox.uids(AND(seen=False)), flags, True)

    # APPEND: add message to mailbox directly, to INBOX folder with \Seen flag and now date
    with open('/tmp/message.eml', 'rb') as f:
        msg = imap_tools.MailMessage.from_bytes(f.read())  # *or use bytes instead MailMessage
    mailbox.append(msg, 'INBOX', dt=None, flag_set=[imap_tools.MailMessageFlags.SEEN])

Actions with folders

BaseMailBox.login/xoauth2 has initial_folder arg, that is "INBOX" by default, use None for not set folder on login.

with MailBox('imap.mail.com').login('[email protected]', 'pwd') as mailbox:

    # LIST: get all subfolders of the specified folder (root by default)
    for f in mailbox.folder.list('INBOX'):
        print(f)  # FolderInfo(name='INBOX|cats', delim='|', flags=('\\Unmarked', '\\HasChildren'))

    # SET: select folder for work
    mailbox.folder.set('INBOX')

    # GET: get selected folder
    current_folder = mailbox.folder.get()

    # CREATE: create new folder
    mailbox.folder.create('INBOX|folder1')

    # EXISTS: check is folder exists (shortcut for list)
    is_exists = mailbox.folder.exists('INBOX|folder1')

    # RENAME: set new name to folder
    mailbox.folder.rename('folder3', 'folder4')

    # SUBSCRIBE: subscribe/unsubscribe to folder
    mailbox.folder.subscribe('INBOX|папка два', True)

    # DELETE: delete folder
    mailbox.folder.delete('folder4')

    # STATUS: get folder status info
    stat = mailbox.folder.status('some_folder')
    print(stat)  # {'MESSAGES': 41, 'RECENT': 0, 'UIDNEXT': 11996, 'UIDVALIDITY': 1, 'UNSEEN': 5}

IDLE workflow

IDLE logic are in mailbox.idle manager, its methods are in the table below:

Method	Description	Arguments
start	Switch on mailbox IDLE mode
poll	Poll for IDLE responses	timeout: Optional[float]
stop	Switch off mailbox IDLE mode
wait	Switch on IDLE, poll responses, switch off IDLE on response, return responses	timeout: Optional[float]

from imap_tools import MailBox, A

# waiting for updates 60 sec, print unseen immediately if any update
with MailBox('imap.my.moon').login('acc', 'pwd', 'INBOX') as mailbox:
    responses = mailbox.idle.wait(timeout=60)
    if responses:
        for msg in mailbox.fetch(A(seen=False)):
            print(msg.date, msg.subject)
    else:
        print('no updates in 60 sec')

Read docstrings and see detailed examples.

Exceptions

Most lib server actions raises exception if result is marked as not success.

Custom lib exceptions here: errors.py.

Release notes

History of important changes: release_notes.rst

Contribute

If you found a bug or have a question, then:

1. Look for answer at: this page, issues, pull requests, examples, source, RFCs, stackoverflow.com, internet.
2. And only then - create merge request or issue.

Reasons

• Excessive low level of imaplib library.
• Other libraries contain various shortcomings or not convenient.
• Open source projects make world better.

Thanks

Big thanks to people who helped develop this library:

shilkazx, somepad, 0xThiebaut, TpyoKnig, parchd-1, dojasoncom, RandomStrangerOnTheInternet, jonnyarnold, Mitrich3000, audemed44, mkalioby, atlas0fd00m, unqx, daitangio, upils, Foosec, frispete, PH89, amarkham09, nixCodeX, backelj, ohayak, mwherman95926, andyfensham, mike-code, aknrdureegaesr, ktulinger, SamGenTLEManKaka, devkral, tnusraddinov, thepeshka, shofstet, the7erm, c0da, dev4max, ascheucher, Borutia, nathan30, daniel55411, rcarmo, bhernacki, ilep, ThKue, repodiac, tiuub, Yannik, pete312, edkedk99, UlisseMini, Nicarex, RanjithNair1980, NickC-NZ, mweinelt, lucbouge, JacquelinCharbonnel, stumpylog, dimitrisstr, abionics, link2xt, Docpart, meetttttt, sapristi, thomwiggers, histogal

Help the project

1. Found a bug or figure out how to improve the library - open issue or merge request 🎯
2. Do not know how to improve library - try to help other open projects that you use ✋
3. Nowhere to put your money - spend it on your family, friends, loved ones, or people around you 💰
4. Star the project ⭐

imap_tools's Issues

Support for getting back all messages in a thread?

First, thank you for this great library, it definitely simplifies using imap over some other libraries in python land.

I've been trying to use this to be able to download all messages in a thread ( reply-to chain ), and the only way I've been able to do this was via searching the Header for a In-Reply-To message-id from the first email, or via the References Header field ( although that has been less reliable ).

I was wondering if imap_tools provided and easier way to do this, and if not whether that's something that would be good to implement?

Receiving and forwarding

Thanks for this awesome library. I have a couple usage questions:

1. Is there a way to enter IDLE mode? I'd like to have a script that is constantly running and act upon any incoming emails.

2. What's the best way I can forward an email? I'm not too familiar with the various email protocols, I'm guessing this is outside the scope of IMAP, but I was wondering if you could point me in the right direction?

Could you implement the IMAP4_SSL starttls() function ?

Could you implement the IMAP4_SSL starttls() function ?

Gmail X-GM-LABELS Fetch

First off, great library so far. Thank you!

Im trying to figure out a way to fetch emails based on if they have a certain label or not. Sadly im a bit lost with IMAP.

can i export eml?

RT
can i export eml?

Adding proxy support?

What would be required to add proxy support?

error when using search function for arcor.de imap

subjects = [msg.text for msg in mailbox.fetch(Q(subject='weather'))]
`UnexpectedCommandStatusError Traceback (most recent call last)
in
1 print(mailbox.folder.get())
----> 2 subjects = [msg.text for msg in mailbox.fetch(Q(subject='weather'))]

in (.0)
1 print(mailbox.folder.get())
----> 2 subjects = [msg.text for msg in mailbox.fetch(Q(subject='weather'))]

~\Anaconda3\lib\site-packages\imap_tools\mailbox.py in fetch(self, criteria, charset, limit, miss_defect, miss_no_uid, mark_seen, reverse)
90 """
91 search_result = self.box.search(charset, self._criteria_encoder(criteria, charset))
---> 92 check_command_status('box.search', search_result)
93 message_id_set = search_result[1][0].decode().split(' ') if search_result[1][0] else ()
94 # first element is string with email numbers through the gap

~\Anaconda3\lib\site-packages\imap_tools\utils.py in check_command_status(command, command_result, expected)
45 raise UnexpectedCommandStatusError(
46 'Response status for command "{command}" == "{typ}", "{exp}" expected, data: {data}'.format(
---> 47 command=command, typ=typ, data=str(data), exp=expected))
48
49

UnexpectedCommandStatusError: Response status for command "box.search" == "NO", "OK" expected, data: [b'[SERVERBUG] Internal error occurred. Refer to server log for more information. [2020-05-06 15:40:57] (0.001 + 0.000 secs).']`

logout() error

Hi, trying to logout(), i've got errors on both Linux and Windows:
Connection is established with:

with MailBox('mail.domain.com', port=993, ssl=True).login('username', 'password',initial_folder='Sent') as sent:

On Linux (MailBox() as sent):

Traceback (most recent call last):
File "imap.py", line 8, in
sent.logout()
File "/root/imapcrm/venv/lib/python3.5/site-packages/imap_tools/mailbox.py", line 69, in logout
result = self.box.logout()
File "/usr/lib/python3.5/imaplib.py", line 614, in logout
self.shutdown()
File "/usr/lib/python3.5/imaplib.py", line 312, in shutdown
self.sock.shutdown(socket.SHUT_RDWR)
File "/usr/lib/python3.5/ssl.py", line 967, in shutdown
socket.shutdown(self, how)
OSError: [Errno 9] Bad file descriptor

On Windows (MailBox() as mailbox):

Traceback (most recent call last):
File "c:/Users/Todor/python/imapjson/imap.py", line 8, in
mailbox.logout()
File "c:\Users\Todor\python\imapjson\venv\lib\site-packages\imap_tools\mailbox.py", line 69, in logout
result = self.box.logout()
File "C:\Users\Todor\AppData\Local\Programs\Python\Python38-32\lib\imaplib.py", line 633, in logout
typ, dat = self._simple_command('LOGOUT')
File "C:\Users\Todor\AppData\Local\Programs\Python\Python38-32\lib\imaplib.py", line 1205, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "C:\Users\Todor\AppData\Local\Programs\Python\Python38-32\lib\imaplib.py", line 983, in _command
raise self.abort('socket error: %s' % val)
imaplib.abort: socket error: [WinError 10038] An operation was attempted on something that is not a socket

Search intro spam folder or deleted folder

I have tried this code

mailbox = MailBox(myimap)
mailbox.login(data[0], data[1] , initial_folder='Spam')
subjects = [msg.text for msg in mailbox.fetch(Q(text='david',date_gte=dt.date(2020,3, 30)))]

I have read all documentation I can't find a way to search in spam and Trash folder and inbox in the same time _

AttributeError when listing the folders

Hello,

I got AttributeError when trying to do

    with MailBox(server).login(username, password) as mailbox:
        print(mailbox.folder.list())

= >

  File "/tools/env/lib/python3.7/site-packages/imap_tools/folder.py", line 126, in list
    folder_dict = folder_match.groupdict()
AttributeError: 'NoneType' object has no attribute 'groupdict'

I found out that the response from the mail server (line 121 in folder.py - variable data) is:

[b'(\\HasNoChildren) NIL "INBOX"', b'(\\HasNoChildren \\Sent) NIL "sent"', b'(\\HasNoChildren \\Drafts) NIL "drafts"', b'(\\HasNoChildren) NIL "newsletters"', b'(\\HasNoChildren \\Archive) NIL "archive"', b'(\\HasNoChildren \\Junk) NIL "spam"', b'(\\HasNoChildren \\Trash) NIL "trash"']

Leading me that the matching regexp has extra quotes requirement around the delimiter.

Not working:
re.compile(r'\((?P<flags>[\S ]*)\) "(?P<delim>[\S ]+)" (?P<name>.+)'
Working:
re.compile(r'\((?P<flags>[\S ]*)\) (?P<delim>[\S ]+) (?P<name>.+)'

can i export eml?

thanks for your answer
but i want to export file from message .
not read message from file~
issue61

Cloning message

I'm having trouble cloning a message. My intent is to serialize and access on a different service. When testing the code below I set at least one message to unread in my native mail client.

from imap_tools import MailBox, Q, MailMessage

mailbox = MailBox(os.getenv('MAIL_HOST'))
mailbox.login(os.getenv('MAIL_USER'), os.getenv('MAIL_PASS'), initial_folder='INBOX')

msgs_gen = mailbox.fetch(Q(seen=False))
msgs = list(msgs_gen)

test = msgs[0]
test_copy = MailMessage.from_bytes(test.obj.as_bytes())

print(test.uid)
print(test_copy.uid)

Results in the error when accessing test_copy.uid:

     60         """Message UID"""
     61         # zimbra, yandex, gmail, gmx
---> 62         uid_match = re.search(r'UID\s+(?P<uid>\d+)', self._raw_uid_data.decode())
     63         if uid_match:
     64             return uid_match.group('uid')

AttributeError: 'NoneType' object has no attribute 'decode'

Many of the email properties make their way to the copy, but not all, including uid and flags.

My mail server is hosted on Microsoft Office 365, which is an Exchange server.

Encrypted by default? SSL

Hi,

just wondering, are connections SSL encrypted by default?
If not, how do you activate encryption?

Best regards

Constructing imap_tools.MailMessage object from string export

Hello,

This is the first time I've contributed on someone else's GitHub project, so my apologies if I have broken established etiquette in my ignorance.

Having spent a good few weeks wrestling with imaplib in vain, I am now using imap_tools into my project (an IMAP client which bulk downloads and archives emails in JSON format). I have quickly come to appreciate how much simpler it makes everything!

I am writing two classes, a JSON Encoder and Decoder. The encoder class currently encodes imap_tools.MailMessage.obj property as a string using the as_string() method which imap_tools.MailMessage.obj inherits as an instance of the email.message.Message class.

However, when it comes to the Decoder class, I'd like this to recreate the MailMessage object using a constructor method, but currently, the only available options are constructing this from bytes.

As a result, I currently have to create an email.message.Message object from the string export, immediately convert it back into bytes, and then call the MailMessage constructor function. This seems very hack-y and not an efficient way of decoding emails in bulk.

import email, imap_tools

message_as_string = my_imap_mailmessage.obj.as_string()
...
bytes_string = email.message_from_string(message_as_string).as_bytes()
b = imap_tools.MailMessage([tuple(['', bytes_string])])

Would there be any support from the devs for creating a constructor method that would recreate a MailMessage object from its string representation? I would be happy to contribute to implementing this (although I am relatively unexperienced).

Thanks!

Filter emails with attachment

Hi, it would be nice to add an option to filter emails with attachment.

I am facing issues to read email from another mailbox which is added to my email account

[Feature Request] Ability to filter by datetime instead of just date

This is a great project that makes integrating imap capabilities so much simpler.

I was wondering if you'd be willing to consider adding the ability to filter by datetime instead of just date. This could be implemented by utilizing RCF2822's time entry that provides GWT hours, minutes, seconds, and timezone offset.

Use-case for this proposed functionality:

• Objective - Email cronjob that runs every 10 minutes to log all new emails for communication record-keeping and statistical analysis.

from imap_tools import MailBox, Q, AND
from datetime import datetime, timedelta

mailbox = MailBox('imap.mail.com')
mailbox.login('[email protected]', 'password', initial_folder='INBOX')

# Using system received time
mailbox.fetch(Q(AND(new=True, datetime_gte=datetime.now() - timedelta(minutes=10))))
# And using sent date
mailbox.fetch(Q(AND(new=True, sent_datetime_gte=datetime.now() - timedelta(minutes=10))))

mailbox.logout()

Cannot import any class from the lib

After a pip install imap_tools (so getting the version 0.16.0, I cannot use the library as I cannot import any class.

/ # pip install imap_tools
Collecting imap_tools
  Downloading imap_tools-0.16.0-py3-none-any.whl (51 kB)
     |████████████████████████████████| 51 kB 298 kB/s
Installing collected packages: imap-tools
Successfully installed imap-tools-0.16.0
/ # python
Python 3.7.7 (default, Mar 11 2020, 01:01:39)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from imap_tools import Mailbox
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Mailbox' from 'imap_tools' (/usr/local/lib/python3.7/site-packages/imap_tools/__init__.py)

How to do exact from_ and to search

Hi,

This might be the configuration of my IMAP server (Dovecot backed by Solr) or the IMAP protocol itself, but when i search for...

client = '[email protected]'
for msg in imap.fetch(OR(to=client, from_=client)):
    ...

... i get the results for [email protected], but also for "User <[email protected]>" like the search string is split into parts and they match.

Is this expected? How do i get the results where msg.from_ or msg.to is exactly what i need?

Fetched email without uid.

Why some emails I fetched without the uid? But I can get its uid by the mailbox.search func.

loss msg.html

msg.html is Empty
but
msg.text has a value

Is this a problem?

How can I fix it
or
How to judge

HTML charset issue

If the charset is not utf-8 as below. there are some decoding issue.

<message.py>
def text(self) -> str:
...
return part.get_payload(decode=True).decode('utf-8', 'ignore')
def html(self) -> str:
...
return part.get_payload(decode=True).decode('utf-8', 'ignore')

This should be changed as below.
return part.get_payload(decode=True).decode(part.get_content_charset(), 'ignore')

message.from_ returns empty string

It is what the title says, also message.from_values return this {'email': '', 'name': '"Lopez', 'full': '"Lopez'}

does not extract attached EML files

As .EML files have these headers:

`--00000000000094809105878daac0
Content-Type: message/rfc822; name="Totally legit email (2).eml"
Content-Disposition: attachment; filename="Totally legit email (2).eml"
Content-Transfer-Encoding: base64
Content-ID: <f_jv0apprb0>
X-Attachment-Id: f_jv0apprb0

MIME-Version: 1.0
Date: Fri, 25 Jan 2019 10:37:47 -0800
Message-ID: [e-mail]
Subject: Totally legit email
From: REDACTED [email protected]
To: REDACTED [email protected]
Content-Type: multipart/alternative; boundary="000000000000a30bb605804c9ff2"

--000000000000a30bb605804c9ff2
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable`

imap-tools does not return this as an attachment and completely disregards the attachment itself along with the filename.

Limit issue?

Let say your inbox has 90 emails. And your code is

for msg in mailbox.fetch(limit=100):
    print(msg.uid)

It seems to stuck in the loop.

Here is the trace when i stop the debug:

 line 99, in fetch
    fetch_result = self.box.fetch(message_id, "(BODY[] UID FLAGS)" if mark_seen else "(BODY.PEEK[] UID FLAGS)")
  File "/usr/lib/python3.6/imaplib.py", line 534, in fetch
    typ, dat = self._simple_command(name, message_set, message_parts)
  File "/usr/lib/python3.6/imaplib.py", line 1196, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python3.6/imaplib.py", line 1019, in _command_complete
    typ, data = self._get_tagged_response(tag)
  File "/usr/lib/python3.6/imaplib.py", line 1139, in _get_tagged_response
    self._get_response()
  File "/usr/lib/python3.6/imaplib.py", line 1094, in _get_response
    data = self.read(size)
  File "/usr/lib/python3.6/imaplib.py", line 305, in read
    return self.file.read(size)
KeyboardInterrupt

mailbox.move doesn't work

import logging
from imap_tools import MailBox

## logger setup
log = logging.getLogger('mailClient')
log.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
log.addHandler(ch)

## creds
hostname = ""
login = ""
password = ""

## connect to imap
mbox = MailBox(hostname)
mbox.login(login, password, 'test')

log.debug("%s", [{'id': msg.id, 'uid':msg.uid, 'subj':msg.subject, 'date':msg.date} for msg in mbox.fetch()])
mbox.move(mbox.fetch(), 'processed')

er = mbox.expunge()
log.debug(er)
lo = mbox.logout()
log.debug(lo)

Outputs:
2018-12-27 18:38:14,064 - mailClient - DEBUG - [{'id': '1', 'uid': '1', 'subj': 'test', 'date': 'Thu, 27 Dec 2018 18:28:31 +0300'}]
2018-12-27 18:38:14,184 - mailClient - DEBUG - ('OK', [None])
2018-12-27 18:38:14,186 - mailClient - DEBUG - ('BYE', [b'Microsoft Exchange Server 2016 IMAP4 server signing off.'])

But message stays in the 'test' folder.
Could you please advise?

Thanks,
Andrey

Unable to access GSuite mail

Is there anything in particular required to get imap_tools working with a GSuite (Google) email account? I followed Google's instructions by enabling IMAP on my account and created an "App password" which allows programs like this access.

from imap_tools import MailBox, MailMessage, Q

mailbox = MailBox("imap.gmail.com")
mailbox.login(
    "[email protected]",
    "my16charapppswrd",
    initial_folder="INBOX",
)

imaplib.error: b'[AUTHENTICATIONFAILED] Invalid credentials (Failure)'

I have also used this App Password technique with http://improvmx.com where it was able to properly connect to Gmail.

TypeError from email if from is None

My inbox contains a spam message for which obj["from"] is None.
from_values() calls decode_header(self.obj['from']), which fails with

/usr/lib/python3.7/email/header.py in decode_header(header)
     78                     for string, charset in header._chunks]
     79     # If no encoding, just return the header with no charset.
---> 80     if not ecre.search(header):
     81         return [(header, None)]
     82     # First step is to parse all the encoded parts into triplets of the form

TypeError: expected string or bytes-like object

I am not sure what the correct action is - is it an upstream bug, or is it a bug here because imap_tools should never call this function when from is missing? If the second of these is true, what should happen? Probably the bug is with the client that sent the spam, but well behaved users shouldn't suffer for this. I leave it in your capable hands because for my current simple requirement I will just remove the email and carry on. Thank you.

Attachment filename is being partially lost

Hi, I've detected a problem with the filename property of the Attachment class.

Sometimes I get the following result while decoding attachment filename:

In [37]: decode_header(attach_message.get_filename())
Out[37]: [(b'Contract ', 'utf-8'), (b'27 04 20.docx', None)]

The filename property decodes and returns the first element from the list which is

In [24]: mails[0].attachments[0].filename
Out[24]: 'Contract '

I suggest the change in the following pull request: #32

Thank you!

Bulk fetch

Hi. Is there a way to fetch a header in bulk thus not query imap for each message separately?

An example with "plain" imaplib would be (snippet below takes 15 seconds for 6000 emails):

with IMAP4_SSL('imap.gmail.com') as mb:
  mb.login('foo', 'bar')
  mb.select('INBOX')
  
  _, data = mb.search(None, 'ALL')
  f = ",".join([x.decode('ASCII') for x in data[0].split()])
 
  mb.fetch(f, 'BODY[HEADER.FIELDS (Subject)]')

Snippet below takes 18 seconds for just 100 emails as (I guess) it does 100 separate queries.

with MailBox('imap.gmail.com').login('foo', 'bar', initial_folder='INBOX') as mailbox:
	mailbox.fetch()
	[msg.uid for msg in mailbox.fetch(headers_only=True, limit=100)]

I'd love to use your lib for all the parsing beauties

Thanks

space in folder names causing issues

i'm running into errors in handling folder names with spaces.

with MailBox('imap.mail.com').login('[email protected]', 'pwd') as mailbox:
    mailbox.folder.exists("INBOX/foo/bar baz")

returns

error: LIST command error: BAD [b'Error in IMAP command LIST: Extra arguments (0.001 + 0.000 secs).']

this is not a problem with .set and .create, which makes it very interesting to role through and create based on criteria, but only if a folder doesn't exist.

thanks!

'MailMessage' object has no attribute 'reply_to'

Hi,

On the PyPI doc I can see there is a "reply_to" into Mailbox object. But when I try to access it I have the following error :

'MailMessage' object has no attribute 'reply_to'

Any ideas ?

Thanks

requesting a new public attachment attribute of "content_disposition"

First, let me thank you for your well designed imap_tools python library. It is very elegant and effective. Thank you for all of your efforts.

I have a code request for the attachments section of messages.py.

Specifically, I am requesting a new public attachment attribute of "content_disposition". When I look at messages.py, there are three attributes for attachments: filename, content_type, and payload. All are located around line 243 of messages.py. And I see you using 'Content-Disposition' during your attachment walk on line 215. So maybe there is a way that I am missing. However, I have not figured out a way to call that attribute of content_disposition. The reason I need this is pretty simple. I am looking for a way to determine if an attachment is inline or attached so I can download the files as needed.. Here is an example of one of my emails.

--000000000000f6524605af620d7c
Content-Type: image/jpeg; name="image001.jpg"
Content-Disposition: inline; filename="image001.jpg"
Content-Transfer-Encoding: base64
Content-ID: <17493fb11c24ce8e91>
X-Attachment-Id: 17493fb11c24ce8e91

verses

--000000000000f6524705af620d7d
Content-Type: application/pdf; name="SKMBT_C65420091513480.pdf"
Content-Disposition: attachment; filename="SKMBT_C65420091513480.pdf"
Content-Transfer-Encoding: base64
Content-ID: <17493fb11c332794c592>
X-Attachment-Id: 17493fb11c332794c592

The inline images shows up in the HTML. It is an image of the person's email signature. The other is an attachment to the email. I am looking for a way to differentiate between the two.

Thank you in advance and I appreciate you taking the time

Folders with '/' character in folder name

There's an issue with how the '/' characters are treated by the mailbox.folder.list and mailbox.folder.set functions.

When you list all the folders with the mailbox.folder.list function the folders with '/' in their names are returned without the '/' character. For example if I have a nested folder structure like Inbox -> test -> test1/test2 the mailbox.folder.list method returns this as Inbox/test/test1test2.

This (Inbox/test/test1test2) raises an error when passed as an argument to the mailbox.folder.set function, so not sure if this is a problem with the set function or the list function.

Implement Proxy Support

Proxy support would be a great feature. I've found that piece of code which would add pysocks as a third party dependency but it might be worth it.

import imaplib
import socks
import ssl

class SocksIMAP4(imaplib.IMAP4):
    PROXY_TYPES = {"socks4": socks.PROXY_TYPE_SOCKS4, "socks5": socks.PROXY_TYPE_SOCKS5, "http": socks.PROXY_TYPE_HTTP}

    def __init__(
        self,
        host: str = "",
        port: int = imaplib.IMAP4_PORT,
        proxy_addr: str = None,
        proxy_port: int = None,
        username: str = None,
        password: str = None,
        proxy_type: str = "http",
    ):

        self.proxy_addr = proxy_addr
        self.proxy_port = proxy_port
        self.username = username
        self.password = password
        self.proxy_type = SocksIMAP4.PROXY_TYPES[proxy_type.lower()]

        imaplib.IMAP4.__init__(self, host, port)

    def _create_socket(self):
        return socks.create_connection(
            (self.host, self.port),
            proxy_type=self.proxy_type,
            proxy_addr=self.proxy_addr,
            proxy_port=self.proxy_port,
            proxy_username=self.username,
            proxy_password=self.password,
        )


class SocksIMAP4SSL(SocksIMAP4):
    def __init__(
        self,
        host: str = "",
        port: int = imaplib.IMAP4_SSL_PORT,
        proxy_addr: str = None,
        proxy_port: int = None,
        username: str = None,
        password: str = None,
        proxy_type: str = "http",
    ):

        SocksIMAP4.__init__(
            self,
            host,
            port,
            proxy_addr=proxy_addr,
            proxy_port=proxy_port,
            username=username,
            password=password,
            proxy_type=proxy_type,
        )

    def _create_socket(self):
        sock = SocksIMAP4._create_socket(self)
        server_hostname = self.host if ssl.HAS_SNI else None
        return ssl._create_stdlib_context(certfile=None, keyfile=None).wrap_socket(
            sock, server_hostname=server_hostname
        )

    def open(self, host: str = "", port: int = imaplib.IMAP4_PORT):
        SocksIMAP4.open(self, host, port)```

message.from_ parsing error

When the from header (the value in message.headers['From']) contains the following:
('"Firstname, Lastname"' [email protected])

the message.from_ only contains -> "Firstname

I got around this by sending this message.headers['From'] value as an argument to the email.utils.getaddresses function as a workaround.

Paging

I love the simplicity of your imap_tools. It works very well.

What I wanted to find out how would I page through a huge amount of emails. I see you can limit the amount of emails it should bring back, but I can not find where to page through it. For e.g. Page 1 Limit 100, Page 2 Limit 100 etc.

Exception handling, interfaces

Hi,

your lib is quite nice already, as it solves a couple of ugly issues with imaplib.

OTOH, some areas remain improvable.

Example: for an imap reorganization tool, I did:

    def _select(self, mailbox):
        try:
            resp, data = self._imap.folder.set(mailbox)
        except imap_tools.utils.UnexpectedCommandStatusError as e:
            log.debug('select: %s', e)
            return False, 0
        if resp == 'OK':
            return True, int(data[0].decode('utf-8'))
        return False, 0

As you can see, this interface is no beauty as such. I needed a way to deal with non-existing folders properly (e.g. in order to create them later on), but this is obvious only after reading the source.

My tool relocates mails of a certain year into a sub folder. That way, mailboxes for mailing lists like LKML keep manageable.
Hence, I need to fetch uids, and move them. Unfortunately, search is implicitly combined in the fetch method of your lib.

Therefor:

        # fetch matching mails
        resp, items = self._imap.box.uid('SEARCH', None, '(SINCE 01-Jan-%s BEFORE 01-Jan-%s)' % (year, year + 1))
        if ret == 'OK':
            log.debug('items: %s', items)
        else:
            log.debug('search: %s %s', ret, items)
            raise ImapReorgError(ERRPROT, 'searching in mailbox <%s> failed' % mailbox)

        # copy matching messages
        nr = 0
        uids = [uid.decode('utf-8') for uid in items[0].split()]
        uidset = []
        limit = self._limit if self._limit is not None else len(uids)
        while uids:
            uidset = uids[:limit]
            log.info('move %s of %s messages to %s', len(uidset), len(uids), dest)
            uids = uids[limit:]
            nr += len(uidset)
            copyres, delres = self._imap.move(','.join(uidset), dest)
            if copyres[0] != 'OK':
                raise ImapReorgError(ERRPROT, 'copying failed: %s' % str(copyres))
            if delres[0][0] != 'OK':
                raise ImapReorgError(ERRPROT, 'deleting failed: %s' % str(delres))

        # messages copied, expunge mailbox
        if nr:
            ret, data = self._imap.expunge()
            if ret != 'OK':
                raise ImapReorgError(ERRPROT, 'expunge <%s> failed' % mailbox)
            log.info('moved %s messages from <%s> to <%s>', nr, mailbox, dest)
        else:
            log.debug('no message dates matched %s in <%s>', year, mailbox)

Please note, that the uid handling is not exactly pretty as well. Also, converting to a list in order to properly handle some limit only to generate a string from it, which you in turn split into a list again, is improvable as well.

While that it, the third line of utils.py:cleaned_uid_set is, hmm, unpalatable, to put it nicely.

What I've missed is an improved error handling. See the call to self._imap.move above, and its error handling.
Your documentation reflects this by mostly ignoring any return values.

My full code is available here.

I hope, that you don't misunderstand this note. I would like to show you real world application of your lib and shed some light on the rough corners of your lib.

Working with read only mailboxes

Hello,

I'm trying out your tool, it's nifty :) Thank you!

On fastmail.com you have the option to generate an "app" password, and to give it restrictive permissions, like read-only.

When I use a read-only access, I cannot do a mailbox.fetch(). It raises the readonly error.

I tried with mark_seen=False, without success.

Is it possible to access a mailbox in read-only? I just want to get the subjects, nothing more. I'm using the default basic example.

Error in parsing recipient email addresses containing non-ASCII characters

I'm reporting a bug that I'm getting using imap_tools. I am trying to download an email using IMAP where the names of several recipients contain ü and ö.

The error message is as follows (email_to_json.py is my own file):

File "/Users/Andrew/PycharmProjects/imapEmailArchiver/email_to_json.py", line 40, in default
    "to": obj.to,
  File "/Users/Andrew/PycharmProjects/imapEmailArchiver/venv/lib/python3.7/site-packages/imap_tools/message.py", line 118, in to
    return tuple(i['email'] for i in self.to_values)
  File "/Users/Andrew/PycharmProjects/imapEmailArchiver/venv/lib/python3.7/site-packages/imap_tools/message.py", line 112, in to_values
    return parse_email_addresses(self.obj['To'] or '')
  File "/Users/Andrew/PycharmProjects/imapEmailArchiver/venv/lib/python3.7/site-packages/imap_tools/utils.py", line 72, in parse_email_addresses
    for raw_name, email in getaddresses([raw_header]):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/email/utils.py", line 112, in getaddresses
    all = COMMASPACE.join(fieldvalues)
TypeError: sequence item 0: expected str instance, Header found

Attempting to inspect the MailMessage.to property while debugging results in the following error:

UnicodeEncodeError: 'utf-8' codec can't encode characters in position 1737-1738: surrogates not allowed

Because the To header contains recipient names containing these illegal characters, decoding it as UTF-8 seems to fail (despite the fact these characters appear in the UTF-8 spec).

This seems to be a problem with imap_tools itself rather than my implementation of it; please correct me if I'm mistaken

Ability to fetch only certain components instead of entire message?

Fetching messages is great, but sometimes I only want certain parameters, and it seems like a waste to download the entire message if I only want a few parameters. It's also much faster to only download say, the subject, instead of the entire message if I'm only interested in the subject.

Does imap_tools support only fetching certain parameters for speed purposes?

I ideally would want the equivalent of:

a = mailbox.box.uid('fetch', "$UIDS", "BODY.PEEK[HEADER.FIELDS (SUBJECT)]")

"to_value" method is implemented with wrong parsing logic

The implementation now is not compatible with multi-recipients(in "to") situation:

    def to_values(self) -> list:
        """The addresses of the recipients (all data)"""
        if 'to' in self.obj:
            msg_to = decode_header(self.obj['to'])
            return [self._parse_email_address(part) for part in
                    self._decode_value(msg_to[0][0], msg_to[0][1]).split(',')]
        return []

The correct logic should be:

    def to_values(self) -> list:
        """The addresses of the recipients (all data)"""
        if 'to' in self.obj:
            msg_to = [decode_header(i) for i in self.obj['to'].split(',')]
            rtn_list = []
            for to in msg_to:
                if len(to) == 2:
                    raw_to_name, raw_to_addr = to[0], to[1]
                    to_name = self._decode_value(raw_to_name[0], raw_to_name[1])
                    to_addr = self._decode_value(raw_to_addr[0], raw_to_addr[1])
                    rtn_list.append(self._parse_email_address(to_name + to_addr.replace('\r\n\t', '')))
                else:
                    to_addr = self._decode_value(to[0][0], to[0][1])
                    rtn_list.append(self._parse_email_address(to_addr.replace('\r\n\t', '')))
            return rtn_list
        return []

Move messages unread

Hi!
I'm using mailbox.move to move some messages from one folder to another, but all messages become read after moving. Can we move them with unread status? Or maybe set unread status again to that messages after moving?
Thanks!

SOCKS proxy support?

Any plans to add proxy support in the future? That would be very handy.

Attachment without a name is not recovered

When using the outlook feature "Forward email as an attachment", the attached mail did not get a name.

So, as https://github.com/ikvk/imap_tools/blob/master/imap_tools/message.py#L191 cannot find a name, the attachment is skipped.

I proposed to give a default name to this attachment that could be "mail_attachment_XX" with XX being the index of the unnamed attachment found (the first one found will be mail_attachment_1, etc.).

Moreover, once we encounter this unnamed attachment (with Content-Type: message/rfc822, what is parsed is not the initial mail but a forwarded one. And the following attachments will be attachment to this second mail, not the initial one.

Questions about the fetch method

Hello!

I have some problems with the following code. Where am I wrong? I'm expecting to see the Subject of the most recent Unseen message but it is not working. Thanks in advance!

from imap_tools import MailBox

with MailBox('imap-mail.outlook.com').login('<mymail>', '<mypassword>') as mailbox:
                messages = mailbox.fetch('(UNSEEN)', mark_seen=False)
                for msg in messages:
                	print(msg.subject)

Intermittent UnexpectedCommandStatusError from mailbox.fetch()

When looping over fetch, every few or dozens of passes will result in error.

with MailBox('outlook.office365.com').login('xxx', 'xxx') as mailbox:
	for message in mailbox.fetch():
		# do something

quickly or eventually results in...

UnexpectedCommandStatusError(imap_tools.utils.UnexpectedCommandStatusError: Response status for command "box.fetch" == "NO", "OK" expected, data: [b'The specified message set is invalid.']

• Python 3,8.1
• imap-tools==0.13.1

I have added a while loop with catch to try again on error, but I want to make sure there isn't something else going on.

Are there any known issues with certain services or reasons this might be happening?

Problem with PEC email attachments

Hi and thank you for this best project :)
I created a ticket-service email based and it's a very lucky escape for imap lib python :D

So i've a problem when I receive a PEC email, all the information are visibile like date, from and subject but when I try to read the attachments my script throws this exception...

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/imap_tools/message.py", line 226, in attachments
    result.append((filename, codecs.decode(payload_item_bytes, cte)))
LookupError: unknown encoding: 7bit

This is a semplification of the script that I use and throws this exception :

import datetime
from imap_tools import MailBox
import itertools

mailbox = MailBox('imap.gmail.com')
email_username = '******************'
email_password = '******************'

mailbox.login(email_username, email_password)

imap_criteria = "UNSEEN"

# suppose only one PEC email
mails = mailbox.fetch(imap_criteria)

# x.id      -> OK
# x.subject -> OK
# x.from_   -> OK

len(x.attachments)

I'm using Python3 and imap-tools (0.8.0)

I think it was a codec problem, what can I do to solve it?
Thanks in advance

Not getting all results when using headers_only

Hi,

I wanted to get a quick list of the uids of messages in a mailbox. Using:

msg_uids = [msg.uid for msg in mailbox.fetch()]

I get a list with the uids for all messages in the mailbox, in length equal to:

mailbox.folder.status()['MESSAGES']

But when I do the same with:

msg_uids = [msg.uid for msg in mailbox.fetch(headers_only=True)]

I only get to see a few of them...

I would expect either all uids to be returned or none at all, since the uid is not a part of the headers, right?

Similarly, something like:

for msg in mailbox.fetch(Q(uid=xxx),headers_only): print(msg.headers)

only works for the uids from the list returned by the previous command.
Without the headers_only, it works for all the uids from that mailbox.

I have not yet looked into the imap_tools code, but I wanted to mention it here already anyway since this looks like strange behavior to me. At least, this is not what I would expect, and even if the usage seems strange (why would I need to use the uids when I can use Q to query), using the headers_only might perhaps also have other side-effects...

Best regards, Franky

How to read only headers?

Hi, can you provide an example in the documentation on this topic:
How to fetch only email headers in a loop?
I was unable to figure how to use the query language with fetch() to get it
Thank you